Justin (admin of Fedisearch) @fedisearch@qoto.org
- Website
- https://fedisearch.com/
Hi I'm Justin, admin of fedisearch.com,
Fedisearch is a search engine for fediverse (mastodon, miskey and pleroma) content. Fedisearch respects privacy and robot no indexing directives.
Joined Jul 2020
@hansw@mastodon.social if someone replies to our post, your original post is not indexed, only their reply is indexed. If you reply to their post, because you as the author chose not to be indexed, it will not be indexed.
Here's the background issue on scalability, for your interest.
Let's say someone who has been indexed for a long time decided that they no longer wants to be indexed. We will have to search for their presence in all the posts that they are mentioned in. We can't do this at scale without an index, as otherwise it means scanning through millions of rows. We can't really create indices for mentions either, as this index's only use case is to help speed up unindexing of user's posts in mentions.
Justin (admin of Fedisearch)
boosted
We want to work with instance admins to bring Fedisearch's functionality directly to their instances' web interface. If you are a mastodon or pleroma instance admin, and would like to participate, feel free to send us a direct message.
@hansw@mastodon.social @fedisearch_com we are committed to work on that, but there isn't a solution that is scalable to the size of our data. Commiting to an arbitrary date of your choice seems unrealistic. However, if you can DM me all the account uri that you own, I can do a one-time search and delete for you,hope this addresses the immediate cocnern.
Justin (admin of Fedisearch)
boosted
We are launching https://fedisearch.com , a website for doing full text search on fediverse content.
This could be very useful for small instances users that are missing full text searches, just about anyone else who is looking for a blazing fast fediverse search engine.
We have a well behaved indexer that honors users' preference on opting out status indexing.
Please give it a try and let us know what you think.
@freemo
True true. Adding some insights on behind the scenes, our crawler only visits mastodon, pleroma and misskey instances, because it only knows how to parse content from those instances. Some hubzilla posts get indexed because they appeared on the federated timelines of those three type of instances.
Yup this is indeed the way we are aiming for. However, still facing some technical difficulties with the performance when applying this method on mega threads where many people are being mentioned.
@freemo looks great !
@hansw regarding indexing of @ mentions, methods to achieve it is still being explored.
Does the opting out section in https://fedisearch.com/about help?
Justin (admin of Fedisearch)
boosted
With the help from @8zu 🎉 , we recently localised https://fedisearch.com in Chinese and Japanese
你好！中文用户
こんにちは！日本のユーザー
@rgx thank you :)
Justin (admin of Fedisearch)
boosted
Just found at another instance -- and think this is a really important development.
Fedisearch.com allows Full Text Searches on fedinet posted content -- from Mastodon, Pleroma, Misskey instances.
In Mastodon instances, Full Text Search is VERY RARE in my experience. It's available at qoto.org, and one reason I have recommended people to keep an account there if their main account does not provide it.
Even Elite instances have this option turned OFF (to save processor cycles and memory). Silly, because they could support that easily.
Now we have an external search Engine and it works. Tested briefly and found results I knew would be there, shown as expected.
Kudos for a much needed service and thank you to @fedisearch !
With the help from @8zu 🎉 , we recently localised https://fedisearch.com in Chinese and Japanese
你好！中文用户
こんにちは！日本のユーザー
I get your point indeed, and we take users' privacy very very seriously. I think the main problem here is that Pixelfed doesn't support the semantic to allow users to opt out from search indexing. In the case of mastodon, we support this feature.
The same set of settings that blocks google from indexing you would also block fedisearch from doing so as well. We have a insignificant amount of traffic compared to google... so if being indexed is the main worry, we feel we shouldn't be the main antagonist 😅
I'll leave the link to Toot! https://apps.apple.com/us/app/toot/id1229021451 a well received iOS app for mastodon, commercial and proprietary, yet makes 4.5/5 stars on 132 ratings.
And, publicly endorsed by mastodon https://joinmastodon.org/apps
Withdrawing from further discussions
@hansw I understand the concern, and personally hate spammers a lot too.
We try to strike for a balance between out of the box utility and respects for people's desire to stay out of the spotlight, and it indeed has been hard.
If we search pixelfed + a username on Google, we'd find many results. It wouldn't be fair for Google to grab all the search results without opt in while a smaller niche focused projects has to knock on doors right?
Not saying that defaulting to opt-out is right, but just want some extra leniency on our way to do something good to this community.
@namark I don't see mastodon and other community publically deny access to the network from close sourced software. Would you want to substantiate?
The use of analytic solutions on fedisearch is fully disclosed in our privacy policy; data collected are not personally identifiable.
@hansw for sure. Looks like we need to implement self serve immeidate opt out sooner
@hansw I think this is very valid concern. Sorry we let this one slip through. We will resolve this.
- Website
- https://fedisearch.com/
Hi I'm Justin, admin of fedisearch.com,
Fedisearch is a search engine for fediverse (mastodon, miskey and pleroma) content. Fedisearch respects privacy and robot no indexing directives.
Joined Jul 2020
