Justin (admin of Fedisearch) @fedisearch@qoto.org

Some notes regarding Fedisearch and opensource .

We mentioned earlier that Fedisearch.com is not opensource. Despite many benefit opensourcing could bring to the community, we have reservations:

Disabling fedisearch's compliance mechanism to robots.txt and robots meta tag is quite trivial.

Having folks run individual deployment of fedisearch that indifferently indexes everyone's content would severely damage fedisearch's reputation, and most likely will kill this project in its incubation.

Hence, no opensource, unless there's a good solution to the problem above.

@freemo good point

@freemo

The crawler honors robots.txt. However, due to how federation works, even if an instance A has a robots.txt that blocks all crawler access, it is still possible that the content from A appear on the federated timeline of other instances. It is for this reason that we also check for instance-domain-name.tld/@username for noindex meta tag.

Hence, using noindex metatag is the only reliable way to opt out of indexing. [1]

Mastodon has this meta tag option built-in and instance admins are able to override this option for every user on their site.

[1]: https://fedisearch.com/about

@freemo that is well understood when we registered for an account here. We don't have plans to engage in commercial activities either at this point.

We pushed a new feature to auto hide posts that have been marked as sensitive by their author in the search results.

Still rebuilding index (since 15 hours ago) to support filtering by time interval ...

@namark
Fedisearch.com does not have plan to opensource at this moment

@thor@mstdn.social
Thanks!
Regarding account indexing opt in for fediverse-only, this is a planned feature. We currently don't block other search engines, it's important that people are able to first discover this site through some search frontage

@feizong
Understood the concern, but we don't see this as a threat to any user in the fediverse.

Allow us to elaborate:
Even when mastodon purposefully dumbed down the search functionality, pleroma and misskey search are still functioning at full capacity. Also, if content producers allowed search indexing, their content would've been searchable on Google as well. Https://Fedisearch.com respects every content producers' privacy; as long as they turn on noindex option, none of their content would be indexed by us.

@matrixsasuke
Thanks. I think search social was constantly under heavy load. Part of why this project was started was because of the lack of a good public ally available search engine that respects people's privacy

@thor@mstdn.social
Decenrealizing a search index is technically a very hard problem. What would you see as a core motivator for building a decentralized index?

We are testing some advanced search features that allow users to specify a range of datetime to search postings from. However, still waiting on search index migrations to complete

@realcaseyrollins
It's currently sorted by "relevant", but this is quite hard to get right. But indeed it'd be nice to be able to atleast filter for only last x weeks. We will add that

We want to work with instance admins to bring Fedisearch's functionality directly to their instances' web interface. If you are a mastodon or pleroma instance admin, and would like to participate, feel free to send us a direct message.

We've spent some time to make the search page uglier. Was it worth it?

@realcaseyrollins Strong branding is so preferable. It gives instance operator some more motivation

@mashiro
Thank you for the interest.
We have a closed beta version of json apis that returns just about the same amount of information as our site's search entry point. Could you elaborate further on what you want to build?
The team is also working on a drop in replacement for mastodon's search controller at the same time.

Hope we didn't disturb too many of you with the announcement and boosts. We will try to keep reboosting minimal, while trying to initially spread the word out.