🇳🇱
If an instance like hubzilla doesnt allow individuals to opt-out then the sys admin still has the ability to opt-in or opt-out the entire server. The users know about that. I agree it is unfair to the users they dont have the power to control that, but the fault lies with the hubzilla developers, not with fedisearch.
I assume by verified opt-in you mean someone going to fedisearch and only being indexed if they specifically ask to be by submitting a form. To me, that is not a good solution I would recommend. Most people either want to be indexed by all search engines, or none at all. By using a system like that it would make it impractical for someone who wishes to be indexed to have the burden of reaching out and contacting every search enginer in existence.
I'd much prefer to stick to standards and where those standards arent implemented, then put the pressure on the people to implement them rather than to put the blame ont he search engines who honor the standards and those standards, when implemented, give you complete power and control over if you are indexed.
@freemo
True true. Adding some insights on behind the scenes, our crawler only visits mastodon, pleroma and misskey instances, because it only knows how to parse content from those instances. Some hubzilla posts get indexed because they appeared on the federated timelines of those three type of instances.
