look, we get it - it’s possible to scrape, index, proxy, and otherwise manipulate a huge amount of public-facing data in the fediverse.
that doesn’t mean you should. it’s bad. it violates the authors’ consent in many cases, and frankly it’s rude.
this isn’t your dev sandbox.
note I called it public-facing. that doesn’t mean public-owning. posts in the fediverse do not suddenly fall into the public domain by any legal mechanism I’ve been made aware of to date.
I post on my server. I choose to federate a subset of my posts to other servers at my discretion. if you abuse my open and trusting format for posting my thoughts publicly by repurposing my posts without my prior authorization I will absolutely do whatever I can to make you stop, also at my discretion.
please stop and think before you build software. full stop. any software at all. stop and think "am I even potentially putting others in harm’s way by building this? could this be used maliciously? would this be secure for people to have access to?" and unless you have strong answers, just don’t. don’t build the thing.
@djsundog This approach would have stopped Google from being built. Google received hate-mail from server administrators who had wired up GET endpoints to things that did stuff to their servers (in violation of the HTTP standard, but you can do whatever you want when you're writing your own code). Google's search crawler put those users in harm's way---in some cases, "delete post" was one of the buttons incorrectly flagged as a GETtable URL, and the crawler would blow away a blog as it crawled the blog. Crawler had no way to know it was doing this.
... but fundamentally, "am I even potentially putting others in harm's way" is too high a bar. A better bar is a standard people can agree upon collectively (such as robots.txt... Which *still* isn't good enough to *protect* things people don't want seen, you really do need auth/auth for that, but it's a good standard for flagging content as "don't be a dick").
One problem the fediverse has right now is there is no standard for individual users to flag their toots something equivalent to "noindex, nofollow" so we're not even at a place yet where there's a standard to opt-in or opt-out of being search-indexed. I think the system needs that (and lacking it, I think we ask too much of the world to demand that nobody build any indexers, even the non-malicious among us).
... it's possible that Mastodon's protocol is simply default-too-open to feel comfortable using it in its current iteration.
