@olives @jdp23 using Microsoft PhotoDNA, Google's SafeSearch APIs, and Thorn's service for detecting CSAM is in fact an industry standard when in comes to trust and safety on user generated content.

You might not like that they're US based or not know these tools, but we can surely work towards tools that work for the fediverse and within a privacy framework.

We don't yet have data on how quickly reports of CSAM or similar content are actioned on. Ideally we prevent publishing CSAM up front.

@olives @jdp23 Also, at the end of the day, if you want to run a small instance, and you know your members are absolutely not going to post any content that's illegal (e.g., CSAM), then you don't have to use any of those tools to scan for potentially harmful content.

But, other admins may go "yeah, I'd rather play it safe", and then employ tools to assist them in moderation.

@olives @jdp23 further, the report itself doesn't actually name and shame instances at all.

The information about "which" instances came later, because instance admins wanted to make sure they were blocking instances spreading CSAM, as to avoid that making its way onto their servers.

To me, several things are true simultaneously:

- the report called attention to a problem that Mastodon collectively hasn't paid enough attention to, and had some useful suggestions for improving moderation tools

- by eliding important details, including that the source of much of the CSAM material has been known for this since 2017, is widely defederated, and that reject_media was developed specifically in 2017 specifically to deal with this problematic instance (and does so effectively for sites that turn it on), it painted an inaccurate picture of the situation.

- focusing only on the report's shortcomings shifts attention away from real problems, including that Mastodon installations by default don't block instances that are known sources of CSAM, that Mastodon gGmbH hasn't prioritized addressing this or improving moderation tools, and that the mobile apps and SpreadMastodon direct newcomers to a site where the moderators don't take action on clearly illegal content. Mastodon gGmbH's has a track record of not prioritizing user safety, and it's a huge problem. Hopefully the reaction to this report leads to positive changes.

- then again, the report doesn't take a "positive deviance" approach of looking at what works (tier0 blocklists, existing mechanisms like silencing and reject_media) and the possibilities for making a decentralized approach work. Instead the report concludes that centralization will be required, and suggests collaboration with Threads and others "to help bring the trust and safety benefits currently enjoyed by centralized platforms to the wider Fediverse ecosystem." But wait a second, trust and safety SUCKS for most people on Threads, why won't these supposed "benefits" lead to the same situation in the fediverse?

@thisismissem@hachyderm.io @olives@qoto.org

@jdp23 @olives I think I agree on that last point, in that centralisation is bad & not the correct approach, that's why we're trying to figure out how to make tech that was previously only existent in centralised spaces available to the decentralized social web.

Early approaches will almost certainly be centralised to some degree, whilst we work with partners to evolve systems towards decentralisation.

@jdp23 @olives but I will say that the fediverse has a LOT to learn from centralised social media as to trust & safety (even if we do some things differently already)

As it is, we're seeing new fediverse software being launched in a mainstream way without attention paid to even the most basic of moderation tools, and that's a huge problem.

@jdp23 @olives So far the fediverse's way of tackling CSAM & other horrendous content has been defederation, rather than prevention; we've usually been moderating in a reactive manner, rather than proactively.

@jdp23 @thisismissem As far as I know, the real child porn issue is far more recent. It was initially defederated by instances for containing objectionable content.

What's frustrating to me (about that instance, not you) is that other similar instances moderate such content better in a bid to offer a space for free artistic expression.

At the end of the day, I don't think these kinds of activities are good for any instance. It drags everyone down.

Some degree of collaboration might help here. Even if someone wants to have a different standard from another, it would be nice to have a baseline (i.e. real child porn should have no place anywhere).

While I suspect David is cherry-picking some of these posts (given the dates), it would be nice to see some of these other iffy accounts go away too.

The spammy ones which hint at engaging in illicit activity. I'm not sure if you've had to deal with them, however, a few instances have had to.

If they're appearing elsewhere (i.e. on the main fediverse), then that is quite a disturbing development. These accounts are also present on mainstream social media.

@olives @jdp23 yeah, probably the first step to dealing with them is to start collecting data & signals on content that may be harmful & starting to either process that content for review or to build better automated tools based on the learnings — the real thing highlighted in the report is that large instances are currently flying blind, which is dangerous for everyone

@olives@qoto.org Thanks for the correction on the timing. It's long enough ago that I don't remember, I thought that back in 2017 there were reports of stuff that would be illegal in the US and/or Western Europe (as well as the content that was acceptable by Japanese standards but objectionable elsewhere) but maybe it was just concerns. More recently I've heard that site and another site owned by the same corporation are essentially unmoderated, not sure how long that's been the case for. In any case this is stuff that should have been in the report!

Agreed about the value of a baseline and collaboration, but sometimes the source of the problems isn't interested in collaborating. It's weird because it's a corporate-owned instance, you'd think they'd have incentive to deal with it! More positively the parent corporation has been taking steps to deal with CSAM on their other sites so hopefully this report will convince them to pay attention to their federated sites as well.

@thisismissem@hachyderm.io Totally agreed that the fediverse has a lot to learn about trust and safety, including from centralized media. But also, centralized media's approach to trust and safety has discrimination baked in at multiple levels, relies on exploitative arrangements with outsourced content moderators, leads to results that favor authoritarian governments and ethnonationalists, and embodies a "surveillance is safety" approach. Admittedly at the scale they're dealing with it's hard to picture other approaches that wouldn't destroy their business model, but that doesn't make it a good template for the fediverse!

So it's certainly worth learning from the things they do well, and there's clearly at least some technology that can be adapted to apply in the decentralized space without bringing the other bad effects -- the report's recommendation for grayscale and blurring technology for moderators is a clear example of this.

In terms of earlier approaches being centralized to some degree ... I can certainly understand the argument for making incremental improvements on the way to the more difficult decentralized solution, but there are also arguments the other way.

Centralized systems tend to reinforce themselves, and there are several factors that make it likely to happen in this case as well:

- Intiial centralized solutions will be useful but imperfect, so resources will go to improving them (instead of creating decentrailzed solutions); and as the centralized solutions improve, the bar to get equivalently-good decentralized solutions becomes higher and higher.

- whoever's got a stake in the initial centralized solutions has a strong incentive to to favor continuing down that path instead of shifting to a decentralized solution.

- Meta's arrival, and the approach they're taking of working with large instance admins and platform providers who also run large instances, is likely to magnify these dynamics. There are reports that they'll require agreements from instances that federate with them, which certainly makes sense -- they'd be remiss not to! But it would be very natural to put in specific requirements for using some of the initial centralized tools and services, which become additional roadblocks for future decentralized tools. And it would also be natural for them to provide some of these services themselves, and/or partner with OpenAI or wholever to provide services ... in either case, these services become another way for them to harvest data.

So it's complex, and it'll be interesting to see how things play out.

@jdp23 I'm not gunna respond to everything individually, but just wanna say that I'm directly in the room with the folks working on these matters and doing very strong vocalisation of things that are discriminatory, particularly to LGBTQI+ folks and Sex Workers, and as a possible implementor of said moderation tools, you can guarantee I'll fight as hard as possible towards decentralized approaches, even once we've something roughly working that's centralised.

@jdp23 At least there's a seat being made available at the tablet now, and we can hopefully learn from centralised platforms mistakes (in our eyes) and from what they did really well.

@thisismissem I know that you're in the room and fighting hard -- I really respect what you do! As usual it bugs me that cis allies aren't fighting equally hard but it is what it is. And you having a seat at the table is a big deal!

@jdp23@blahaj.zone

@jdp23@indieweb.social @jdp23@blahaj.zone they're at least helping boost voices that need to be heard, so that's at least something, especially given the door has previously been completely shut, being able to tackle these issues with folks who have long-term experience, but also who are willing to listen to fresh ideas & advocate for doing better, that's quite something!

@thisismissem fair enough, that is indeed something -- and it's ver unusual, so credit where credit is due! @jdp23@blahaj.zone

@jdp23 @thisismissem "that would be illegal in the US" In the U.S., free expression is protected by the First Amendment (and honestly, that's a good thing).

In Japan, it would fall under Article 21 of the Constitution, which in this case, is seen as protecting free expression.

It would be inaccurate to suggest that this expression was "illegal in Western Europe".

Naturally though, real child porn is not protected by any of these.

Also, the E.U. e-Commerce Directive (similar to the U.S. Section 230) limited intermediary liability (which is a good thing for innovation / online expression) for communicating with another service.

There are additional measures which could be taken, such as not storing external content.

Of course, if real child porn appears, that can be dealt with accordingly, but it's important to give services a bit of breathing room.

As for that instance, I think it was acquired by a different company, at some point. I can't comment further on their moderation.

I've seen a post on the fedi that when it comes to moderation, you can't just look at the world the way it is today, you have to adapt to changes which come along.

@jdp23 @thisismissem Noting that at one point, a Japanese friend of mine who runs an instance did tell me that people in the West tend to send a lot of bogus reports, because they don't like one kind of expression or another.

So, it might be preferable, if those were limited to ones which are actually actionable. Likely not relevant though.

@thisismissem @jdp23 Our admin caught sight of the smoke and defederated that specific instance around a month ago.

From an administrative perspective, it's hard to say he made the wrong decision (although, hard to say it's good either). It wasn't made lightly.

If there was better collaboration, you wouldn't be learning that there might be a bit of a problem from an outsider's report (who probably only discovered the fediverse not long ago).

Some parts of the network do not cache images.

This is probably done for peace of mind, more than it being likely that something will happen. It seems the person who developed Mastodon (but not other tools) didn't think about this at all.

Scanning discussions were held within communities for other software years ago, although that didn't go anywhere as it didn't look like a problem.

@thisismissem @jdp23 Ideally, the less of this content the better, although I tend to worry about the process (removing it at all costs).

That is one area where I haven't been very fond of the large platforms.

As for the statistics, both of them are potential. I don't think he looks at the hits for legal reasons, although I'd lean towards probably for the first one.

The second one is for an algorithm of poorer repute (it's far newer and fuzzier).

Hope that helps clarify what I mean.

@olives @jdp23 yeah, it's really that first one that's needing action for now, critically, as that's for known images of CSAM, the rest are all just potentially CSAM/child trafficking, which is important to look at but not automatically removable matter.

Second would be more like a signal amongst other factors that leads to manual review & takedowns.