The Scale Of Moderating Facebook: It Turns Off 1 Million Accounts Every Single Day

(Mis)Uses of Technology

from the not-an-easy-issue dept

For years now, we’ve discussed why it’s problematic that people are demanding internet platforms moderate more and more speech. We should be quite wary of internet platforms taking on the role of the internet’s police. First, they’re really bad at it. As we noted in a recent post, platforms are horrendously bad at distinguishing abusive content from those documenting abusive content and that creates all sorts of unfortunate and bizarre results, with those targeted by harassing content often having their own accounts shut down. On top of that, the only way to actually moderate content at scale requires a set of rules, and any such set of rules, as applied, will create hysterically bad results. And that’s because the scale of the problem is so massive. It’s difficult for most people to comprehend even slightly the scale involved here. As a former Facebook employee who worked on this stuff once told me, “Facebook needs to make one million decisions each day — one million today, one million tomorrow, one million the next day.” The idea that they won’t make errors (both of the Type 1 and Type 2 category) is laughable.

And it appears that the scale is only growing. Facebook has now admitted that it shuts off 1 million accounts every single day — which means that earlier number I heard is way low. If it’s killing one million accounts every day, that means it’s making decisions on way more accounts than that. And, the company knows that it gets things wrong:

Still, the sheer number of interactions among its 2 billion global users means it can’t catch all “threat actors,” and it sometimes removes text posts and videos that it later finds didn’t break Facebook rules, says Alex Stamos.

“When you’re dealing with millions and millions of interactions, you can’t create these rules and enforce them without (getting some) false positives,” Stamos said during an onstage discussion at an event in San Francisco on Wednesday evening.

That should be obvious, but too many people think that the answer is to just put even more pressure on Facebook — often through laws requiring it to moderate content, takedown content and kill accounts. And, when you do that, you actually make the false positive problem that much worse. Assuming, for the sake of argument, that Facebook has to kill 10% of all the accounts it reviews, that’s 10 million accounts every day. If the punishment for taking down content that should have been left up is public shame/ridicule, that acts as at least some defense to get Facebook to be somewhat careful about not taking down stuff that it shouldn’t. But, on the flip side, if you add a law (such as the new one in Germany) that puts criminal penalties on social media companies for leaving up content that it wants taken down, you’ve changed the equation.

Now, the choice isn’t between “public ridicule vs. bad person on our platform” it’s “public ridicule v. criminal charges and massive fines.” So the incentive for Facebook, and other platforms changes such that it’s now encouraged to kill a hell of a lot more accounts, just in case. So suddenly the number of “false positives” is going to sky rocket. That’s not a very good solution — especially if you want platforms to support free speech. Again, platforms have every right to moderate content on their platforms, but we should be greatly concerned when governments are forcing them to moderate in a way that may have widespread consequences on how people speak, and where those policies can tilt the scales in often dangerous ways.

Filed Under: alex stamos, choices, intermediary liability, moderation, scale
Companies: facebook