Content-Oblivious Trust and Safety Techniques

We present the results of a survey about the trust and safety techniques of a group of online service providers that collectively serve billions of users. We classify techniques that require the provider to be able to access the contents of users’ files and communications at will as content dependent, and content oblivious otherwise. We find that more providers use abuse-reporting features (which are content oblivious) than other abuse-detection techniques, but that participants’ abuse-reporting tools do not consistently cover the types of abuse that users may encounter. We also find that, despite strong consensus among participating providers that automated content scanning (which is content dependent) is the most useful means of detecting child sex abuse imagery, they do not consider it to be nearly as useful for other kinds of abuse. These results indicate that content-dependent techniques do not constitute a silver bullet to protect users against abuse. They also demonstrate that the impact of end-to-end encryption (which, controversially, impedes outside access to user content) on abuse detection may vary by abuse type. These findings have implications for policy debates over the regulation of online service providers’ anti-abuse obligations and their use of end-to-end encryption.