Developing a Photo Recognition Module to Detect Explicit Content

In today’s digital landscape, user-generated content (UGC) plays a pivotal role in engaging communities, but it also comes with risks. Ensuring that platforms remain safe from explicit content is crucial for both user protection and the well-being of moderation teams. This case study examines the implementation of an automated photo recognition module designed to identify and flag explicit content within images uploaded to an application.

The Challenge

As platforms grow, monitoring and filtering user-uploaded media for explicit material—such as hate speech, weapons, drug use, nudity, and sexual content—becomes an essential requirement. Traditional manual review processes are not scalable and can expose administrators to harmful media, creating a stressful and potentially unsafe work environment. Therefore, automation of content moderation is necessary to ensure:

1. User safety: Preventing explicit content from being displayed to users.
2. Operational efficiency: Automating the detection process to handle a large volume of media uploads.
3. Admin protection: Safeguarding content moderators from exposure to harmful or offensive images.

Approach

To tackle these challenges, we adopted a hybrid approach, combining automation with manual oversight, to ensure efficient and accurate content moderation. Below are the steps taken:

1. Media Status Lifecycle

A media lifecycle was introduced within our system to handle uploaded images. This lifecycle includes a custom status called “PROCESSING,” which is triggered as soon as any image is uploaded to the platform. The “PROCESSING” status ensures the image is temporarily unavailable until the system has verified its safety.

2. Integration with AWS Rekognition

We integrated the AWS Rekognition service, a machine learning-based image and video analysis tool. AWS Rekognition scans images for explicit or inappropriate content and tags the media across several sensitive categories, including:

• Hate speech
• Weapons
• Drug use
• Nudity
• Sexual content

3. Automated Violation Detection

After the image has been analyzed by AWS Rekognition, the system evaluates the results:

• If the image passes all checks, it proceeds to a “PUBLISHED” status and is made visible to users.
• If any violation is detected, the media status is changed to “VIOLATION,” effectively preventing it from being displayed to users.

4. Accuracy and Moderation Efficiency

AWS Rekognition provides a high level of accuracy in detecting explicit content, significantly reducing the need for manual review. However, to ensure no inappropriate content slips through and that innocent images aren’t falsely flagged, we have implemented an override system.

5. Admin Override Feature

While automation handles the bulk of the filtering, human intervention remains available through the media status override. This feature allows site administrators to:

• Manually remove any media that violates community guidelines but may have been missed by the detection system.
• Correct cases of false positives, where media was flagged in error (for instance, artistic or medical content may be misinterpreted as explicit).

Written by Simon Hughes

Head of development

Back