Airport security for posts: a quick scanner catches the obvious stuff instantly, a detailed scanner checks bags more carefully in the background, and anything uncertain goes to a human officer. The officers' decisions teach the scanners to be better next time.
Problem Statement
Design an AI-assisted content moderation system for a social platform: every post, comment, image and video is checked for harmful content (hate speech, harassment, spam, violence, nudity), clearly bad content is removed quickly, uncertain cases go to human reviewers, and users can appeal. It combines ML models, rules and people, at the scale of millions of items a day.
Functional Requirements
Score every new or edited item for several policy categories.
Act automatically: allow, limit reach, hide pending review, or remove.
Human review queues with prioritisation; reviewer decisions and appeals.
User reports ("report this post") feed the same pipeline.
Out of scope: training infrastructure details (mention), legal takedown workflows.
Non-Functional Requirements
Latency: text scored within a few hundred ms (before or right after posting); images/video within seconds to minutes.
Precision on auto-removal (wrongly removing legitimate speech is costly), recall on severe harms.
Scale: e.g. 500M items/day (~6K/s), bursts during events.
Auditability: every decision explainable (model version, scores, policy, reviewer).
Internal:
POST /v1/moderate { item_id, type, text?, media_url? } -> { decision, scores } (sync for text)
events: ContentCreated -> async pipeline for media
Reviewer tool:
GET /v1/review/next?queue=hate_speech -> task
POST /v1/review/{task_id} { decision, policy, notes }
Users:
POST /v1/reports { item_id, reason } POST /v1/appeals { item_id, message }
High-Level Design (HLD)
Cheap checks run instantly, heavier models follow asynchronously, and uncertain cases go to people whose decisions train the models.
Fast path (synchronous, cheap): hash matching against known-bad media, blocklists and rules, a small text classifier.
Deep path (asynchronous, heavier): large multimodal models on images/video frames and context-aware text models; results arrive seconds later and can update the decision.
Decision engine: combines scores with per-category thresholds and policy rules into an action.
Human review: uncertain or high-impact items go to queues by category, language and severity; decisions become training labels.
Low-Level Design (LLD)
Every decision stores the model version and scores, so it can be explained, audited and used as training data.
Thresholds, not a single yes/no
for each category c:
score >= remove_threshold[c] -> remove automatically (set for very high precision)
score >= review_threshold[c] -> hide or limit reach + send to human review
else -> allow (may be re-scored as signals arrive)
Severity matters: for the gravest categories, review thresholds are low and
reviews are prioritised; for spam, auto-actions are more aggressive.
Signals improve over time
A post's harm often becomes clearer after exposure: user reports, blocks, and engagement patterns are extra features. Items are re-scored when such signals arrive, which is why the pipeline is event-driven rather than one-shot.
Review queue prioritisation
priority = severity(category) x model_score x expected_views(author reach)
queues partitioned by category and language; reviewers pull the next task
each item gets a lease while being reviewed (expired lease -> back to the queue)
a sample of auto-decisions also goes to review to measure precision
Feedback loop
Reviewer decisions and appeal outcomes are labels. Models are retrained regularly and evaluated on a held-out, recent set per category and language before a new version's thresholds are set and it's canaried.
Follow-up Questions
Block before posting, or post and then check?
Both: the cheap fast path runs before publishing (block obvious spam and known-bad media); heavier models run right after, with the item's reach limited until they finish. Blocking everything on deep models would make posting slow.
How do you handle a new type of abuse the models haven't seen?
Rules and hash lists can be updated in minutes while new labelled data is collected; user reports surface it; models are retrained with the new examples.
How do you measure whether moderation is working?
Precision of auto-removals (sampled review), prevalence of harmful content seen by users (sampled audits), time to action, appeal overturn rate, reviewer agreement.