Text written by users serves two different teams. Product and operations want to learn what is broken. Community and trust teams want to know what should not stay published.
Both jobs run on the same stream, and both fail the same way most often: by treating the text at the word level.
Why word lists fall short
Content moderation usually starts with a list of banned words. What the list catches in the first few days looks satisfying. A few weeks in, two problems arrive at once.
First, users route around the list quickly. Repeated letters, symbols in place of letters, spaces or dots inserted mid-word, dropped vowels, ASCII substitutions for accented characters. Writing a rule for each turns into maintenance work with no end.
Second, and more importantly, lists cannot see context. The same word can appear in a complaint that quotes someone, in a joke, in a citation, and in an actual insult. A complaint that begins "the agent said this to me" and is deleted for the phrase it quotes means the system has punished the user for complaining.
Context is not a technical detail. A slur aimed directly at another user and a general expression of anger deserve different actions on the same platform.
What Turkish adds
Agglutinative morphology matters as much here as it does in search. The same root shows up in dozens of surface forms, and word-level matching cannot catch them all. Models working at the subword level are noticeably more robust.
There are also regional differences in usage. Some expressions read as serious insults in one context and as familiarity in another. Setting thresholds without knowing your platform's user profile leads either to over-censorship or to moderation that is too loose.
Unintended bias in classifiers
There is a known risk with toxicity classifiers: sentences containing certain identity terms can score highly even when the content is not abusive, because those terms appear predominantly in abusive contexts in training data. Dixon and colleagues measured this effect and discussed mitigation (Measuring and Mitigating Unintended Bias in Text Classification, AIES 2018).
The practical consequence: before going live, build a test set of non-abusive sentences that contain identity terms. Run that set again whenever you change models.
Graded actions instead of a single threshold
Building moderation as a binary, block or allow, creates cost in both directions.
Defining different actions per category works better:
- Low-risk content is published and labelled for statistics only
- Medium-risk content is published but flagged into the moderation queue
- High-risk content goes to human review before publication
- Threats, harassment and unlawful content are blocked and an incident record is opened
Setting up that ladder requires deciding which error is more expensive in each category. Missing a threat costs far more than a handful of false flags, so recall takes priority there. In mild profanity the reverse holds; false flags drive users off the platform.
The threshold is not a technical parameter. It is the numerical expression of platform policy. Who set it and on what grounds should be written down.
An appeals mechanism is not optional
For platforms operating in the EU this is now regulated. The Digital Services Act requires a statement of reasons for content restriction decisions and an internal complaint-handling system, with complaints submittable electronically and free of charge for at least six months. One provision matters particularly here: decisions on complaints may not be taken solely by automated means, and qualified staff must supervise them (DSA Article 17 and Article 20).
In Turkey, content removal and access blocking run under Law No. 5651, with separate obligations defined for social network providers.
Regulation aside, the mechanism earns its keep. Appeal records are the cleanest data you will get on where the model is wrong. The categories where appeals succeed are the categories where your threshold is too tight.
Common mistakes on the review analysis side
Extracting operational insight from the same stream has its own recurring errors.
Making sentiment the only metric. "Negative reviews are 18 per cent" says nothing on its own. A ratio moves when volume moves. During a campaign, review volume can triple; the negative share may look lower while the absolute count of negative reviews has gone up. Report the ratio and the absolute number together.
Assuming reviewers are representative. Review-writing clusters at the extremes: the very satisfied and the very annoyed write, the broad middle does not. The picture from review data is not the general view of your customer base. Use it to set priorities, not to measure satisfaction. How to build satisfaction measurement is covered in the survey methodology article.
Freezing the label set. When a new product, campaign or problem appears, the system squeezes it into old categories. If the share of "other" is climbing, the label set needs updating. Tracking that share is a good early warning for drift.
Not wiring the output to a queue. The most common ending for review analysis is a weekly report nobody opens. Analysis produces value only when specific labels land on specific teams' work lists. Having critical reviews automatically open a record in the support or product queue is far more effective than reporting.
How we measure quality
The measurement logic is similar for moderation and review analysis.
- Recall and precision per category, since a single overall accuracy figure is misleading
- Agreement between automatic labels and human coding
- Number of appeals and the rate at which they succeed
- Waiting time in the moderation queue
- Share of the "other" category in the total
A few hundred records sampled monthly is enough for the human comparison. Without it, quality degradation is typically discovered only when user complaints rise.
Deployment checklist
- Does detection rely on a word list or a contextual model
- Have spelling variants and character substitutions been tested
- Is there a test set of harmless sentences containing identity terms
- Are action tiers defined per category, and who owns the threshold decision
- Do users receive a statement of reasons
- Is there an appeals mechanism, and are appeals concluded by humans
- In review analysis, are ratio and absolute count reported together
- Are labels wired to a team queue, and is the "other" share tracked
To see how these services connect over API, look at Review Analysis Services and Profanity and Abuse Detection Services.
This article is general information and is not legal advice.
References
- Dixon, L., Li, J., Sorensen, J., Thain, N., Vasserman, L. (2018). Measuring and Mitigating Unintended Bias in Text Classification. AIES '18. dl.acm.org
- Digital Services Act (Regulation (EU) 2022/2065), Article 17, Statement of Reasons. eu-digital-services-act.com
- Digital Services Act, Article 20, Internal Complaint-Handling System. eu-digital-services-act.com
- Law No. 5651 on Regulation of Publications on the Internet (Türkiye). mevzuat.gov.tr