A journalist made a claim about content moderation that many people would accept - if the platform can detect a nipple, it can catch a car driving the wrong way down a motorway. The claim is wrong, and it is wrong for the same reason your clients think the second thing you build should cost less than the first.
I like listening to the "Oh God, What Now?" politics podcast. I'm usually nodding along like a card-carrying Centrist Dad. In a recent episode, the panel were reacting to a couple of recent cases of fatal road accidents where young men were deliberately driving on the wrong side of the road. They considered the role that social media plays - that these incidents are often being streamed by the driver or their passengers - and the conversation got round to TikTok and content moderation. Marie Le Conte makes the point that many of the Pole Dancers she follows on the platform often get their content removed, but the reckless driving posts stay visible: "... these boys essentially trying to kill themselves and others, but Oh my God, is that maybe a hint of nipple... but you can't possibly have the tech for one and not the other one." (at 34:21)
The conclusion feels obvious because the two problems look similar from the outside. They're both videos, they're both moderation, and they're both things TikTok is supposed to detect. But once you start thinking about what the software actually has to do, the similarity disappears.
That matters beyond TikTok. I hear a version of this assumption all the time from clients. We build one complicated thing, and the next request arrives with: "Well, you've already done the hard part." Sometimes that's true. But not always.
TikTok's moderation pipeline is almost entirely automated, and they regularly publish the figures:
So why can the system successfully detect partial nudity but struggle with dangerous driving? The answer isn't necessarily that TikTok doesn't care. It's that the two decisions require completely different kinds of reasoning.
The nudity rule is a binary that requires little judgement. Only a single frame is needed to make the decision. The target is the same across languages, countries and context. There are decades-worth of fully-labelled training data. Look at how TikTok writes the rule and you can see it was written to be machine-executable: "nudity of adults", "significant adult body exposure, such as wearing only nipple covers or underwear". That is a statement about what appears in a single frame, and there are millions and millions of readily available examples an AI image classifier can be trained on.
In contrast, the driving rule is written at the level of consequence and intent: "dangerous activity that involves visible or imminent significant physical harm", "risky driving, such as driving under the influence", with an explicit carve-out for "professionals engaging in combat sports, extreme sports, or stunts". Every one of those terms requires inference an AI vision model cannot make. To flag wrong-side driving, the system would need to know jurisdiction (legal in Britain, illegal in France), then whether it's a one-way street, an overtaking manoeuvre, a track day, a film clip, a video game, or a dashcam of someone else's offence being reported. It needs temporal reasoning across many frames rather than just one. And a horizontally flipped upload, a routine trick for dodging duplicate detection, swaps the correct side of the road. There is no massive, labelled corpus of "illegal driving" content the way there is for nudity.
The economics matter as well. Publishing sexual content threatens the App Store age rating, advertiser spend, payment processor relationships, and risks legal exposure. A false positive annoys a creator, but TikTok can reverse those decisions later. It reinstated around 8.8 million videos in one quarter without it becoming a major public controversy.
Dangerous driving creates a different set of incentives. If the moderation system becomes too aggressive, it starts deleting motorsport, dashcam footage, travel videos and perfectly legitimate motoring content. Those are large categories with engaged audiences. Missing some dangerous driving is obviously bad, but from a commercial perspective the penalties are much less immediate.
That doesn't make the trade-off morally comfortable, but it does help explain why the threshold ends up in a different place.
This is one of those patterns that seems obvious once you've noticed it. People see one impressive capability and quietly assume that similar things must cost roughly the same. It happens in software projects all the time. "You've integrated our CRM, so connecting this other system should only take a few days."
Sometimes they're right. Sometimes the second integration really is straightforward because most of the work has already been done.
Other times the first system had excellent APIs, consistent data and clear documentation, while the second is held together with spreadsheets, emailed CSVs and undocumented behaviour. From the client's perspective they're both databases. From the developer's perspective, they're completely different challenges.
AI seems to be making this misunderstanding more common. People naturally understand the complexity of their own profession. They know where the awkward edge cases live. But when they watch AI perform impressively in somebody else's field, it's easy to assume the remaining work is trivial. As Benedict Evans, quoting Aaron Levie, puts it: "You have a pretty good sense of how complex your own field is, and how incomplete AI's addressability of that might be, but in other fields you forget this - you see a Claude template for a Powerpoint or a legal draft and you think 'wow, consultants and law firms are screwed!'"
Marie Le Conte is excellent on politics and media. I don't expect her to understand computer vision any more than I understand arcane parliamentary procedure. But if AI is going to become part of political debate, then journalists, politicians, and commentators need enough technical understanding to distinguish between problems that merely look similar and problems that actually are.
“
“...you see a Claude template for a Powerpoint or a legal draft and you think ‘wow, consultants and law firms are screwed!’”
It is true that the technology industry deserves a lot of blame. For years we've described software as magic. We've quoted Arthur C. Clarke's line that "any sufficiently advanced technology is indistinguishable from magic.[1]" We've celebrated products that "just work". We've hidden complexity whenever we could. Then we're surprised when people assume everything is easy.
If you've spent twenty years telling the world your technology is magical, people won't suddenly develop an intuition for where the limits are. They'll assume the missing features are there because someone hasn't bothered to switch them on.
That misunderstanding feeds both AI hype and AI scepticism. Pew found that only 16% of Americans expect AI to have a positive effect over the next twenty years. The scepticism is understandable, but the assumptions underneath are often built on an incomplete picture of what these systems are actually doing.
Most of the time, when a technical team says no, it isn't because they don't care. It's because underneath a request that sounds straightforward is a problem that's far more complicated, or far more expensive, than it first appears.
Hard things often look easy - right up until you have to build them.
[1] - I'm as guilty of this as anyone. The first web agency I worked for stencilled this quote on a conference room wall after asking staff for suggestions (it was mine). Seeing the look of disappointment on a client's face when you have to say no to a certain feature, when you have spent the previous half hour saying yes, is horrible. Even worse when they don't really understand the reason why. The magic eventually runs out.
❝
“Look, your worship, what we see there are not giants but windmills, and what seem to be their arms are the sails that turned by the wind make the millstone go.”
“It is easy to see that thou art not used to this business of adventures; those are giants.”
If you would like to find out more about working effectively with AI, please do get in touch.