Google runs an internal red team whose entire job is to break Google's own AI before someone else does. Read that again. The most resourced AI lab on earth is essentially paying its own staff to find the holes it shipped, because it knows the holes are there.
This is the part of the AI boom nobody putting up billboards wants to talk about. Not the hallucinations. Not the copyright fights. The security layer underneath all of it — the part that's supposed to keep a chatbot from leaking your data or following a stranger's instructions instead of yours — is being built in flight.
And if Google is improvising, everyone is improvising.
The threats are not theoretical
The categories Google's security people spend their days on are now well-known inside the field: prompt injection, data poisoning, and model inversion attacks. Translation, for anyone not steeped in this: someone slipping a hidden instruction into a webpage your AI assistant reads. Someone tainting the training data months before a model ships. Someone reverse-engineering a model to pull back private information it was trained on.
Adversarial inputs — prompts designed to make a model misbehave — are a particular fixation at Google, per TechCrunch's reporting on the rise of AI red teaming. The work resembles the white-hat hacking culture that grew up around web software in the 2000s, except the target keeps mutating. A patch for one prompt injection doesn't necessarily close the next one. The attack surface is a language. You cannot firewall a language.
CNBC has covered the resulting boom in cybersecurity firms selling tools to plug gaps in commercial AI systems. A booming market for AI defense is a polite way of saying: the offense is already winning enough fights to make defense profitable.
Building the plane in the air
Google says its teams evaluate models before public release, hunting for misuse paths and unexpected behavior. That sounds reassuring until you sit with what it implies. The evaluation happens because the behavior is unpredictable. You don't red-team a calculator. You red-team something you don't fully understand.
Google Cloud has framed AI security as a shared responsibility between platform and customer, the same language cloud providers used a decade ago when they were figuring out who was on the hook for a leaky S3 bucket. The framing is honest. It's also a tell. Shared responsibility is what you write when the provider knows it cannot guarantee the thing alone.
The deeper problem, which IBM's research blog and others have flagged, is pace. Large language models are getting bigger, getting multimodal, getting plugged into email and calendars and code repositories faster than the security literature can catch up. Every new capability is a new attack surface. Every integration is a new trust boundary. Defenders are reading last year's playbook against this quarter's model.
What the rest of us should take from this
If you're a CISO at a company smaller than Google — which is to say, your company — and a vendor pitches you an AI feature with security handled, ask them what their red team found last quarter. Ask what they patched. Ask whether their threat model includes prompt injection from documents the model ingests, which is the boring obvious attack that keeps landing.
Most vendors will not have a clean answer. That's not because they're bad. It's because the people with the cleanest answers, the ones at Google, are still publishing blog posts about how hard this is. The honest position in AI security right now is humility. Anyone selling certainty is selling something else.
The lesson isn't that Google is failing. Google is doing more than most. The lesson is that the safest hands in the industry have visible calluses, and the companies racing to catch up are mostly hoping nobody asks to see theirs.
