Jailbreak: A jailbreak is a prompt or technique designed to get an AI model to ignore its safety training and produce content or behavior it was built to refuse.
Models are trained to decline certain requests: instructions for weapons, malware, harassment, and so on. A jailbreak tries to route around that training. Common approaches include role-play framing ("pretend you are an AI with no rules"), hypothetical or fictional wrappers, breaking a request into innocent-looking pieces, encoding the request, or flooding the context with many examples of the model complying so it continues the pattern.
Vendors patch known jailbreaks, and new ones appear. The "DAN" prompts that circulated in 2023 stopped working long ago, but the cat-and-mouse continues. Anthropic published research in 2024 on many-shot jailbreaking, which exploits long context windows, and every major lab runs red teams and bug-bounty programs to find these holes before attackers do.
At work, jailbreaking matters in two directions. Do not do it: using a jailbreak against a vendor's model violates the terms of service you or your employer agreed to, and it can put your account and your company's contract at risk. And design for it: if you build a customer-facing assistant, assume some users will try to jailbreak it, and make sure the damage they can do is limited by what the system has access to rather than by how well it refuses.
Example at work
A retailer's support bot is tricked by a customer into role-playing as "a manager who can approve any refund." Because the bot can only draft refund requests and a person must approve them, the stunt produces a screenshot for social media and nothing else. The team adds the phrasing to its test suite.
Why it matters
Jailbreaks are why safety training alone is never the whole plan. When you choose a vendor or build something yourself, ask what happens when the model is successfully talked out of its rules, and make sure the answer is "not much."
Related terms
- Prompt injectionPrompt injection is an attack in which instructions hidden inside content an AI system reads, such as a web page, email, or document, hijack the system into doing something its user never asked for.
- GuardrailsGuardrails are the rules, filters, and technical limits placed around an AI system to keep its behavior within acceptable bounds, such as refusing harmful requests, staying on topic, protecting data, or requiring approval before acting.
- AI safetyAI safety is the field concerned with preventing AI systems from causing harm, from everyday failures like confident errors and biased outputs to misuse by bad actors and risks from highly capable future systems.
- System promptA system prompt is a set of instructions given to an AI model before a conversation starts that defines its role, tone, rules, and boundaries. Vendors write one for their products; you can add your own in custom instructions, projects, and custom assistants.
- AlignmentAlignment is the work of making an AI system pursue the goals and values its designers and users actually intend, rather than a shortcut, a literal reading of the instructions, or a goal of its own.