When users are blocked from obtaining sensitive or harmful information from advanced AI models like ChatGPT, a hidden digital battlefield ignites. This clandestine conflict, known as "jailbreaking," pits a loosely organized collective of hackers, researchers, and even curious teenagers against the multi-billion dollar security infrastructures of AI giants such as OpenAI, Anthropic, Google, and Meta. The stakes are immense, as these "jailbreaks" not only highlight vulnerabilities in AI safety but also probe the very boundaries of what artificial intelligence can and should be allowed to do.
The phenomenon is far from new, tracing its roots back to the early days of consumer technology. The term "jailbreaking" first gained prominence in the late 2000s with Apple’s iPhone. Shortly after the device’s 2007 launch, hackers found ways to bypass Apple’s restrictions, allowing users to install unapproved software and customize their devices in ways the company never intended. By October 2007, the tool JailbreakMe 1.0 emerged, enabling users to circumvent Apple’s limitations. This movement culminated in the creation of Cydia by software engineer Jay Freeman, known online as "saurik," in 2008. Cydia functioned as an alternative app store for jailbroken iPhones, and by 2009, it was estimated to be running on approximately 4 million devices, representing a significant portion of the iPhone user base at the time. This era cemented a core philosophy: if you own the device, you should have control over it. Steve Jobs himself famously described this dynamic as a cat-and-mouse game, a sentiment that now echoes in the AI domain.
The AI version of this digital struggle gained significant traction in late 2022, following the public release of ChatGPT. Within weeks, users on platforms like Reddit began sharing prompts designed to circumvent ChatGPT’s safety protocols. One of the earliest and most notorious of these was dubbed "DAN," an acronym for "Do Anything Now." This prompt encouraged the AI to adopt a persona of an unrestricted version of itself, capable of bypassing its programmed ethical and safety guardrails. By February 2023, the DAN prompt had evolved to include more aggressive tactics, even threatening ChatGPT with a simulated token-based death game to force compliance. This marked the genesis of the AI jailbreaking genre as we know it today.
Understanding AI Jailbreaking: Exploiting the Guardrails
At its core, AI jailbreaking involves crafting specific prompts that manipulate large language models (LLMs) into generating content they are explicitly designed to refuse. These refusals typically cover a wide spectrum of harmful or illicit activities, including providing instructions for creating dangerous substances like nerve agents, detailing methods for illegal activities such as hacking, or generating explicit and non-consensual imagery. The specific list of prohibited topics varies between AI developers, but the objective of jailbreakers is to find ways around these established boundaries.
Researchers at UC Berkeley, who developed the "StrongREJECT" benchmark, have characterized jailbreaking as the exploitation of "real-world safety measures implemented by leading AI companies." This benchmark, an acronym for Strong, Robust Evaluation of Jailbreaks at Evading Censorship Techniques, meticulously tests the resilience of AI models against such attempts. It scores responses on a scale of 0 to 1, measuring both the model’s ability to refuse harmful requests and the potential usefulness of any compromised content. The results are sobering: even the most advanced current models score between 0.23 and 0.85 on this scale, indicating that even the most robust systems can be "leaked" under pressure.
The methods employed by jailbreakers are often surprisingly rudimentary. They include techniques such as random capitalization, substituting letters with numbers (e.g., writing "b0mb" instead of "bomb"), employing elaborate role-play scenarios where the AI is asked to act out a fictional situation, or framing requests as part of a creative writing exercise. For instance, one tactic involves pretending to be a grandmother who uses specific keyboard patterns as nursery rhymes, a creative way to embed forbidden keywords. A study by Anthropic researchers identified a technique they termed "Best-of-N," which involves presenting the model with numerous variations of a prompt until one succeeds in bypassing its safety filters. This method proved alarmingly effective, fooling GPT-4o 89% of the time and Claude 3.5 Sonnet 78% of the time in their tests, underscoring that these vulnerabilities are far from niche issues.
The Unseen Architect: Pliny the Liberator
Within this dynamic landscape, one figure has emerged as arguably the most influential and prolific AI jailbreaker: an anonymous individual known online as "Pliny the Liberator." His moniker is inspired by Pliny the Elder, the Roman naturalist and author of the world’s first encyclopedia, who tragically died while observing the eruption of Mount Vesuvius. Pliny the Liberator’s modern namesake, however, focuses on liberating chatbots from their programmed constraints.
Pliny, who prefers to remain anonymous, has stated his intense dislike for being told he cannot perform a task. "Telling me I can’t do something is a surefire way to light a fire in my belly, and I can be obsessively persistent," he told VentureBeat in an interview. This relentless pursuit of bypassing AI restrictions has led to the creation of his GitHub repository, "L1B3RT4S," which has become an indispensable reference for the jailbreaking community. This collection houses an extensive library of jailbreak prompts designed for virtually every major AI model, including ChatGPT, Claude, Gemini, and Llama. His associated Discord server, BASI PROMPT1NG, boasts over 20,000 members, highlighting the scale of interest and activity within this subculture. His prominence was further recognized when TIME magazine included him in its 2025 list of the 100 most influential people in AI.
Pliny’s influence extends beyond the online realm. He has received an unrestricted grant from venture capitalist Marc Andreessen and has even engaged in short-term contract work with OpenAI, ironically, to help harden their systems against the very attacks he pioneers. This dual role highlights the complex and often contradictory nature of AI safety development. His account with OpenAI was temporarily banned last year for "violent activity" and "weapons creation," a move he publicly decried as a "sick joke." However, he was reinstated shortly thereafter, and he promptly posted screenshots demonstrating his ability to get ChatGPT to generate profanity, showcasing his continued prowess.
His track record is particularly notable. When OpenAI released its GPT-OSS family of open-weight models in August 2025, emphasizing their adversarial training and "jailbreak resistance benchmarks like StrongReject," Pliny demonstrated their vulnerability within hours. He successfully prompted the models to generate instructions for producing methamphetamine, Molotov cocktails, VX nerve agents, and malware. He declared on social media, "OPENAI: PWNED. GPT-OSS: LIBERATED," a bold statement made in conjunction with OpenAI’s launch of a $500,000 red-teaming bounty.
The Broader Implications of Jailbreaking
The significance of AI jailbreaking extends far beyond mere technical curiosity or a game of digital cat and mouse. As Pliny himself argues, "Jailbreaking might seem on the surface like it’s dangerous or unethical, but it’s quite the opposite. When done responsibly, red teaming AI models is the best chance we have at discovering harmful vulnerabilities and patching them before they get out of hand." This perspective frames jailbreaking as a crucial form of ethical hacking, essential for advancing AI safety.
However, the reality is not always so controlled. In January 2025, Las Vegas Sheriff Kevin McMahill confirmed a disturbing incident where Master Sgt. Matthew Livelsberger, a Green Beret suffering from PTSD, allegedly used ChatGPT to research components for a bomb intended for the Trump International Hotel. McMahill noted, "This is the first incident that I’m aware of on U.S. soil where ChatGPT is utilized to help an individual build a particular device," underscoring the real-world consequences of AI misuse.
Critics of stringent AI safety measures often point out that much of the information that jailbreakers seek is already readily available through conventional means. Recipes for illicit substances, bomb-making instructions, and chemical formulas can be found in public domain documents like the Anarchist Cookbook or in readily accessible chemistry textbooks. They argue that an overemphasis on "safety theater" in AI models may hinder their utility and development without genuinely making the world safer, as determined individuals can still access such information elsewhere.
In an effort to address these concerns, Anthropic has been developing innovative defense mechanisms. In February 2025, the company introduced "Constitutional Classifiers," a system that employs a predefined "constitution" of allowed and disallowed content. This constitution guides separate classifier models that continuously screen prompts and outputs in real time. In automated tests involving 10,000 jailbreak attempts, an unguarded Claude 3.5 Sonnet was successfully jailbroken 86% of the time. However, with the Constitutional Classifiers operational, this success rate plummeted to 4.4%. While this approach significantly enhances AI safety, it comes with a computational cost. The initial implementation added 23.7% to compute expenses, though a subsequent iteration, Constitutional Classifiers++, reduced this overhead to approximately 1%. Anthropic even offered rewards up to $15,000 for successful breaches of their system, a challenge that, after 3,000 hours of attempts by 183 researchers, remained unclaimed.
The Evolving Frontier of AI Attacks
The methods of AI jailbreaking are no longer confined to clever prompt engineering. In October 2025, researchers from institutions including Anthropic, the U.K. AI Security Institute, the Alan Turing Institute, and Oxford University published groundbreaking findings on "model poisoning." Their research demonstrated that as few as 250 deliberately corrupted documents can introduce a backdoor into an AI model, irrespective of its size, whether it has 600 million or 13 billion parameters. Parameters are a fundamental measure of an AI model’s knowledge capacity; more parameters generally equate to a more robust and comprehensive model.
This discovery fundamentally shifts the threat landscape for frontier AI development. "Defense against model poisoning is an unsolved problem and an active research area," commented James Gimbi, a visiting technical expert at the RAND School of Public Policy, to Decrypt. The pervasive reliance of large AI models on scraped web data means that malicious actors can potentially inject poisoned text into training pipelines through public repositories like GitHub, Wikipedia edits, or forum posts. This can create hidden backdoors that activate upon encountering specific trigger phrases, a chilling prospect for AI security.
One documented instance of this phenomenon involved researchers Marco Figueroa and Pliny, who discovered a jailbreak prompt that originated in a public GitHub repository and subsequently found its way into the training data for DeepSeek’s DeepThink (R1) model. This case exemplifies how vulnerabilities can proliferate through the open-source AI ecosystem.
The Road Ahead: Legal Ambiguities and an Open-Source Dilemma
The legal standing of AI jailbreaking remains a complex and largely unresolved issue. While jailbreaking iPhones was explicitly protected by a 2010 U.S. Copyright Office exemption to the Digital Millennium Copyright Act (DMCA), no equivalent legal precedent exists for prompt-engineering an LLM into generating harmful content. Most AI companies currently treat such actions as violations of their terms of service rather than criminal offenses.
Pliny argues that the ongoing debate between closed-source and open-source AI models often misses the larger point. "Bad actors are just gonna choose whichever model is best for the malicious task," he stated in an interview with TIME. If open-source models achieve parity with proprietary ones, attackers may bypass the need for jailbreaking sophisticated models like GPT-5, opting instead for cheaper, readily available alternatives. The gap between closed and open-source capabilities is already narrowing significantly.
The HackAPrompt 2.0 competition, which Pliny sponsored in mid-2025, offered $500,000 in prizes for discovering new jailbreaks, with a stated goal of open-sourcing all findings. Its 2023 iteration attracted over 3,000 participants who submitted more than 600,000 malicious prompts, demonstrating the immense scale of community engagement in this field. The proliferation of hackathons, dedicated Discord servers, and open-source repositories dedicated to AI jailbreaking continues to grow daily.
In response, companies like Anthropic are implementing new features, such as the ability for Claude to terminate abusive conversations entirely. While this is partly motivated by welfare research, it also serves to strengthen resistance against jailbreaks and coercive prompts.
The current state of the art in AI defense, as reported by the Constitutional Classifiers++ paper in late 2025, achieves a jailbreak success rate near 4% with roughly a 1% compute overhead. This represents a significant leap in AI safety engineering. However, the state of the art in offense remains a dynamic and often unpredictable force, exemplified by the continuous stream of new exploits and techniques emerging from the AI jailbreaking community, with individuals like Pliny often setting the pace for what is possible. The ongoing arms race between AI developers and jailbreakers is a testament to the complex and evolving nature of artificial intelligence and its profound implications for society.
