OpenAI Enhances Protections Against Adversarial Distillation
As frontier AI models advance, the need for safeguards in their responsible use becomes critical. Adversarial distillation enables malicious actors to replicate these capabilities without proper protections, allowing them to train new models that can perform tasks the original models would refuse due to safeguards.
OpenAI is sharing its strategies for detecting and disrupting adversarial distillation, enhancing protections for its models, and providing insights to strengthen defenses across the industry. This approach combines adversarial testing, model training, classifiers, and API-level protections, along with investigations into coordinated attempts to extract model capabilities, aligning with safeguards used in biology and cybersecurity.
A primary focus is on preventing attackers from extracting protected reasoning, the internal steps a model takes to arrive at an answer. These traces can provide valuable training data for another model, containing information omitted from the final answer. Defenses against protected reasoning extraction have been significantly improved in GPT-6.2 Astra and GPT-6.2 Sol. Internal evaluations showed the average success rate for attackers attempting to recover protected reasoning dropped from 54.0% in GPT-6 Astra to 0.2% in GPT-6.2 Astra.
Understanding the Risks of Distillation
Distillation is a common machine-learning technique where one model learns from the outputs of another. While it raises intellectual property concerns, unauthorized distillation also poses safety risks, as a second model could acquire advanced capabilities without the original model’s safeguards. This is particularly critical in dual-use domains like cybersecurity and biology, where models could be misused to exploit vulnerabilities or develop harmful pathogens.
Protecting Reasoning Traces for Cyber Safety
OpenAI’s models assist users with coding and defensive cybersecurity tasks. A second model trained on benign outputs may inadvertently acquire offensive cybersecurity skills. OpenAI studied how distillation involving reasoning traces could transfer risky cyber capabilities, assessing how much capability transfers when the student model has access to the teacher’s exact reasoning traces versus summaries. Results indicated that the student trained on exact reasoning traces outperformed the one trained on summaries, maintaining a higher score across various benchmarks.
Complementary Defense Strategies
OpenAI employs a dual-layered defense against adversarial distillation. At the request level, model behavior, classifier checks, and API controls limit reasoning trace extraction. At the campaign level, patterns in request and account signals are monitored to detect coordinated extraction attempts. Defensive training teaches models to recognize reasoning extraction attempts and refuse to disclose protected reasoning while responding to legitimate requests. Input classifiers detect extraction attempts, while output classifiers prevent leaks in assistant messages.
API-level protections restrict how protected reasoning can be reused across interactions and models. In GPT-6.2 Astra and GPT-6.2 Sol, a model’s protected reasoning is tied to the conversation history that produced it, ensuring earlier reasoning is dropped if a prior message is rewritten.
Investigating Unauthorized Campaigns
When unauthorized distillation campaigns are detected, OpenAI investigates the responsible accounts and takes enforcement actions against those involved. Insights from these investigations inform upgrades to defenses, enhancing detection capabilities for similar activities. This effort extends beyond OpenAI’s platform, as the company collaborates with other model providers to establish shared standards for safeguards and coordinated enforcement.
Measuring Progress Against Distillation
OpenAI tests the effectiveness of request-level safeguards by evaluating whether attackers can extract protected reasoning in scenarios mimicking production deployments. Recent evaluations showed that the average attack success rate for extracting protected reasoning from earlier requests was below 1% for GPT-6.2 Astra, significantly lower than for GPT-6 Astra. OpenAI is applying selected improvements to GPT-6 Astra as well.
OpenAI has observed multiple unauthorized distillation campaigns this year, with enforcement actions affecting thousands of accounts. The company tracks the prevalence of distillation abuse and aims to increase the cost of distillation campaigns while reducing response times.
National Security Implications
Adversarial distillation poses national security risks, as foreign actors reproducing advanced capabilities from U.S. models could deploy them without the original protections. OpenAI has briefed Congress on activities indicating attempts to distill U.S. frontier models and supports legislation to enhance the U.S. government’s ability to identify and sanction foreign actors involved in illicit capability extraction.
Guidance for Other Developers
OpenAI emphasizes the importance of supporting the safe proliferation of frontier capabilities through its anti-distillation work. The company suggests several steps for other model developers:
- Protect private reasoning across interactions and check for leaks in assistant messages.
- Maintain logs of successful extraction attacks for regression testing.
- Combine request-level mitigations with campaign-level investigations.
- Coordinate safeguard requirements with deployment partners.
OpenAI continues to share findings through the Frontier Model Forum and appropriate government channels to help strengthen defenses while keeping sensitive details private.
Source: OpenAI News