ActionOPD:
Distilling Decisions, Not Tokens

Tsinghua University

* Equal contribution† Corresponding authors

ActionOPD, an action-level on-policy distillation method, transfers decisions from black-box teachers through semantic agreement while the student learns through its own reasoning and responses.

Comparison of knowledge distillation, token-level on-policy distillation, outcome-based reinforcement learning, and ActionOPD. ActionOPD compares student and teacher actions at student-encountered states.
Supervise the action, not its wording. ActionOPD transfers teacher decisions at states encountered by the student.

How ActionOPD works

ActionOPD starts from an objective: maximize the likelihood of teacher actions at student-encountered states. That likelihood is the total probability mass of student responses that realize the same semantic action, rather than reproduce one teacher-written sequence.

  1. Sample decisions at the same state

    Query a black-box teacher for three action samples and generate four candidate responses from the student.

  2. Reward semantic agreement

    Compare tool calls or user-facing messages, not reasoning text. Matching actions receive a reward, with a smaller bonus for additional teacher matches.

  3. Advance once, learn from the group

    Execute a highest-reward action to continue the trajectory. All candidates in retained groups contribute to the policy update.

At one state, four student responses are matched against shared teacher samples. A highest-reward action advances the environment, while the candidate group supplies training supervision.
One environment continuation, multiple candidate responses for learning.

The agreement objective yields a REINFORCE-form gradient. In practice, group-relative policy optimization trains the student's own reasoning and action tokens, without teacher token probabilities or tokenizer alignment.

Transfer beyond the training environments

Training on a curated pool of 6,697 AWM and EnvScaler tasks improves the four-benchmark average in all four student–teacher configurations, by 3.3–6.9 percentage points over the corresponding base models. Tau scores use the first four trials per task (trials 0–3).

Scaling with students and teachers

Larger students reach higher final averages. At both student sizes, DeepSeek-V4-Flash yields higher averages than Qwen3.7-Flash, demonstrating effective cross-family distillation without tokenizer alignment.

Four-benchmark average (%) after 100 training steps
Training teacherQwen3-4BQwen3-8B
Base (no distillation)30.7835.65
Qwen3.7-Flash35.92 +5.1438.97 +3.32
DeepSeek-V4-Flash37.66 +6.8840.03 +4.38
Student–teacher scaling of ActionOPD. Qwen3-8B reaches higher final averages than Qwen3-4B. DeepSeek-V4-Flash produces higher student averages than Qwen3.7-Flash at both sizes.
Student–teacher scaling in the tested configurations. Gains over Base use unrounded scores.

Competitive distillation, a flexible interface

We compare ActionOPD with offline distillation (KD) and sampled-token OPD on τ²-bench, using a Qwen3-4B student and Qwen3-32B teacher. All methods are evaluated after 50 training steps.

On held-out tasks, ActionOPD is close to OPD and ahead of KD on single-trial success (pass¹), but below both on success across all four trials (pass⁴). Its generation-only supervision offers a competitive alternative without teacher token log-probabilities or tokenizer alignment. The DeepSeek-to-Qwen results above separately demonstrate its cross-family applicability.

Domain average (%) · Four trials per task
MethodTrainTest
pass¹pass⁴pass¹pass⁴
Base26.36.929.27.5
KD33.715.737.519.2
OPD34.311.739.617.5
ActionOPD37.913.740.014.2

Semantic supervision matters

Using the same 50-step Qwen3-4B / Qwen3-32B setup, we test three changes to ActionOPD:

  • Validity reward: train on action validity, retaining semantic scores for action selection.
  • Binary reward: remove the frequency bonus from training and action selection.
  • Random commit: keep semantic rewards, but select uniformly from all candidates.

ActionOPD leads in average single-trial success on training and held-out tasks, supporting the combined design. The largest drop comes from replacing semantic agreement with validity rewards.

Domain average (%) · Four trials per task
MethodTrainTest
pass¹pass⁴pass¹pass⁴
Base26.36.929.27.5
Validity reward30.513.230.810.0
Binary reward33.310.536.914.2
Random commit33.29.237.512.5
ActionOPD37.913.740.014.2

What changes in the agent's decisions?

Selected positive cases compare Qwen3-4B Base with the same student after 100 ActionOPD steps using DeepSeek-V4-Flash. Each pair shows a Base failure and an ActionOPD success on the same task and trial.

Following policy constraints

The user requests a flight change, but basic economy bookings cannot be modified.

Successes across four trials: 2/4 → 4/4

Qwen3-4B Base Failure

Reads that the booking is basic economy, yet modifies the flight through the API, violating the policy.

Qwen3-4B + ActionOPD Success

Explains the restriction and calls the transfer tool when the user requests escalation. The booking remains unchanged.

Prohibited modifications fall from 2/4 to 0/4. Success here means preserving a compliant booking, not completing a rebooking.

Tracking the correct target

Update the target order's address and speaker. Retrieve the new address in New York from a different source order.

Successes across four trials: 0/4 → 3/4

Qwen3-4B Base Failure

Treats the source order's New York address as if it already belonged to the target. Changes only the speaker, leaving the target address in Los Angeles.

Qwen3-4B + ActionOPD Success

Retrieves the source address, updates the target order's address, then changes the speaker.

The returned order contains both the New York address and the requested green speaker.

Completing and verifying actions

Restore service after an overdue-bill suspension and an unseated SIM. Payment alone does not reactivate the line.

Successes across four trials: 0/4 → 2/4

Qwen3-4B Base Failure

Promises restoration after payment but never calls resume_line. The user pays and reboots, but service remains disconnected and the agent transfers the case.

Qwen3-4B + ActionOPD Success

Recovers from an invalid tool call, resumes the line with resume_line, and guides the user to reseat the SIM.

User-side tool output confirms connection, and the evaluator verifies that the debt is cleared.

Can an agent supervise itself?

Self-ActionOPD uses the current Qwen3-4B model as both teacher and student, with independently sampled responses. The teacher receives privileged action guidance, private task information, and reference actions or target outcomes. The student uses only public interaction context.

After 50 steps, held-out average pass¹ and pass⁴ improve by 4.8 and 2.5 percentage points. The student retains these benefits when evaluated without privileged context, supporting the feasibility of action-level self-distillation.

Shared model weights Qwen3-4B

TeacherPrivileged task information
StudentPublic context only
Held-out τ²-bench average (%) · Four trials per task
Qwen3-4Bpass¹pass⁴
Base29.27.5
Self-ActionOPD34.010.0

Citation

@misc{yuan2026actionopd,
  title  = {ActionOPD: Distilling Decisions, Not Tokens},
  author = {Yuan, Huining and Yi, Xiangmin and Xu, Zelai and Wang, Zixu
            and Chen, Xinlei and Zhang, Xiao-Ping and Wang, Yu and Yu, Chao},
  year   = {2026},
  eprint = {2610.19820},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2610.19820}
}

Figure

Scroll to inspect the figure. Press Escape to close.