MARSHAL: Incentivizing Multi-Agent Reasoning
via Self-Play with Strategic LLMs

Overview

We introduce MARSHAL, an end-to-end reinforcement learning framework designed to incentivize Multi-Agent Reasoning through Self-play witH strAtegic LLMs in a diverse range of competitive and cooperative games.

MARSHAL addresses the challenge of credit assignment in multi-agent multi-turn self-play through two core mechanisms:

• Turn-level Advantage Estimator: Enables fine-grained credit assignment, allowing the model to accurately attribute long-term outcomes to individual actions.
• Agent-specific Advantage Normalization: Stabilizes the training process by calibrating advantage estimates relative to the performance of each agent.

Key Results

By leveraging self-play across strategic games, MARSHAL (based on Qwen3-4B) demonstrates notable generalization capabilities:

• Strategic Games: Achieves up to 28.7% performance improvement on held-out games.
• Reasoning Benchmarks: When integrated into leading multi-agent systems (MASs), MARSHAL yields consistent gains of up to
  • +10.0% on AIME
  • +7.6%^* on GPQA-Diamond
  • +3.5% on average across all tested benchmarks.

  ^*Note: Due to a typo, this value is reported as 6.6% in the current arXiv preprint (v2). It will be corrected in the camera-ready version.

Evaluation of MARSHAL and two baselines on strategic games and reasoning benchmarks.

Game Replay

Click 'Next Step' to view a self-play demonstration of the MARSHAL agent in Tic-Tac-Toe. Observe the agent's reasoning process regarding game states and strategies as it controls both Player 1 (X) and Player 0 (O).

Method

Algorithmic Design

We introduce two modifications to GRPO [1] for multi-agent self-play: a turn-level advantage estimator for fine-grained credit assignment, and agent-specific advantage normalization to stabilize training across heterogeneous roles.

Overview of MARSHAL. Left column: generating player trajectories through self-play in strategic games. Middle column: naive advantage estimation by GRPO. Right column: advantage estimation by MARSHAL for accurate credit assignment in multi-turn, multi-agent setting.

Game Selection

We use a portfolio of six strategic games, split into training and held-out testing sets to evaluate generalization:

• Perfect-information competitive: For deterministic planning and role adaptation, train on Tic-Tac-Toe; test on Connect Four.
• Imperfect-information competitive: For robust reasoning under uncertainty, train on Kuhn Poker; test on Leduc Hold'em.
• Imperfect-information cooperative: For intent recognition and theory of mind, train on Mini Hanabi; test on Simple Hanabi.

Game selection. From left to right, Tic-Tac-Toe, Kuhn Poker, Hanabi, Connect Four, Leduc Hold'em.

Reward Design

The reward signal consists of three components:

• Intrinsic game rewards: +/-1 for win/loss/draw or chips won; shared reward for cooperative games.
• Action format regularization: Small reward for valid actions; large penalty for invalid ones.
• Response length penalty: Turn-level penalty for verbosity to encourage conciseness.

Main Results

Based on Qwen3-4B [2], we train two model types: specialist agents on single games (Tic-Tac-Toe, Kuhn Poker, Mini Hanabi) and a generalist agent on all three simultaneously.

Strategic Ability

Specialist agents generalize effectively to their more complex, held-out counterparts (e.g., from Tic-Tac-Toe to Connect Four). The generalist model achieves high performance across all settings, with 28.7% improvement on Leduc Hold'em and 22.9% on Simple Hanabi, demonstrating robust skill transfer.

Average normalized game returns.

Generalization to Multi-Agent Systems

We evaluate the transfer of capabilities to reasoning benchmarks within multi-agent systems, including the competitive MAD [3] and cooperative AutoGen [4]. In MAD, the generalist agent improves by 3.51% on average. Notably, it achieves striking gains on challenging benchmarks, including +6.57% on GPQA-Diamond in MAD and +10.00% on AIME in AutoGen.

Evaluation results on downstream reasoning benchmarks within multi-agent systems.

Pattern Analysis

Qualitative

Analysis of reasoning traces reveals emergent multi-agent skills cultivated in games and transferred to general-purpose systems.

Qualitative analysis of emergent reasoning patterns.

Quantitative

Following the taxomony of Cemri et al. (2025) [5], failure mode analysis on GPQA-Diamond shows a 11.5% reduction in Inter-Agent Misalignment. This confirms that MARSHAL agents actively listen to peers and maintain focus, validating game-theoretic self-play cultivates transferable multi-agent reasoning skills.

Percentage of different failure modes in GPQA-Diamond.

Ablation

Self-Play vs. Fixed-Opponent

Training against fixed opponents leads to overfitting and poor generalization. Self-play provides an adaptive curriculum essential for developing robust, generalizable policies.

Generalization comparison between MARSHAL (self-play) and its fixed-opponent variant. For competitive games, entries indicate first-move / second-move returns. Underlined scores indicate performance degradation compared to the standard MARSHAL model.

Analysis of Algorithmic Design

Ablation studies confirm that both the turn-level advantage estimator and agent-specific normalization are critical. Removing either component significantly degrades performance, especially in long-horizon and competitive games.

Ablation results for algorithmic design. For competitive games, entries indicate first-move / second-move returns. Underlined scores indicate performance degradation compared to the standard MARSHAL model.

Scaling to Larger Models

We extended training to the larger Qwen3-8B model. Results demonstrate that MARSHAL scales stably, consistently unlocking cooperative and competitive reasoning capabilities at larger scales.

Strategic Ability (8B)

Strategic ability comparison on the larger Qwen3-8B.

Generalization to MAS (8B)

Generalization to multi-agent systems using the larger Qwen3-8B.

Citation

@misc{yuan2025marshal,
      title={MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs}, 
      author={Huining Yuan and Zelai Xu and Zheyue Tan and Xiangmin Yi and Mo Guang and Kaiwen Long and Haojia Hui and Boxun Li and Xinlei Chen and Bo Zhao and Xiao-Ping Zhang and Chao Yu and Yu Wang},
      year={2025},
      eprint={2510.15414},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2510.15414}, 
}

References

[1] Shao, Zhihong, et al. "Deepseekmath: Pushing the limits of mathematical reasoning in open language models." arXiv:2402.03300.

[2] Yang, An, et al. "Qwen3 technical report." arXiv:2505.09388.

[3] Liang, Tian, et al. "Encouraging divergent thinking in large language models through multi-agent debate." EMNLP 2024.

[4] Wu, Qingyun, et al. "Autogen: Enabling next-gen llm applications via multi-agent conversation." COLM 2024.

[5] Cemri, Mert, et al. "Why do multi-agent llm systems fail?." NeurIPS 2025.