MARSHAL: Incentivizing Multi-Agent Reasoning
via Self-Play with Strategic LLMs

Overview

We introduce MARSHAL, an end-to-end reinforcement learning framework designed to incentivize Multi-Agent Reasoning through Self-play witH strAtegic LLMs in a diverse range of competitive and cooperative games.

MARSHAL addresses the challenge of credit assignment in multi-agent multi-turn self-play through two core mechanisms:

• Turn-level Advantage Estimator: Enables fine-grained credit assignment, allowing the model to accurately attribute long-term outcomes to individual actions.
• Agent-specific Advantage Normalization: Stabilizes the training process by calibrating advantage estimates relative to the performance of each agent.

Key Results

By leveraging self-play across strategic games, MARSHAL (based on Qwen3-4B) demonstrates notable generalization capabilities:

• Strategic Games: Achieves up to 28.7% performance improvement on held-out games.
• Reasoning Benchmarks: When integrated into leading multi-agent systems (MASs), MARSHAL yields consistent gains of up to
  • +10.0% on AIME
  • +7.6% on GPQA-Diamond
  • +3.5% on average across all tested benchmarks.

Figure 1. Evaluation of MARSHAL and two baselines on strategic games and reasoning benchmarks.

Game Replay

Click 'Next Step' to view a self-play demonstration of the MARSHAL agent in Tic-Tac-Toe. Observe the agent's reasoning process regarding game states and strategies as it controls both Player 1 (X) and Player 0 (O).

Method

Algorithmic Design

We introduce two modifications to GRPO for multi-agent self-play: a turn-level advantage estimator for fine-grained credit assignment, and agent-specific advantage normalization to stabilize training across heterogeneous roles.

Figure 2. Overview of MARSHAL. Left column: generating player trajectories through self-play in strategic games. Middle column: naive advantage estimation by GRPO. Right column: advantage estimation by MARSHAL for accurate credit assignment in multi-turn, multi-agent setting.

Game Selection

We use a portfolio of six strategic games, split into training and held-out testing sets to evaluate generalization:

• Perfect-information competitive: For deterministic planning and role adaptation, train on Tic-Tac-Toe; test on Connect Four.
• Imperfect-information competitive: For robust reasoning under uncertainty, train on Kuhn Poker; test on Leduc Hold'em.
• Imperfect-information cooperative: For intent recognition and theory of mind, train on Mini Hanabi; test on Simple Hanabi.

Figure 3. Game selection. From left to right, Tic-Tac-Toe, Kuhn Poker, Hanabi, Connect Four, Leduc Hold'em.

Reward Design

The reward signal consists of three components:

• Intrinsic game rewards: +/-1 for win/loss/draw or chips won; shared reward for cooperative games.
• Action format regularization: Small reward for valid actions; large penalty for invalid ones.
• Response length penalty: Turn-level penalty for verbosity to encourage conciseness.

Main Results

Based on Qwen3-4B, we train two model types: specialist agents on single games (Tic-Tac-Toe, Kuhn Poker, Mini Hanabi) and a generalist agent on all three simultaneously.

Strategic Ability

Specialist agents generalize effectively to their more complex, held-out counterparts (e.g., from Tic-Tac-Toe to Connect Four). The generalist model achieves high performance across all settings, with 28.7% improvement on Leduc Hold'em and 22.9% on Simple Hanabi, demonstrating robust skill transfer.

Figure 4. Average normalized game returns.

Generalization to Multi-Agent Systems

We evaluate the transfer of capabilities to reasoning benchmarks within multi-agent systems, including the competitive MAD and cooperative AutoGen. In MAD, the generalist agent improves by 3.51% on average. Notably, it achieves striking gains on challenging benchmarks, including +6.57% on GPQA-Diamond in MAD and +10.00% on AIME in AutoGen.

Table 1. Evaluation results on downstream reasoning benchmarks within multi-agent systems.

Pattern Analysis

Qualitative

Analysis of reasoning traces reveals emergent multi-agent skills cultivated in games and transferred to general-purpose systems.

Table 2. Qualitative analysis of emergent reasoning patterns.

Quantitative

Following the taxomony of Cemri et al. (2025), failure mode analysis on GPQA-Diamond shows a 11.5% reduction in Inter-Agent Misalignment. This confirms that MARSHAL agents actively listen to peers and maintain focus, validating game-theoretic self-play cultivates transferable multi-agent reasoning skills.

Figure 5. Percentage of different failure modes in GPQA-Diamond.

Ablation

Self-Play vs. Fixed-Opponent

Training against fixed opponents leads to overfitting and poor generalization. Self-play provides an adaptive curriculum essential for developing robust, generalizable policies.

Table 3. Generalization comparison between MARSHAL (self-play) and its fixed-opponent variant. For competitive games, entries indicate first-move / second-move returns. Underlined scores indicate performance degradation compared to the standard MARSHAL model.

Analysis of Algorithmic Design

Ablation studies confirm that both the turn-level advantage estimator and agent-specific normalization are critical. Removing either component significantly degrades performance, especially in long-horizon and competitive games.

Table 4. Ablation results for algorithmic design. For competitive games, entries indicate first-move / second-move returns. Underlined scores indicate performance degradation compared to the standard MARSHAL model.

Scaling to Larger Models

We extended training to the larger Qwen3-8B model. Results demonstrate that MARSHAL scales stably, consistently unlocking cooperative and competitive reasoning capabilities at larger scales.

Strategic Ability (8B)

Table 5. Strategic ability comparison on the larger Qwen3-8B.

Generalization to MAS (8B)

Table 6. Generalization to multi-agent systems using the larger Qwen3-8B.

BibTeX

@article{yuan2025marshal,
  title={MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs},
  author={Yuan, Huining and Xu, Zelai and Tan, Zheyue and Yi, Xiangmin and Guang, Mo and Long, Kaiwen and Hui, Haojia and Li, Boxun and Chen, Xinlei and Zhao, Bo and others},
  journal={arXiv preprint arXiv:2510.15414},
  year={2025}
}