Long-horizon LLM agents · Context compression · Preference optimization

DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents

Tianle Chen1,2, Pushkar Singh1,†, Ketan Todi1, Biswajit Paria1, Qijia He1,3, Wenqi Wang1,2, Yuwen Tan1,2, Jike Zhong4, Sugato Basu1, Deepti Ghadiyaram2, Arjun Akula5,†
1Google 2Boston University 3University of Washington 4University of Southern California 5Google DeepMind †Project lead
Left: a compressed agent repeats FindUnvisited(3,0) and gets 3,0 back each time; the repeated call becomes the rejected action and the agent's later MarkVisited(3,0) becomes the chosen action, and DPO trains the agent. Right: on CodeGym, DeLoop has far fewer tasks with loops and fewer steps per task than the compressed baseline and the full-history agent.
Training an agent on its own loops breaks the loop. Left: a repeated search and the later action that marks the server as visited form a preference pair, and DPO trains the agent to prefer the recovery. Right: Qwen3-8B on CodeGym; DeLoop and the baseline summarize at 4,096 tokens, and full history is the uncompressed reference.Swipe to see the whole figure.

TL;DR An agent that repeats a tool call and gets the same observation back has given us a free training pair: the repeat is the negative, and its own later recovery is the positive. Training on these self-mined pairs cuts loops by 9.4× on Qwen3-8B, needs about one third of the steps, and raises task success on Qwen3-14B and Qwen3-32B by about 7 points.

Abstract

Language-model agents that run long tasks often redo work they have already done: they re-issue a tool call they made earlier and get back exactly the same observation, a step that adds no new information. We identify this failure as a loop. Loops appear natively in Qwen, Llama, and Gemini agents and become far more common once the agent's history is compressed to fit its context window.

To reduce these loops, we propose DeLoop, a self-supervised training method. DeLoop builds on a simple observation: the agent's own rollout can provide the supervision needed to learn from its loops. We use a looping action as a negative example and the first later non-looping action in the same rollout as a positive example. DeLoop learns from these pairs through direct preference optimization and does not require task rewards, gold answers, or LLM judges.

On CodeGym under compression, DeLoop reduces the share of Qwen3-8B tasks containing loops from 57.3% to 6.1%. It also uses about one third as many steps per task, with similar task success. On Qwen3-32B, DeLoop reduces loops and steps while raising task success from 37.7% to 44.8%.

What Is a Loop?

A loop occurs when the agent repeats a previous tool call with identical arguments and receives the same observation, so the step adds no new information. Play the example below to see how compression hides an earlier call, how the agent repeats it, and how DeLoop turns the repeat into a training pair.

Agent rollout · CodeGym, count clusters of online servers
    Context window
    0 / 4,096 tokens
    Tool calls
    0
    Looping actions
    0

    The baseline rollout follows the real Qwen3-8B episode in the paper's appendix (earlier turns abridged). The DeLoop rollout is a schematic of the trained behavior on the same state.

    Loops Are Costly, and Compression Makes Them More Common

    Task failure rate with and without a loop for Qwen3-8B, Qwen3-14B, and four Gemini models; runs with a loop fail more often in every model.
    Runs that enter a loop fail more often in every model we evaluate (task failure rate, CodeGym under compression).
    (a) Qwen3-8B loop rate rises from 29.0% with no compression to 57.3% at a 4,096-token summarization budget and 64.7% at 2,048. (b) Three Gemini models loop more when compression is triggered.
    (a) Qwen3-8B loops more as the summarization budget shrinks. (b) Gemini models loop more once compression is triggered.

    Loops waste interaction. On CodeGym under summarization at 4,096 tokens, looping Qwen3-8B runs contain 6.5 looping actions on average, execute 10.5 more actions per task than loop-free runs, and carry 11k more tokens. On average over six Qwen3 and Gemini models, 59.5% of looping runs fail, compared with 33.4% of loop-free runs.

    Compression makes them more common. Qwen3-8B loops in 29.0% of runs with the full history, 57.3% when summarizing at 4,096 tokens, and 64.7% at 2,048 tokens. When we remove the record of the most recent completed tool call from 300 compressed histories, the agent repeats that call in 48.8% of samples, against 3.3% with the record present.

    Better context management does not close the gap. Summaries that keep an explicit trace or ledger of past actions still loop in 52.0% to 62.7% of runs, and the ACON and LLMLingua-2 compressors reach 46.3% and 46.7%, all well above the 29.0% with the full history. The agent often fails to use a record it has, so we train the policy itself.

    Method: Repeated Actions as Self-Supervision

    DeLoop runs the compressed agent on training tasks, finds its loops with a structural detector, and turns each loop it later recovers from into a preference pair.

    1. A. Detect loops. A step is a repeat if its canonicalized tool call and observation already appeared earlier in the rollout. A repeated call with a new observation is not a loop.
    2. B. Mine pairs. The looping action is the rejected response . The first later action that is not a repeat and differs from it is the chosen response . Both are paired with the history before the loop, . A loop the agent never recovers from gives no pair.
    3. C. Train with DPO. The policy learns to prefer the recovery over the repeat under the same context, with the frozen base policy as the reference.
    An agent rollout where FindUnvisited(3,0) returns 3,0 three times; the repeated call becomes the rejected action and the later MarkVisited(3,0) becomes the chosen action; the pair trains DeLoop with DPO.
    Mining a preference pair from one rollout.

    Pair construction uses only the actions and observations: no task reward, gold label, or LLM judge. A recovery gives a training pair even when the rollout ultimately fails the task.

    Results

    Main comparison on CodeGym

    Agent Qwen3-8B · Setting summarization at 4,096 tokens · Tasks 300 held-out

    DeLoop cuts the loop rate 9.4×, from 57.3% to 6.1%, the lowest of all methods. Steps per task fall from 15.00 to 4.67 and tokens per task by 47%, while task success holds. Adding cross-domain pairs (DeLoop-Mix) raises task success from 25.1% to 30.3%.

    • DeLoop (ours)
    • Compressed baseline
    • Full-history reference
    • Other methods
    Full table
    MethodSupervision (labels needed)Loop (%) ↓Success (%) ↑Steps ↓Tokens/task (k) ↓

    Qwen3-8B on CodeGym under summarization at 4,096 tokens unless noted. Steps and estimated tokens are per-task averages. Parentheses: change from the compressed baseline.

    Scaling to larger models

    Agents Qwen3-8B, Qwen3-14B, Qwen3-32B · Pairs mined from each model's own rollouts

    Loops fall at every scale, 3.5× on Qwen3-14B and 3.0× on Qwen3-32B, and tokens per task fall by 35% and 29%. At these scales the repair also raises task success: from 45.1% to 52.3% on Qwen3-14B and from 37.7% to 44.8% on Qwen3-32B.

    Three panels: task success versus loop rate, steps per task, and tokens per task for Qwen3-8B, 14B, and 32B; DeLoop (stars) moves each model to fewer loops, fewer steps, and fewer tokens than its compressed baseline (squares).
    Task success versus loop rate, steps, and estimated tokens per task. Squares: compressed baseline; stars: DeLoop; colors: model size.Swipe to see the whole figure.

    Cross-domain data, context settings, and memory policies

    DeLoop-Mix adds 4,635 cross-domain pairs from ALFWorld, Open-AgentRL, and Terminal-Bench, mined with the same label-free rule. It raises success both with and without compression, and the loop repair holds across five context settings and seven memory policies, including summaries augmented with an action ledger (62.7% → 11.3% loops).

    (a) Task success with and without compression for the base agent, DeLoop, and DeLoop-Mix. (b) Loop rate across five context settings: DeLoop stays low while the base agent rises. (c) Loop rate across seven memory policies: DeLoop-Mix is far lower under every policy.
    (a) Task success with and without compression. (b) Loop rate across context settings. (c) Loop rate across memory policies.Swipe to see the whole figure.

    Other agent environments and benchmarks

    With each environment's native loop and success metrics, loops fall in all four further environments, by up to 32.8 points on AppWorld, and tokens per task fall wherever recorded. In native harnesses, DeLoop cuts unproductive duplicate calls on Terminal-Bench from 16.6 to 6.9 per task, and DeLoop-Mix lowers the BrowseComp-Plus loop rate from 22.4% to 1.2% while matching the untrained model's accuracy.

    EnvironmentLoop (%) ↓Success (%) ↑Tokens (k) ↓
    BaseOursBaseOursBaseOurs
    ALFWorld55.227.637.338.19.36.2
    Open-AgentRL79.073.543.945.5––
    AppWorld73.740.97.05.817.914.1
    ScienceWorld96.583.53.51.56.56.0

    Base: untrained Qwen3-8B. Ours: DeLoop-Mix. Native metrics per environment; bold blue marks an improvement.

    What the training signal does

    • The looping action matters. Imitating only the loop-breaking action leaves a loop rate of 37.7%; contrasting it with the looping action brings it to 6.1%.
    • The signal works across objectives. KTO, IPO, and online GRPO on the same signal all cut the loop rate to 15.0% or below; DPO gives the lowest loop rate and the highest task success.
    • The remaining repeats are informative. After training, the share of repeated calls that return new information rises from 29.9% to 69.8%.
    • General ability is preserved on LiveCodeBench, MT-Bench, and IFEval.

    Conclusion

    Long-horizon agents that compress their history fall into loops: they re-issue an action they already ran and receive the same result. The loop labels itself: the looping action is the rejected action, and the agent's later loop-breaking action is the chosen one. Trained on these self-mined pairs, DeLoop cuts the loop rate 9.4× at about one third of the steps per task while maintaining task success, and raises task success by 7.2 points at 14B and 7.1 points at 32B. An agent's own loops become its supervision: break the loop, keep the progress.

    Citation

    @article{chen2026deloop,
  title   = {DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents},
  author  = {Chen, Tianle and Singh, Pushkar and Todi, Ketan and Paria, Biswajit and
             He, Qijia and Wang, Wenqi and Tan, Yuwen and Zhong, Jike and
             Basu, Sugato and Ghadiyaram, Deepti and Akula, Arjun},
  journal = {arXiv preprint arXiv:2610.19376},
  year    = {2026}
}