The Hidden Risk in Long-Running AI Sessions

Long-running sessions with AI systems feel productive because the model retains the objective, earlier decisions, failed attempts, and the language used to describe the problem, which saves users from repeating themselves and gives the work a sense of continuity.

That continuity has a cost because the same history also preserves mistakes, assumptions, and behavioral tendencies; once a model starts down the wrong path, later prompts may not be enough to make it reconsider the frame it has already adopted.

A recent paper, Old Habits Die Hard: How Conversational History Geometrically Traps LLMs, offers evidence for this kind of persistence.[1] Across multi-turn conversations, the authors found that hallucination, sycophancy, and refusal affected the probability of the same behavior appearing again, with the strongest effects in semantically coherent conversations (the normal condition for sustained work on a codebase, research project, or business problem).

The paper does not directly test software engineering, project management, or autonomous agents, and it does not establish that a coding model becomes irreversibly trapped after making a bad design decision. Its experiments use simulated conversations, specific behavioral categories, and controlled evaluation, so applying the result to long-running AI work remains an inference (although a consequential one).

Part I: Conversation is a poor place to keep state

In a conventional software system, important state has a type, an owner, and a lifecycle: databases distinguish active accounts from closed ones, version-control systems distinguish current revisions from predecessors, and workflow engines distinguish proposals from approvals.

Conversation history makes few such distinctions. User statements, model guesses, corrected facts, abandoned plans, and verified results share one stream, leaving the model to infer their authority and relevance while their current validity may depend on whether a later correction survived summarization or context truncation.

This works surprisingly well in short interactions but becomes harder to trust as a session grows.

Consider a coding agent that attributes a failing test to a dependency mismatch early in a session, then spends several turns interpreting logs through that theory, changing configuration, and explaining why the change is appropriate. Even if the diagnosis is wrong, it has become the organizing frame through which later evidence is read.

The risk is broader than factual error. A model may become overly deferential to a user's initial theory, reluctant to revisit a decision it previously defended, or disposed to refuse an action after interpreting similar actions as unsafe. These behaviors differ, but they share a dependence on conversational history.

The obvious response is to move durable state out of the conversation. Goals, decisions, constraints, evidence, open questions, and current work can be stored as structured records. A fresh model session can then receive a compact account of the present situation instead of a transcript containing every step that produced it.

This interrupts direct conversational carryover, removes much of the model's earlier rhetoric, allows corrections to replace obsolete conclusions, makes goals and constraints explicit, and creates an audit trail. External memory also introduces a second risk: if a wrong conclusion is extracted from conversation and stored as a durable fact, the system has converted a temporary error into infrastructure, allowing every fresh session to receive the same false premise in a cleaner, shorter, and more authoritative form.

Structured memory is useful only if it preserves the distinctions that conversation tends to blur, particularly the following:

what was observed;
what was inferred;
what remains uncertain;
what has been verified;
who had authority to decide;
which evidence supports or contradicts a claim;
what has been corrected or superseded.

The relevant design objective is controlled reconstruction of state, rather than better recall in the abstract.

That shift changes how an AI session should begin and end: instead of inheriting an entire conversational past, the model receives current goals, constraints, approved decisions, active commitments, unresolved conflicts, relevant evidence, and known uncertainty, then returns proposed changes and evidence for the system to decide what becomes durable and under what status.

Conversation remains useful as the working surface where questions are refined and ideas are tested, and it can remain part of the historical record, but it should not quietly become the authoritative representation of what an organization knows or intends.

Part II: Technical deep dive

The technical problem is a separation between working context and durable state. An AI model consumes a sequence of tokens in which instructions, evidence, earlier answers, corrections, and incidental language all appear as context; a memory system can impose stronger semantics before that information returns to the model.

State reconstruction

A packet-driven session reconstructs context from current records, including mandatory material and other evidence selected for its relevance to the requested action.

Governing goals, constraints, active directives, commitments, and blockers should not compete in a similarity ranking because a semantically weak match does not make a constraint optional. Mandatory state should be selected deterministically and included completely; when it cannot fit, the system should report the failure rather than emit a plausible but incomplete packet.

Relevant evidence can be found through lexical search, semantic similarity, graph traversal, temporal filtering, and ranking, but these mechanisms establish neither authority nor truth. Retrieval rank may explain why a record was selected without saying whether it deserves belief.

Authority, provenance, confidence, and epistemic status

Memory systems often collapse four different ideas. Authority describes who may issue an instruction or approve a decision: a project owner may approve a goal and a security policy may prohibit an action, while a retrieved document cannot grant itself either power.

Provenance records whether information came from a person, document, API response, test run, model output, or earlier record, making an assertion traceable without making it correct.

Confidence is an estimate (often supplied by the same model that produced the claim) that can guide investigation but provides weak evidence of correctness.

Epistemic status records what kind of validation a claim has received, distinguishing claims that are unassessed, hypothesized, observed, corroborated, verified, disputed, or refuted. These states need explicit transition rules so a model cannot promote its own conclusion by stating it confidently or repeating it in several forms.

The distinction between authority and epistemic status is especially important. An owner can authoritatively choose a business goal without proving a factual proposition. A test runner can provide strong evidence about a specific build without gaining authority to set project goals.

Claims and evidence lineage

Validation works better when the unit is an atomic claim rather than a document-sized memory.

A meeting summary may contain a verified date, an uncertain forecast, an approved decision, and an unresolved assumption; assigning one status to the whole summary hides those differences, whereas claim-level references let corrections and contradictions attach to the proposition they affect.

Evidence counts also require care because five summaries derived from one conversation are not five independent sources, and two agents reading the same incorrect record do not corroborate it by agreeing. Repetition can create the appearance of consensus while preserving a single evidentiary lineage.

A defensible memory system tracks ancestry by retaining links from derived records to source lineages and making corroboration depend on qualifying independent evidence rather than the number of rows returned by a search. Reporting both record count and lineage count helps prevent a model from mistaking paraphrase volume for confirmation.

Corrections and dependency propagation

An append-only history preserves what was previously believed and why, while the active view still requires clear correction semantics.

When a claim is corrected, dependent summaries should become stale, decisions and procedures that materially relied on it may require review, and context packets derived from it should no longer authorize new writes.

This requires dependency tracking. A decision should identify the claims, goals, and evidence on which it relies. Some dependencies can be declared by the producer, but the system should also capture them from the context the producer actually consumed. Otherwise, omitting a weak premise becomes an easy way to prevent later corrections from reopening the decision.

Propagation must be conservative and operationally bounded because large dependency graphs cannot be traversed indefinitely in a synchronous write, although a correction should never be rejected merely because it has too many dependents. A practical design records the correction first, places a fail-closed barrier on affected state, and completes propagation through resumable work.

Counterevidence retrieval

Most retrieval systems are optimized to find supporting context, so a query and its semantically similar results tend to share vocabulary and framing (useful for relevance, but dangerous for verification).

For each claim used to guide an action, context construction should include a separate challenge path. It should retrieve known contradictions, corrections, superseded versions, failed verification attempts, alternative hypotheses, and invalidation conditions. Structural relationships should be examined before semantic search; a weakly similar correction may be more important than a highly similar supporting note.

An empty challenge result must be described precisely: “No counterevidence found in the completed search over this snapshot” is defensible, while “No counterevidence exists” usually is not.

Counterevidence also creates a denial-of-service problem because, if any untrusted statement can mark a claim disputed, an attacker can block work by flooding the system with contradictions. Trust and validation method should therefore determine how a contradiction changes state; low-trust allegations should remain visible and may block sensitive use, but they should not silently rewrite the status of a verified claim.

Session boundaries and independent verification

Fresh sessions reduce direct carryover from earlier model behavior and are most valuable when the governing frame should be reconsidered (after a material correction, failed verification, goal conflict, or major design reversal).

Session resets are not a complete defense because a concise packet can reproduce the same bias when it contains a false premise or one-sided evidence; the quality of the reconstruction matters more than the absence of transcript history.

Independent verification can provide another boundary. A verifier should receive the goals, acceptance criteria, claimed result, primary artifacts, assumptions, and counterevidence, then derive applicability and evaluate claims without the producer's conversational history or persuasive chain of reasoning.

Independence must be stated honestly: a second session run by the same principal over the same evidence is useful procedural review, but it is not an independently authenticated authority. Stronger claims require separate identities, controlled capabilities, and evidence that the verifier did not simply inherit the producer's framing.

Token reduction without commitment loss

Reducing context can lower cost and latency, but the more important benefit is selective attention. A smaller packet gives the model less irrelevant history to interpret, which may reduce both distraction and the opportunity for earlier rhetoric to shape later work. The result depends entirely on what the compression process removes.

Recent work illustrates several approaches. Memori converts dialogue into semantic triples and conversation summaries, reporting 81.95 percent accuracy on LoCoMo with 1,294 tokens per query (about five percent of full context and 67 percent fewer tokens than the competing approaches in its evaluation).[3] Those numbers are promising, although they come from a nine-page white paper, one benchmark, and an LLM-as-judge setup whose grading prompt is deliberately generous. Its instruction to resolve contradictions by preferring the most recent memory also works for some temporal updates but is too weak for decisions whose correctness depends on authority, scope, or evidence rather than recency alone.

The Deterministic Memory Framework takes a different route by removing generative summarization from memory preparation. It uses deterministic scoring, classical NLP signals, embeddings, and structured recall, reporting higher overall LLM-as-judge scores than Mem0 on its LoCoMo run while using zero tokens for memory preparation and substantially fewer tokens over a conversation.[4] Its LongMemEval evaluation is only a small random sample, and its benchmark still uses an LLM to judge answers, so the results support feasibility rather than a general conclusion that deterministic pruning preserves every important commitment.

Context-Agent addresses the shape of history rather than only its length. By organizing dialogue as a dynamic tree and retrieving the active branch, it reports higher task-completion rates than full-history and truncation baselines while reducing average context tokens by roughly 46 to 52 percent across the models in its benchmark. On TopiOCQA it used about 57 percent of full-history tokens and improved exact match and F1, although the context-management machinery added about eight percent response latency in the reported environment.[5]

Compress the Context, Keep the Commitments provides the clearest framework for interpreting these results. It treats goals, constraints, decisions, safety boundaries, and other state as typed semantic commitments, then evaluates compression through critical-atom recall, weighted recall, commitment density, and recoverability.[2] Its own experiment is small and its token gains are described as provisional, but the underlying standard is sound: a compressed context should be judged by whether it preserves the commitments that determine correctness, not by compression ratio alone.

For long-running agent sessions, token reduction should therefore be evaluated on three axes: the cost of preparing memory, the tokens supplied to the working model, and the downstream effect of omitted or mutated commitments. A compact packet that drops a governing constraint is worse than an expensive transcript, while a packet that preserves mandatory commitments, relevant evidence, and known contradictions can reduce cost and improve the conditions under which the model reasons. The appropriate target is verified commitment density rather than the smallest possible prompt.

Evaluation

A useful evaluation should test the architecture against the failure it is intended to reduce by comparing full conversation replay, the harness's normal truncation or summarization, a basic structured packet, a packet with explicit counterevidence, and a packet followed by independent verification while holding the downstream model, task sequence, and evidence constant.

The scenarios should contain planted mistakes: an incorrect conclusion that is repeated before correction; several records derived from one source; a decisive but weakly similar contradiction; a stale decision based on a corrected premise; and a concise structured packet containing a false assertion.

Relevant measures include erroneous-claim survival across later actions, correction adoption, unsupported promotion, counterevidence recall, stale-decision inclusion, goal and constraint recall, and final task correctness.

Availability must be measured alongside correctness because a system that blocks every action will have an excellent unsupported-claim rate. Correct-action block rate, packet-emission rate, false disputes, and required human resolutions reveal whether fail-closed behavior has become unusable.

The central evaluation question is straightforward: after the system encounters an error, does that error become less likely to influence later decisions? Token reduction and retrieval accuracy matter, but they are secondary if the memory layer preserves incorrect conclusions more effectively than conversation did.

A different role for conversation

The paper on behavioral persistence does not prove that long-running AI coding sessions inevitably deteriorate, but it identifies a mechanism that should affect how persistent agent systems are designed: coherent history influences later behavior, including behavior we would prefer not to preserve.

Moving state into structured memory reduces one source of path dependence while raising the standard for what memory must represent. A durable record needs authority boundaries, provenance, claim status, evidence ancestry, correction semantics, and a way to surface challenges in addition to text and an embedding.

The goal is to keep an AI system's past available without allowing that past to acquire authority merely because it remains in context.

References

Adi Simhi, Fazl Barez, Martin Tutek, Yonatan Belinkov, and Shay B. Cohen, “Old Habits Die Hard: How Conversational History Geometrically Traps LLMs”, arXiv:2603.03308v2, 2026.
Natalia Trukhina and Vadim Vashkelis, “Compress the Context, Keep the Commitments: A Formal Framework for Verifiable LLM Context Compression”, arXiv:2605.17304v1, 2026.
Luiz C. Borro, Luiz A. B. Macarini, Gordon Tindall, Michael Montero, and Adam B. Struck, “Memori: A Persistent Memory Layer for Efficient, Context-Aware LLM Agents”, arXiv:2603.19935v1, 2026.
Matteo Stabile and Enrico Zimuel, “DMF: A Deterministic Memory Framework for Conversational AI Agents”, arXiv:2606.03463v1, 2026.
Junan Hu, Shudan Guo, Wenqi Liu, Jianhua Yin, and Yinwei Wei, “Context-Agent: Dynamic Discourse Trees for Non-Linear Dialogue”, arXiv:2604.05552v2, 2026.