Abstract
Conversational memory changes during use, so endpoint question answering alone cannot establish how a persistent state accumulates, ages, or incorporates revisions. We introduce LSREP, a Longitudinal State-Replay Evaluation Protocol combining ordered replay, explicit lifecycle schedules, repeated probes, evolving reference answers, and mechanism-fidelity checks. Its architectural case study is ICE v2, a local-first memory middleware with typed stores, retrieval fusion, and dynamic context budgets. The private, single-user instantiation contains 1,985 turns, 219 distinct probes, and 1,211 probe-checkpoint observations across 52 checkpoints. On three ordinary-density datasets, ICE v2 has a near-zero mean quality difference from vector-RAG while selecting 32% fewer fragments but using 6.6% more estimated prompt tokens. A fourth, dense dataset exposes catastrophic failures of the unbudgeted baseline. The fidelity audit limits attribution: procedural retrieval is defective, several mechanisms are unexercised, and graph utility is not established. In a complementary matched public diagnostic, ICE v2 loses decisively to pure vector-RAG on LongMemEval: 50.8% versus 72.8% in the evidence-only oracle and 43.0% versus 69.5% in full-S. Paired differences are -22.0 points (95% CI [-26.6, -17.4]) and -26.5 ([-31.3, -21.8]). Conservative abstention accompanies severe multi-session and temporal failures. ICE uses less context in this diagnostic, establishing a quality-cost trade-off rather than superior efficiency. Together, replay, fidelity auditing, and public endpoint testing expose distinct failure modes that neither architectural descriptions nor aggregate scores identify alone.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Sonar, D. (2026). LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture. https://omanscience.com/en/articles/lsrep-a-longitudinal-state-replay-protocol-for-evaluating-conversational-memory-with-ice-v2-as-an-audited-local-first-architecture
MLA 9
Sonar, Deepesh. "LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture." https://omanscience.com/en/articles/lsrep-a-longitudinal-state-replay-protocol-for-evaluating-conversational-memory-with-ice-v2-as-an-audited-local-first-architecture.
Chicago (author–date)
Sonar, Deepesh. 2026. "LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture." https://omanscience.com/en/articles/lsrep-a-longitudinal-state-replay-protocol-for-evaluating-conversational-memory-with-ice-v2-as-an-audited-local-first-architecture.
Harvard
Sonar, D. (2026) 'LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture', Available at: https://omanscience.com/en/articles/lsrep-a-longitudinal-state-replay-protocol-for-evaluating-conversational-memory-with-ice-v2-as-an-audited-local-first-architecture.
Vancouver
Sonar D. LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture. https://omanscience.com/en/articles/lsrep-a-longitudinal-state-replay-protocol-for-evaluating-conversational-memory-with-ice-v2-as-an-audited-local-first-architecture
IEEE
D. Sonar, "LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture," https://omanscience.com/en/articles/lsrep-a-longitudinal-state-replay-protocol-for-evaluating-conversational-memory-with-ice-v2-as-an-audited-local-first-architecture.