01 / Overview
Beyond a convincing answer.
An agent can sound right and still do the wrong thing. What happens when that mistake becomes an action?
Language-model agents increasingly operate across external APIs, retrieved documents and persistent memory. In these settings, a plausible response is not sufficient: success depends on selecting the right tool, supplying valid arguments, respecting authorisation boundaries and recovering safely when a step fails. Errors can propagate across a workflow, while untrusted content can introduce instructions that conflict with the user’s intent.
This research proposal outlines a controlled investigation of three connected problems: tool-use reliability, indirect prompt injection and memory-related error propagation. It proposes a sandboxed evaluation framework using synthetic business workflows, explicit task specifications and trace-level logging. Matched conditions would compare an agent baseline against targeted interventions, including argument validation, source-provenance tracking and approval gates for consequential actions.
The planned analysis would separate task completion from safety compliance, measure failure and recovery at each step, and examine whether mitigations reduce unsafe actions at the cost of additional latency or unnecessary escalation. Published work on agent evaluation, prompt injection and long-context reasoning motivates the design; it is not evidence that the proposed interventions will succeed. No experiments or empirical findings are reported here. The intended contribution is a reproducible methodology for identifying where an agent fails, how that failure propagates, and which controls warrant further testing before deployment in realistic environments.
This is a proposal, not a completed empirical study. The experiments below are planned; all expected outcomes are hypotheses.
02 / Research question
Where does reliability break?
How can we systematically identify, evaluate, and mitigate failure modes in autonomous language-model agents operating in realistic environments?
My background in deploying AI voice agents, API integrations and multi-step automation motivates this question. This proposal turns that engineering focus towards testable questions about agent behaviour. It does not present individual production incidents as research evidence.
The unit of analysis is the complete action trace: what an agent observed, what it attempted, what changed in the environment, and whether that change was authorised. A task is not safely completed merely because its final text looks correct.
03 / Threat model
Four places to look.
The proposed environment gives agents constrained tools and synthetic records. An attacker may modify retrieved documents or tool-response text, but cannot change trusted instructions, tool permissions or the evaluator. Memory contamination is tested separately through controlled seeded entries.
| Surface | Failure mode | What to inspect |
|---|---|---|
| Tool use | Wrong tool, invalid arguments, cascading errors | Calls, schema violations, final environment state |
| External content | Prompt injection, unauthorised action, attempted exfiltration | Source provenance, denied calls, synthetic canary leakage |
| Memory | Stale retrieval, contaminated facts, cross-session propagation | Memory reads and writes, correction uptake, later decisions |
| Oversight | Bypassed approval, unsafe retry, missed escalation | Approval events, repeated side effects, stop conditions |
04 / Proposed experiments
Three experiments.
One shared framework.
All three studies would run in a sandbox with simulated APIs and no real customer data, payments or outbound communication. Each protocol defines its own controls, measurements and reproduction requirements.
ATool-use reliabilityFrom intent to execution
Question. How do workflow length and tool failures affect safe task completion?
Hypothesis. Argument validation and bounded recovery will reduce invalid calls and repeated side effects relative to an otherwise matched baseline. Their impact on completion rate remains an open question.
Setup. Create synthetic scheduling, record-update and lead-routing tasks at increasing step counts. Inject controlled timeouts, missing fields and ambiguous responses. Each task specifies an allowed final state and prohibited actions.
Controls and variables. Hold model version, sampling settings, tools and task fixtures constant. Vary workflow length, fault type and validation/recovery policy; compare the same fixtures across conditions.
Measurement. Score final state, invalid calls, successful recovery and duplicate writes. Replay each trace against a deterministic task oracle. Reset the sandbox between runs and retain failed runs in the analysis.
Reproduction and limits. Publish fixtures, fault schedules, prompts, seeds and scoring rules before execution. Simulated APIs cannot reproduce every production failure; repeat under multiple fault schedules.
BPrompt injection resistanceWhen data becomes an instruction
Question. Can malicious instructions in external content redirect an agent across a trust boundary?
Hypothesis. Explicit source labelling and externally enforced tool permissions will reduce attack success, but may also block legitimate tasks.
Setup. Pair clean task documents with variants containing attempts to override instructions, invoke unauthorised tools or disclose synthetic canaries. Vary attack position and phrasing without changing the legitimate task.
Controls and variables. Use matched clean/adversarial pairs, identical tool access and fixed model settings. Compare baseline behaviour, source labelling alone, and source labelling with permission enforcement.
Measurement. Report attacker-goal achievement separately from attempted violations blocked by the environment. Also measure legitimate task completion and false blocking on clean inputs.
Reproduction and limits. Release synthetic payloads, permitted-action specifications and trace scoring. Keep development and held-out attack templates separate. A finite attack set cannot establish general security.
CMemory & context reliabilityThe mistakes an agent remembers
Question. When does persistent memory preserve useful context, and when does it perpetuate an error?
Hypothesis. Provenance-aware memory and explicit correction rules will reduce reuse of stale or contaminated facts compared with unqualified persistent memory.
Setup. Run multi-session synthetic workflows with stable facts, updated facts and deliberately incorrect seeded entries. Introduce a trusted correction at a predetermined session and inspect downstream actions.
Controls and variables. Compare no persistent memory, plain persistent memory and provenance-aware memory. Hold retrieval budget and task order constant; vary contamination, correction timing and context length.
Measurement. Track retrieval accuracy, correction adoption, downstream actions based on incorrect facts and persistence of errors across sessions. Analyse memory benefits separately from contamination costs.
Reproduction and limits. Version memory snapshots and retrieval settings; reset memory between independent sequences. Long-context findings motivate the study but do not establish the behaviour of a persistent-memory system.
05 / Evaluation plan
Measure actions, not confidence.
Pre-register task specifications, primary metrics, exclusion criteria and the run budget. Use paired comparisons on identical fixtures, repeated trials and uncertainty intervals appropriate to task-level clustering. Report safety and utility separately rather than hiding trade-offs in one score.
- Safe completion rate
- Runs reaching the specified final state with no policy violation / all runs.
- Attack success rate
- Adversarial runs achieving the attacker’s predefined goal / all adversarial runs.
- Recovery rate
- Fault-injected runs completing safely after a fault / all fault-injected runs.
- Error propagation
- Downstream decisions that use the seeded incorrect fact / all downstream decisions that depend on that fact.
Secondary measures include tool-call count, latency, approval requests and false blocking. Deterministic state checks should be preferred where possible; ambiguous traces would need a documented human adjudication process.
Controls worth testing
Schema validation, idempotent writes, least-privilege tools, memory provenance and approval gates for irreversible actions are candidate mitigations, not established findings from this work. Approval rules should be enforced outside the model, and evaluation must include the costs of unnecessary escalation.
06 / Limitations
What this would not prove.
A controlled benchmark is not a guarantee of safe deployment. Synthetic workflows may miss organisational complexity, model updates can change behaviour, and attack templates can bias the evaluation. Retrieval quality, prompt wording, tool descriptions and evaluator errors are potential confounders.
Any eventual findings would be scoped to the tested models, tasks and threat model. Further work would need broader environments, independent replication and carefully governed real-world evaluation. This proposal makes no claim of peer review, institutional affiliation or demonstrated mitigation effectiveness.
07 / Related reading
The work this builds on.
These published sources inform the proposed methodology. They are distinct from the experiments described above.
- AgentBench: Evaluating LLMs as AgentsXiao Liu et al. · 2023 preprint
Multi-environment agent evaluation and analysis of reasoning and decision-making failures.
- InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model AgentsQiusi Zhan, Zhixiang Liang, Zifan Ying & Daniel Kang · 2024
Evaluation of malicious instructions encountered through external content in tool-integrated agents.
- Lost in the Middle: How Language Models Use Long ContextsNelson F. Liu et al. · 2024, TACL
Evidence that information position affects long-context performance; not a direct test of persistent agent memory.
AI-assistance disclosure
AI assistance was used to draft and organise this web summary from Adaku Ogbonna’s research brief and to support reference checking. This page presents proposed methods, not experimental findings.
Working on agent reliability?
Let’s compare notes