Tag
Everything tagged LLM Ops & Eval, most recent first.
10 entries
Trustworthy AI Agents: Test in CI, Enforce at Runtime
Talk at API:World 2026 on catching unsafe agent behaviour before release and stopping it at runtime, using policy tests, grounded-answer checks, trajectory tests, and versioned tool contracts.
AI Engineer World's Fair 2026 talk: agents need feature flags. We ship the most behavior-changing systems we have ever built, agents that send email, move money, and modify databases, with none of the canary, segment-targeting, and kill-switch discipline web teams settled a decade ago. The talk lays out six flag types for the six behavior surfaces an agent has, a rollout playbook with the numbers to track, and why a stop button is now required by law.
AI Engineer World's Fair 2026 talk: your coding agent is creating review debt, the accumulating gap between the code an agent produces and the code humans have actually reviewed, trusted, and understood. Five signal families and ten deterministic checks turn that gap into a single defensible score, validated across 524 real pull requests.
A three-layer evaluation framework in Java + LangChain4j for AI-agent trust testing: policy compliance, judge-based answer safety, and tool-trajectory validation, wired into CI as a build-time gate.
Build-time trust tests for LLM agents
A JUnit 5 framework that catches enterprise AI-agent failures before release by turning allowed-tools, grounded-answers, and golden-trajectory expectations into regression tests that run in CI.
Operating an LLM system means making a fuzzy, non-deterministic thing measurable.
LLM systems are hard to operate because outputs are open-ended and non-deterministic. LLM ops and eval is the discipline that makes them measurable: evals as tests, tracing, regression gates, and a flywheel that turns production failures into your next test set.
What an eval is, the types of eval, and the step-by-step for running one that tells you something true.
Eval is how you turn a fuzzy, open-ended LLM system into something you can measure. This is a practical guide to the types of eval, from exact match to LLM-as-judge to human review, and the six-step procedure for running one you can actually trust.
An incident-remediation agent behind a security envelope
A self-healing microservices agent that reasons over telemetry, deploy history, and logs via MCP and remediates production incidents behind an RBAC, ACL, OAuth, and human-approval envelope.
Blast radius and incident-to-deploy correlation
A platform that maps service topology, indexes logs and traces, correlates incidents to deployments, and computes blast radius, giving evidence-grounded answers to 'why did this deploy fail?' and 'what breaks if I change this?'
Hallucination-free PPT to narrated video
Converts PowerPoint decks into narrated explainer videos where every narrated claim traces back to specific slide content, using a content-addressable evidence index, a closed-loop verifier, and a dual-provenance knowledge graph.
