{"id":17458,"date":"2026-10-25T18:05:49","date_gmt":"2026-10-25T18:05:49","guid":{"rendered":"https:\/\/dmsretail.com\/RetailNews\/introducing-agentic-evaluation-in-faith\/"},"modified":"2026-10-25T18:05:49","modified_gmt":"2026-10-25T18:05:49","slug":"introducing-agentic-evaluation-in-faith","status":"publish","type":"post","link":"https:\/\/dmsretail.com\/RetailNews\/introducing-agentic-evaluation-in-faith\/","title":{"rendered":"Introducing Agentic Evaluation in FAITH"},"content":{"rendered":"

\"Retail<\/a><\/p>
\n<\/p>\n

\n

As AI agents become more capable and their use expands across cybersecurity and other domains, the need for effective evaluation grows. An agent investigating a codebase makes a sequence of decisions: where to look, which tools to use, how to interpret the evidence, and when to submit a finding. Evaluating that work requires clear task definitions, consistent execution conditions, and a way to examine both the process and the outcome.<\/span>\u00a0<\/span><\/p>\n

We\u2019re introducing <\/span>Agentic FAITH<\/span><\/b>, an extension of Cisco Foundation AI\u2019s <\/span>FAITH evaluation framework<\/span>. It brings tasks that require multiple steps and tool use into the same framework teams already use for question-answering benchmarks. Researchers and engineers can now use one workflow to run cybersecurity evaluations, grade outcomes, and analyze results across both kinds of tasks.<\/span>\u00a0<\/span><\/p>\n

\"\"

One workflow for cybersecurity evaluations<\/span><\/b>\u00a0<\/span><\/h2>\n

FAITH already brings cybersecurity benchmarks for threat intelligence, vulnerability classification, and security knowledge into a common evaluation workflow. Agentic support extends that workflow to tasks such as exploring a repository with terminal tools, giving teams one framework for running and analyzing both traditional and agentic evaluations.<\/span>\u00a0<\/span><\/p>\n

The workflow stays familiar: <\/span>query<\/span> runs the benchmark and collects records, <\/span>eval<\/span> grades the outcomes, and <\/span>summarize<\/span> reports the results. The <\/span>run-all<\/span> command combines these stages. Teams using FAITH can run and analyze their new agentic benchmarks through these same commands.<\/span>\u00a0<\/span><\/p>\n

For agentic tasks, FAITH manages the interaction between the model and its tools. The model chooses an action, receives the tool\u2019s output, and uses that result to decide what to do next. A run continues until the agent submits an answer or reaches a stopping condition defined by the benchmark.<\/span><\/p>\n

\"\"

See it in action with VulnLoc<\/span><\/b>\u00a0<\/span><\/h2>\n

The first integrated agentic benchmark is <\/span>VulnLoc<\/span>, which evaluates how well agents locate vulnerable files in a repository and avoid false alarms on patched code.<\/span>\u00a0<\/span><\/p>\n

Each task gives the agent a repository snapshot and a description of a vulnerability class. The agent explores the code through terminal tools, searches for relevant logic, and follows what it finds. It then submits the files it suspects contain the vulnerability or explicitly reports that it found no matching vulnerability.<\/span>\u00a0<\/span><\/p>\n

VulnLoc can run locally with Docker for trusted inputs or on Kubernetes through Agent Sandbox. The execution environment enforces read-only access to the repository.<\/span>\u00a0<\/span><\/p>\n

FAITH integrates both phases of the benchmark:<\/span>\u00a0<\/span><\/p>\n