AWS describes UAEF for evaluating AI agents across frameworks – Unite.AI

AWS on November 11, 2026 published a technical introduction to the Universal Agent Evaluation Framework (UAEF), an open source system licensed under Apache-2.0 that evaluates AI agents using 36 metrics built into seven dimensions.

UAEF is designed to keep ratings comparable across LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore. These frameworks log messages, tool calls, and agent-to-agent handoffs differently, so evaluation code tied to a format may require new metrics, scoring logic, and result storage after a framework change. This makes it more difficult to determine whether a change to the prompt, template, or tools improved the agent.

AWS reports that customers at 12 organizations in the telecommunications, healthcare, and financial services industries have used UAEF to enforce consistent metrics and compare agent versions before release. The awslabs/uaef repository contains the Python library, framework adapters, parameters, AWS deployment infrastructure, web interface, and executable notebooks.

Canonical tracks enable shared scoring

Adapters normalize each agent’s messages, tool calls, and steps into a canonical format AgentTracethe only representation used by the scoring engine. UAEF can adapt an existing trace from a past execution, a log, or an observability store, or invoke an agent via a runtime Amazon Resource Name or HTTP endpoint and convert the resulting execution. A generic JSON adapter extends the process to custom frameworks.

The metrics catalog groups integrated capabilities into tool call, response quality, responsible AI, performance, multi-turn conversation, coordination, and multi-agent reasoning. Scores use a normalized scale of 0 to 1, with 1 being best. Deterministic metrics calculate directly from traces and reality, while LLM metrics as a judge uses a model on Amazon Bedrock for qualities like relevance and security. Custom metrics, RAGAS and DeepEval integrations are also supported.

Concrete answers are overall optional, although reference comparison metrics require them. Evaluating tool calls can also define the expected tools, their order, and their arguments. When a metric does not have the necessary input, it returns no score with a warning, and the empty result is excluded from the dimension averages rather than counted as zero. Partially labeled datasets and unlabeled production traffic can therefore still be evaluated with metrics whose inputs are available.

Stored experiments exhibit regressions

Persistent assessments become experiments. Amazon DynamoDB stores metadata for listing and comparison, including rating counts and average scores, while Amazon S3 stores complete JSON results with scores for metrics and judges’ reasoning. Individual runs can record the framework, model, tools, system prompt, parameters, configuration snapshot, and Git commit, linking each score to the design of the agent under test. UAEF analyzes traces for personal information before archiving, and redaction is available for archived findings.

For parameters shared by two runs, the comparison engine calculates absolute and percentage changes from a reference. A configurable threshold, set at 5% by default, separates improvements, regressions, and unchanged results; regressions are classified as minor, moderate or severe and grouped by size. This grouping can show whether a timely review influenced the instrument call without changing the quality of the response.

Reporting keeps reproducible statistics separate from model-generated interpretation. The web interface shows results by test case and by dimension, and an optional statistics report covers score distributions, confidence intervals, outliers, metric correlations, and failed test cases without using an LLM. A separate analysis generated by Bedrock can propose root causes and prioritize next steps for an execution or comparison.

The Python library and low-code web service use the same evaluation engine and repositories. Teams can run evaluations in notebooks, scripts, continuous integration and continuous delivery pipelines, or through a browser without changing how results are evaluated and compared.

AWS dependencies and isolation limits

The installation guide requires Python 3.10 or later and the uv package manager. AWS credentials are required for Bedrock-based judgment and DynamoDB/S3 persistence, so that teams using these judgment and storage capabilities remain connected to AWS services even when their agents use different frameworks.

UAEF treats the evaluated agent output as untrusted input for judging prompts. It submits the rubric separately, excludes content, rigorously analyzes judges’ responses, and requires a period of verbatim testing before accepting a score. AWS says judges’ scores should not limit security checks; The runtime application belongs to a separate Amazon Bedrock Guardrails call path.

The repository describes UAEF as a reference architecture designed to run within a platform that provides user and tenant isolation. Direct multi-tenant deployment without host enforcement is not supported. Each tenant needs a separate deployment with its own AWS account or, at a minimum, its own stack, DynamoDB tables, S3 buckets, and Cognito user pool.

For companies using multiple agent frameworks, UAEF offers a stable scoring foundation that can survive changes in requests, models, tools, or implementation. Understand that continuity requires more than just installing the library: Useful comparisons depend on appropriate metrics and datasets, while multi-tenant use depends on the isolation provided outside of the UAEF.