LLM Evaluation Metrics

Measures whether the generated answer addresses the specific question asked, not just related topics

Measures how relevant retrieved context chunks are to the user's query

Whether highest-ranked retrieved chunks are most relevant to the query

Percentage of all expected relevant context that was successfully retrieved

Measures the computational cost of processing each query

Measures customer satisfaction with LLM system responses

Measures the percentage of queries resolved without human intervention

Uses evaluator LLM with explicit rubrics to score subjective quality dimensions

Measures whether the output fully resolves the underlying user need (actionability, tone, focus)

Measures effectiveness of PII detection and redaction in outputs

Decomposes output into atomic claims, generates closed-ended questions, and verifies against context

Measures the time taken to generate a response from query to final output

Measures adherence to safety guidelines and prevention of harmful outputs

End-to-end success rate from full conversation trace to goal achievement

Agent calls expected tools with correct parameters and in proper sequence