Inter-annotator agreement

Inter-annotator agreement is a measure of how consistently different human reviewers label or rate the same items, used to check that evaluation guidelines are clear and that labels are reliable.

Inter-annotator agreement sits in the Evaluation & safety part of the Agentik {OS} glossary, which defines the words used to build and run AI agent systems.

Also called Inter-rater reliability.