
MACE (Multi-Annotator Competence Estimation)
When evaluating redundant annotations (like those from Amazon's MechanicalTurk), we usually want to
- aggregate annotations to recover the most likely answer
- find out which annotators are trustworthy
- evaluate item and task difficulty