MACE (Multi-Annotator Competence Estimation)


When evaluating redundant annotations (like those from Amazon's MechanicalTurk), we usually want to
  1. aggregate annotations to recover the most likely answer
  2. find out which annotators are trustworthy
  3. evaluate item and task difficulty
MACE solves all of these problems, by learning competence estimates for each annotators and computing the most likely answer based on those competences.

Click MACE in the sidepanel to get started.








MACE

Select the input parameters below

Select the MACE parameters below

Run Mace