Articles

Truth Is a Lie:
Crowd Truth and the
Seven Myths of
Human Annotation

Lora Aroyo, Chris Welty

n Big data is having a disruptive impact amar prestar aen … “The world is changed.” In the past
across the sciences. Human annotation of
semantic interpretation tasks is a critical
part of big data semantics, but it is based
on an antiquated ideal of a single correct
I decade the amount of data and the scale of computation
available has increased by a previously inconceivable
amount. Computer science, and AI along with it, has moved
truth that needs to be similarly disrupted.
solidly out of the realm of thought problems and into an
We expose seven myths about human empirical science. However, many of the methods we use
annotation, most of which derive from predate this fundamental shift, including the ideal of truth.
that antiquated ideal of truth, and dispel Our central purpose is to revisit this ideal in computer sci-
these myths with examples from our ence and AI, expose it as a fallacy, and begin to form a new
research. We propose a new theory of
theory of truth that is more appropriate for big data seman-
truth, crowd truth, that is based on the
intuition that human interpretation is
tics. We base this new theory on the claim that, outside
subjective, and that measuring annota- mathematics, truth is entirely relative and is most closely
tions on the same objects of interpretation related to agreement and consensus.
(in our examples, sentences) across a Our theories arise from experimental data that has been
crowd will provide a useful representation published previously (Aroyo and Welty 2013a, Soberon et al.
of their subjectivity and the range of rea- 2013, Inel et al. 2013, Dumitrache et al. 2013), and we use
sonable interpretations.
throughout the article examples from our natural language

Copyright © 2015, Association for the Advancement of Artificial Intelligence. All rights reserved. ISSN 0738-4602 SPRING 2015 15
Articles

processing (NLP) work in medical relation extraction; The Seven Myths

however, the ideas generalize across all of semantics,
including semantics from images, audio, video, sen- The need for human annotation of data is very real.
sor networks, medical data, and others. We begin by We need to be able to measure machine performance
on tasks that require empirical analysis. As the need
looking at the current fallacy of truth in AI and where
for machines to handle the scale of the data increas-
it has led us, and then examine how big data, crowd-
es, so will the need for human annotated gold stan-
sourcing, and semantics can be combined to correct dards. We have found that, just as the sciences and
the fallacy and improve big data analytic systems. humanities are reinventing themselves in the pres-
ence of data, so too must the collection of human
Human Annotation annotated data.
We have discovered the following myths that
In the age of big data, machine assistance for empir- directly influence the practice of collecting human
ical analysis is required in all the sciences. Empirical annotated data. Like most myths, they are based in
analysis is very much akin to semantic interpreta- fact but have grown well beyond it, and need to be
tion: data is abstracted into categories, and meaning- revisited in the context of the new changing world:
ful patterns, correlations, associations, and implica- Myth One: One Truth
tions are extracted. For our purposes, we consider all Most data collection efforts assume that there is one
semantic interpretation of data to be a form of empir- correct interpretation for every input example.
ical analysis. Myth Two: Disagreement Is Bad
In NLP, which has been a big data science for more To increase the quality of annotation data, disagree-
than 20 years, numerous methods for empirical ment among the annotators should be avoided or
analysis have been developed and are more or less reduced.
standards. One of these methods is human annota- Myth Three: Detailed Guidelines Help
tion to create a gold standard or ground truth. This When specific cases continuously cause disagree-
method is founded on the ideal that there is a single, ment, more instructions are added to limit interpre-
universally constant truth. But we all know this is a tations.
fallacy, as there is not only one universally constant Myth Four: One Is Enough
truth. Most annotated examples are evaluated by one per-
Gold standards exist in order to train, test, and eval- son.
uate algorithms that do empirical analysis. Humans Myth Five: Experts Are Better
perform the same analysis on small amounts of exam- Human annotators with domain knowledge provide
ple data to provide annotations that establish the better annotated data.
truth. This truth specifies for each example what the Myth Six: All Examples Are Created Equal
correct output of the analysis should be. Machines The mathematics of using ground truth treats every
can learn (in the machine-learning sense) from these example the same; either you match the correct
examples, or human programmers can develop algo- result or not.
rithms by looking at them, and the correctness of Myth Seven: Once Done, Forever Valid
their performance can be measured on annotated Once human annotated data is collected for a task, it
examples that weren’t seen during training. is used over and over with no update. New annotat-
The quality of annotation gold standards is estab- ed data is not aligned with previous data.
lished by measuring the interannotator agreement,
which is roughly the average pairwise probability
Debunking the Myths
that two people agree, adjusted for chance (Cohen
1960). This follows from the ideal of truth: a higher- Now let us revisit these myths in the context of the
quality ground truth is one in which multiple new changing world, and thus in the face of a new
humans provide the same annotation for the same theory of truth.
examples. Again, we all know this is a fallacy, as there One Truth
is more than one truth for every example. In the
Our basic premise is that the ideal of truth is a falla-
extreme case, if we want to interpret music, or a
cy for semantic interpretation and needs to be
poem, we would not expect all human annotations
changed. All analytics are grounded in this fallacy,
of it to be the same. In our experiments we have and human annotation efforts proceed from the
found that we don’t need extreme cases to see clear- assumption that for each task, every example has a
ly multiple human perspectives reflected in annota- “correct” interpretation and all others are incorrect.
tion; this has revealed the fallacy of truth and helped With the widespread use of classifiers as a tool, this
us to identify a number of myths in the process of “one truth” myth has become so pervasive that is is
collecting human annotated data. assumed even in cases where it obviously does not

16 AI MAGAZINE
 Articles

No. Sentence
ex1 [GADOLINIUM AGENTS] used for patients with severe renal failure show signs of [NEPHROGENIC
SYSTEMIC FIBROSIS].
ex2 He was the first physician to identify the relationship between [HEMOPHILIA] and [HEMOPHILIC
ARTHROPATHY].
ex3 [Antibiotics] are the first line treatment for indications of [TYPHUS].
ex4 With [Antibiotics] in short supply, DDT was used during World War II to control the insect vectors of
[TYPHUS].
ex5 [Monica Lewinsky] came here to get away from the chaos in [the nation’s capital].
ex6 [Osama bin Laden] used money from his own construction company to suport the [Muhajadeen] in
Afganistan against Soviet forces.
def1 MANIFESTATION links disorders to the observations that are closely associated with them; for example,
abdominal distension is a manifestation of liver failure
def2 CONTRAINDICATES refers to a condition that indicates that drug or treatment SHOULD NOT BE USED, for
example, patients with obesity should avoid using danazol
def3 ASSOCIATED WITH refers to signs, symptoms, or findings that often appear together; for example, patients
who smoke often have yellow teeth

Table 1. Example Sentences and Definitions

hold, like analyzing a segment of music for its mood erate the same ground truth. Rather than accepting
(Lee and Hu 2012). In our research we have found this as a natural property of semantic interpretation,
countless counterexamples to the one truth myth. disagreement has been considered a measure of poor
Consider example ex1 in table 1. Annotators were quality in the annotation task, either because the task
asked what UMLS1 relation was expressed in the sen- is poorly defined or because the annotators lack suf-
tence between the highlighted terms, and they dis- ficient training. However, in our research we found
agreed, some choosing the side-effect relation, others that disagreement is not noise but signal, and at the
choosing cause. Looking closely at the sentence, level of individual examples can indicate that the
either interpretation looks reasonable; in fact one sentence or the relation at hand is ambiguous or
could argue that in general the cause relation sub- vague, or that the worker is not doing a good job. The
sumes the side-effect relation, and as a result this isn’t sentence ex4 in table 1 provides a good illustration of
disagreement at all. However, the definition of the ambiguity in a sentence, where we found annotators
relations are that cause is a strict sufficient causality disagreed on what relation was expressed between
and side-effect represents the possibility of a condi- the highlighted terms. In a very deep reading of the
tion arising from a drug. We might rule in favor of sentence, one may conclude that Antibiotics treat
one or the other relation being appropriate here, but Typhus because why else would its shortage cause
in actuality most experts are unable to make the dis- you to eliminate the carriers of the disease? However,
tinction in reading the sentence, and it seems quite in a more shallow reading the sentence does not
reasonable to suppose that the semantics of the rela- clearly express any relation between the two argu-
tions, while they may be ontological, are not linguis- ments. In example ex3, the sentence is quite precise
tic: they are difficult or at the very least uncommon and clear about the relationship, and we see this at
to express in language. The fact of the matter seems the level of annotator disagreement: it is high for the
to be, from experts and nonexperts alike, they have sentence ex4, and nonexistant for sentence ex3. This
varying degrees of difficulty understanding why the corresponds well with what we consider to be the
side-effect relation and the cause relation are differ- suitability of each sentence for lexical-based relation
ent, but they are uniformly unable to tell when a sen- extraction. Disagreement is giving us information.
tence expresses one or the other. This clearly indi-
cates that the “correct” interpretation of sentence ex1 Detailed Guidelines Help
is a matter of opinion; there is not one true interpre- The perceived problem of low annotator agreement is
tation. typically addressed through development of detailed
guidelines for annotators that help them consistent-
Disagreement Is Bad ly handle the kinds of cases that have been observed,
When empirically grounded AI work began, it was through practice, to generate disagreement. We have
noticed that if you give the same exact annotation found that increasingly precise annotation guidelines
task to two different people, they will not always gen- do eliminate disagreement but do not increase qual-

SPRING 2015 17
Articles

140

10 workers
120
20 workers
30 workers
100

80

60

40

20

0
TB T MT PB P MP BD D MD CO C MC LO HM DF SS OTH NO

Figure 1: Comparison of Worker Distribution

on a Set of 16 Medical Relations.
Comparison (plus NONE and OTHER) was made across 30 sentences for 10, 20, and 30 workers per sentence.

ity, perfuming the agreement scores by forcing annotation guidelines, and we are forced to keep
human annotators to make choices they may not instructions simple. This turns out to drastically
actually think are valid, and removing the potential reduce the design period for annotation tasks, which
signal on individual examples that are vague or can easily drag on for months (the ACE 2002 guide-
ambiguous. For example, the ACE 2002 RDC guide- lines took more than a year). Simplifying guidelines
lines V2.3 say that “geographic relations are assumed allows annotators to make choices they are more
to be static,” and claim that sentence ex5 expresses comfortable with, drastically reduces development
the located relation between the two arguments, and training time, and allows for disagreement to be
even though one clear reading of the sentence is that used as a signal.
Monica Lewinsky is not in the capital. A further prob-
lem with overly specifying the guidelines is that it One Is Enough
often leads to crisp definitions of relations that make Due to the time and cost required to generate human
sense from an ontological perspective (that is, the annotated data, standard practice is for the vast
relations exist in the world) but are never expressed majority, often more than 90 percent, of annotated
in language. Consider the definitions in table 1: the examples to be seen by a single annotator, with a
manifestation relation from UMLS has a very precise small number left to overlap among all the annota-
definition, but we were unable to find examples of it tors so that agreement can be measured. We see
in medical texts. When we turn to crowdsourcing as many examples where just one perspective isn’t
a potential source of cheaper and more scalable enough, in some cases there are five or six popular
human annotated data, we are faced with the reality interpretations of a sentence and they can’t be cap-
that microtask workers won’t read long complex tured by one person. In several experiments on rela-

18 AI MAGAZINE
 Articles

Relation Abbreviation Crowd Accuracy SME Accuracy

TREATS sT .81 .88
PREVENTS sP .88 .84
DIAGNOSE sD .72 .89
CAUSES sC .69 .70
LOCATION sL .83 .79
SYMPTOM sS .63 .79
MANIFESTATION sM .77 .71
CONTRAINDICATES sCI .92 .93
ASSOCIATED WITH sAW .87 .31
SIDE EFFECT sSE .92 .88
IS A sIA .82 .88
PART OF sPO .86 .93
ALL .81 .79
RANK .73 .98
TOP .74 1.00

Table 2. Subject Matter Expert Versus Crowd Accuracy on UMLS Relation Annotation Task.

tion annotation for examples like those shown in crowd annotations, the expert annotators reach
table 1, we saw that between 15 and 20 workers per agreement only on 30 percent of the sentences, and
sentence yielded the same relative disagreement the most popular vote of the crowd covers 95 percent
spaces as any higher amount up to 50. Figure 1 shows of this expert annotation agreement. Table 2 further
the results of one such experiment on 30 sentences shows the relative accuracy of the crowd to medical
for a different set of 16 medical relations, in which subject matter experts (SME). In our analysis, mis-
we ran the same sentences through the annotation takes by the crowd were not surprising, but experts
process with 10, 20, and 30 workers annotating each were far more likely than nonexperts to see relations
sentence. What the specific relations are doesn’t mat- where none were expressed in a sentence, when they
ter; the graph shows the accumulated results for each knew the relation to be true. In sentence ex2 (table
relation across all the sentences, and we see that the 1), medical experts annotated a causes relation
relative distribution of the worker’s annotations look between the two arguments, because they knew it to
the same for 20 and 30 workers per sentence, but are be true. Crowd annotators did not indicate a rela-
different at 10. In further experiments we found 15 tion, and in general tended to read the sentences
workers per sentence to be the lowest point where more literally, which made them better suited to pro-
the relative distribution stabilizes; it is very likely this vide examples to a machine.
number depends on other factors in the domain, but We have also found that multiple perspectives on
we have not investigated it deeply. We can learn sev- data, beyond what experts may believe is salient or
eral things from figure 1, for example that since the correct, can be useful. The Waisda? video tagging
most popular choice across the sentences is OTHER, game (Gligorov et al. 2011) study shows that only 14
the set of relations given to the workers was not well percent of tags searched for by lay users could be
suited to this set of sentences. We could not learn this found in the professional video annotating vocabu-
from only one annotation per sentence, nor could we lary (GTAA), indicating that there is a huge gap
learn individual properties of sentences such as ambi- between the expert and lay users’ views on what is
guity (discussed above). important. Similarly, the steve.museum project (Lea-
son 2009) studied the link between a crowdsourced
Experts Are Better user tags folksonomy and the professionally created
Conventional wisdom is that if you want medical museum documentation. Again in this separate
texts annotated for medical relations you need med- study only 14 percent of lay user tags were found in
ical experts. In our work, experts did not show sig- the expert-curated collection documentation.
nificantly better quality annotations than nonex-
perts. In fact, with 30 microworkers per sentence for All Examples Are Created Equal
the UMLS relation extraction task, we found 91 per- In typical human annotation tasks, annotators are
cent of the expert annotations were covered by the asked to say whether some simple binary property

SPRING 2015 19
Articles

Rel: 15 Workers/sent pair

Sentence ID sT sP sD sCA sL sS sM sCI sAW sSE sIA sPO sNONE sOTH

225527731 0 0 0 1 0 11 0 0 0 0 0 0 0 0
225527732 0 0 0 0 0 7 2 0 2 2 0 1 0 0
225527733 0 0 0 1 0 7 1 0 1 0 0 0 0 1
225527734 0 0 0 0 0 1 0 0 2 0 0 0 0 9
225527735 0 0 0 0 0 13 0 0 0 0 0 0 0 0
225527736 0 0 0 2 0 2 0 0 1 0 0 0 3 4
225527737 0 0 0 2 0 6 2 0 3 1 1 0 0 0
225527738 0 0 0 2 0 0 1 0 0 1 8 1 0 0
225527739 0 0 0 10 0 0 0 0 0 0 0 1 0 0
225527740 0 0 0 10 0 2 1 0 1 0 0 0 0 1
225527741 1 0 0 5 0 3 3 0 1 0 1 0 1 1
225527742 0 0 0 4 0 0 0 0 3 0 0 0 0 4
225527743 0 0 0 1 0 1 2 0 1 0 0 0 0 8
225527744 0 0 0 3 0 1 0 0 1 8 0 0 0 1
225527745 0 0 0 5 0 2 3 0 1 4 0 0 0 0
225527746 0 0 1 1 5 2 0 0 1 0 0 0 2 0
225527747 0 0 0 1 8 2 2 0 1 0 0 0 1 1
225527748 0 0 0 1 7 1 0 0 1 0 0 0 2 1
225527749 0 0 0 0 0 0 0 0 3 0 1 1 4 2
225527750 0 0 0 1 0 4 2 0 3 0 1 2 0 0

Figure 2. Sentence Vectors Representing Crowd Annotations.

The figure shows annotations on 20 sentences, 10–15 workers per sentence. Rows are individual sentences, columns are the relations from
table 2. Cells contain the number of workers that selected the relation for the sentence, that is, 8 workers selected the sIA relation for sen-
tence 738. The cells are heat-mapped on the number of annotations.

holds for each example, like whether sentence ex2 of them but not clearly, and as described above this
(table 1) expresses the cause relation. They are not is an obvious example of a sentence with a high lev-
given a chance to say that the property may partial- el of vagueness. In sentence ex3, all annotators indi-
ly hold or holds but is not clearly expressed. Individ- cate the treats relation with no disagreement. The dis-
ual humans are particularly bad at uniformly choos- agreement allows us to weight the latter sentence
ing from scales of choices (like high, medium, low), higher than the former, giving us the ability to both
but we can find by recording disagreement on each train and evaluate a machine in a more flexible way.
example that poor quality examples tend to generate
high disagreement. In sentence ex4, we find a mix Once Done, Forever Valid
between treats and prevents in the crowd annota- Perspectives change over time, which means that
tions, indicating that the sentence may express either training data created years ago might contain exam-

20 AI MAGAZINE
 Articles

ples that are not valid or only partially valid at a lat- worker-worker disagreement score, which is calcu-
er point in time. Take for example the sentence ex6, lated by constructing a pairwise confusion matrix
and a task in which annotators are asked to identify between workers and taking the average agreement
mentions of terrorists. In the 1990s, [Osama bin for each worker. The first metric gives us a measure of
Laden] would have been labeled as “hero” and after how much a worker disagrees with the crowd on a
2001 would have been labeled as “terrorist.” Consid- sentence basis, and the second gives us an indication
ering the time, both types would be valid, and they as to whether there are consisently like-minded
introduce two roles for the same entity. We are only workers. While we encourage disagreement, if a
just beginning to investigate this particular myth, worker tends to disagree with the crowd consistent-
but our approach includes continuous collection of ly, and does not generally agree with any other work-
training data over time, allowing the adaptation of ers, that worker will be labeled low quality. Before
gold standards to changing times. We can imagine computing worker measures, the sentences with the
cases such as popularity of music and other clearly lowest clarity scores (see below) are removed from
more subjective properties of examples that would be the disagreement calculations, to ensure that work-
expected to change, but even cases that may seem ers are not unfairly penalized if they happened to
more objective could benefit from continuous col- work on a bad batch of sentences. Our experiments
lection of annotations as, for example, relative levels show that these worker metrics are more than 90 per-
of education shift. cent effective in identifying low quality workers
(Soberon et al. 2013).
Crowd Truth Sentence scores include sentence clarity and the
Crowd truth is the embodiment of a new theory of core crowd truth metric for relation extraction, sen-
truth that rejects the fallacy of a single truth for tence-relation score (SRS). SRS is measured for each
semantic intepretation, based on the intuition that relation on each sentence as the cosine of the unit
human interpretation is subjective and that measur- vector for the relation with the sentence vector. The
ing annotations on the same objects of interpretation relation score is used for training and evaluation of
(in our examples, sentences) across a crowd will pro- the relation extraction system; it is viewed as the
vide a useful representation of their subjectivity and probability that the sentence expresses the relation.
the range of reasonable interpretations. Crowd truth This is a fundamental shift from the traditional
has allowed us to identify and dispel the myths of approach, in which sentences are simply labelled as
human annotation and to paint a more accurate pic- expressing, or not, the relation, and presents new
ture of human performance on semantic interpreta- challenges for the evaluation metric and especially
tion for machines to attain. for training. In our experiments we have seen that
The key element to crowd truth is that multiple the sentence-relation score is highly correlated with
workers are presented the same object of interpreta- clearly expressing a relation. Sentence clarity is
tion, which allows us to collect and analyze multiple defined for each sentence as the max relation score
perspectives and interpretations. To facilitate this, we for that sentence. If all the workers selected the same
represent the result of each worker’s annotations on relation for a sentence, the max relation score will be
a single sentence as a vector in which each interpre- 1, indicating a clear sentence. In figure 2, sentence
tation that is possible is a dimension in the vector 735 has a clarity score of 1, whereas sentence 736 has
space. In the case of relation extraction, the crowd a clarity score of 0.61, indicating a confusing or
truth vector has n + 2 dimensions, where n is the ambiguous sentence. Sentence clarity is used to
number of relations + options for NONE and OTHER weight sentences in training and evaluation of the
(allowing a worker to indicate that a sentence does relation extraction system, since annotators have a
not express a relation at all or does not express any of hard time classifying them, the machine should not
the given relations). In these vectors, a 1 is given for be penalized as much for getting it wrong in evalua-
each relation the worker thought was being tion, nor should it treat such training examples as
expressed (workers can indicate multiple relations), strong exemplars.
and we use them to form sentence disagreement vec- Relation scores include relation similarity, relation
tors for each sentence by summing all the worker ambiguity, and relation clarity. Similarity is a pair-
vectors for the sentence. An example set of disagree- wise conditional probability that if relation Ri is
ment vectors is shown in figure 2. We use these vec- annotated in a sentence, relation Rj is as well. Infor-
tors to compute metrics on the workers (for low qual- mation about relation similarity is used in training
ity and spam), on the sentences (for clarity and what and evaluation, as it roughly indicates how confus-
relations may be expressed), and on the relations (for able the linguistic expression of two relations are.
clarity and similarity) as follows: This would indicate, for example, that relation
Worker measures include the worker-sentence dis- colearning (Carlson et al. 2009) would not work for
agreement score, which is the average of all the similar relations. Ambiguity is defined for each rela-
cosines between each worker’s sentence vector and tion as the max relation similarity for the relation. If
the full sentence vector (minus that worker), and the a relation is clear, then it will have a low score. Since

SPRING 2015 21
Articles

Sign

Referent Interpreter

Figure 3. Triangle of Reference.

techniques like relation colearning have proven nitive process attempts to find the referent of that
effective, it may be useful to exclude ambiguous rela- sign (an object, an idea, a class of things, and so on).
tions from the set. Clarity is defined for each relation This process of interpretation is what we generally
as the max sentence-relation score for the relation mean when we talk about semantics.
over all sentences. If a relation has a high clarity In crowd truth for relation extraction, sentences
score, it means that it is atleast possible to express the are the signs, workers are the interpreters, and the ref-
relation clearly. We find in our experiments that a lot erents are provided by the semantics of the domain;
of relations that exist in structured sources are very in our examples the set of relations are the possible
difficult to express clearly in language and are not fre- referents. Adapting crowd truth to a new problem
quently present in textual sources. Unclear relations involves substituting for the sentences the objects to
may indicate unattainable learning tasks. be interpreted, and identifying the possible semantic
The three kinds of scores hold up well in our exper- space of referents. Once the semantic space is identi-
iments for building a medical relation extraction fied, it is mapped into a vector space and the same
gold standard (Aroyo and Welty 2013b). We believe measures can be applied. For example, if the inter-
the idea of crowd truth generalizes to other big data pretation task were to identify the predominant col-
semantics tasks quite easily. The three kinds of meas- ors in an image, the vector space could be the range
ures we introduce correspond directly to the three of possible (relevant) colors.
corners of the triangle of reference (see figure 3) The most work in adaptation of crowd truth to a
between a sign, something the sign refers to, and the new problem lies in determining a useful vector space
intepreter of the sign (Ogden and Richards 1923). for representing the disagreement. It is important for
The intepreter perceives the sign (a word, a sound, an the dimensionality to be relatively low, so that there
image, a sentence, and so on) and through some cog- is reasonable opportunity for workers to agree as well

22 AI MAGAZINE
 Articles

as disagree. In the case of event processing (Inel et al. Notes

2013) we mapped disagreement about geospatial 1. www.nlm.nih.gov/research/umls/.
location of events into a predefined spatial contain-
ment hierarchy (continent, country, city) to allow References
the disagreement space to be consistent across sen- Alonso, O., and Baeza-Yates, R. 2011. Design and Imple-
tences. If each sentence had its own space, it would mentation of Relevance Assessments Using Crowdsourcing.
not be possible to compute aggregate metrics. In Advances in Information Retrieval. Lecture Notes in Com-
Crowd truth is already being used to gather anno- puter Science Volume 6611, 153–164. Berlin: Springer-Ver-
tated data for a variety of NLP tasks, and we believe lag. dx.doi.org/10.1007/978-3-642-20161-5_16
it generalizes to big data problems for which a gold Aroyo, L., and Welty, C. 2013a. Crowd Truth: Harnessing
standard is needed for training and evaluation. Our Disagreement in Crowdsourcing a Relation Extraction Gold
experiments indicate the crowd truth approach can Standard. In Web Science 2013. New York: Association for
be faster, cheaper, and more scalable than traditional Computing Machinery.
ground truth approaches involving dedicated human Aroyo, L., and Welty, C. 2013b. Measuring Crowd Truth for
annotators, while improving certain quality dimen- Medical Relation Extraction. In Semantics for Big Data:
sions, by exploiting the disagreement between crowd Papers from the AAAI Fall Symposium, ed. F. van Harmelen, J.
workers as a signal, rather than trying to eliminate it. A. Hendler, P. Hitzler, and K. Janowicz. AAAI Technical
Report FS-13-04, Palo Alto, CA.
Bozzon, A.; Brambilla, M.; Ceri, S.; and Mauri, A. 2013.
Related Work Reactive Crowdsourcing. In Proceedings of the 22nd Interna-
tional Conference on World Wide Web, 153–164. New York:
When dealing with crowdsourcing, there is a growing Association for Computing Machinery.
literature on observing and analyzing workers’ Carlson, A.; Betteridge, J.; Hruschka, Jr., E. R.; and Mitchell,
behaviour (Mason and Suri 2012) for ultimately T. M. 2009. Coupling Semi-Supervised Learning of Cate-
being able to detect and eliminate spam (Bozzon et gories and Relations. In Proceedings of the North American
al. 2013; Kittur, Chi, and Suh 2008; Ipeirotis, Provost, Chapter of the Association for Computation Linguistics Human
and Wang 2010), and analyze workers perfomance Language Technologies 2009 Workshop on Semi-Supervised
for quality control and optimization of the crowd- Learning for Natural Language Processing, 1–9. Stroudsburg,
sourcing processes (Singer and Mittal 2013). This PA, USA: Association for Computational Linguistics.
becomes even more challenging when the goal is to Chklovski, T., and Mihalcea, R. 2003. Exploiting Agreement
harness the disagreement between annotators, as we and Disagreement of Human Annotators for Word Sense
need to distinguish between the good disagreement Disambiguation. Paper presented at the International Con-
and the spam-based one. Our worker metrics relate ference on Recent Advances in Natural Language Process-
ing, 10–12 September 2003. Available from University of
to the approach proposed by Sheng, Provost, and
North Texas Scholarly Works Digital Library, Denton, TX.
Ipeirotis (2008) for improving data quality for super-
Cohen, J. 1960. A Coefficient of Agreement for Nominal
vised learning. The novelty that sets crowd truth
Scales. Educational and Psychological Measurement 20(1): 37–
apart is that we reject the “one truth” fallacy that for
46. dx.doi.org/10.1177/001316446002000104
each annotation there is a single correct answer that
Difallah, D. E.; Demartini, G.; and Cudré-Mauroux, P. 2012.
enables distance and clustering metrics to detect out-
Mechanical Cheat: Spamming Schemes and Adversarial
liers more easily (Alonso and Baeza-Yates 2011; Techniques on Crowdsourcing Platforms. In Crowdsourcing
Raykar and Yu 2012; Difallah, Demartini, and Cudré- the Semantic Web: Proceedings of the 1st International Work-
Mauroux 2012) . shop on Crowdsourcing the Semantic Web. CEUR Workshop
Some existing research efforts bear similarities to Proceedings Volume 842, 26–30. Aachen, Germany: RWTH
crowd truth. In Markines et al. (2009), an approach Aachen University.
to finding the similarity between folksonomies is pro- Dumitrache, A.; Aroyo, L.; Welty, C.; Sips, R.-J.; and Levas,
posed, which also breaks up the problem according A. 2013. “Dr. Detective”: Combining Gamification Tech-
to the triangle of reference (figure 3). In Chklovski niques and Crowdsourcing to Create a Gold Standard in
and Mihalcea (2003), disagreement in the crowd was Medical Text. In Crowdsourcing the Semantic Web: Proceedings
harnessed for word sense disambiguation (WSD), and of the 1st International Workshop on Crowdsourcing the Seman-
tic Web. CEUR Workshop Proceedings Volume 1030.
the results showed WSD to be a task humans can not
Aachen, Germany: RWTH Aachen University.
do consistently. Most recently, in Plank, Hovy, and
Gligorov, R.; Hildebrand, M.; van Ossenbruggen, J.;
Sgaard (2014), an approach to dealing with particu-
Schreiber, G.; and Aroyo, L. 2011. On the Role of User-Gen-
larly hard examples of part-of-speech tagging is pro-
erated Metadata in Audio Visual Collections. In Proceedings
posed, using an idea similar to our disagreement of the 6th International Conference on Knowledge Capture (K-
approach. We believe these efforts add further evi- CAP 2011), 145–152. New York: Association for Computing
dence to our basic hypothesis, that semantic intepre- Machinery.
tation is subjective, and gathering a wide range of Inel, O.; Aroyo, L.; Welty, C.; and Sips, R.-J. 2013. Exploit-
human annotations is desirable. We have outlined ing Crowdsourcing Disagreement with Various Domain-
steps to bring this together into a more general theo- Independent Quality Measures. Paper presented at the 3rd
ry of truth. International Workshop on Detection, Representation, and

SPRING 2015 23
Articles

Methods 44(1): 1–23. dx.doi.org/10.3758/s13428-011-0124-6

$QLQWHUQDWLRQDOIRUXPRQGHFODUDWLYHORJLFSURJUDPPLQJQRQ Ogden, C. K., and Richards, I. 1923. The Meaning of Meaning.
PRQRWRQLFUHDVRQLQJDQGNQRZOHGJHUHSUHVHQWDWLRQ London: Trubner & Co.
Plank, B.; Hovy, D.; and Sgaard, A. 2014. Learning Part-of-
Speech Taggers with Inter-Annotator Agreement Loss. In
Proceedings of the 14th Conference of the European Chapter of
the Association for Computational Linguistics (EACL-2014).
Stroudsburg, PA: Association for Computational Linguistics.
Raykar, V. C., and Yu, S. 2012. Eliminating Spammers and
Ranking Annotators for Crowdsourced Labeling Tasks. Jour-
nal of Machine Learning Research 13: 491–518.
/H[LQJWRQ.<86$6HSWHPEHU 
KWWSOSQPUPDWXQLFDOLW Sheng, V. S.; Provost, F.; and Ipeirotis, P. G. 2008. Get Anoth-
er Label? Improving Data Quality and Data Mining Using
Multiple, Noisy Labelers. In Proceedings of the 14th ACM
7KH WK HGLWLRQ RI /RJLF 3URJUDPPLQJ DQG 1RQPRQRWRQLF 5HDVRQLQJ
SIGKDD International Conference on Knowledge Discovery and
&RQIHUHQFH /3105  IHDWXUHV
Data Mining, 614–622. New York: Association for Comput-
‡ &ROORFDWLRQ ZLWK WKH WK ,QWHUQDWLRQDO $OJRULWKPLF 'HFLVLRQ 7KHRU\
ing Machinery. dx.doi.org/10.1145/1401890.1401965
&RQIHUHQFH $'7 
‡ 7KUHH LQYLWHG WDONV JLYHQ E\ OHDGHUV LQ WKHLU ILHOGV Singer, Y., and Mittal, M. 2013. Pricing Mechanisms for
‡ )RXU ZRUNVKRSV WKH WK $QVZHU 6HW 3URJUDPPLQJ &RPSHWLWLRQ DQG Crowdsourcing Markets. In Proceedings of the 22nd Interna-
D MRLQW /3105$'7 'RFWRUDO &RQVRUWLXP tional Conference on World Wide Web, 1157–1166. New York:
‡ 3URFHHGLQJV # /HFWXUH 1RWHV LQ $UWLILFLDO ,QWHOOLJHQFH 6SULQJHU Association for Computing Machinery.
‡ 7ZR EHVW SDSHUV RI JHQHUDO $, LQWHUHVW LQYLWHG WR $UWLILFLDO Soberon, G.; Aroyo, L.; Welty, C.; Inel, O.; Overmeen, M.;
,QWHOOLJHQFH -RXUQDO (OVHYLHU and Lin, H. 2013. Content and Behaviour Based Metrics for
‡ 7ZR EHVW SDSHUV IRFXVLQJ RQ ORJLF SURJUDPPLQJ LQYLWHG WR 7KHRU\ Crowd Truth. In Crowdsourcing the Semantic Web: Proceedings
DQG 3UDFWLFH RI /RJLF 3URJUDPPLQJ of the 1st International Workshop on Crowdsourcing the Seman-
‡ $ JUHDW YHQXH /H[LQJWRQ LV D SOHDVDQW PHGLXP VL]H XQLYHUVLW\ WRZQ tic Web. CEUR Workshop Proceedings Volume 1030, 56–69.
LQ WKH KHDUW RI WKH EHDXWLIXO %OXHJUDVV 5HJLRQ LQ &HQWUDO .HQWXFN\ Aachen, Germany: RWTH Aachen University
0DLQ VSRQVRUV
(&&$,$UWLILFLDO ,QWHOOLJHQFH MRXQUQDO $VVRFLDWLRQ IRU/RJLF 3URJUDPPLQJ.5RUJ
8QLYHUVLW\ RI.HQWXFN\86$8QLYHUVLW\ RI&DODEULD,WDO\
Lora Aroyo is an associate professor at the Web and Media
,QFRRSHUDWLRQ ZLWK$$$, Group, Department of Computer Science, VU University
Amsterdam, The Netherlands. Her research work has
focused on semantic technologies for modeling user and
context for recommendation systems and personalized
access of online multimedia collections. She was a scientific
Exploitation of Events in the Semantic Web. 21 October, coordinator of the NoTube project, dealing with the inte-
Sydney, Australia. gration of web and TV data using semantic technology, and
a number of nationally funded projects. Aroyo has served as
Ipeirotis, P. G.; Provost, F.; and Wang, J. 2010. Quality Man-
program chair for the European and the International
agement on Amazon Mechanical Turk. In Proceedings of the
Semantic Web Conferences, as conference chair for the
ACM SIGKDD Workshop on Human Computation, 64–67. New
ESWC 2010 Conference, on the editorial board of the
York: Association for Computing Machinery.
Semantic Web Journal, as vice president of User Modeling
dx.doi.org/10.1145/1837885.1837906
Inc., and on the editorial board of the Journal of Human-
Kittur, A.; Chi, E. H.; and Suh, B. 2008. Crowdsourcing User Computer Studies and the User Modeling and User-Adapted
Studies with Mechanical Turk. In Proceedings of the SIGCHI Interaction Journal. In 2012 and 2013 she won IBM Faculty
Conference on Human Factors in Computing Systems, 453–456. Awards for her work on crowd truth: Crowdsourcing for
New York: Association for Computing Machinery. ground truth data collection for adapting the IBM Watson
Leason, T. 2009. Steve: The Art Museum Social Tagging Proj- system to the medical domain. For more information visit
ect: A Report on the Tag Contributor Experience. In Muse- crowdtruth.org, or follow Twitter: @laroyo.
ums and the Web 2009. Silver Spring, MD: Museums and the
Web LLC.
Lee, J. H., and Hu, X. 2012. Generating Ground Truth for Chris Welty is a research scientist at Google Research in
Music Mood Classification Using Mechanical Turk. In Pro- New York, having left IBM’s Watson Group in the summer
ceedings of the 12th ACM/IEEE-CS Joint Conference on Digital of 2014. He holds a Ph.D. in computer science from Rens-
Libraries, 129–138. New York: Association for Computing selaer Polytechnic Institute. Welty was a member of the
Machinery. dx.doi.org/10.1145/2232817.2232842 technical leadership team for Watson and IBM’s Jeopardy!
Markines, B.; Cattuto, C.; Menczer, F.; Benz, D.; Hotho, A.; Challenge and is a recipient of AAAI’s Feigenbaum Prize. He
and Stumme, G. 2009. Evaluating Similarity Measures for is also known for his work in the semantic web and ontol-
Emergent Semantics of Social Tagging. In Proceedings of the ogy communities. He serves on the editorial board of this
18th International Conference on World Wide Web. New York: magazine, as well as the Journal of Applied Ontology, and is
Association for Computing Machinery. the natural language processing area editor for the Journal of
dx.doi.org/10.1145/1526709.1526796 Web Semantics.
Mason, W., and Suri, S. 2012. Conducting Behavioral
Research on Amazon’s Mechanical Turk. Behavior Research

24 AI MAGAZINE