Active Learning with Uncertain Annotators

Calma, Adrian

dc.date.accessioned	2020-12-23T15:30:36Z
dc.date.available	2020-12-23T15:30:36Z
dc.date.issued	2020
dc.identifier	doi:10.17170/kobra-202009171814
dc.identifier.uri	http://hdl.handle.net/123456789/12382
dc.description	Zugleich: Dissertation, Universität Kassel, 2020	ger
dc.language.iso	eng	eng
dc.publisher	kassel university press
dc.rights	Namensnennung - Weitergabe unter gleichen Bedingungen 4.0 International	*
dc.rights.uri	http://creativecommons.org/licenses/by-sa/4.0/	*
dc.subject	collaborative interactive learning	eng
dc.subject	uncertain annotators	eng
dc.subject	active learning	eng
dc.subject.ddc	004
dc.title	Active Learning with Uncertain Annotators	eng
dc.type	Buch
dcterms.abstract	In the digital age, many applications can benefit from collecting data. Classification algorithms, for example, are used to predict the class labels of samples (also termed data points, instances or observations). However, these methods require labeled instances to be trained on. Active learning is a machine learning paradigm where an active learner has to train a model (e.g., a classifier) which is in principle trained in a supervised way. Active learning has to be done by means of a data set where a low fraction of samples are labeled. To obtain labels for the unlabeled samples, the active learner has to ask an annotator (e.g., a human expert), generally called oracle, for labels. In most cases, the goal is to maximize some metric assessing the task performance (e.g., the classification accuracy) and to minimize the number of queries at the same time. Therefore, active leaning strategies aim at acquiring the labels of the most useful instances. However, many of those strategies assume the presence of an omniscient annotator providing the true label for each instance. But humans are not omniscient, they are error-prone. Thus, the previous assumption is often violated in real-world applications, where multiple error-prone annotators are responsible for labeling. First, the concept of dedicated collaborative interactive learning is described with focus on the first two research challenges: uncertain and multiple uncertain oracles. Next, the state-of-the-art in the field of active learning is presented by an extended literature review. As there is a lack of publicly available data sets that contain information regarding the degree of belief (confidence) of an annotator regarding the provided labels, methods for realistically simulating uncertain annotators are introduced. Then, a first approach that considers the confidences provided by an annotator and transforms them into gradual labels is presented. The suitability of the gradual labels is evaluated in a case study with two annotators that label 30 000 handwritten images. Afterward, the meritocratic learning is introduced, which adopts the merit principle to select annotators for labeling an instance and to weigh their provided labels. By preferring superior annotators, a better label quality is reached at smaller labeling costs. These important steps pave the way to future dedicated collaborative interactive learning, where many experts with different expertise collaborate, label not only samples but also supply knowledge at a higher level such as rules, with labeling costs that depend on many conditions. Moreover, human experts may even profit by improving their own knowledge when they get feedback from the active learner.	eng
dcterms.accessRights	open access
dcterms.creator	Calma, Adrian
dcterms.dateAccepted	2020-01-24
dcterms.extent	vi, 150 Seiten
dcterms.isPartOf	Intelligent Embedded Systems ;; Band 16	eng
dc.contributor.corporatename	Kassel, Universität Kassel, Fachbereich Elektrotechnik / Informatik
dc.contributor.referee	Sick, Bernhard (Prof. Dr.)
dc.contributor.referee	Krempl, Georg (Prof. Dr.)
dc.publisher.place	Kassel
dc.relation.isbn	978-3-7376-0874-9
dc.subject.swd	Maschinelles Lernen	ger
dc.subject.swd	Lernendes System	ger
dc.subject.swd	Datenerfassung	ger
dc.title.subtitle	Towards Dedicated Collaborative Interactive Learning	eng
dc.type.version	publishedVersion
dcterms.source.series	Intelligent Embedded Systems
dcterms.source.volume	Band 16
kup.iskup	true
kup.order	https://www.genialokal.de/Produkt/Adrian-Calma/Active-Learning-with-Uncertain-Annotators_lid_44489410.html
kup.price	39,00
kup.series	Intelligent Embedded Systems
kup.subject	Naturwissenschaft, Technik, Informatik, Medizin
kup.typ	Dissertation
kup.institution	FB 16 / Elektrotechnik / Informatik
kup.binding	Softcover
kup.size	DIN A5