Article CC BY 4.0
refereed
published

An extensive review of tools for manual annotation of documents

ORCID
0000-0002-6488-2394
Affiliation
German Federal Institute for Risk Assessment (BfR), Department 9 - Experimental Toxicology and ZEBET, Unit 94 - Animal Welfare and Knowledge Transfer, Berlin, Germany
Neves, Mariana;
ORCID
0000-0001-7426-0508
Affiliation
German Centre for the Protection of Laboratory Animals (BfR), German Federal Institute for Risk Assessment (BfR), Germany
Ševa, Jurica

MOTIVATION:
Annotation tools are applied to build training and test corpora, which are essential for the development and evaluation of new natural language processing algorithms. Further, annotation tools are also used to extract new information for a particular use case. However, owing to the high number of existing annotation tools, finding the one that best fits particular needs is a demanding task that requires searching the scientific literature followed by installing and trying various tools.
METHODS:
We searched for annotation tools and selected a subset of them according to five requirements with which they should comply, such as being Web-based or supporting the definition of a schema. We installed the selected tools (when necessary), carried out hands-on experiments and evaluated them using 26 criteria that covered functional and technical aspects. We defined each criterion on three levels of matches and a score for the final evaluation of the tools.
RESULTS:
We evaluated 78 tools and selected the following 15 for a detailed evaluation: BioQRator, brat, Catma, Djangology, ezTag, FLAT, LightTag, MAT, MyMiner, PDFAnno, prodigy, tagtog, TextAE, WAT-SL and WebAnno. Full compliance with our 26 criteria ranged from only 9 up to 20 criteria, which demonstrated that some tools are comprehensive and mature enough to be used on most annotation projects. The highest score of 0.81 was obtained by WebAnno (of a maximum value of 1.0).

Cite

Citation style:
Could not load citation form.

Access Statistic

Total:
Downloads:
Abtractviews:
Last 12 Month:
Downloads:
Abtractviews:

Rights

Use and reproduction: