EValuation of Events aNd Temporal Information (EVENTI)
The purpose of EVENTI is to promote research in Temporal Processing for Italian. This evaluation exercise will provide the chance to test a newly developed language resource, the Ita-TimeBank (Caselli et al., 2011a) while making it available to the NLP community as one of the largest manually annotated data sets for Temporal Processing.
Following the experience of the previous TempEval Evaluation Exercises, the EVENTI task will consist of 4 subtasks:
- Task A: determine the extent and the normalization of temporal expressions (i.e. timex) in a text according to the TimeML TIMEX3 tag. Empty TIMEX3 tags, as specified in the TimeML Annotation Guidelines, will be taken into account as well.
- Task B: determine the extent of the events in a text according to the TimeML EVENT tag definition adapted to Italian. In addition, determine the values of the feature CLASS.
- Task C: determine temporal relations from raw text. This task involves performing Task A and Task B and subsequently identifying pairs of elements (event\event and event\timex pairs in the same sentence) which stand in a temporal relation (TLINK) and classifying temporal relations among them. As in TempEval-3, all temporal relation values are used.
- Task D: determine the temporal relations given two gold items. In particular, given the pair of elements (event\event or event\timex pairs) that have a temporal link, classify the temporal relation between them with the same values as in task C.
Pilot Task: Temporal Processing of Historical Texts
Participants will be asked to run their systems on a corpus of about 5,000 tokens following the same sub-task defined for the main task.
No training data will be provided, while the test set will consist of newspaper articles published in “Il Trentino” by Alcide De Gasperi in 1914.
Participants will be required to process the texts with the same system used for the main task. The results obtained will be used to analyze how well systems built for contemporary languages perform on historical texts. In addition, participants can use any freely available training data for building a new system or for adapting an already existing one.
Participants in both tasks may use any additional resources (lexica, Wikipedia, knowledge basis, etc.) a description is included in the final report.
Detailed guidelines, task materials and data sets for development, training and testing will be made available on the Task Website.
Dataset: Training, test and gold data are available.
Tommaso Caselli (TrentoRISE)
Rachele Sprugnoli (Fondazione Bruno Kessler, University of Trento)
Manuela Speranza (Fondazione Bruno Kessler)
Monica Monachini (ILC-CNR)