TASK PROPOSAL SUBMISSION

Tasks proposals should be no longer than 4 pages and should include:

  • task title and acronym;
  • names and affiliation of the organizers (minimum 2 organizers);
  • brief task description, including motivations and state of the art showing the international relevance of the task;
  • description and examples of the data, including information about their availability, development stage, and issues concerning privacy and data sensitivity;
  • expected number of participants and attendees;
  • names and contact information of the organizers.

In submitting your proposal, please bear in mind that we encourage:

  • challenging tasks involving linguistic analysis, e.g., beyond “simple” classification problems;
  • tasks focused on multimodality, e.g., considering both textual and visual information;
  • tasks characterized by different levels of complexity, e.g., with a straightforward main subtask and one or more sophisticated additional subtasks;
  • the re-annotation of datasets from previous years with new annotation types, and texts from publicly available corpora;
  • both new tasks and re-runs. For new tasks, you will have to specify in the proposal why it would attract a reasonable number of participants, and why it is needed;
  • application-oriented tasks, that is tasks that have a clearly defined end-user application showcasing.

The organizers of the accepted tasks should take care of planning, according to the scheduled deadlines (see below):

  • the development and distribution of data sets needed for the contest, i.e. data for training and development, and data for testing;
  • the scorer to be used to evaluate the submitted systems should be included in the release of development data;
  • the development of task guidelines, where all the instructions for the participation are made clear together with a detailed description of data and evaluation metrics applied for the evaluation of the participant results;
  • the collection of participants results;
  • the evaluation of participants results according to standard metrics and baseline(s);
  • the solicitation of submissions;
  • the reviewing process of the papers describing the participants approach and results (according to the template to be made available by the EVALITA-2020 chairs);
  • the production of a paper describing the task (according to the template to be made available by the EVALITA-2020 chairs).

*** Email your proposal in PDF format to evalita2020@gmail.com with "Evalita 2020 TASK Proposal" as the subject line by the submission deadline (February 7th 2020). ***

Please feel free to contact the EVALITA-2020 chairs at evalita2020@gmail.com in case of any questions or suggestions.

Deadlines of the task proposal:

  • February 7th 2020: submission of task proposals
  • March 6th 2020: notification of task proposal acceptance

Tentative timelines of Evalita 2020:

  • 29th May 2020: development data available to participants
  • 4th September 2020: test data available, registration closes
  • 4th - 24th September 2020: evaluation window
  • 2nd October 2018: assessment returned to participants
  • 6th November 2018: technical reports due to organizers (camera ready)