Textual Entailment

In this evaluation exercise systems have to detect inferential relationships between pairs of short texts. We will focus on the entailment relation. In terminology and evaluation method we follow by and large the Pascal RTE exercises for English.

A pair of texts consists of T (for text) and H (hypothesis). Systems have to decide whether T entails H, which is the case when the truth of H can be inferred from T within the context induced by T. Another way of thinking about this is whether H contains any new information with respect to T: if it doesn’t, H is entailed by T.

Here are two examples to illustrate the task:

  • Example 1: not entailed T: Il primo acquisto immobiliare fu un terreno in via Alciati a Milano, per 190 milioni di lire. H: Il primo acquisto è un terreno in via Alciati a Milano.
  • Example 2: entailed T: Il sito ufficiale parla di attività nei panni di direttore commerciale e, dopo sei mesi, di direttore generale. H: Il sito ufficiale parla di attività di direttore commerciale e, dopo sei mesi, di direttore generale.

The training and test data will consist of several hundred examples of such pairs, equally divided into positive and negative examples. Performance is measured by accuracy (number of correctly recognised pairs divided by the total number of pairs).

Task materials

Detailed Guidelines

Data Distribution

The test data are now available. The development set for the Textual Entailment task is now available. The full Italian RTE Corpus (with gold test data).

Organizers

  • Johan Bos (Uni. Roma “La Sapienza”)
  • Marco Pennacchiotti (Saarland University)
  • Fabio Massimo Zanzotto (Uni. Roma “Tor Vergata”)