The data was extracted from news articles published by Xinhua News Agency and Agence France Presse and was obtained using the automatic parallel sentence identification method described in the following publication: Dragos Stefan Munteanu, Daniel Marcu, The corpus contains 1,, sentence pairs; the word count on the English side is approximately 31M words.
The sentences in the parallel corpus preserve the form and encoding of the texts in the original Gigaword corpora. For each sentence pair in the corpus the authors provide the names of the documents from which the two sentences were extracted, as well as a confidence score between 0. The parallel sentence identification approach is designed to judge sentence pairs in isolation from their contexts, and can therefore find parallel sentences within document pairs which are not parallel.
The fact that two documents share several parallel sentences does not necessarily mean the documents are parallel. In order to make this resource useful for research in Machine Translation MT , the authors made efforts to detect potential overlaps between this data and the standard test and development data sets used by the MT community. Sentence pairs in this distribution that have a 7-gram overlap with a sentence pair in a NIST MT evaluation set or sentence pairs coming from documents whose names are similar to those in the NIST MT sets are marked with a negative confidence score.
For each sentence pair in the corpus the authors provide the names of the documents from which the two sentences were extracted, as well as a confidence score between 0.
The parallel sentence identification approach is designed to judge sentence pairs in isolation from their contexts, and can therefore find parallel sentences within document pairs which are not parallel.
The fact that two documents share several parallel sentences does not necessarily mean the documents are parallel. In order to make this resource useful for research in Machine Translation MT , the authors made efforts to detect potential overlaps between this data and the standard test and development data sets used by the MT community.
Sentence pairs in this distribution that have a 7-gram overlap with a sentence pair in a NIST MT evaluation set or sentence pairs coming from documents whose names are similar to those in the NIST MT sets are marked with a negative confidence score. Web Download. Philadelphia: Linguistic Data Consortium, Improving Machine Translation Performance by Exploiting Non-parallel Corpora, Computational Linguistics, 31 4 The corpus contains , sentence pairs the word count on the English side is approximately 16M words.
0コメント