Findings presented at the Sixth Conference on Machine Translation (WMT21) in November 2021 confirm that traditional lexical overlap metrics like BLEU correlate poorly with human judgments on high-quality NMT, driving industry adoption of neural metrics like COMET.

The Limitations of N-Gram Overlap (BLEU) in the Neural Era

Since its introduction by IBM researchers in 2002, the BLEU (Bilingual Evaluation Understudy) score has served as the default benchmark for machine translation. BLEU operates by counting exact word and n-gram overlaps between a machine translation output and a human reference translation. However, as demonstrated extensively at the WMT21 Metrics Shared Task in November 2021, surface n-gram matching penalizes valid synonyms, legitimate paraphrases, and flexible word order in morphologically rich languages, while failing to penalize critical errors such as dropped negations or inverted named entities.

How COMET Leverages Cross-Lingual Embeddings and Human MQM Data

Developed by Unbabel and academic partners, COMET (Crosslingual Optimized Metric for Evaluation of Translation) uses large pretrained multilingual language models (such as XLM-RoBERTa) fine-tuned directly on human quality judgments, Direct Assessments (DA), and Multidimensional Quality Metrics (MQM) error annotations. Because COMET evaluates both the source sentence and the target translation in a shared semantic vector space, it correlates far more closely with professional human revisers—enabling LSPs to benchmark NMT engines accurately and predict post-editing effort.

Benötigen Sie zertifizierte mehrsprachige Fachübersetzungen, beglaubigte Urkundenübersetzungen oder ISO-konforme Lokalisierung für Ihre internationalen Projekte? Fordern Sie ein kostenloses maßgeschneidertes Angebot von Alphatrad an und erhalten Sie innerhalb von 24 Stunden einen detaillierten Ablaufplan.