Why Neural Metrics Like COMET Are Replacing BLEU in Machine Translation Evaluation (WMT21)

Findings presented at the Sixth Conference on Machine Translation (WMT21) in November 2021 confirm that traditional lexical overlap metrics like BLEU correlate poorly with human judgments on high-quality NMT, driving industry adoption of neural metrics like COMET.

The Limitations of N-Gram Overlap (BLEU) in the Neural Era

Since its introduction by IBM researchers in 2002, the BLEU (Bilingual Evaluation Understudy) score has served as the default benchmark for machine translation. BLEU operates by counting exact word and n-gram overlaps between a machine translation output and a human reference translation. However, as demonstrated extensively at the WMT21 Metrics Shared Task in November 2021, surface n-gram matching penalizes valid synonyms, legitimate paraphrases, and flexible word order in morphologically rich languages, while failing to penalize critical errors such as dropped negations or inverted named entities.

How COMET Leverages Cross-Lingual Embeddings and Human MQM Data

Developed by Unbabel and academic partners, COMET (Crosslingual Optimized Metric for Evaluation of Translation) uses large pretrained multilingual language models (such as XLM-RoBERTa) fine-tuned directly on human quality judgments, Direct Assessments (DA), and Multidimensional Quality Metrics (MQM) error annotations. Because COMET evaluates both the source sentence and the target translation in a shared semantic vector space, it correlates far more closely with professional human revisers—enabling LSPs to benchmark NMT engines accurately and predict post-editing effort.

Hai bisogno di traduzioni multilingue certificate, asseverazioni legali o localizzazione conforme agli standard ISO per i tuoi progetti internazionali? Richiedi un preventivo gratuito personalizzato ad Alphatrad e ricevi una risposta dettagliata entro 24 ore.

Editorial & Copyright Attribution Note: This industry analysis has been independently synthesized and adapted by Alphatrad’s editorial team based on verified regulatory documentation and industry research originally published by ACL Anthology – WMT21 Conference on Machine Translation. For the original publication and primary documentation, please consult:
Source: https://aclanthology.org/volumes/2021.wmt-1/

Share

Share

Follow Us