Quality score formula
How Quality Score is calculated?
Quality score reflects the number of mistakes made per 1000 words of translated text.
The formula is: Q=E/V*1000, where
- Q - quality score
- E - the number of mistake points scored for the selected piece of text
- V - the volume of text in which the mistakes were found.
E.g. the text contains 500 words and contains 12 mistake points, the quality score will be 12/500*1000=24
Quality score reflects the number of mistakes made per 1000 words of translated text.
The formula is:
Quality score = Σ(mistake_severity.score * mistake_type_spec.weight) * project.evaluation_corrected_word_count / project.evaluation_sample_word_count / project.evaluation_total_word_count * 1000, where
- Quality score - the number of "base" mistakes per 1000 words ("base" mistake - the mistake that has the severity score of 1 and weight coefficient of 1).
- Σ - the sum of all the products of the mistake severity score and weight coefficient of mistake type per specialization.
- mistake_severity.score - mistake severity score.
- mistake_type_spec.weight - weight coefficient of mistake type per project specialization.
- project.evaluation_corrected_word_count - source words in corrected units of evaluation sample.
- project.evaluation_sample_word_count - total source words of evaluation sample.
- project.evaluation_total_word_count - total source words after evaluation start filter.
How the number of translated words is selected?
Before starting the evaluation, the reviewer selects the number of words to evaluate:
To make it simpler, let’s make an example:
Imagine that the whole translated text contains 10000 words, and the editor corrects the segments containing 2000 words. You select random 300-words sample out of 2000 corrected words, and score 10 mistake points in these 300 words. The system calculates the proportion and count that 2000 words would contain 67 mistake points (10/300*2000). So you have 67 mistake points per 10000 words of translated text, which means 6.7 points per 1000 words. So the quality score is 6.7 mistake points. You’ll be able to see these numbers on the evaluation page:
And the system will use the total number of words as V in the formula above.
Why is it made that way?
We came to this through several stages of evolution.
First, the system was selecting just the beginning of the text, and we found out that the translators started to translate first 1000 words better than the rest of the text. So we decided to select the random part of the text. But then the system sometimes selected the pieces containing no corrections while skipping heavily corrected parts. So we changed the logic, and now the system returns the required number of corrections to the evaluator, but remembers how much text it took to find these corrections.
