Teaching a computer Chinese made it excellent at English

The algorithm able to understand English best is one designed to learn Chinese.

As noted by Karen Hao of the MIT Technology Review, a computer model built by Chinese search engine and commerce firm Baidu now leads all other models on a benchmark of general English-sentence comprehension known as GLUE.

The model is named ERNIE, and its core advantage is that it was trained to guess phrases, a key to understanding Chinese.

Classic models for natural language understanding, such as GPT-2, are trained to guess a next word based on what preceded it. So, for example, predicting the missing word here:

Harry Potter is a series of fantasy novels written by J. K. ███

Models such as Google’s BERT are trained instead to guess words—known in the business as “tokens”—based on what came before and afterward, taking advantage of the additional context:

███ Potter is a series ███ fantasy novels ███ by J. ███ Rowling.

The Baidu researchers trained ERNIE to predict sets of missing tokens. This is essential for understanding Chinese, in which individual characters rarely work alone, as Hao explains:

While certain characters do have inherent meaning, like fire (火, huŏ), water (水, shuĭ), or wood (木, mù), most do not until they are strung together with others. The character 灵 (líng), for example, can either mean clever (机灵, jīlíng) or soul (灵魂, línghún), depending on its match. And the characters in a proper noun like Boston (波士顿, bōshìdùn) or the US (美国, měiguó) do not mean the same thing once split apart.

So training the ERNIE becomes a guessing game of phrases. This skill, it turns out, transfers nicely to English, enabling it to do well at predicting entire sets of words, such as those missing here:

Harry Potter is ███ ███ ███ fantasy novels ███ by ███ ███ ███.

And that, for that moment, helps make ERNIE English’s next top model.