Crossing the Language Chasm: Extracting Information from Foreign-Language Text

Just a few years ago, text analytics was unknown to most business people, even most data analysts. Today, more and more organizations are not only aware of text analytics, but trying it themselves. With increasing awareness comes expanding expectations: identifying the subject matter of text isn’t enough, the pressure is on to identify sentiment, to find significance in tiny snips of text, to mine massive quantities of text in real time, and to analyze text written in languages that the analyst cannot read.

Just a few years ago, text analytics was unknown to most business people, even most data analysts. Today, more and more organizations are not only aware of text analytics, but trying it themselves. With increasing awareness comes expanding expectations: identifying the subject matter of text isn’t enough, the pressure is on to identify sentiment, to find significance in tiny snips of text, to mine massive quantities of text in real time, and to analyze text written in languages that the analyst cannot read.

Think about that last item one moment: analyze text written in languages the analyst cannot read.  

The main drive behind text analytics so far has been speed – conquering large quantities of text that human investigators simply don’t have time to read and interpret. That’s about to change. As text analytics enters the toolkits of businesses and governments, they are asking for something more – cross-lingual text analytics, tools that enable analysts to mine text written in languages that they themselves do not understand.

Who wants cross-lingual analytics?

Market researchers from Western countries studying consumer preferences in Asia, and vice-versa.

English speakers in the U.S. interested in the political concerns of recent immigrants. 

Intelligence agencies in Freedonia searching for signs of terrorist activity in Moosylvania.

… And so on.

How are they going about this? Some are translating the text to their own language and then applying their usual text analysis process to the translated text. That approach has major problems. If you’ve ever read machine translated text, you know it’s less than perfect, much less than perfect. If text mining is hard with text in its original form, imagine the quality issues that crop up when the text has been machine translated. What’s more, even machine translation can be costly when applied to massive quantities of text. High cost for poor quality is bad news; the good news is that there is better way to go about it.

Cross-lingual text analytics is more accurate and less costly when as much of the process is performed in the original language of the text as possible.  Keep translation to a bare minimum, and use tools built for working with the language you require. Read this outline of the process, and notice how translation is minimized:

The Cross-Lingual Text Analytics Process

Define the topic .

What is the subject matter of the investigation? If the topic is, say, soft drinks, what terms are used to describe soft drinks in English? The list must be complete – so it would include not just the term “soft drinks,” but also “soda,” “pop,” “lemonade” and many others. Some of the terms will be brand names, others slang. And the specific sense of the each term must be identified; you are interested in soda as a fizzy drink, not an ingredient for baking.

Translate the topic

When a thorough and unambiguous definition of the subject matter is complete, it is translated to the target language.

Search

Translated terms form the basis for a text search to identify and retrieve relevant text.

Categorize

Working within the original language of the text, extract information such as subject matter and sentiment in the text. This information can be used to categorize or score text as required.

Summarize

Summarization includes tasks such as calculating the proportion of the text mentioning specific terms or estimating the percentages of positive and negative sentiment.

Tag

Tagging is the annotation of text, adding codes to denote specific functional parts such as titles or names.

Translate text

When all other steps have been performed, the analyst may choose to translate some portion of the original text for examination.

(Note that many investigations do not require all of these steps.)

That’s the right way to go about it. Carefully define and translate search concepts, rather than sloppily translate a mass of text. Use the most efficient search process available. Evaluate text in its original form, not a poorly translated version. And translate only a minimal quantity of relevant text, or none at all. This cross-lingual text analytics process is efficient and accurate.

In a climate of global interaction and competition, much of the information we need is embedded in languages we don’t understand. Extracting that information is not optional, it’s a necessity. Cross-lingual text analytics is in your future, perhaps your near future. Go about it in the wrong way, and you’ll waste resources and produce nothing of value. Understanding and using the right process maximizes the chance of success for your project and your organization.

More on text analytics:

Text Analytics WIIFM (What’s in it for Me?) 

©2011 Meta S. Brown