Concept Trending : A Glimpse into the future?

In the previous post some ideas were presented on the trends of Text Analytics. Analyzing and extracting knowledge from text is a hard thing, whether this involves Sentiment Analysis, Text Classification, Cluster Analysis or Information Extraction.

A particularly interesting application of Text Analytics is the identification of trends for specific concepts. In contrast with simple keyword trending, this type of trending attempts to disambiguate keywords according to their context and use co-reference resolution to identify the subjects for which the sentiment relates to.

In the previous post some ideas were presented on the trends of Text Analytics. Analyzing and extracting knowledge from text is a hard thing, whether this involves Sentiment Analysis, Text Classification, Cluster Analysis or Information Extraction.

A particularly interesting application of Text Analytics is the identification of trends for specific concepts. In contrast with simple keyword trending, this type of trending attempts to disambiguate keywords according to their context and use co-reference resolution to identify the subjects for which the sentiment relates to.

To better understand concept trending let’s look at an example : Suppose that one wishes to identify the trend of negative characterizations -and even swear words- that exist on the Greek web. The first step would be to collect the information from various blogs and forums whenever a negative keyword is found. A Text analysis toolkit could then provide the means of identifying the subject(s) of negative characterizations on the Greek web such as Politicians, the Economy or the International Monetary Fund which recently came in to the rescue.

From a post dated December 28th, 2009 :

“Over the past month there has been a considerable amount of increase in negative economy sentiment, crime-related incidents and/or terms that communicate future social instability and uneasiness.”

Although not stated on purpose, the country which the article addressed was Greece and the trend increase on negative sentiment was found to be starting in the beginning of December 2009. This is a photo of a Greek newspaper taken on February 4, 2010

The title shown writes about the “Fear of Social Explosion”. On May 6th 2010 after clashes in the center of Athens, mentions about “Social Explosion” in Greece started appearing on the Web. The following Google search uses a timeline for “Social Unrest”. The increase of mentions appears to be starting on February 2010.

Although concept trending has significant challenges it is a process which in my experience has proven itself many times. A recent article at NewScientist suggests that by capturing the sentiment of the crowds we are able to predict the moves of S&P 500 or by looking at keyword searches such as “job search engine” we can predict coming changes of the US unemployment rate.