New Insights from Text Analytics

Text Analytics has gained the attention it deserves in the past few years. Sentiment Analysis is perhaps the most frequently discussed type of analysis but there will be always new ways to analyze and gain insights from text data.  

Examples of new types of analysis – and they have a vast potential – are in my opinion two:  Sequence Detection and Concept Mining. I am not aware whether these types of analysis are currently being implemented by any Text Mining practitioner at the moment and if there is one, feel free to add your comments below.

So what is Sequence Detection and Concept Mining ? Some examples :

Suppose that you receive several similar e-mails sent from customers as the one seen below :

“I have been trying repeatedly to solve my billing problem through customer care. I first talked with someone called  Mrs Jane Doe. She said she should transfer my call to another representative from the sales department. Yet another rep from the sales department informed me that i should be talking with the Billing department instead. Unfortunately my bad experience of being transferred through various representatives was not over because the Billing department informed me that i should speak to the……“

Currently Text Analytics software will identify key elements of the above text but a very important piece of information goes unnoticed. It is the sequence of events which takes place :

 (Jane Doe => Sales Dept =>Billing  Dept =>…)

Being able to detect the sequence of events is an important element in understanding customer interaction. In our example above, imagine the possibility of detecting similar sequences through thousands of e-mails or call center transcripts and running a sentiment analysis, a process which then could correlate sentiment with specific event sequences.

Next,  is the usage of Concept Mining (this is just a phrase i coined for this post) : Being able to analyze information to different conceptual levels. A very powerful technique indeed and let’s see why this is so.

People that have attended the 7th annual Text Analytics Summit in Boston had the opportunity to listen to several presentations regarding Semantics. The discussions between experts from the Semantics Panel and the attendees revealed that people could not find Semantics practical for several reasons. Yet, in Semantics lies the power of being able to find patterns on different conceptual levels.

As a -very basic- example, if we use Information Extraction to annotate -say- the Tweets containing mentions of American Telcos we can tag each one as a more general category called TELCOS. We can also tag individual prepaid packages as a more general category called PREPAID_PACKAGES. By doing that we can then search for patterns in a more general conceptual level than searching for patterns only at a Telco Brand level or a specific Telco’s prepaid package. As an example we can run  a sentiment analysis on all prepaid packages mentions,  identify patterns of negative or positive sentiment and see which Telco is the winner of positive sentiment at a conceptual level.

The possibilities are endless.