I recently posted an article, Metadata, Connection, and the Big Data Story, covering the big-data analysis process as applied to “human data” that is communicated in intentionally expressive sources such as text, video, and social likes and shares and in implicit expressions of sentiment.
I recently posted an article, Metadata, Connection, and the Big Data Story, covering the big-data analysis process as applied to “human data” that is communicated in intentionally expressive sources such as text, video, and social likes and shares and in implicit expressions of sentiment.
The article is spun out from Q&A interview of four industry figures: Fernando Lucini (HP Autonomy), Marie Wallace (IBM), Elliot Turner (AlchemyAPI), and Stephen Pulman (University of Oxford and TheySay). (AlchemyAPI is sponsoring my up-coming market study, “Text Analytics 2014: User Perspectives on Solutions and Providers.”) Read each interview by clicking or tapping on the name of the interviewee.
This interview is —
1) What’s the really interesting information content that we’re not really getting at yet, and what’s interesting about it?
Companies always want to know what consumers will buy and when they will buy it. But Intent Mining is still in its infancy. Today, it is easy to classify the topics and sentiment of an individual post, or roll up how Twitter feels about the top entities extracted from a real-time hashtag feed. The performance of these classification and ranking processes will get better, but they are just first components of future solutions that seek to predict an individual’s purchasing behavior. Success in this task will combine other elements like a person’s interests, relationships, geography – and ultimately their identity, purchase history and privacy preferences – so that applications can plot where a person is in their “buyer’s journey” and provide the best offers at the best times.
Going beyond text, other areas for big progress are in the mining of audio, speech, images and video. These are interesting because of their incredible growth. For example, we will soon see over 1 billion photos/day taken and shared from the world’s camera phones. Companies with roots in unsupervised deep-learning techniques should be able to leverage their approaches to dramatically improve our ability to correctly identify the content contained in image data.
2) How well are we doing with Natural Language Processing, noting that formally, “processing” includes both understanding and generation, two parts of a conversation?
Google has trained us to search using keywords, and this won’t change overnight. But the trend is easy to spot: the interactive question-answering capabilities made famous by IBM’s Watson will become commonplace, offered at a fraction of today’s costs and made available as easy-to-integrate web services.
The goal of interactive search is to let us get faster answers to the detailed questions we ask every day. Early, mass adopters of new Question Answering applications include large support organizations with complex products where the entire business model can be improved with faster time-to-answer. As interactive Question Answering systems move beyond professional analysts we will see search and retrieval transform to become dialog-based and be highly aware of an ongoing context – machines will “stay in conversation” and not treat each search as a unique event, but encourage ad-hoc drill down or expansion of answers already obtained.
3) And how well are we able to mine and automate understanding of affective states, of mood, emotion, attitudes, and intent, in the spectrum of sources available to us?
Work is on-going to move beyond a binary classification of sentiment to identifying richer emotional states, and applying these results to the best way a company should engage an individual. But gaining a deeper, more valuable understanding of an individual’s intent, and tying this intent to the specific call-to-action a company could make, are big tasks that are just getting off the ground.
A never-ending challenge to understanding text is staying current with emerging slang and phrases. Unsupervised learning can enable machines to discover new words without human-curated training sets.
4) Deep learning, active learning, or maybe some form of machine learning that’s being cooked up in a research lab: What business benefits are delivered by these technologies, and what are the limits to their usefulness, technical or other?
Few NLP companies are truly deploying deep learning methods, even though deep learning can give us a much richer representation of text than is possible with traditional NLP approaches. Deep learning can produce more robust text and vision systems that hold their accuracy when analyzing data far different from what they were trained on. Plus, unsupervised techniques make it practical to keep up with the rapid evolution of everyday language. Did you know that 15 percent of the daily queries submitted to Google – over 500 million – have never been seen before by Google’s search engine? And this rate of previously unknown phrases is unchanged in the past 15 years? The ability to understand today’s short, jargon-filled phrases, and keep up with tomorrow’s new words, are predicated on mastering unsupervised, deep learning approaches.
With respect to limitations, unsupervised deep learning isn’t magic. It isn’t cheap or easy, either. It first requires some new skills for driving continued innovation in machine learning. Second, we need massive training data sets in order to build hierarchical representations of data without human involvement. In fact, we believe AlchemyAPI’s algorithms are trained on data corpora orders of magnitude larger than most competitive solutions. Finally, processing all of this information at such huge scale requires technical innovations all their own, coupled with significant amounts of affordable computing power. Thankfully, all these limitations are being addressed today to help bring about the next generation of intelligent machines.
5) Mobile’s growth is only accelerating, complicating the data picture, accompanied by a desire for faster, more accurate, and more useful, situational insights delivery. How is your company keeping up?
Almost overnight, Mobile has enabled an explosion in image capture to where we will soon surpass 1 billion photos taken and shared each day. AlchemyAPI is investing heavily in computer vision technologies used to help machines quickly understand the content of all these photos in order to drive a host of new commerce, content and social applications.
Response time is perhaps THE critical driver for mobile applications. Our obsession with speed of results and low latency through our cloud infrastructure leads to AlchemyAPI being consistently chosen over competitors whenever NLP-based applications require near real-time processing of unstructured data.
6) Where does the greatest opportunity reside, for your company as a solution provider? Internationalization? Algorithms, visualization, or other technical advances? In data integration and synthesis and expansion to new data sources? In monetizing data, that is, yourselves, or via partners, or assisting your customers? In untapped business domains or in greater uptake in the domains you already serve?
Today, AlchemyAPI is the world’s most popular text analysis cloud service. Our obsession with speed and accuracy of our results has resulted in being selected by designers of innovative, near real-time NLP applications to process billions documents each month. We will continue to advance our algorithms while adding new deep learning-based features including a new classifier, a computer vision API and advanced question-answering solutions.
Our greatest opportunity is to provide development teams a programmable Internet platform that makes it easy for their companies to create and deploy NLP applications in the cloud. The Alchemy platform can include built-in data sources, NLP functionality, storage of results, data analysis and reporting. Customer apps will run on a secure service that scales, tunes, and backs up data automatically. Over time, visual tools, sample UIs and our library of components can minimize custom coding and streamline the assembly of NLP apps using a building-block approach. These apps then integrate easily into a customer’s product lines and business systems using open APIs.
7) Anything to add, regarding the 2014 outlook for analytical and semantic and sensemaking technologies?
Being a provider of NLP-based Web services gives AlchemyAPI a unique market perspective. We help our customers build new applications that serve nearly every industry vertical and run on every computing device. Because we are a subscription service, AlchemyAPI is also able to get a sense of which applications are succeeding and how fast they are growing.
What we see leads us to this conclusion: despite our industry’s penchant for hype, there is an arms race brewing in NLP-driven applications and we see 2014 as a year where much more of this energy moves from experimentation to building and deploying new apps into production.
Two types of businesses we talk to every day include large public companies who are exploring how to incorporate modern text analysis capabilities into their established product lines. The second type are innovative startups who are charging at full speed with business plans aimed squarely at disrupting the established order. It’s this second group that’s adding excitement to our industry. They are intent on making inroads into business intelligence, customer support, advertising and publishing, sales and marketing automation, enterprise search and many other markets. They won’t all make it, but they are generating a lot of new interest, a lot of new customers, and forcing everyone to innovate faster.
Thank you to Elliot!