Recommendation engines are the digital version of the pot of gold at the end of the Big Data rainbow. Imagine being able to know, in near-time, what your customers really want and when they want it. For a subscription service, the gold is even brighter: You can deliver your customers’ desires directly to them, leading to a dramatic reduction in churn.
But that’s assuming you manage to create a recommendation engine that actually works. We’ve all had the experience, as consumers, of seeing websites offer us “personalized recommendations” for products and services that have nothing to do with what we really want, need, or could be persuaded to try.
The Keys to Recommendation Success
Recommendation engines sound like they should be easy to build—you’re simply filtering available offerings based on a customer’s past likes. The problem is that what your customer likes today may be of no interest to them tomorrow.
Good recommendation engines weigh the big picture against sudden interests—does your customer really want to know about puppy training or listen to Mississippi Blues, or was their interest in a topic or genre a passing whim, or research for a now-finished project? And do you want to push more of the same on customers, or would they prefer to explore different, but related, aspects of their primary interests?
The decisions that need to be made are many, but one thing is certain: The more data you have, the better you can segment customers to compare their likes and dislikes and produce valid, interesting recommendations for the content that they truly want to consume.
Basic Building Blocks
A recommendation engine will obviously work best if a business has a sizable subscriber user base and a significant inventory of content. Assuming that you have both, your first big decision is whether you wish to use a collaborative filter approach or the content-centric approach.
A collaborative filtering algorithm utilizes user ratings and other user behavior to make predictive recommendations on what other users within the same segment might like. The recommendation engine has no understanding of what it is recommending—whether a movie is really interesting, or the music is catchy— instead relying on millions of bits of data to make suggestions. The more data, the better the recommendations. And the more customers rank the suitability of the recommendations, the smarter the engine gets. This is probably the most effective approach, but is not suitable for a brand new business that lacks the data to power it.
A content-based approach requires excellent tagging skills for the content the subscription service is offering, which will be analyzed against user data such as ratings, behavior, and their specific interests. To deliver the recommended content in real-time, you’ll almost certainly need to budget a serious chunk of your budget of both money and time to algorithmic development, as it’s unlikely you’d be able to acquire anything suitable as a commodity product. Personalized recommendation services need to be … personalized.
Getting Personal with Your Customers
After deciding whether your engine will be fueled by a content or collaborative approach, you’ll want to mine customer profile and usage information to develop persona-based segments based on the content you already know that specific group of users enjoys/values. For example, a news subscription service may have a segment of subscribers who are intensely interested in political news, while another group is fully focused on the financial markets. With some users, you may want to push breaking news, while others might enjoy long-form content that they can leisurely peruse during a long commute.
When you have your segments in place, you can repurpose the data to develop tempting offerings tailored for each group to reward best customers and lure new ones. You can also utilize user data in tandem with customer usage patterns, support interactions, and social media conversations to predict when a major churn event is looming on the horizon—perhaps due to a competitor’s new offer, or dissatisfaction with your offering. This same data will reveal what a business is doing right so that you can do more of it.
Put Your Data to Work
The issue with Big Data is that it’s BIG. Storing and processing it can eat up network resources and budgets. Segregated silos can slow the flow of the data stream, impacting availability. Long waits for IT to prepare data for analysis will thwart any attempt at serving up recommendations in real-time—and real-time capacity is critical for recommendation engines. These and other complications result in data—and a recommendation engine—that doesn’t live up to its business potential.
Apache Hadoop addresses all of the problems that plague Big Data-driven initiatives. It provides affordable storage on commodity machines and fully supports data integration from multiple sources across different data storage technologies. Used in conjunction with tools such as Apache Drill, it enables self-service data exploration using unstructured, semi-structured, and structured data.
If you’re interested in learning more about machine learning, I suggest reading the free ebook, Practical Machine Learning: Innovations in Recommendation by Ted Dunning.
