RulesFest 2011 – Kenny Shi: Scalability in a Real-Time Decision Platform

My friend Kenny has decided to also talk about performance.  Given the magnitude of the traffic on eBay and the number of rules they need to activate, I can’t think of a best person to talk about scalability issues.  Disclaimer, Kenny will be my co-presenter next week at Business Rules Forum 😉

So many ways to fraud…  We know quite a bit about credit card and debit card fraud, and a little about insurance fraud, from our FICO days.  Fraud for online e-commerce is quite a bit more diverse.  Those fraudsters are incredibly creative when it comes to finding weaknesses in the system.  Kenny’s examples of fraud always impressed me as I do not believe I could have come up with nearly half of those scams!

The fraud detection architecture applies rules and models as a service, invoked either synchronously (SOAP interface to the application) or asynchronously (via a message bus).  The presentation focuses on making faster and more scalable real-time decisions.

From a developer / architect perspective, it would be more comfortable to preocess the cases offline of course but the nature of the business requires more timely detection – both in terms of revenue protection and customer satisfaction.

Fraud detection is the result of expert rules and data-driven analytics.  Neural nets can detect trends that are buried in past transactions.  Rules address flash fraud detection, e.g. patterns identified by fraud experts before models have had a chance to learn about those.  As expected both of them require a ton of data to make good decisions, including some sophisticated computations.  Data retrieval and normalization takes most of the transaction time.  In the end, neural net processing represents only 2% of the time, and rules 3%.

Logically, many performance efforts by Kenny’s team have been focused on data.  Data access is one area that Kenny covered.  More interestingly, he analyzed other related avenues that relate more directly to decisioning “optimizations”:

• Unnecessary variables
• Excessive logic
• Conflicting logic

Independently, optimization techniques:

• Pre-fetch — although it goes against the very flexible and dynamic nature of rules
• Lazy load: only when needed, with smart ordering of conditions from most discriminating / cheaper to least — not always easy to estimate though
• Pre-aggregation of data — freshness can be addressed by aggregating up to “yesterday” for example and then look at real-time data for the current day only

Distributed locality could be addressed with distributed hash tables and other architecture tactics that Kenny quickly went through.

Horizontal scability with server clusters, etc.  Database partition can help, based on data access patterns — per seller, per IP address.

Scalability for Rule Authoring

Sharing variables implies srong capabilities to catalog and describe variables so that all modelers and rules writers can safely and conveniently reuse those characteristics.

Great plug for Sparkling Logic — Thanks!  But seriously, Kenny really appreciated the fluid metaphors we came up with, which help address that very set of problems.

Scalability for Rule Deployment

The continuous integration of rules allows to detect problems earlier and more frequently.  As an added bonus the deployment is ready when it is time to deploy.

Statistical testing is similar to Champion Challenger in the sense that it allows the experts to compare the Production rules with the new Testing rules.  Kenny went a little fast

Other techniques like smoketests and incremental deployment allow to test without impacting the entire workload.

Dog-food approach: Kenny also uses rules to monitor rules.  Smart.

As you can imagine, constant monitoring allows to ensure that they are on-target with their performance objectives.

Closing remark: Fraudsters are very creative so catching them is like a game of Catcha Mouse!

Wow…  What a fast speaker!

Check out the abstract here.