By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    customer experience analytics
    Using Data Analysis to Improve and Verify the Customer Experience and Bad Reviews
    6 Min Read
    data analytics and CRO
    Data Analytics is Crucial for Website CRO
    9 Min Read
    analytics in digital marketing
    The Importance of Analytics in Digital Marketing
    8 Min Read
    benefits of investing in employee data
    6 Ways to Use Data to Improve Employee Productivity
    8 Min Read
    Jira and zendesk usage
    Jira Service Management vs Zendesk: What Are the Differences?
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: RulesFest 2011 – Kenny Shi: Scalability in a Real-Time Decision Platform
Share
Notification Show More
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Business Intelligence > Business Rules > RulesFest 2011 – Kenny Shi: Scalability in a Real-Time Decision Platform
Business Rules

RulesFest 2011 – Kenny Shi: Scalability in a Real-Time Decision Platform

CMatignon
Last updated: 2011/10/25 at 4:55 PM
CMatignon
5 Min Read
SHARE

My friend Kenny has decided to also talk about performance.  Given the magnitude of the traffic on eBay and the number of rules they need to activate, I can’t think of a best person to talk about scalability issues.  Disclaimer, Kenny will be my co-presenter next week at Business Rules Forum 😉

My friend Kenny has decided to also talk about performance.  Given the magnitude of the traffic on eBay and the number of rules they need to activate, I can’t think of a best person to talk about scalability issues.  Disclaimer, Kenny will be my co-presenter next week at Business Rules Forum 😉

Kenny ShiSo many ways to fraud…  We know quite a bit about credit card and debit card fraud, and a little about insurance fraud, from our FICO days.  Fraud for online e-commerce is quite a bit more diverse.  Those fraudsters are incredibly creative when it comes to finding weaknesses in the system.  Kenny’s examples of fraud always impressed me as I do not believe I could have come up with nearly half of those scams!

The fraud detection architecture applies rules and models as a service, invoked either synchronously (SOAP interface to the application) or asynchronously (via a message bus).  The presentation focuses on making faster and more scalable real-time decisions.

From a developer / architect perspective, it would be more comfortable to preocess the cases offline of course but the nature of the business requires more timely detection – both in terms of revenue protection and customer satisfaction.

Fraud detection is the result of expert rules and data-driven analytics.  Neural nets can detect trends that are buried in past transactions.  Rules address flash fraud detection, e.g. patterns identified by fraud experts before models have had a chance to learn about those.  As expected both of them require a ton of data to make good decisions, including some sophisticated computations.  Data retrieval and normalization takes most of the transaction time.  In the end, neural net processing represents only 2% of the time, and rules 3%.

Logically, many performance efforts by Kenny’s team have been focused on data.  Data access is one area that Kenny covered.  More interestingly, he analyzed other related avenues that relate more directly to decisioning “optimizations”:

  • Unnecessary variables
  • Excessive logic
  • Conflicting logic

Independently, optimization techniques:

  • Pre-fetch — although it goes against the very flexible and dynamic nature of rules
  • Lazy load: only when needed, with smart ordering of conditions from most discriminating / cheaper to least — not always easy to estimate though
  • Pre-aggregation of data — freshness can be addressed by aggregating up to “yesterday” for example and then look at real-time data for the current day only

Distributed locality could be addressed with distributed hash tables and other architecture tactics that Kenny quickly went through.

Horizontal scability with server clusters, etc.  Database partition can help, based on data access patterns — per seller, per IP address.

Scalability for Rule Authoring

Sharing variables implies srong capabilities to catalog and describe variables so that all modelers and rules writers can safely and conveniently reuse those characteristics.

Great plug for Sparkling Logic — Thanks!  But seriously, Kenny really appreciated the fluid metaphors we came up with, which help address that very set of problems.

Scalability for Rule Deployment

The continuous integration of rules allows to detect problems earlier and more frequently.  As an added bonus the deployment is ready when it is time to deploy.

Statistical testing is similar to Champion Challenger in the sense that it allows the experts to compare the Production rules with the new Testing rules.  Kenny went a little fast

Other techniques like smoketests and incremental deployment allow to test without impacting the entire workload.

Dog-food approach: Kenny also uses rules to monitor rules.  Smart.

As you can imagine, constant monitoring allows to ensure that they are on-target with their performance objectives.

Closing remark: Fraudsters are very creative so catching them is like a game of Catcha Mouse!

 

Wow…  What a fast speaker!

Check out the abstract here.

 

Powered By WizardRSS.com | Full Text RSS Feed | Amazon Plugin | Settlement Statement

CMatignon October 25, 2011 October 25, 2011
Share This Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

ai can help with nurse burnout
Breakthroughs in AI Are Helping to Prevent Nurse Burnout
Artificial Intelligence Exclusive
AI in marketing
AI Can’t Replace Creativity When Crafting Digital Content
Artificial Intelligence
ai in furniture design
Top 5 AI-Driven Furniture Engineering Design Applications
Artificial Intelligence
data protection regulation
Benefits of Data Management Regulations for Consumers & Businesses
Data Management

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

big data and accounting
Business RulesData ManagementITSoftware

Strategies to Make Better Profits for CPAs During Tax Season

10 Min Read
Business,Analytics,(ba),Technology,Using,Big,Data,,Cloud,Computing,And
Analytics

What to Consider When Choosing a Masters in Business Analytics

15 Min Read
Benefits of IoT in Small Business
Internet of Things

Understanding the Tremendous Benefits of IoT for Small Businesses

8 Min Read
cybersecurity tips for data centric businesses
Security

5 Essential Cybersecurity Tips For Data Centric Businesses In 2021

5 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
ai chatbot
The Art of Conversation: Enhancing Chatbots with Advanced AI Prompts
Chatbots

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?