By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    data analytics in sports industry
    Here’s How Data Analytics In Sports Is Changing The Game
    6 Min Read
    data analytics on nursing career
    Advances in Data Analytics Are Rapidly Transforming Nursing
    8 Min Read
    data analytics reveals the benefits of MBA
    Data Analytics Technology Proves Benefits of an MBA
    9 Min Read
    data-driven image seo
    Data Analytics Helps Marketers Substantially Boost Image SEO
    8 Min Read
    construction analytics
    5 Benefits of Analytics to Manage Commercial Construction
    5 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: RulesFest 2011 – Kenny Shi: Scalability in a Real-Time Decision Platform
Share
Notification Show More
Latest News
data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security
ai in software development
3 AI-Based Strategies to Develop Software in Uncertain Times
Software
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Business Intelligence > Business Rules > RulesFest 2011 – Kenny Shi: Scalability in a Real-Time Decision Platform
Business Rules

RulesFest 2011 – Kenny Shi: Scalability in a Real-Time Decision Platform

CMatignon
Last updated: 2011/10/25 at 4:55 PM
CMatignon
5 Min Read
SHARE
- Advertisement -

My friend Kenny has decided to also talk about performance.  Given the magnitude of the traffic on eBay and the number of rules they need to activate, I can’t think of a best person to talk about scalability issues.  Disclaimer, Kenny will be my co-presenter next week at Business Rules Forum 😉

My friend Kenny has decided to also talk about performance.  Given the magnitude of the traffic on eBay and the number of rules they need to activate, I can’t think of a best person to talk about scalability issues.  Disclaimer, Kenny will be my co-presenter next week at Business Rules Forum 😉

Kenny ShiSo many ways to fraud…  We know quite a bit about credit card and debit card fraud, and a little about insurance fraud, from our FICO days.  Fraud for online e-commerce is quite a bit more diverse.  Those fraudsters are incredibly creative when it comes to finding weaknesses in the system.  Kenny’s examples of fraud always impressed me as I do not believe I could have come up with nearly half of those scams!

More Read

big data and accounting

Strategies to Make Better Profits for CPAs During Tax Season

What to Consider When Choosing a Masters in Business Analytics
Understanding the Tremendous Benefits of IoT for Small Businesses
5 Essential Cybersecurity Tips For Data Centric Businesses In 2021
How To Interact With Power BI Data In A PowerPoint Presentation

The fraud detection architecture applies rules and models as a service, invoked either synchronously (SOAP interface to the application) or asynchronously (via a message bus).  The presentation focuses on making faster and more scalable real-time decisions.

From a developer / architect perspective, it would be more comfortable to preocess the cases offline of course but the nature of the business requires more timely detection – both in terms of revenue protection and customer satisfaction.

Fraud detection is the result of expert rules and data-driven analytics.  Neural nets can detect trends that are buried in past transactions.  Rules address flash fraud detection, e.g. patterns identified by fraud experts before models have had a chance to learn about those.  As expected both of them require a ton of data to make good decisions, including some sophisticated computations.  Data retrieval and normalization takes most of the transaction time.  In the end, neural net processing represents only 2% of the time, and rules 3%.

Logically, many performance efforts by Kenny’s team have been focused on data.  Data access is one area that Kenny covered.  More interestingly, he analyzed other related avenues that relate more directly to decisioning “optimizations”:

  • Unnecessary variables
  • Excessive logic
  • Conflicting logic

Independently, optimization techniques:

  • Pre-fetch — although it goes against the very flexible and dynamic nature of rules
  • Lazy load: only when needed, with smart ordering of conditions from most discriminating / cheaper to least — not always easy to estimate though
  • Pre-aggregation of data — freshness can be addressed by aggregating up to “yesterday” for example and then look at real-time data for the current day only

Distributed locality could be addressed with distributed hash tables and other architecture tactics that Kenny quickly went through.

Horizontal scability with server clusters, etc.  Database partition can help, based on data access patterns — per seller, per IP address.

Scalability for Rule Authoring

Sharing variables implies srong capabilities to catalog and describe variables so that all modelers and rules writers can safely and conveniently reuse those characteristics.

Great plug for Sparkling Logic — Thanks!  But seriously, Kenny really appreciated the fluid metaphors we came up with, which help address that very set of problems.

Scalability for Rule Deployment

The continuous integration of rules allows to detect problems earlier and more frequently.  As an added bonus the deployment is ready when it is time to deploy.

Statistical testing is similar to Champion Challenger in the sense that it allows the experts to compare the Production rules with the new Testing rules.  Kenny went a little fast

Other techniques like smoketests and incremental deployment allow to test without impacting the entire workload.

Dog-food approach: Kenny also uses rules to monitor rules.  Smart.

As you can imagine, constant monitoring allows to ensure that they are on-target with their performance objectives.

Closing remark: Fraudsters are very creative so catching them is like a game of Catcha Mouse!

 

Wow…  What a fast speaker!

Check out the abstract here.

 

Powered By WizardRSS.com | Full Text RSS Feed | Amazon Plugin | Settlement Statement

CMatignon October 25, 2011
Share this Article
Facebook Twitter Pinterest LinkedIn
Share
- Advertisement -

Follow us on Facebook

Latest News

data analytics in sports industry
Here’s How Data Analytics In Sports Is Changing The Game
Big Data
data analytics on nursing career
Advances in Data Analytics Are Rapidly Transforming Nursing
Analytics
data analytics reveals the benefits of MBA
Data Analytics Technology Proves Benefits of an MBA
Analytics
anti-spoofing tips
Anti-Spoofing is Crucial for Data-Driven Businesses
Security

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

big data and accounting
Business RulesData ManagementITSoftware

Strategies to Make Better Profits for CPAs During Tax Season

10 Min Read
Business,Analytics,(ba),Technology,Using,Big,Data,,Cloud,Computing,And
Analytics

What to Consider When Choosing a Masters in Business Analytics

15 Min Read
Benefits of IoT in Small Business
Internet of Things

Understanding the Tremendous Benefits of IoT for Small Businesses

8 Min Read
cybersecurity tips for data centric businesses
Security

5 Essential Cybersecurity Tips For Data Centric Businesses In 2021

5 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai in ecommerce
Artificial Intelligence for eCommerce: A Closer Look
Artificial Intelligence
AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?