We use cookies, including third-party cookies from Google to serve personalized ads through AdSense, to operate this site and understand how it is used. By continuing to browse, you accept this use. See our Privacy Policy and Terms of Use for details, including how to opt out of personalized advertising.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
    Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
    23 Min Read
    Flat editorial illustration: The article contrasts passive follower totals with a behavioral path from attention and trust to cli
    Creator Analytics Beyond Follower Count: Seven Metrics That Measure Audience Behavior
    11 Min Read
    Flat editorial illustration: The article connects AI and marketing analytics with managerial judgment grounded in consumer behavi
    How an Online MBA in Marketing Connects AI and Data Analytics
    11 Min Read
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers -- AI-generated illustration
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers
    7 Min Read
    chatgpt image jul 21, 2026, 04 34 30 pm
    4 Core Benefits of Predictive Maintenance after Vibration Analysis
    10 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Population health management with RevoScaleR
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Population health management with RevoScaleR
Data MiningR Programming Language

Population health management with RevoScaleR

DavidMSmith
DavidMSmith
5 Min Read
Population health management with RevoScaleR
Illustration generated with FLUX.2 [Klein 9B] via local ComfyUI.
SHARE

This guest post is by Douglas McNair MD PhD, Engineering Fellow & President, Cerner Math Inc. — ed.

RevoScaleR scaling big-data modeling performance for real-time health data analysis at Cerner

The size of data sets is increasing much more rapidly than the speed of cores, of RAM, and of disk drives. This is particularly true of electronic health records (EHRs) and other health care data, where multi-billion-row relational tables with dozens to thousands of columns are common in data warehouses derived from EHRs. Analyzing a data set that contains N observations on P variables frequently entails operating on several terabyte arrays — too large for main memory, beyond the statistical packages’ typical limit of a maximum vector length of 231 – 1 (around two billion cells), and too large even for pagefile virtual memory on a single machine.

Statistical software able to use multi-core processors is a necessary, but not sufficient requirement to deal with data of this size. For statistical analysis of large data, you also need:

  1. parallelized external memory algorithms (PEMAs) that do not require all data to be in RAM, and
  2. efficient multi-threading and inter-process communication across massively parallel clusters of computers.

Like other statistical packages, the R system is memory-limited. Revolution Analytics’ RevoScaleR and bigmemory and a few other big-data packages overcome the memory limitation of R by providing a backing file for processing blocks of data at a time or by providing an ODBC or other streaming service for processing blocks of data.

More Read

Apache Spark Use Cases
Apache Spark Use Cases
Perform Data Mining With Web Scrapers to Track Prices
Interview: Dr Graham Williams
Examining Big Data’s Potential In Predictive Marketing
Top 10 analytics mistakes

Since cardinality N is large, especially for our pharmacovigilance (drug safety) and other use-cases that involve modeling low-eventrate low-prevalence outcomes, only tasks with complexity that is approximately O(N) are feasible.  But not all big-memory and parallelized architectures offer comparable performance or scalability.

For example, Lumley’s biganalytics package in R provides a method for passing big.matrix objects to the biglm function, which computes ordinary least squares regressions on large datasets, like one of Cerner’s cohorts of Type 2 diabetics:

library(biglm)
library(foreach)
biglm.big.matrix( los.outlier ~ gender + age.grp + medicaid + last.HbA1c +
 max.creat + min.glu
 + nbr.adm24mo + nbr.comorbid_dx + nosocomial.inf,
 family=binomial(), data=”type2_diabetic” )

And Revolution R Enterprise can do the same with RevoScaleR:

rxLogit( los.outlier ~ gender + age.grp + medicaid + last.HbA1c +
  max.creat + min.glu + nbr.adm24mo + nbr.comorbid_dx + nosocomial.inf,
  data=”type2_diabetic.xdf” )

Happily, the biglm regression takes less than 20 seconds to run on this cohort of 2 million diabetic patients and 10 variables.  But even more remarkably, the rxLogit regression takes less than 0.5 seconds to run on this dataset. (Both analyses ran on a Windows 7, 64-bit quadcore 2.5 GHz laptop with 8 MB of RAM.) And the RevoScaleR regression runs far faster than this on a multi-node cluster.

The difference might seem inconsequential if our use-case involved a one-time execution of a regression at model-development-time. However, a majority of our run-time applications instead involve near-realtime kmeans ‘clustering’ or ‘classification’ or ‘prediction’ use-cases with high cardinality and dimensionality, where close matches to a particular patient and her attributes are sought within very large cohorts, and this task is repeated for successive different patients every second. 

For high-value, high-performance applications like these, RevoScaleR and related tools are fast becoming an indispensable part of Cerner’s armamentarium for addressing many use-cases in health services and population health management.

Douglas McNair, MD, PhD, is president of Cerner Math, Inc., and one of three Cerner Engineering Fellows and is responsible for innovations in decision support and very-large-scale datamining. McNair joined Cerner in 1986, first as VP of Cerner’s Knowledge Systems engineering department; then as VP of Regulatory Affairs; then as General Manager for Cerner’s Detroit and Kansas City branches. Subsequently, he was Chief Research Officer, responsible for Cerner’s clinical research operations. In 1987, McNair was co-inventor and co-developer of Discern Expert®, a decision-support engine that today is used in more than 2,000 health care facilities around the world. Between 1977 and 1986, McNair was a faculty member of Baylor College of Medicine in the Departments of Medicine and Pathology. He is a diplomate of the American Board of Pathology and the American Board of Internal Medicine.

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
Big Data Business Intelligence Exclusive Web Analytics
Flat editorial illustration: A SaaS-editorial scene showing product usage, support patterns, and renewal feedback informing a fle
Using Product Usage Analytics and Support Data to Build a B2B Product Roadmap
Business Intelligence Infographic
From Hiring To Payroll: Managing HR Data Across Business Systems -- AI-generated illustration
From Hiring To Payroll: Managing HR Data Across Business Systems
Big Data Business Intelligence Infographic
Flat editorial illustration: The article is about automating the collection, preparation, and delivery of marketing data across p
From Data to Dashboards: 5 Marketing Reporting Automation Software Picks
Big Data Business Intelligence Exclusive

Stay Connected

1.2KFollowersLike
33.7KFollowersFollow
222FollowersPin

You Might also Like

Data Mining in the New Economy and How to Get Started
Data Mining

Data Mining in the New Economy and How to Get Started

12 Min Read
Analytic Applications are Built by Data Scientists
Data QualityPredictive AnalyticsR Programming Language

Analytic Applications are Built by Data Scientists

3 Min Read
Freakonomics and Your Data
AnalyticsCloud ComputingData MiningData QualityKnowledge ManagementStatistics

Freakonomics and Your Data

6 Min Read
Interview – Anne Milley, SAS,  Part 1
Data MiningPredictive Analytics

Interview – Anne Milley, SAS, Part 1

15 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Chatbots and SEO: How Can Chatbots Improve Your SEO Ranking?
Artificial Intelligence Chatbots Exclusive
5 Great Tips for Using Data Analytics for Website UX
5 Great Tips for Using Data Analytics for Website UX
Big Data

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-26 SmartData Collective. All Rights Reserved.
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?