By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData Collective
  • Analytics
    AnalyticsShow More
    predictive analytics in dropshipping
    Predictive Analytics Helps New Dropshipping Businesses Thrive
    12 Min Read
    data-driven approach in healthcare
    The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
    6 Min Read
    analytics for tax compliance
    Analytics Changes the Calculus of Business Tax Compliance
    8 Min Read
    big data analytics in gaming
    The Role of Big Data Analytics in Gaming
    10 Min Read
    analyst,women,looking,at,kpi,data,on,computer,screen
    Promising Benefits of Predictive Analytics in Asset Management
    11 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Four Really Real Meanings of Real-Time
Share
Notification Show More
Latest News
ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
ai in omnichannel marketing
AI is Driving Huge Changes in Omnichannel Marketing
Artificial Intelligence
ai for small business tax planning
Maximize Tax Deductions as a Business Owner with AI
Artificial Intelligence
ai in marketing with 3D rendering
Marketers Use AI to Take Advantage of 3D Rendering
Artificial Intelligence
How Big Data Is Transforming the Maritime Industry
How Big Data Is Transforming the Maritime Industry
Big Data
Aa
SmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Mining > Four Really Real Meanings of Real-Time
Big DataData ManagementData Mining

Four Really Real Meanings of Real-Time

Paige Roberts
Last updated: 2016/08/22 at 12:27 PM
Paige Roberts
11 Min Read
SHARE

Our director of engineering told me that she had a customer ask if we could do real-time data processing with Syncsort DMX-h. Knowing that real-time means different things to different people, the engineer asked what exactly the customer meant by real-time. He said, “We want to be able to move our data out of the database and into Hadoop in real-time every two hours.”

Our director of engineering told me that she had a customer ask if we could do real-time data processing with Syncsort DMX-h. Knowing that real-time means different things to different people, the engineer asked what exactly the customer meant by real-time. He said, “We want to be able to move our data out of the database and into Hadoop in real-time every two hours.”

When she told me that story, I wanted to quote Inigo Montoya from “The Princess Bride.” You keep using that word, “real-time.” I do not think it means what you think it means.

More Read

How Big Data Is Transforming the Maritime Industry

How Big Data Is Transforming the Maritime Industry

Utilizing Data to Discover Shortcomings Within Your Business Model
Small Businesses Use Big Data to Offset Risk During Economic Uncertainty
The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas
Analytics Changes the Calculus of Business Tax Compliance

But what does real-time actually mean? And what do you really mean when you say real-time? What do other people usually mean when they say real-time? How can you tell which meaning people are using? And what the heck is near real-time?

Here are four different things that I believe real-time really means, and how to determine which meaning you’re using.

You Keep Using that Word, Real-Time

Sub-Second Response

Generally, when engineers say “real-time”, they are usually referring to sub-second response time. In this kind of real-time data processing, nanoseconds count. Extreme levels of performance are key to success.

Example quotes:

“Our cyber-security process has to respond in real-time to stop automated attacks from stealing customer data.”

“This stock exchange application has to bid in real-time or we’ll lose money.”

If this is what you mean when you say “real-time data processing”, then you need the data to come in, the condition for response to be evaluated, and the response to happen  ̶  all generally in less than a second. And if someone else’s system can do it a few nanoseconds faster, you might lose out. In this kind of real-time, pushing the limits of performance isn’t a bonus; it’s a necessity.

Human Comfortable Response Time

What this kind of real-time processing comes down to is a commandment: “Thou shalt not bore or frustrate the users.” The performance requirement for this kind of processing is usually a couple of seconds.

Example quotes:

“We need real-time drill down on visualizations for our business intelligence team, no matter how big the data.”

“This website needs to respond to user requests in real-time or we’ll lose sales.”

If this is what you mean when you say “real-time”, then performance matters, but it may not be the number one criteria. In some cases, a difference of a single second can be critical. For instance, if a person clicks on an ad on a web page, and the page takes 4 seconds to load, the user is likely to get bored and go look at a different web page. If that same page had loaded in 3 seconds, that user might have bought something on that web page.

For the most part, however, as long as the data gets crunched and the application responds before the user decides to go surf somewhere else, or check email or something, then the performance requirement is met.

Event-Driven

If when you say “real-time”, you mean the opposite of scheduled, then you mean event-driven. Instead of happening in a particular time interval, event-driven data processing happens when a certain action or condition triggers it. The performance requirement for this is usually before another event happens.

Example quotes:

“As changes are made to the database, the replication process copies them out to the cluster in real-time.”

“A listener watches for data to arrive from our customers in this location, then loads it into the system in real-time.”

In some cases, you don’t know precisely when you’ll need data processing done, but as soon as a certain thing happens, that’s when the need for data processing is triggered. Common event examples are changes in the data or user actions.

There are actually two different performance requirements for event-driven data processing. First, the data processing system has to be finished working and ready to start again before the next event happens. So, if on average, the events happen no closer together than five minutes, a data processing time frame of 2-3 minutes is excellent. If the events tend to happen an average of 10 seconds apart, then clearly, a 2-3 minute processing time would be unacceptable.

The second performance requirement may be more arbitrary. It’s the business SLA. If for example, you want to be able to assure the CEO that his dashboards have the most current data up to the minute, then the data processing has to be able to complete within a minute of any data change in order to meet that deadline.

Streaming Data Processing

If when you say “real-time”, you mean the opposite of batch processing, then you mean streaming data processing. In batch processing, data is gathered together, and all records or other data units are processed in one big bundle until they’re all done. In streaming data processing, the data is processed as it flows in, one unit at a time. And once the data starts coming in, it generally doesn’t end. The performance requirement for streaming data processing is you must process data as fast as the data flows in.

Example quotes:

“We’re sifting through Twitter data in real-time for mentions of our company to keep an eye on sentiment.”

“The server information in this data center is monitored in real-time to catch problems early.”

More and more, when people say “real-time data processing” these days, they are most likely referring to streaming data processing. Streaming data processing has some very specific, and sometimes tricky to implement requirements. You have to be able to process the data continuously, without start-up or clean-up overhead. Micro-batch streaming data processing frameworks like Spark Streaming have found a way to handle start-up and clean-up needs while still keeping up with streaming performance speed requirements. Streaming data processing also requires a way to deal with occasional system failures without massive data loss. In some cases, data loss is acceptable, but in others, it isn’t.

Near Real-Time

The takeaway from all these different meanings of real-time is not that you’re using the word wrong, or one definition is more right than another. It’s that when you’re thinking about implementing a real-time data processing application, it’s important to consider what kind of real-time you really mean. Based on that, you can determine what level of performance you will require.

No That's Near Real-Time

That’s not real-time. That’s near real-time. Duh!

So, what does near real-time mean? Well, near real-time is essentially something engineers say because they’re cringing inside about how ambiguous the word “real-time” has become. Seriously, why does one little-hyphenated word suddenly have four different meanings? What’s up with this English language drift thing? It’s so imprecise!

Near real-time basically means any one of the definitions I mentioned, aside from sub-second response time. Although, I have heard an engineer or two use real-time to mean streaming.

So, What Does Real-Time Really, Really Mean?

At the recent Spark Summit East, Syncsort GM, Tendu Yogurtcu was asked, “What trends do you see coming up?” to which she responded, “A lot more customers are moving to real-time data processing.” Ali Ghodsi, CEO of DataBricks had the same opinion. “We’re seeing a real push for real-time.” He also saw trends toward breaking data silos and doing more advanced analytics, “all in real-time.” Tendu also said that the future lies in combining streaming and batch on one platform.

In the same interview, Tendu and  Dave Vallente of the CUBE delved into the question that spurred this blog post – what is the real definition of real-time? Dave came up with a great one that covers all four of the above meanings, and Tendu called back to it: “Respond before you lose the customer.” This, in some ways, is the best possible way to think about real-time when designing data processing systems.

Regardless of what level of performance your system has in any given situation, if you end up losing the customer, then it’s simply too slow. Maybe you should try moving up to real-time data processing.

Paige Roberts August 22, 2016
Share this Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

ai software development
Key Strategies to Develop AI Software Cost-Effectively
Artificial Intelligence
ai in omnichannel marketing
AI is Driving Huge Changes in Omnichannel Marketing
Artificial Intelligence
ai for small business tax planning
Maximize Tax Deductions as a Business Owner with AI
Artificial Intelligence
ai in marketing with 3D rendering
Marketers Use AI to Take Advantage of 3D Rendering
Artificial Intelligence

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

How Big Data Is Transforming the Maritime Industry
Big Data

How Big Data Is Transforming the Maritime Industry

8 Min Read
utlizing big data for business model
Big Data

Utilizing Data to Discover Shortcomings Within Your Business Model

6 Min Read
big data use in small businesses
Big Data

Small Businesses Use Big Data to Offset Risk During Economic Uncertainty

7 Min Read
data-driven approach in healthcare
Analytics

The Importance of Data-Driven Approaches to Improving Healthcare in Rural Areas

6 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data
ai is improving the safety of cars
From Bolts to Bots: How AI Is Fortifying the Automotive Industry
Artificial Intelligence

Quick Link

  • About
  • Contact
  • Privacy
Follow US

© 2008-23 SmartData Collective. All Rights Reserved.

Removed from reading list

Undo
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?