Apache Spark, hosted on Hadoop, is great for processing large amounts of data quickly, but wouldn’t it be even better if you could process data in real time?
Apache Spark, hosted on Hadoop, is great for processing large amounts of data quickly, but wouldn’t it be even better if you could process data in real time? If your business depends on making decisions quickly, you should definitely consider the MapR distribution, which includes the complete Spark stack including Spark Streaming.
Here are some amazing, of-the-moment, game-changing uses for real-time Big Data processing.
Credit Card Fraud Detection
Your credit card is swiped, the receipt is signed, you buy something. Only it wasn’t you. Perhaps your wallet was stolen. Perhaps some hackers stole your information from another site. Maybe a credit card skimmer at the local gas station got your number. However it happens, it’s credit card fraud (see how big data reduces financial fraud), and credit card companies want to know when and where it’s happening so they can stop it.
Banks and credit card companies are on the hook for fraudulent charges, so they want to block as many of them as possible. They already have sophisticated mathematical models to detect possibly bogus transactions, but these models are typically only updated once a year.
Apache Spark Streaming, running on Hadoop, makes it possible for banks to process transactions and detect fraud in real time against previously identified fraud footprints. At the same time, they can continuously update their models. This means that the more they detect fraud, the more accurate they’ll be in the future.
Network Security
Network security is at the top of nearly every business’s agenda, especially after all the high-profile data breaches of the last few years. Modern networks carry a lot of data. Though most traffic is benign, some people on the Internet are up to no good. Hackers can commandeer thousands and millions of computers over the Internet into “botnets” to cause Distributed Denial of Service (DDoS) attacks, steal credit card information, and otherwise wreak havoc with information.
Wouldn’t it be nice if there were a way to detect security problems before they happened? That’s exactly what a global managed security services provider is doing with its security service running on Hadoop.
The provider uses different components of the Spark stack to examine packets for traces of malicious activity in real time. At the front end, it uses Spark Streaming to check against known threats before passing the packets on to the storage platform where the data will be further processed using other packages such as GraphX and MLlib.
Hackers move quickly, always trying to stay one step ahead of IT departments. With machine learning and stream-processing solutions, systems can keep learning about new threats as they evolve, protecting clients in real time (see how big data can boost security).
Genomic Sequencing
The 20th century saw a staggering reduction in the number of people dying of various diseases. It was also the century that medicine discovered DNA. In the 21st century, genomic engineering looks to offer a new renaissance of medicine (see how geneticists are using big data to accelerate research).
The only problem is that DNA sequencing genomes require vast amounts of computing power. For instance, the latest “next-gen” DNA sequencer, the Illumina XTen, produces 6 terabytes of data per day. For medical-grade data, scientists need to sequence 300 billion base pairs. That really puts the “big” in “Big Data”.
Not only is there a lot of data, it takes a long time to process, even on the biggest and fastest supercomputers available. NextGen Genomic companies are using Spark’s capabilities to drastically reduce the time it needs to process genomes. It used to take them several weeks to align chemical compounds with genes. Now it only takes geneticists a few hours.
Real-Time Ad Processing
In the series Mad Men, Harry Crane, the bespectacled, neglected head of the Media Department for Sterling, Cooper, and Partners, constantly complains about the lack of a computer of the firm’s own???. The company eventually gets an IBM mainframe, but Crane would be even more jealous of the real-time processing capabilities that advertisers have today—and are far beyond what anyone in the 1960s could have imagined.
One advertising firm uses Spark, on MapR-DB, to build a real-time ad targeting platform. The system looks at user data and decides which ads to show users on the Internet based on demographic data. Since advertising is so time-sensitive, advertisers have to move fast if they want to capture mindshare. Spark Streaming is one way to help them do that.
Medical
While genomics offers one way to revolutionize the healthcare industry, providers are always looking to make healthcare more efficient. One way is to try to prevent hospital readmittance. One provider uses Spark to examine patient records combined with clinical information to find out who’s most likely to have complications after being released from the hospital. Then they can deploy home healthcare services to prevent readmission, saving on costs for both patients and hospitals.
Conclusion
As you can see, there is a diverse range of real-time uses for the Spark stack with the MapR distribution. Big Data isn’t just hype—it’s solving real problems to make the world better, faster, and cheaper to live in. If these examples have whetted your appetite, why not see what the MapR distribution of Spark can do for you?
For a more in-depth introduction to Spark, read Getting Started with Spark: From Inception to Production, a free interactive eBook by James A. Scott.
