As I have explained in previous blog posts, I am almost as obsessive-compulsive about literature and philosophy as I am about data and data quality, because I believe that there is much that the arts and the sciences can learn from each other.
As I have explained in previous blog posts, I am almost as obsessive-compulsive about literature and philosophy as I am about data and data quality, because I believe that there is much that the arts and the sciences can learn from each other.
Therefore, I really enjoyed recently reading the book Proust Was a Neuroscientist by Jonah Lehrer, which shows that science is not the only path to knowledge. In fact, when it comes to understanding the brain, art got there first.
Without doubt, I will eventually write several blog posts that use references from this book to help me explain some of my perspectives about data quality and its many related disciplines.
In this blog post, with help from Jonah Lehrer and the composer Igor Stravinsky, I will explain The Data-Decision Symphony.
Data, data everywhere
Data is now everywhere. Data is no longer just in the structured rows of our relational databases and spreadsheets. Data is also in the unstructured streams of our Facebook and Twitter status updates, as well as our blog posts, our photos, and our videos.
The challenge is can we somehow manage to listen for business insights among the endless cacophony of chaotic data volumes, and use those insights to enable better business decisions and deliver optimal business performance.
Whether you choose to measure it in terabytes, petabytes, or how much reality bites, the data deluge has commenced—and you had better bring your A-Game to D-Town. In other words, you need to find innovative ways to derive business insight from your constantly increasing data volumes by overcoming the signal-to-noise ratio encountered during your data analysis.
The Music of the Data
This complex challenge of filtering out the noise of the data until you can detect the music of the data, which is just another way of saying the data that you need to make a critical business decision, is very similar to how we actually experience music.
As Jonah Lehrer explains, “music is nothing but a sliver of sound that we have learned how to hear. Our sense of sound is a work in progress. Neurons in the auditory cortex are constantly being altered by the songs and symphonies we listen to.”
“Instead of representing the full spectrum of sound waves vibrating inside the ear, the auditory cortex focuses on finding the note amid the noise. We tune out the cacophony we can’t understand.”
“This is why we can recognize a single musical pitch played by different instruments. Although a trumpet and violin produce very different sound waves, we are designed to ignore these differences. All we care about is pitch.”
Instead of attempting to analyze all of the available data before making a business decision, we need to focus on finding the right data signals amid the data noise. We need to tune out the cacophony of all the data we don’t need.
Of course, this is easier in theory than it is in practice.
But this is why we need to always begin our data analysis with the business decision in mind. Many organizations begin with only the data in mind, which results in performing analysis that provides little, if any, business insight and decision support.
“But a work of music,” Lehrer continues, “is not simply a set of individual notes arranged in time.”
“Music really begins when the separate pitches are melted into a pattern. This is a consequence of the brain’s own limitations. Music is the pleasurable overflow of information. Whenever a noise exceeds our processing abilities . . . [we stop] . . . trying to understand the individual notes and seek instead to understand the relationship between the notes.”
“It is this psychological instinct—this desperate neuronal search for a pattern, any pattern—that is the source of music.”
Although few would describe analyzing large volumes of data as a “pleasurable overflow of information,” it is our search for a pattern, any pattern in the data relevant to the decision, which allows us to discover a potential source of business insight.
The Data-Decision Symphony
“When we listen to a symphony,” explains Lehrer, “we hear a noise in motion, each note blurring into the next.”
“The sound seems continuous. Of course, the physical reality is that each sound wave is really a separate thing, as discrete as the notes written in the score. But this isn’t the way we experience the music.”
“We continually abstract on our own inputs, inventing patterns in order to keep pace with the onrush of noise. And once the brain finds a pattern, it immediately starts to make predictions, imagining what notes will come next. It projects imaginary order into the future, transposing the melody we have just heard into the melody we expect. By listening for patterns, by interpreting every note in terms of expectations, we turn the scraps of sound into the ebb and flow of a symphony.”
This is also how we arrive at making a critical business decision based on data analysis.
We discover a pattern of business context, relevant to the decision, and start making predictions, imagining what will come next, projecting imaginary order into the data stream, turning bits and bytes into the ebb and flow of The Data-Decision Symphony.
However, our search for the consonance of business context among the dissonance of data, could cause us to draw comforting, but false, conclusions—especially if unaware of any confirmation bias—resulting in bad, albeit data-driven, business decisions.
The musicologist Leonard Meyer, in his 1956 book Emotion and Meaning in Music, explained how “music is defined by its flirtation with—but not submission to—expectations of order. Although music begins with our predilection for patterns, the feeling of music begins when the pattern we imagine starts to break down.”
Lehrer explains how Igor Stravinsky, in The Rite of Spring, “forces us to generate patterns from the music itself, and not from our preconceived notions of what the music should be like.”
Therefore, we must be vigilant when we perform data analysis, making sure to generate patterns from the data itself, and not from our preconceived notions of what the data should be like—especially when we encounter less than perfect data quality.
As Jonah Lehrer explains, “the brain is designed to learn by association: if this, then that. Music works by subtly toying with our expected associations, enticing us to make predictions and then confronting us with our prediction errors.”
“Music is the sound of art changing the brain.”
The Data-Decision Symphony is the sound of the art and science of data analysis enabling better business decisions.