The Heisenberg Principle states that for certain things accuracy and certainty in knowing one quality ( say position of an atom) has to be a trade off with certainty of another quality (like momentum). I was drawn towards the application of this while in an email interaction with Edith Ohri, who is a leading data mining person in Israel and has her own customized GT solution. Edith said that it seems it is impossible to have data that is both accurate (data quality) and easy to view across organizations (data transparency). More often than not the metrics that we measure are the metrics we are forced to measure due to data adequacy and data quality issues.
Now there exists a tradeoff in the price of perfect information in managerial economics, but is it really true that the Business Intelligence we deploy is more often than not constrained by simple things like input data and historic database tables. And that more often than not Data quality is the critical constraint that determines speed and efficacy of deployment.
I personally find that much more of the time in database projects goes in data measurement, aggregation, massaging outliers, missing value assumptions than in the “high value” activities like insight generation and business issue resolution.
Is it really true? Analysis is easy; it’s the data that’s tough?
What do you think in terms of the uncertainty inherent in data quality and data transparency-
