Maybe text mining SHOULD be playing a bigger role in data warehousing
464 Views
When I chatted last week with David Bean of Attensity, I commented to him on a paradox:Many people think text information is important to analyze, but even so data warehouses don’t seem to wind up holding very much of it.
My working theory explaining this has two parts, both of which purport to show why text data generally doesn’t fit well into BI or data mining systems. One is that it’s just too messy and inconsistently organized. The other is t…
When I chatted last week with David Bean of Attensity, I commented to him on a paradox:Many people think text information is important to analyze, but even so data warehouses don’t seem to wind up holding very much of it.
My working theory explaining this has two parts, both of which purport to show why text data generally doesn’t fit well into BI or data mining systems. One is that it’s just too messy and inconsistently organized. The other is that text corpuses generally don’t contain enough information.
Now, I know that these theories aren’t wholly true, for I know of counterexamples. E.g., while I’ve haven’t written it up yet, I did a call confirming that a recently published SPSS text/tabular integrated data mining story is quite real. Still, it has felt for a while as if truth lies in those directions
.Anyhow, David offered one useful number range: If you do exhaustive extraction on a text corpus, you wind up with 10-20X as much tabular data as you had in text format in the first place. (Comparing total bytes to total bytes.)
So how big are those corpuses? I think most text mining installations usually have at least 10s of thousands of documents or verbatims to play with. Special cases aside, the upper bound seems to usually be about two orders of magnitude higher.
And most text-mined documents probably tend to be short, as they commonly are just people’s reports on a single product/service experience – perhaps 1 KB or so, give or take a factor of 2-3? So we’re probably looking at 10 gigabytes of text at the low end, and a few terabytes at the high end, before applying David’s 10-20X multiplier.
Hmm – that IS enough data for respectable data warehousing …
Obviously, special cases like national intelligence or very broad-scale web surveys could run larger, as per the biggest Marklogic databases. Medline runs larger too.” title=”http://feeds.feedburner.com/~r/MonashInformationServices/~3/430910676/”>Link to original post
Trending Now
1
Big Data
1,748 views
1,748 views
Is Your Data At Risk? Here’s How To Protect Your Information Online
Rehan Ijaz - January 11, 20192
Big Data
51 shares2,464 views
51 shares2,464 views
5 Genius App Data Monetization Strategies That Can Make You Bank
Annie Qureshi - January 10, 20193
Big Data
21 shares1,853 views
21 shares1,853 views
Fascinating Ways Big Data and DMPs Are Disrupting Online Publishing
Annie Qureshi - January 10, 20194
Analytics
37 shares3,292 views
37 shares3,292 views
How To Build A High-Performing Data Analytics Platform In The Cloud
Namita Awasthi - January 9, 20195
Big Data
66 shares2,446 views
66 shares2,446 views
Growing Data Privacy Concerns Highlight A Need For VPNs In 2019
Sean Mallon - January 9, 20196
Big Data
55 shares2,212 views
55 shares2,212 views
You must log in to post a comment.