Maybe text mining SHOULD be playing a bigger role in data warehousing
516 Views
When I chatted last week with David Bean of Attensity, I commented to him on a paradox:Many people think text information is important to analyze, but even so data warehouses don’t seem to wind up holding very much of it.
My working theory explaining this has two parts, both of which purport to show why text data generally doesn’t fit well into BI or data mining systems. One is that it’s just too messy and inconsistently organized. The other is t…
When I chatted last week with David Bean of Attensity, I commented to him on a paradox:Many people think text information is important to analyze, but even so data warehouses don’t seem to wind up holding very much of it.
My working theory explaining this has two parts, both of which purport to show why text data generally doesn’t fit well into BI or data mining systems. One is that it’s just too messy and inconsistently organized. The other is that text corpuses generally don’t contain enough information.
Now, I know that these theories aren’t wholly true, for I know of counterexamples. E.g., while I’ve haven’t written it up yet, I did a call confirming that a recently published SPSS text/tabular integrated data mining story is quite real. Still, it has felt for a while as if truth lies in those directions
.Anyhow, David offered one useful number range: If you do exhaustive extraction on a text corpus, you wind up with 10-20X as much tabular data as you had in text format in the first place. (Comparing total bytes to total bytes.)
So how big are those corpuses? I think most text mining installations usually have at least 10s of thousands of documents or verbatims to play with. Special cases aside, the upper bound seems to usually be about two orders of magnitude higher.
And most text-mined documents probably tend to be short, as they commonly are just people’s reports on a single product/service experience – perhaps 1 KB or so, give or take a factor of 2-3? So we’re probably looking at 10 gigabytes of text at the low end, and a few terabytes at the high end, before applying David’s 10-20X multiplier.
Hmm – that IS enough data for respectable data warehousing …
Obviously, special cases like national intelligence or very broad-scale web surveys could run larger, as per the biggest Marklogic databases. Medline runs larger too.” title=”http://feeds.feedburner.com/~r/MonashInformationServices/~3/430910676/”>Link to original post
Trending Now
1
Best Practices
74 shares2,774 views
74 shares2,774 views
Cloud Security: Practical And Effective Ways To Protect Your Data
Ryan Kh - March 15, 20192
Best Practices
44 shares2,368 views
44 shares2,368 views
Factors Behind Data Storage Security: Is Your Business Vulnerable?
Diana Hope - March 15, 20193
Big Data
55 shares2,384 views
55 shares2,384 views
Is Big Data The Key To Our Culture’s Relationship Crisis?
Andrej Kovacevic - March 15, 20194
Big Data
56 shares2,573 views
56 shares2,573 views
The Effect of The Data Revolution in Enterprise Software Development
Ryan Kh - March 13, 20195
Artificial Intelligence
66 shares2,738 views
66 shares2,738 views
You must log in to post a comment.