CACM Article on DB/IR

In my rush to finish writing my book this month, I haven’t had much time for reading. But I did notice an article in the April ‘09 issue of Communications of the ACM that caught my attention: “Database and Information-Retrieval Methods for Knowledge Discovery” (no subscription necessary for online access). I’m not sure what sort of IR system my brain uses, but that title certainly excited a lot of neurons!

It’s a worthwhile read, especially for people unfamiliar with the artificial dichotomy between database and information retrieval research. It’s a bit too academic for my taste–I would have liked to see at least some mention of the commercial efforts to bridge this gap between unstructured and structured information access (hint, hint). And of course there’s too much emphasis on ranking and nary a mention of interactive or exploratory interfaces.

But enough quibbling. Here are a few excerpts to when your appetite:

DB and IR are separate fields in computer science due to historical accident. Both investigate concepts, models, and computational methods for managing large amounts of complex information, though each began almost 40 years ago with very different application areas as motivations and technology drivers; for DB it was accounting systems (such as online reservations and banking), and for IR it was library systems (such as bibliographic catalogs and patent collections). Moreover, these two directions and their related research communities emphasized very different aspects of information management; for DB it was data consistency, precise query processing, and efficiency, and for IR it was text understanding, statistical ranking models, and user satisfaction.
…
Structured and unstructured search conditions are combined in a single query, and the query results must be ranked. The queries must be evaluated over very large data sets that exhibit high update rates…A programmer can build such an application through two separate platforms—a DB system for the structured data and an IR search engine for the textual and fuzzy-matching issues. But this widely adopted approach is a challenge to application developers, as many tasks are not covered by the underlying platforms and must be addressed in the application code. An integrated DB/IR platform would greatly simplify development of the application and largely reduce the cost of maintaining and adapting it to future needs.
…
With a knowledge base that sublimates valuable content from the Web, we could address difficult questions beyond the capabilities of today’s keyword-based search engines. For example, a user might ask for a list of drugs that inhibit proteases and obtain a fairly comprehensive list of drugs for this HIV-relevant family of enzymes. Such advanced information requests are posed by knowledge workers, including scientists, students, journalists, historians, and market researchers. Although it is possible to find relevant answers, the process is laborious and time-consuming, as it often requires rephrasing queries and browsing through many potentially promising but ultimately useless result pages.

Enjoy!

More Read

The Case Against Collaboration, Part I

In-Memory Analytic Databases Are so Last Century

Social Networking: Theory and Practice

What it means to me to be a free agent

4 Types of Data That Will Boost Your Marketing ROI

Link to original post