A Scaling Challenge for Twitter Search

March 10, 2009
54 Views

The other day, I explained why, as far as I can tell, Twitter’s existing search functionality isn’t that hard to implement. In a subsequent post, I argued that Twitter is not a search engine, an opinion that seems to place me in a minority of the blogosphere, albeit a substantial one.

Today I see that Twitter is have some trouble with what strikes me as a core constituency, SXSW attendees.

Daniel Terdiman at CNET writes that “At SXSW, attendees confront Twitter saturation“:

At SXSW, the standard is for everyone to include the tag “#sxsw” in their tweets. For example, on Friday, I was looking for sources for a different story and tweeted, “If you are launching an iPhone app at #sxsw, or know someone who is, please let me know. Thanks!”

That’s a great convention because it allows anyone wanting to know what’s going on to search Twitter for posts using any search term important to them.

I did a search for the “#sxsw” tag on Saturday afternoon and found that there had been 392 tweets with the term in just the previous 10 minutes. That number mushroomed to more than 1,500 in the previous hour.

Large volumes

The other day, I explained why, as far as I can tell, Twitter’s existing search functionality isn’t that hard to implement. In a subsequent post, I argued that Twitter is not a search engine, an opinion that seems to place me in a minority of the blogosphere, albeit a substantial one.

Today I see that Twitter is have some trouble with what strikes me as a core constituency, SXSW attendees.

Daniel Terdiman at CNET writes that “At SXSW, attendees confront Twitter saturation“:

At SXSW, the standard is for everyone to include the tag “#sxsw” in their tweets. For example, on Friday, I was looking for sources for a different story and tweeted, “If you are launching an iPhone app at #sxsw, or know someone who is, please let me know. Thanks!”

That’s a great convention because it allows anyone wanting to know what’s going on to search Twitter for posts using any search term important to them.

I did a search for the “#sxsw” tag on Saturday afternoon and found that there had been 392 tweets with the term in just the previous 10 minutes. That number mushroomed to more than 1,500 in the previous hour.

Large volumes of results wouldn’t be such a problem if users had a way to summarize, navigate, and explore them. But that will take more than a search engine that offers more than reverse-date ordering of Boolean queries.

I wonder if Abdur Chowdhury and his team are working on this problem. Perhaps if Twitter is willing to make some of the historical logs available for download (they’re already public, just not easily downloaded), some of us HCIR wonks could implement interfaces on top of it to explore the possibilities. Abdur, if you read this and are interested, please let us know!

Link to original post

You may be interested

How SAP Hana is Driving Big Data Startups
Big Data
298 shares2,919 views
Big Data
298 shares2,919 views

How SAP Hana is Driving Big Data Startups

Ryan Kh - July 20, 2017

The first version of SAP Hana was released in 2010, before Hadoop and other big data extraction tools were introduced.…

Data Erasing Software vs Physical Destruction: Sustainable Way of Data Deletion
Data Management
44 views
Data Management
44 views

Data Erasing Software vs Physical Destruction: Sustainable Way of Data Deletion

Manish Bhickta - July 20, 2017

Physical Data destruction techniques are efficient enough to destroy data, but they can never be considered eco-friendly. On the other…

10 Simple Rules for Creating a Good Data Management Plan
Data Management
69 shares627 views
Data Management
69 shares627 views

10 Simple Rules for Creating a Good Data Management Plan

GloriaKopp - July 20, 2017

Part of business planning is arranging how data will be used in the development of a project. This is why…