We use cookies, including third-party cookies from Google to serve personalized ads through AdSense, to operate this site and understand how it is used. By continuing to browse, you accept this use. See our Privacy Policy and Terms of Use for details, including how to opt out of personalized advertising.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
    Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
    23 Min Read
    Flat editorial illustration: The article is about automating the collection, preparation, and delivery of marketing data across p
    From Data to Dashboards: 5 Marketing Reporting Automation Software Picks
    12 Min Read
    Flat editorial illustration: The article contrasts passive follower totals with a behavioral path from attention and trust to cli
    Creator Analytics Beyond Follower Count: Seven Metrics That Measure Audience Behavior
    11 Min Read
    Flat editorial illustration: The article connects AI and marketing analytics with managerial judgment grounded in consumer behavi
    How an Online MBA in Marketing Connects AI and Data Analytics
    11 Min Read
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers -- AI-generated illustration
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers
    7 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Big Data
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Business Intelligence > Big Data
Business Intelligence

Big Data

Steve Bennett
Steve Bennett
6 Min Read
Big Data
Illustration generated with FLUX.1 [schnell] via Cloudflare Workers AI.
SHARE

Several month’s ago a short video appeared on YouTube with an interview of LinkedIn’s Chief Scientist DJ Patil. In it he discusses how ‘Big Data‘ impacts the practise of analytics. I’ve only just got around to posting about it but I am doing so now because he has some insights that I agree with and would like to share as they are still relevant. 

Big data is today most often associated with the internet superstars like Google, eBay and Amazon. There are 3 other areas with lower profiles where big data is important: intelligence (spooks, the military, etc.), scientific and academic research, and the financial markets.

Big data’s future is much bigger than this because more and more areas of human activity are going to be faced with vast data sets. When you hear people talking about the growth of knowledge and statements like ‘if this data were printed then the stack would grow faster than NASA’s fastest rocket‘, you have to remember that there is a good chance that each page of new data is adding to someone’s analytic data set.

    I’m not quoting the guy verbatim but here’s what I heard and my takeouts to his comments:

    • Open source ‘big data ready’ technologies like Hadoop (see my earlier blog or here) have come into their own now. Look to people with these skills over those only with SQL if you are facing big data challenges.
    • We have reached a tipping point in the use of open source for commercial solutions to big data problems.
    • If you want good analysts then the best place is to look is in occupations where people will already have the practical skills in manipulating big data sets: scientific fields like meteorology, oceanography and the like. I agree but this is not the only place as in my experience I also need analysts that relate well to business decision makers – i.e. those people that make commercial decisions based on the analytics. This is perhaps less important in pure tech plays like LinkedIn.
    • Open source will transform the practise of analytics in the next 3 – 5 years. I think it will take longer than this to really impact the more traditional industries. I’m not happy about this but I am realistic about the difficulty in convincing business leaders that open source is a superior solution to proprietary ones. The money behind the big vendors will keep them going for a number of years yet.

    One potential qualifier to DJ Patil’s perspective is that although he has a very impressive big data background as a mathematician, US Department of Defence analyst (‘Threat Anticipation’), and former eBay Director of Strategy and Analytics, his current employer is LinkedIn.

    The core of LinkedIn’s big data is structured and fairly static: profiles of people. So I’m not sure how similar their big data challenges would be to, say, those faced with processing, understanding and predicting large streams of real time data from financial markets or very large sensor arrays. On the other hand, the growth of LinkedIn communities and their related activities must generate large amounts of semi-structured data.

    I also have no idea what LinkedIn’s own analytic goals are beyond what DJ mentions on his own profile where he says his analytics drives product features like:

    • “People You May Know”
    • “Who Viewed My Profile”
    • “Groups You Might Like”

    Maybe somebody reading this blog knows more?

    The video is on YouTube and I embed it here for convenience:

    Or you can download ‘DJ Patil on How Big Data Impacts Analytics’ directly from this blog.

    TAGGED:data qualityhadoopopen source
    Share This Article
    Facebook Pinterest LinkedIn
    Share

    Follow us on Facebook

    Latest News

    Flat editorial illustration: The article compares proxy MCP servers by what an AI agent can do through them: manage proxy account
    Proxy MCP Servers: 7 Options and What Each Lets an AI Agent Do
    Artificial Intelligence Big Data Data Collection
    Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
    Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
    Big Data Business Intelligence Exclusive Web Analytics
    Flat editorial illustration: A SaaS-editorial scene showing product usage, support patterns, and renewal feedback informing a fle
    Using Product Usage Analytics and Support Data to Build a B2B Product Roadmap
    Business Intelligence Infographic
    From Hiring To Payroll: Managing HR Data Across Business Systems -- AI-generated illustration
    From Hiring To Payroll: Managing HR Data Across Business Systems
    Big Data Business Intelligence Infographic

    Stay Connected

    1.2KFollowersLike
    33.7KFollowersFollow
    222FollowersPin

    You Might also Like

    dataFUD: Manage your information resources
    Predictive Analytics

    dataFUD: Manage your information resources

    6 Min Read
    The Fragility of Knowledge
    Uncategorized

    The Fragility of Knowledge

    5 Min Read
    Flat editorial illustration: The article connects duplicate and decayed CRM records to distorted attribution; a SaaS-editorial sc
    Business IntelligenceExclusive

    Your Attribution Model Is Fine. Your CRM Is Lying to It

    17 Min Read
    Energy Department Announces New SunShot Projects to Harness the Power of Big Data
    AnalyticsBig Data

    Energy Department Announces New SunShot Projects to Harness the Power of Big Data

    6 Min Read

    SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

    The Art of Conversation: Enhancing Chatbots with Advanced AI Prompts
    The Art of Conversation: Enhancing Chatbots with Advanced AI Prompts
    Chatbots
    ai chatbot
    How AI Website Chatbots Improve Customer Support and Lead Generation
    Chatbots Exclusive

    Quick Link

    • About
    • Contact
    • Privacy
    Follow US
    © 2008-26 SmartData Collective. All Rights Reserved.
    Welcome Back!

    Sign in to your account

    Username or Email Address
    Password

    Lost your password?