By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    data Analytics instagram stories
    Data Analytics Helps Marketers Make the Most of Instagram Stories
    15 Min Read
    analyst,women,looking,at,kpi,data,on,computer,screen
    What to Know Before Recruiting an Analyst to Handle Company Data
    6 Min Read
    AI analytics
    AI-Based Analytics Are Changing the Future of Credit Cards
    6 Min Read
    data overload showing data analytics
    How Does Next-Gen SIEM Prevent Data Overload For Security Analysts?
    8 Min Read
    hire a marketing agency with a background in data analytics
    5 Reasons to Hire a Marketing Agency that Knows Data Analytics
    7 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-23 SmartData Collective. All Rights Reserved.
Reading: Clearest Introductions to Information Theory
Share
Notification Show More
Aa
SmartData CollectiveSmartData Collective
Aa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Uncategorized > Clearest Introductions to Information Theory
Uncategorized

Clearest Introductions to Information Theory

Editor SDC
Last updated: 2009/05/08 at 6:40 PM
Editor SDC
4 Min Read
SHARE

I’ve previously written notes about an information theory betting “paradox” and Kelly’s optimal betting fraction.

Information theory formalized the duality between information transmission and probability. Many interesting results come from thinking of probability in terms of information, such as using Kelly betting to determine the maximum profit from private information about a stock or how to correctly price an option. 
The following are the best three papers I’ve found on information theory for someone who has just started learning. (One other, Elements of Information Theory, is also good but not free except at a library; I have the solution manual in electronic format if anyone would like it)
1) Shannon’s original 1948 paper. Page 3-7 or so is the good bit if you don’t like math, it explains the information rate of English (also good if you do like math of course). 
2) Information Theory, Inference, and Learning Algorithms by David MacKay. I really like how this one is written, with exercises embedded in the text to work as you go. This is exactly how I teach myself things when I’m reading independently. For anyone interested in machine learning, this will be …


I’ve previously written notes about an information theory betting “paradox” and Kelly’s optimal betting fraction.

Information theory formalized the duality between information transmission and probability. Many interesting results come from thinking of probability in terms of information, such as using Kelly betting to determine the maximum profit from private information about a stock or how to correctly price an option. 
The following are the best three papers I’ve found on information theory for someone who has just started learning. (One other, Elements of Information Theory, is also good but not free except at a library; I have the solution manual in electronic format if anyone would like it)
1) Shannon’s original 1948 paper. Page 3-7 or so is the good bit if you don’t like math, it explains the information rate of English (also good if you do like math of course). 
2) Information Theory, Inference, and Learning Algorithms by David MacKay. I really like how this one is written, with exercises embedded in the text to work as you go. This is exactly how I teach myself things when I’m reading independently. For anyone interested in machine learning, this will be a fascinating book. Start from page 1. (this book is also available online from the author’s site)
3) Lecture notes: “An Introduction to Information Theory and Entropy” by Tom Carter. This is the best one of the three for anyone reading about info theory for the first time. It’s intuitively explained and goes relatively slowly- it’s the first one I would recommend to a friend. 
Please feel free to leave a comment, ask a question, or email me about anything, even unrelated to the above.

More Read

big data improves

3 Ways Big Data Improves Leadership Within Companies

IT Is Not Analytics. Here’s Why.
Romney Invokes Analytics in Rebuke of Trump
WEF Davos 2016: Top 100 CEO bloggers
In Memoriam: Robin Fray Carey

Editor SDC May 8, 2009
Share This Article
Facebook Twitter Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

ai low code frameworks
AI Can Help Accelerate Development with Low-Code Frameworks
Artificial Intelligence
data Analytics instagram stories
Data Analytics Helps Marketers Make the Most of Instagram Stories
Analytics
data breaches
How Hospital Security Breaches Devastate Local Communities
Policy and Governance
analyst,women,looking,at,kpi,data,on,computer,screen
What to Know Before Recruiting an Analyst to Handle Company Data
Analytics

Stay Connected

1.2k Followers Like
33.7k Followers Follow
222 Followers Pin

You Might also Like

big data improves
Big DataJobsKnowledge ManagementUncategorized

3 Ways Big Data Improves Leadership Within Companies

6 Min Read
Image
Uncategorized

IT Is Not Analytics. Here’s Why.

7 Min Read

Romney Invokes Analytics in Rebuke of Trump

4 Min Read

WEF Davos 2016: Top 100 CEO bloggers

14 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Lost your password?