Cookies help us display personalized product recommendations and ensure you have great shopping experience.

By using this site, you agree to the Privacy Policy and Terms of Use.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    sales and data analytics
    How Data Analytics Improves Lead Management and Sales Results
    9 Min Read
    data analytics and truck accident claims
    How Data Analytics Reduces Truck Accidents and Speeds Up Claims
    7 Min Read
    predictive analytics for interior designers
    Interior Designers Boost Profits with Predictive Analytics
    8 Min Read
    image fx (67)
    Improving LinkedIn Ad Strategies with Data Analytics
    9 Min Read
    big data and remote work
    Data Helps Speech-Language Pathologists Deliver Better Results
    6 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Visualizing Lexical Novelty in Literature
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Visualization > Visualizing Lexical Novelty in Literature
Data VisualizationText Analytics

Visualizing Lexical Novelty in Literature

matthewhurst
matthewhurst
4 Min Read
SHARE

Novels are full of new characters, new locations and new expressions. The discourse between characters involves new ideas being exchanged. We can get a hint of this by tracking the introduction of new terms in a novel. In the below visualizations (in which each column represents a chapter and each small block a paragraph of text), I maintain a variable which represents novelty. When a paragraph contains more than 25% new terms (i.e. words that have not been observed thus far) this variable is set at its maximum of 1.

Novels are full of new characters, new locations and new expressions. The discourse between characters involves new ideas being exchanged. We can get a hint of this by tracking the introduction of new terms in a novel. In the below visualizations (in which each column represents a chapter and each small block a paragraph of text), I maintain a variable which represents novelty. When a paragraph contains more than 25% new terms (i.e. words that have not been observed thus far) this variable is set at its maximum of 1. Otherwise, the variable decays. The variable is used to colour the paragraph with red being 1.0 and blue being 0. The result is that we can get an idea of the introduction of new ideas in novels.

In the first book – Austen’s Sense and Sensibility – we can see two things. Firstly, the start of the book keeps a pretty good degree of novelty for the first few chapters. Secondly, each chapter introduces something new.

The second book – Stevenson’s Kidnapped – shows a different pattern. While it starts off with reasonable novelty, this then dies out for most of the book with spurts of interest here and there.

More Read

Seven Misconceptions about Data Quality
Forecast Product Demand with Confidence
How to Stay Out of Cash Flow Crises Using Cash Position Analysis
10 reasons why a grad student should use R
Applying Data Analytics to Customer Experience and Service on Social Media

What is surprising to me (if we take any real meaning from this approach) is that Austen’s Emma – the third book – is strong out of the gate (the first 18 chapters) but fails to break the 25% novelty ceiling thereafter.

[Note that these results are preliminary and I’m going to do more validation and testing.]

Update: see below the original visualization for an updated version with more accurate results.

LexicalNovelty 
Update: After looking at the above results I drilled down on the strange behaviour in Emma. It turns out that Emma as multiple volumes within which the chapter counter reset to I. Consequently I was picking up chapter titles (I, II, III, VI, V, etc.) as novel terms the first go round and this was driving the visualization. I’ve since modified the algorithm to firstly ignore text blocks (paragraphs) with fewer than 5 words and secondly, given it a more dynamic colour scheme.

This improvement still highlights some key differences (again, in as much as the algorithm is correct). However, these differences are now somewhat changed from the first set of observations. Note also that the threshold for novelty has been decreased to 0.1.

LexicalNovelty2 

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

sales and data analytics
How Data Analytics Improves Lead Management and Sales Results
Analytics Big Data Exclusive
ai in marketing
How AI and Smart Platforms Improve Email Marketing
Artificial Intelligence Exclusive Marketing
AI Document Verification for Legal Firms: Importance & Top Tools
AI Document Verification for Legal Firms: Importance & Top Tools
Artificial Intelligence Exclusive
AI supply chain
AI Tools Are Strengthening Global Supply Chains
Artificial Intelligence Exclusive

Stay Connected

1.2kFollowersLike
33.7kFollowersFollow
222FollowersPin

You Might also Like

data visualization techniques to explore
Data Visualization

The Fascinating Role of Data Visualization and Techniques for Assorted Variables

10 Min Read

Start Up Spotlight: UserVoice

5 Min Read

NSA and the Future of Big Data

5 Min Read

Saying Goodbye

5 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots
data-driven web design
5 Great Tips for Using Data Analytics for Website UX
Big Data

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-25 SmartData Collective. All Rights Reserved.
Go to mobile version
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?