We use cookies, including third-party cookies from Google to serve personalized ads through AdSense, to operate this site and understand how it is used. By continuing to browse, you accept this use. See our Privacy Policy and Terms of Use for details, including how to opt out of personalized advertising.
Accept
SmartData CollectiveSmartData Collective
  • Analytics
    AnalyticsShow More
    Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
    Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
    23 Min Read
    Flat editorial illustration: The article is about automating the collection, preparation, and delivery of marketing data across p
    From Data to Dashboards: 5 Marketing Reporting Automation Software Picks
    12 Min Read
    Flat editorial illustration: The article contrasts passive follower totals with a behavioral path from attention and trust to cli
    Creator Analytics Beyond Follower Count: Seven Metrics That Measure Audience Behavior
    11 Min Read
    Flat editorial illustration: The article connects AI and marketing analytics with managerial judgment grounded in consumer behavi
    How an Online MBA in Marketing Connects AI and Data Analytics
    11 Min Read
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers -- AI-generated illustration
    What Kind of Problem-Solving Distinguishes Data Analysts From Software Engineers
    7 Min Read
  • Big Data
  • BI
  • Exclusive
  • IT
  • Marketing
  • Software
Search
© 2008-25 SmartData Collective. All Rights Reserved.
Reading: Visualizing Lexical Novelty in Literature
Share
Notification
Font ResizerAa
SmartData CollectiveSmartData Collective
Font ResizerAa
Search
  • About
  • Help
  • Privacy
Follow US
© 2008-23 SmartData Collective. All Rights Reserved.
SmartData Collective > Big Data > Data Visualization > Visualizing Lexical Novelty in Literature
Data VisualizationText Analytics

Visualizing Lexical Novelty in Literature

matthewhurst
matthewhurst
4 Min Read
Visualizing Lexical Novelty in Literature
Illustrative image generated with OpenAI gpt-image-1.
SHARE

Novels are full of new characters, new locations and new expressions. The discourse between characters involves new ideas being exchanged. We can get a hint of this by tracking the introduction of new terms in a novel. In the below visualizations (in which each column represents a chapter and each small block a paragraph of text), I maintain a variable which represents novelty. When a paragraph contains more than 25% new terms (i.e. words that have not been observed thus far) this variable is set at its maximum of 1. Otherwise, the variable decays. The variable is used to colour the paragraph with red being 1.0 and blue being 0. The result is that we can get an idea of the introduction of new ideas in novels.

In the first book – Austen’s Sense and Sensibility – we can see two things. Firstly, the start of the book keeps a pretty good degree of novelty for the first few chapters. Secondly, each chapter introduces something new.

The second book – Stevenson’s Kidnapped – shows a different pattern. While it starts off with reasonable novelty, this then dies out for most of the book with spurts of interest here and there.

What is surprising to me (if we take any real meaning from this approach) is that Austen’s Emma – the third book – is strong out of the gate (the first 18 chapters) but fails to break the 25% novelty ceiling thereafter.

More Read

Data Analysis Using Relationship Graphs
Data Analysis Using Relationship Graphs
Support Vector Clustering: An Approach to Overcome the Limits of K-means
Alberto’s Business Analytics Predictions for 2012
Location Intelligence: Top 12 industries
New Insights from Text Analytics

[Note that these results are preliminary and I’m going to do more validation and testing.]

Update: see below the original visualization for an updated version with more accurate results.

LexicalNovelty 
Update: After looking at the above results I drilled down on the strange behaviour in Emma. It turns out that Emma as multiple volumes within which the chapter counter reset to I. Consequently I was picking up chapter titles (I, II, III, VI, V, etc.) as novel terms the first go round and this was driving the visualization. I’ve since modified the algorithm to firstly ignore text blocks (paragraphs) with fewer than 5 words and secondly, given it a more dynamic colour scheme.

This improvement still highlights some key differences (again, in as much as the algorithm is correct). However, these differences are now somewhat changed from the first set of observations. Note also that the threshold for novelty has been decreased to 0.1.

LexicalNovelty2 

Share This Article
Facebook Pinterest LinkedIn
Share

Follow us on Facebook

Latest News

Flat editorial illustration: The article compares proxy MCP servers by what an AI agent can do through them: manage proxy account
Proxy MCP Servers: 7 Options and What Each Lets an AI Agent Do
Artificial Intelligence Big Data Data Collection
Flat editorial illustration: The article is about joining public web signals with internal business metrics so BI teams can inter
Alternative Data in BI: Bringing Public Web Data Into Your Dashboards
Big Data Business Intelligence Exclusive Web Analytics
Flat editorial illustration: A SaaS-editorial scene showing product usage, support patterns, and renewal feedback informing a fle
Using Product Usage Analytics and Support Data to Build a B2B Product Roadmap
Business Intelligence Infographic
From Hiring To Payroll: Managing HR Data Across Business Systems -- AI-generated illustration
From Hiring To Payroll: Managing HR Data Across Business Systems
Big Data Business Intelligence Infographic

Stay Connected

1.2KFollowersLike
33.7KFollowersFollow
222FollowersPin

You Might also Like

Examples of Using Kanban Boards with Data Visualization Tools
Data Visualization

Examples of Using Kanban Boards with Data Visualization Tools

6 Min Read
Business Discovery Apps: Data Visualization Plus
AnalyticsBest PracticesBig DataBusiness IntelligenceData VisualizationSoftware

Business Discovery Apps: Data Visualization Plus

3 Min Read
Socializing
Data VisualizationPredictive Analytics

Socializing

14 Min Read
Can Fossil Analysis Software Help Us Plan Curriculum?
Data MiningData VisualizationExclusiveOpen SourceStatistics

Can Fossil Analysis Software Help Us Plan Curriculum?

7 Min Read

SmartData Collective is one of the largest & trusted community covering technical content about Big Data, BI, Cloud, Analytics, Artificial Intelligence, IoT & more.

ai chatbot
How AI Website Chatbots Improve Customer Support and Lead Generation
Chatbots Exclusive
AI chatbots
AI Chatbots Can Help Retailers Convert Live Broadcast Viewers into Sales!
Chatbots

Quick Link

  • About
  • Contact
  • Privacy
Follow US
© 2008-26 SmartData Collective. All Rights Reserved.
Welcome Back!

Sign in to your account

Username or Email Address
Password

Lost your password?