Random paper
 

Characterizing the Google Books corpus: Strong limits to inferences of socio-cultural and linguistic evolution

E. A. Pechenick, C. M. Danforth, and P. S. Dodds

PLoS ONE, 10, e0137041, 2015

Times cited: 430

Logline: The Google Books corpus fails to represent popularity of words and phrases because it is (1) library-like, meaning it contains one of each book; and far more problematically: (2) overwhelmed with scientific and medical literature. The second Fiction corpus may be okay but all preceding work is compromised.

Abstract:

It is tempting to treat frequency trends from Google Books data sets as indicators for the true popularity of various words and phrases. Doing so allows us to draw novel conclusions about the evolution of public perception of a given topic, such as time and gender. However, sampling published works by availability and ease of digitization leads to several important effects. One of these is the surprising ability of a single prolific author to noticeably insert new phrases into a language. A greater effect arises from scientific texts, which have become increasingly prolific in the last several decades and are heavily sampled in the corpus. The result is a surge of phrases typical to academic articles but less common in general, such as references to time in the form of citations. Here, we highlight these dynamics by examining and comparing major contributions to the statistical divergence of English data sets between decades in the period 1800–2000. We find that only the English Fiction data set from the second version of the corpus is not heavily affected by professional texts, in clear contrast to the first version of the fiction data set and both unfiltered English data sets. Our findings emphasize the need to fully characterize the dynamics of the Google Books corpus before using these data sets to draw broad conclusions about cultural and linguistic evolution.
  • This is the default HTML.
  • You can replace it with your own.
  • Include your own code without the HTML, Head, or Body tags.

BibTeX:

@Article{pechenick2015a,
  author = 	 {Pechenick, Eitan A. and Danforth, Christopher M. and Dodds, Peter Sheridan},
  title = 	 {Characterizing the {G}oogle {B}ooks corpus: {S}trong limits to inferences of socio-cultural and linguistic evolution},
  journal = 	 {PLoS ONE},
  year = 	 {2015},
  volume = 	 {10},
  pages = 	 {e0137041},
}

 

Random paper