Books in .txt format for AI training purposes

Re:Books in .txt format for AI training purposes by JoeDaDude on Sunday October 25, 2020 @11:48AM (#24887158)

There is always Project Gutenberg [1]. I suppose there may be pros and cons to each, with Project Gutenberg being generally older books.

[1]. https://www.gutenberg.org/

Fun trivia: I once took an NLP class and discovered that I could predict whether a text was written by Edgar Allan Poe or H. G. Wells with ~86% accuracy based on the presence of one word: "whereupon". The language had changed enough in 60 years such that that single word was no longer used.

[ Parent | on HN ]

Re:Books in .txt format for AI training purposes by pavlov on Sunday October 25, 2020 @12:41PM (#24887583)

I’ve noticed that “presently” (in the meaning “soon”) is one such indicator for post-WWII English.

British novels written in the 1950s consistently use “presently” to indicate a short passage of time. In the following decades it seems to have been removed from editors’ style guides and replaced mostly by plain old “soon”.

[ Parent | on HN ]

Re:Books in .txt format for AI training purposes by MivLives on Sunday October 25, 2020 @03:23PM (#24889108)

Wonder what other words that could be found that do something similar.

[ Parent | on HN ]

Re:Books in .txt format for AI training purposes by milquetoastaf on Sunday October 25, 2020 @04:27PM (#24889610)

You may enjoy this article - https://en.wikipedia.org/wiki/Hapax_legomenon

[ Parent | on HN ]