Clean a Project Gutenberg book

A book downloaded from Project Gutenberg comes wrapped in about 3,000 words of licence and boilerplate: a paragraph at the top and the full Project Gutenberg License at the end. DunneWrite will read all of it if you let it. Your AI provider charges you for every word it reads, and the licence can turn up in Ingestion Review as a few odd entries you then have to reject.

clean-source.mjs removes the wrapper before you import, so only the story is read in. It keeps the book's own details (title, author, language) at the top of the file, so DunneWrite still fills in the author for you when you import it.

You'll need#

• The script. Open clean-source.mjs and press Download raw file (the download arrow above the code). It's a single file with no setup. It's free and open source (MIT), part of dunnewrite-scripts.
• Node.js 18 or newer, free from nodejs.org. To check which version you have, type node --version in a terminal.

You don't need a DunneWrite licence or an AI key to clean a book. The script runs entirely on your computer and doesn't connect to the internet.

Clean and import a book#

What the cleaned file looks like#

The book's details become a short block at the top, and the story follows:

---
title: "Pride and Prejudice"
author: "Jane Austen"
language: "English"
---

[the book, from its first page to its last]

What is removed:

• the licence paragraph at the top,
• everything before the line *** START OF THE PROJECT GUTENBERG EBOOK … ***, apart from the book's own details,
• everything after the line *** END OF THE PROJECT GUTENBERG EBOOK … ***, which is the full licence,
• in older files, a "Produced by …" credit at the very start of the book and an "End of the Project Gutenberg EBook of …" line at the very end.

What is kept: everything between those two lines, word for word. That includes the book's own title page and table of contents, because they are part of the book as published. Delete them by hand if you don't want them read.

Why the Project Gutenberg details go too#

The release date, the eBook number, the link back to gutenberg.org and the volunteer credits are removed on purpose. Project Gutenberg's licence lets you take the licence off a book only if every reference to Project Gutenberg goes with it. In the licence's words (section 1.C), they don't try to stop you copying the book or making derivative works from it "as long as all references to Project Gutenberg are removed." Otherwise the full licence has to stay with the file (section 1.E.4).

So the script leaves you with the plain public-domain book, which you are free to use. It keeps the title, author and language, and any translator, illustrator or original publication details the header gives, because those belong to the book rather than to Project Gutenberg's edition. If the story itself mentions Project Gutenberg (usually in a transcriber's note), the script tells you and leaves the line for you to delete.

Project Gutenberg is run by volunteers, who did the work of turning these books into text. If they made your book possible, consider supporting them.

How much it saves#

Measured on three books:

The wrapper is about the same size whatever the book, so the saving matters most on short books. For Alice it is more than a tenth of the file. The token figure is a rough guide (four characters to a token). DunneWrite's cost screen always shows the exact amount before anything is spent.

If something goes wrong#

• "Not recognised as any known source." The file doesn't have Project Gutenberg's START line. It may be from another site, or the start was cut off. Nothing was written. Import the file as it is.

• "No END marker was found." The download was probably interrupted. The cleaned file keeps everything after the START line. Download the book again if the ending looks cut short.

• Your original file is never changed. The cleaned copy is always a new file, named ….clean.txt by default. To choose its name, use --out:

  node clean-source.mjs pg1342.txt --out "Pride and Prejudice.txt"
  

Project Gutenberg is the first source the script recognises. Sources are added one at a time, and the script always says which source it recognised before it removes anything. Run node clean-source.mjs --help to see the current list.

What you've learned#

• A Project Gutenberg book carries about 3,000 words of licence that you'd otherwise pay to have read.
• clean-source.mjs removes it, with every reference to Project Gutenberg as its licence requires, and keeps the title, author and language so DunneWrite fills in the author.
• Your original file is never changed.

Next#

• Ingest your notes, for reviewing what was found
• Import a PDF novel, for books that come as PDF