How to clean up messy pasted text, in the right order

The order matters more than the tools

Most people clean text in whatever order the damage annoys them. They see duplicate lines, so they de-duplicate. They see ragged spacing, so they trim it. The result looks tidier and is quietly wrong, because several of these operations change what the next one can see.

De-duplication is the clearest example. Two lines that read identically on screen are not identical to a computer if one of them ends in a space. De-duplicate first and both survive. Trim first and one of them goes. Same input, same tools, different answer, entirely because of sequence.

There is a correct order, and it follows a simple principle: work from the characters you cannot see towards the ones you can. Anything invisible has to go before anything that compares text, because comparison is exactly what invisible characters break.

Step one: the characters you cannot see

Copied text carries passengers. Zero-width spaces from a website, byte order marks from a file that started life somewhere else, soft hyphens left over from justified typesetting, and directional marks from anything that has passed through a bilingual document. None of them render. All of them count.

The one that causes the most damage is the non-breaking space. It looks exactly like a normal space, sits in the same place a normal space would, and does not match when you search for a space. This is why find and replace sometimes appears to be broken: you are searching for a character that is not the character in the text.

Run the text through the invisible character remover before you do anything else. It is the only step where you genuinely cannot verify the result by looking, so it goes first, on faith, every time.

Step two: whitespace, then line breaks

Now the visible whitespace. Tabs pretending to be columns, double spaces after full stops, and the trailing spaces at the end of every line that nothing displays and everything compares. Collapse runs of spaces and strip the leading and trailing ones with the whitespace remover.

Only then touch the line breaks. This ordering matters because joining lines welds their edges together: a line ending in three spaces joined to a line beginning with two produces a five-space gap in the middle of a sentence, and once the break is gone you have lost the seam that told you where to look.

The same logic applies to empty lines. Removing them before you fix line breaks and removing them afterwards give different results, because a paragraph broken into six hard-wrapped lines with blank lines between paragraphs is a different shape once those six lines have become one. Fix the breaks with the line break remover first, then tidy the gaps with the empty line remover. Reverse it and your paragraph boundaries dissolve into one continuous block.

The specific horrors of PDF text

A PDF does not contain paragraphs. It contains positioned glyphs, and the line breaks you get when you copy from one are not part of the writing. They are a fossil of the column width the document was laid out at, usually about sixty-five characters, which is why pasted PDF text wraps in a place that has nothing to do with the sentence.

Worse, justified text often hyphenates. Copy it and you get "manage-" at the end of one line and "ment" at the start of the next. Join the lines naively and you get "manage- ment", which no search will ever find. Look for hyphens sitting at the ends of lines and join those pairs deliberately before you flatten anything else.

Then there are smart quotes. Typographic apostrophes and curly double quotes look correct in prose and are actively hostile everywhere else. They break code, they break search matching against straight quotes, and they break CSV parsing, where a curly quote is not a recognised delimiter and a field can run on into the next column. Convert them with find and replace before the text goes anywhere structured.

PDFs also give you accented characters that were composed rather than typed, headers and footers repeated on every page, and page numbers stranded on their own lines. The repeated furniture is the easiest thing in the world to spot once the text is one column: sort it with the line sorter and the fifty copies of the document title arrive in a block.

Step three: duplicates, once and only once

De-duplication is the step everybody wants to run first and should always run late. It compares whole lines, character for character, which means it inherits every flaw still present in the text. An invisible character, a trailing space, or a capital letter in one copy and not the other, and two identical lines look distinct.

By the time you reach the duplicate line remover, the text should already be clean enough that the only differences left are real ones. That is the whole reason it comes fifth rather than first.

Before you delete anything irreversibly, it is worth knowing what you are about to lose. The word frequency counter tells you how repetitive the text actually is, and comparing the before and after with text diff takes ten seconds and has saved a lot of people from silently binning a hundred lines they needed.

Case last, and title case is harder than it looks

Case conversion goes last because it is the one step that destroys information you cannot recover. Lowercase an acronym and nothing in the text tells you it was ever an acronym. Every earlier step is at worst annoying to redo. This one is not.

Upper and lower case are trivial. Sentence case is nearly trivial, though it has to know that a full stop inside "Ltd." is not the end of a sentence. Use the sentence case converter and check anything with abbreviations in it.

Title case is where naive conversion falls apart, because it is a set of editorial conventions rather than a rule. Small words stay lowercase in the middle of a title but not at the start, so it is "The Rise of the Machines" and not "The Rise Of The Machines". Which words count as small varies between style guides. And capitalising every word turns NASA into Nasa, iPhone into IPhone, and McDonald into Mcdonald.

A good title case converter handles the small-word list for you. It cannot know your proper nouns, so read the output. For anything else, the plain case converter covers the mechanical transformations without pretending to have editorial judgement.

The whole sequence

Strip invisible characters. Normalise whitespace. Join hyphenated line-end pairs, then remove the line breaks. Remove empty lines. De-duplicate. Convert case. Seven steps, and the order is the entire point.

You can skip steps safely. What you cannot do is reorder them, because each one narrows the differences the next one has to reason about. Every mistake in this article is the same mistake wearing a different hat: comparing text before you have made it comparable.

One last habit worth forming. Check the word count before and after. If a cleanup you expected to remove a dozen blank lines has removed four hundred words, something upstream went wrong, and you would much rather find that out now than in the document you paste it into.