The characters you cannot see, and the bugs they cause

The string that is not the string

You paste a coupon code and it is rejected. You copy a customer reference from an email into a spreadsheet and the lookup returns nothing. You compare two lines of text side by side and they are, letter for letter, the same. The computer is not being difficult. There is something in one of them that has no shape.

Text is not what you see. Text is a sequence of numbered code points, and plenty of those numbers render as nothing at all, or as something indistinguishable from an ordinary space. Equality is decided on the numbers. Your eyes only get the picture.

This is one of the few problems where the fix is trivial once you can see the cause, and completely opaque until then. So the first move is never to guess. It is to look at the actual code points with something like the unicode inspector, which shows you the sequence rather than the rendering.

The usual offenders

A handful of characters cause the overwhelming majority of these bugs. The non-breaking space, U+00A0, is the most common by a distance. It looks exactly like a space, occupies the same width, and was invented so that "10 kg" would never split across two lines. It is not the space character, so a search for a space misses it and splitting a line on spaces leaves it welded to the word beside it.

The zero-width space, U+200B, is worse because it has no width at all. It was designed to mark a permitted line break inside a long word. Web software sprinkles it into long identifiers and URLs to stop them overflowing their container, and it comes along for the ride when you copy.

The byte order mark, U+FEFF, sits at the very start of a file. It was a signal about byte ordering that most modern text no longer needs, but exporters still write it. Because it is invisible and comes before everything, it silently corrupts the first column heading of a spreadsheet or the first line of a script, which is why a file works perfectly except for one field that nothing can find.

The soft hyphen, U+00AD, shows a hyphen only when a word actually wraps, and hides otherwise. Directional override characters flip the reading order of the text that follows, which is legitimate for mixed Arabic and English but can also make a file called "annexcod.exe" display as something reassuring. The zero-width joiner, U+200D, glues characters into a single glyph, and unlike the rest it is usually doing essential work.

Consequence one: comparisons that quietly fail

Every login form, discount field, spreadsheet lookup and deduplication routine ultimately asks one question: are these two sequences the same? An invisible character makes the answer no while the screen insists the answer is yes.

This is why a password typed from a password manager works and the same password pasted from a note does not. It is why a supplier list has two entries for the same company. It is why a customer swears they entered the reference correctly, and they did, plus one character they could not have known was there.

The tell is almost always length. If a code should be twelve characters and the field reports thirteen, you have your answer without needing to know which character it is. A text diff will often point at a spot where the two lines appear identical, which is itself the diagnosis.

Consequence two: letters that are not the letters they look like

The second problem is the opposite: characters that are perfectly visible but are not what you think. Cyrillic "а" is a different code point from Latin "a" and renders identically in most typefaces. So do Cyrillic "е", "о", "р" and "с", and Greek omicron, and a long tail of lookalikes across dozens of scripts.

This is the mechanism behind lookalike domains. A registered name where one letter comes from a different alphabet displays as the real brand and points somewhere else entirely. Browsers defend against this by showing the underlying encoded form of suspicious names, which you can decode yourself with the punycode converter if a link looks off.

The same trick walks straight past naive filters. A banned word written with one substituted letter is, to a simple text check, a completely different word, while a human reads it exactly as intended. Any filter that compares raw characters is defeated by a single keystroke.

Where they actually come from

Nobody types these. They arrive by transfer. Pasting from a word processor brings non-breaking spaces, smart quotes and the occasional soft hyphen, because the document was designed for print rather than for matching. Copying from a PDF is worse: the layout engine may have inserted hyphens and spacing that were never part of the words.

Autocorrect quietly inserts non-breaking spaces between a number and its unit, and after certain abbreviations, in more than one popular editor. Copying from a rendered web page picks up whatever the page inserted for line breaking, including zero-width spaces inside long strings.

Emoji are the honest case. A family emoji is several people joined by zero-width joiners, and a flag is two regional indicator letters. These sequences are meant to be there, which is exactly why a blanket strip is dangerous: it turns one emoji into three, and mangles text in scripts that use joiners structurally. If you are assembling emoji rather than cleaning them, the emoji picker hands you the whole sequence intact.

The same accent, stored two ways

There is a second kind of invisible difference, and it is not an extra character so much as a different spelling of the same one. The letter é can be a single code point, U+00E9, or it can be a plain "e" followed by a separate combining acute accent, U+0301. Both render identically. Neither is wrong.

They are simply not equal. A name typed on a Mac and the same name typed on a Windows machine can end up in different forms, so a search finds one and misses the other, and a sorted list puts them in surprising places. Character counts disagree too: one form is one character long, the other is two.

The fix is normalisation: converting text to a single agreed form before comparing it, either the composed form where accented letters are single characters, or the decomposed form where the accents are separate. Pick one, apply it to everything on both sides of the comparison, and the problem disappears. Where you want the accents gone entirely, for a URL slug or a filename, remove accents will flatten them to their base letters.

Finding them, and knowing what to keep

Detection first, always. Paste the suspect text into the invisible characters tool to reveal what is hiding in it, or run it through unicode escape to see every character as its code point. Once you know whether you are dealing with a stray non-breaking space or a byte order mark, the fix takes seconds.

For cleaning, be specific rather than aggressive. Replace non-breaking spaces with ordinary spaces, drop the byte order mark from the start of a file, remove soft hyphens and zero-width spaces, and normalise your accents. A targeted find and replace handles most of it, and remove whitespace tidies the ordinary spacing left behind.

Then the caveat that matters more than any of the above: do not strip everything invisible. Zero-width joiners hold emoji together and are grammatically necessary in Persian, Hindi, Malayalam and others. Directional marks are the only thing keeping mixed Arabic and English readable. Blanket removal is not cleaning, it is damage, and it damages exactly the text least likely to be checked by the person who wrote the rule.

The habit worth building is small: when text refuses to match, stop retyping it and look at it properly. Nine times out of ten the answer is one character wide, or no characters wide at all.