Hidden characters
Why AI text contains hidden Unicode characters
Zero-width spaces, word joiners, and non-breaking spaces show up in AI output for predictable reasons. Here is where they come from and how to remove them safely.
AI text contains hidden Unicode characters because language models learn from web text that is full of them, and because the clipboard preserves characters rather than appearance. The common culprits are zero-width spaces (U+200B), word joiners (U+2060), non-breaking spaces (U+00A0), and stray byte-order marks. They are invisible on screen, they survive every copy and paste, and they break search, regex, form validation, and code. Removing them is a targeted filter over specific code points — not a general “strip non-ASCII” pass, which would delete accents and non-Latin scripts.
This guide explains where each one comes from, why the model is not really to blame, and how to clean them without damaging the text.
It starts with the training data
A language model is trained on a very large sample of the public web. That sample is not clean prose. It includes HTML pages, PDFs, forum threads, documentation, and code — all of which contain formatting characters that a browser renders as nothing.
A zero-width space in a web page is often there for a reason: it lets a long URL break across lines, or it marks a legal break point in a script that does not use spaces. When that page is scraped into a training corpus, the character goes with it. The model learns the sequence of tokens, invisible characters included, because nothing in the pipeline tells it that this particular token is noise.
This is the single most important thing to understand: the model is not inserting invisible characters to mark its output. It is reproducing patterns that were already in the text it learned from.
The characters you will actually find
In practice, a handful of code points account for almost everything.
| Code point | Name | Why it appears |
|---|---|---|
| U+200B | Zero-width space | Line-breaking control copied from web text |
| U+200C | Zero-width non-joiner | Required in some Persian and Indic orthography |
| U+200D | Zero-width joiner | Builds emoji sequences and joined letterforms |
| U+2060 | Word joiner | Prevents a line break at that position |
| U+FEFF | Byte-order mark | Encoding marker, sometimes mid-string after a bad copy |
| U+00A0 | Non-breaking space | Prevents wrapping; common in pasted tables and prices |
| U+00AD | Soft hyphen | Hidden hyphenation point |
| U+200E / U+200F | Directional marks | Control reading order around punctuation |
Some of these are legitimate. U+200D is how a family emoji stays one glyph, U+200C is correct in Persian, and U+061C (the Arabic letter mark) is required for right-to-left layout. A cleaner that deletes every invisible character will damage multilingual text. A cleaner that deletes the right ones will not.
Three more sources people miss
Tokenisation of unusual strings. When a model reproduces a long identifier, a URL, or a string of punctuation, the token boundaries do not always align with character boundaries. The output can end up with a joiner or a non-breaking space where the model was reconstructing a chunk it had seen as a single unit.
The rich-text clipboard. When you copy from a chat interface, the browser may write HTML to the clipboard. HTML entities such as become a real U+00A0 when pasted. The character was never in the model’s output at all; it was introduced by the copy layer.
Your own tools. Word processors insert non-breaking spaces after some abbreviations and before units. Design tools insert zero-width characters for text wrapping. A document that passes through three applications can accumulate three kinds of invisible noise.
Why they cause so much trouble
A character with no glyph is still a character. Every comparison in software is a comparison of code points, so an invisible character changes the result of that comparison while changing nothing a human can see.
- Search and filters. A product name with U+200B inside it will not match the same name typed by a customer.
- Regex and validation. Anchored patterns fail when an invisible character sits at the start or end of a field.
- Code. String equality, JSON keys, and environment variables all compare exact bytes.
- Content operations. Duplicate detection, translation memories, and diff tools treat the two strings as different documents.
- Copy protection. Some publishers embed zero-width characters to trace copied text. If you paste their text, the markers come with it.
The failure mode is always the same: something that looks correct does not work, and the error message points somewhere else.
How to tell whether your text has them
You cannot see them, so counting is the only reliable test.
- Copy the text.
- Paste it into the hidden character remover.
- Read the hidden-character count in the cleaning report. A number above zero means the text contained invisible characters, and the number is exactly how many were removed.
- Copy the cleaned text back.
If you prefer a terminal, grep -P "[\x{200B}-\x{200D}\x{2060}]" file.txt lists matching lines on macOS and Linux. In Python, [c for c in text if unicodedata.category(c) == "Cf"] returns every format character, which is a superset of what you usually want to remove.
What a safe cleanup looks like
The correct approach is a targeted filter, followed by whitespace normalisation.
Remove: zero-width space, non-joiner, joiner, word joiner, byte-order mark, left-to-right and right-to-left marks, legacy bidi embeddings and overrides, soft hyphen, combining grapheme joiner, non-emoji variation selectors, and stray C0 and C1 control characters.
Replace: non-breaking spaces, en spaces, thin spaces, and hair spaces with a single ordinary space.
Preserve: emoji presentation selectors (U+FE0F), the Arabic letter mark (U+061C), bidi isolates used by right-to-left scripts, and the ideographic space (U+3000) used in CJK typography.
That combination fixes the breakage without touching anything a reader would notice. It is also why “remove all non-ASCII characters” is the wrong tool: it deletes accented letters, curly quotes you may want, and every character from a non-Latin script along with the actual noise.
Before and after
The two lines below look identical. The first has a zero-width space between “pay” and “ment”, and a non-breaking space before the price.
The payment failed on the £20 plan.
The payment failed on the £20 plan.
After cleaning, both lines are the same string. A search for “payment failed” finds the cleaned version, and a regex anchored to the start of the line matches it.
Should you care about all of this?
If you only paste into a document and read it, invisible characters cost you nothing — until you try to find a phrase, or export the document, or paste it into a CMS that rejects the value.
If you paste into code, JSON, a database, or a form, they are a real source of bugs. And if you manage content, they quietly undermine search, deduplication, and internal linking.
The habit that pays off is simple: when something looks right but does not work, check for invisible characters before you debug anything else. It takes ten seconds and rules out a whole class of problem.
Open the hidden character remover · Read the zero-width space guide
Put this into practice
Clean the text before you paste it
Paste the text into JustCleanText, keep the toggles that match your problem, and copy either rich text or plain text. Everything runs in this browser tab.
Open the cleaner