Skip to content
JustCleanText

Hidden characters

Why AI text contains hidden Unicode characters

Zero-width spaces, word joiners, and non-breaking spaces show up in AI output for predictable reasons. Here is where they come from and how to remove them safely.

JustCleanText5 min readUpdated November 4, 2026

AI text contains hidden Unicode characters because language models learn from web text that is full of them, and because the clipboard preserves characters rather than appearance. The common culprits are zero-width spaces (U+200B), word joiners (U+2060), non-breaking spaces (U+00A0), and stray byte-order marks. They are invisible on screen, they survive every copy and paste, and they break search, regex, form validation, and code. Removing them is a targeted filter over specific code points — not a general “strip non-ASCII” pass, which would delete accents and non-Latin scripts.

This guide explains where each one comes from, why the model is not really to blame, and how to clean them without damaging the text.

It starts with the training data

A language model is trained on a very large sample of the public web. That sample is not clean prose. It includes HTML pages, PDFs, forum threads, documentation, and code — all of which contain formatting characters that a browser renders as nothing.

A zero-width space in a web page is often there for a reason: it lets a long URL break across lines, or it marks a legal break point in a script that does not use spaces. When that page is scraped into a training corpus, the character goes with it. The model learns the sequence of tokens, invisible characters included, because nothing in the pipeline tells it that this particular token is noise.

This is the single most important thing to understand: the model is not inserting invisible characters to mark its output. It is reproducing patterns that were already in the text it learned from.

The characters you will actually find

In practice, a handful of code points account for almost everything.

Code point Name Why it appears
U+200B Zero-width space Line-breaking control copied from web text
U+200C Zero-width non-joiner Required in some Persian and Indic orthography
U+200D Zero-width joiner Builds emoji sequences and joined letterforms
U+2060 Word joiner Prevents a line break at that position
U+FEFF Byte-order mark Encoding marker, sometimes mid-string after a bad copy
U+00A0 Non-breaking space Prevents wrapping; common in pasted tables and prices
U+00AD Soft hyphen Hidden hyphenation point
U+200E / U+200F Directional marks Control reading order around punctuation

Some of these are legitimate. U+200D is how a family emoji stays one glyph, U+200C is correct in Persian, and U+061C (the Arabic letter mark) is required for right-to-left layout. A cleaner that deletes every invisible character will damage multilingual text. A cleaner that deletes the right ones will not.

Three more sources people miss

Tokenisation of unusual strings. When a model reproduces a long identifier, a URL, or a string of punctuation, the token boundaries do not always align with character boundaries. The output can end up with a joiner or a non-breaking space where the model was reconstructing a chunk it had seen as a single unit.

The rich-text clipboard. When you copy from a chat interface, the browser may write HTML to the clipboard. HTML entities such as   become a real U+00A0 when pasted. The character was never in the model’s output at all; it was introduced by the copy layer.

Your own tools. Word processors insert non-breaking spaces after some abbreviations and before units. Design tools insert zero-width characters for text wrapping. A document that passes through three applications can accumulate three kinds of invisible noise.

Why they cause so much trouble

A character with no glyph is still a character. Every comparison in software is a comparison of code points, so an invisible character changes the result of that comparison while changing nothing a human can see.

  • Search and filters. A product name with U+200B inside it will not match the same name typed by a customer.
  • Regex and validation. Anchored patterns fail when an invisible character sits at the start or end of a field.
  • Code. String equality, JSON keys, and environment variables all compare exact bytes.
  • Content operations. Duplicate detection, translation memories, and diff tools treat the two strings as different documents.
  • Copy protection. Some publishers embed zero-width characters to trace copied text. If you paste their text, the markers come with it.

The failure mode is always the same: something that looks correct does not work, and the error message points somewhere else.

How to tell whether your text has them

You cannot see them, so counting is the only reliable test.

  1. Copy the text.
  2. Paste it into the hidden character remover.
  3. Read the hidden-character count in the cleaning report. A number above zero means the text contained invisible characters, and the number is exactly how many were removed.
  4. Copy the cleaned text back.

If you prefer a terminal, grep -P "[\x{200B}-\x{200D}\x{2060}]" file.txt lists matching lines on macOS and Linux. In Python, [c for c in text if unicodedata.category(c) == "Cf"] returns every format character, which is a superset of what you usually want to remove.

What a safe cleanup looks like

The correct approach is a targeted filter, followed by whitespace normalisation.

Remove: zero-width space, non-joiner, joiner, word joiner, byte-order mark, left-to-right and right-to-left marks, legacy bidi embeddings and overrides, soft hyphen, combining grapheme joiner, non-emoji variation selectors, and stray C0 and C1 control characters.

Replace: non-breaking spaces, en spaces, thin spaces, and hair spaces with a single ordinary space.

Preserve: emoji presentation selectors (U+FE0F), the Arabic letter mark (U+061C), bidi isolates used by right-to-left scripts, and the ideographic space (U+3000) used in CJK typography.

That combination fixes the breakage without touching anything a reader would notice. It is also why “remove all non-ASCII characters” is the wrong tool: it deletes accented letters, curly quotes you may want, and every character from a non-Latin script along with the actual noise.

Before and after

The two lines below look identical. The first has a zero-width space between “pay” and “ment”, and a non-breaking space before the price.

The pay​ment failed on the £20 plan.
The payment failed on the £20 plan.

After cleaning, both lines are the same string. A search for “payment failed” finds the cleaned version, and a regex anchored to the start of the line matches it.

Should you care about all of this?

If you only paste into a document and read it, invisible characters cost you nothing — until you try to find a phrase, or export the document, or paste it into a CMS that rejects the value.

If you paste into code, JSON, a database, or a form, they are a real source of bugs. And if you manage content, they quietly undermine search, deduplication, and internal linking.

The habit that pays off is simple: when something looks right but does not work, check for invisible characters before you debug anything else. It takes ten seconds and rules out a whole class of problem.

Open the hidden character remover · Read the zero-width space guide

Put this into practice

Clean the text before you paste it

Paste the text into JustCleanText, keep the toggles that match your problem, and copy either rich text or plain text. Everything runs in this browser tab.

Open the cleaner

FAQ

Common questions

Short answers to what people ask after reading this guide.

Does ChatGPT add invisible characters on purpose?

No. There is no deliberate watermarking step in the text you copy. Invisible characters usually come from the training data the model learned from, from tokenisation of unusual strings, or from the browser and clipboard layer between the chat window and your document.

Are hidden characters a form of AI watermarking?

They can be, in principle — a provider could embed invisible markers in output. But the characters people actually find in pasted text are overwhelmingly ordinary formatting characters that also appear in human-written web content. Treat them as formatting noise to clean, not as proof of authorship.

Why do invisible characters survive copy and paste?

The clipboard carries characters, not appearance. A zero-width space is a real code point with no glyph, so every copy, paste, save, and sync operation faithfully preserves it while nothing visible changes on screen.

Can hidden characters break a website build?

Yes. They break string comparisons, JSON keys, environment variables, and regex patterns. In a build pipeline they can cause a config lookup to fail with an error that points at the wrong line.

Does removing them change the meaning of the text?

Removing format characters does not change words, but removing the wrong ones can. Emoji presentation selectors, the Arabic letter mark, and bidi isolates affect rendering and should be preserved. JustCleanText removes the invisible formatting family and keeps those.

Start cleaning

One paste, then back to writing.

No signup, no upload, and the settings you choose are remembered in this browser only.

100% client-side · No account · Works offline once loaded

Clean & copy