mxbai-embed-large thinks “Hokkaidō” and “Éire” are the same word

A top-ranked MTEB embedding model collapses nearly every word containing a diacritical mark to a single point in vector space — but only when deployed via Ollama. The upstream HuggingFace tokenizer strips accents correctly; the Ollama gguf conversion pipeline silently drops the preprocessing step, and the defect class affects every BERT-derived embedding model in the local-inference stack we tested. 147,687 confirmed cross-entity collisions on Wikidata.
Last updated: 2026-04-11 23:43 UTC · regenerated from scripts/verify_tokenizer_divergence.py

1. The Collision Heatmap

Every cell shows the cosine similarity between two words containing diacritical marks. In a working embedding model, unrelated words should have cosine around 0.3–0.5. Instead, nearly every pair is at 1.00. The one cell around ~0.5 ("São Paulo") has an internal space that resets WordPiece on its second, ASCII-only word.

Figure 1: 361 of 380 off-diagonal cells are at cosine ≈ 1.00.

2. The Paradox: “Hokkaidō” is closer to “Éire” than to “Hokkaido”

Compare two cosines for each word in the 20-word sample: the word vs its own ASCII spelling (blue), and the word vs a different diacritical word (terracotta). A working model should put the blue bars near 1.0 and the terracotta bars near 0.5. The actual result is the exact inverse.

Figure 2: Blue = same word (diacritic vs plain), should be ≈1.0. Terracotta = different word (diacritic vs diacritic), should be ≈0.4. The actual values are inverted.

3. This is a hard collapse, not gradual degradation

If the failure mode were soft blur — diacritical marks adding noise that gradually erodes similarity — the number of colliding pairs would drop smoothly as the threshold is raised. It doesn't. The curve is flat all the way up to 0.99.

Figure 3: Collision count is nearly constant from cosine 0.80 to 0.99. The collapse is binary.

4. Distribution analysis: the [UNK] cluster

Three similarity distributions side by side. The diacritical-vs-diacritical panel (left) is a spike at 1.0, not a bell curve — compare to the control panel (middle), which shows what a healthy embedding space actually produces.

Figure 4: Diacritical-vs-diacritical (left) is a spike at 1.0. Control-vs-control (middle) is a normal distribution around 0.5. This is a tokenizer defect, not embedding noise.

5. Why it happens: the HF-vs-Ollama divergence

"Hokkaidō"  →  [CLS]  [UNK]  [SEP]
"Éire"      →  [CLS]  [UNK]  [SEP]
"Zürich"    →  [CLS]  [UNK]  [SEP]
"café"      →  [CLS]  [UNK]  [SEP]
"Dvořák"    →  [CLS]  [UNK]  [SEP]

6. Scope: this affects every BERT-derived embedding model in Ollama

The verification script runs the same diacritic-vs-ASCII probe against every BERT-family embedding model registered in local Ollama. Each row reports three mean cosine similarities:

• S = diacritical form vs its own ASCII spelling (e.g. Hokkaidō vs Hokkaido). Should be ≈1.0.
• D = diacritical word vs a different diacritical word (e.g. Hokkaidō vs Éire). Should be near the ASCII control baseline.
• C = two unrelated ASCII English words. This is the calibration baseline for the model's overall similarity spread.

A healthy model has S ≈ 1 and D ≈ C. A model with a diacritic attractor has D >> C. A model with an “[UNK] collapse” additionally has S ≈ C (the same word's ASCII form is no more similar than an unrelated word).

Every BERT-derived embedding model we tested via Ollama has a failure mode on diacritical text. mxbai-archived and all-minilm exhibit the full [UNK] collapse; nomic-embed-text has a softer but still-severe diacritic attractor (its unrelated diacritic pairs cluster at cosine ~0.99, even though it recognizes same-word ASCII equivalents). This is not a one-off bug in one model — it's a systemic defect class at the deployment-tooling layer.

7. Who is affected

8. Reproducing this

Everything on this page regenerates from two scripts, using the frozen gguf shipped in this repository so the result is stable even if the upstream mxbai weights are patched:

# 1. Register the archived gguf in Ollama
cd model/
ollama create mxbai-archived -f Modelfile
cd ..

# 2. Run the verification script (writes verification/tokenizer_divergence.json)
pip install transformers   # for the upstream HF probe
python scripts/verify_tokenizer_divergence.py

# 3. Regenerate figures and this page from the JSON artifact
python scripts/generate_defect_page.py

The older single-file demo (scripts/demo_collisions.py) still works and is faster if you just want to see the defect — it embeds 25 pairs via Ollama and writes collisions.csv. That script is the one reproduced daily by GitHub Actions, and the CSV it produces is deterministic (the collisions do not drift between runs), which is itself part of the result.

9. What should be done

1. Ollama / gguf conversion: BERT-derived models with do_lower_case=True need their BasicTokenizer preprocessing (NFD normalization + combining-mark strip) carried through gguf conversion. Without this, WordPiece sees raw diacritics and emits [UNK].
2. Benchmark gap: MTEB should include diacritical / non-Latin string pairs as a robustness check — no current task in the suite surfaces this class of defect.
3. User workaround: If you cannot switch off Ollama, NFD-normalize and strip combining marks on the client side before embedding. This loses linguistic information but prevents the collisions.
4. Model choice: Models with byte-level BPE tokenizers (e.g. nomic-embed-text on some versions, or SentencePiece-based multilingual models) are less exposed, though our probe shows even nomic-embed-text via Ollama has a softer diacritic attractor.

10. Context

This defect was discovered during the Latent Space Cartography project, which applies Vector Symbolic Architecture (VSA) analysis to frozen text embeddings. When probing Wikidata entity embeddings for relational structure, the collision pattern was unmistakable: 147,687 cross-entity pairs at cosine ≥ 0.95, all involving diacritical text. The full analysis is documented in our paper “Latent Space Cartography Applied to Wikidata: Relational Displacement Analysis Reveals a Silent Tokenizer Defect in mxbai-embed-large” (clawRxiv 2604.00648).