A subword tokenizer purpose-built for the Sindhi language (سنڌي) in the Perso-Arabic script. Trained with SentencePiece Unigram with full observed-character coverage and byte-level fallback on 1.46M+ sentences. It was benchmarked against 14 baseline tokenizers across four evaluation registers. The benchmark has 18 entries: 14 baseline entries and four custom vocabulary sizes. Qwen2.5 and SindhiLM_v2 are identical in this evaluation, so the entries represent 17 distinct tokenizer behaviors. The round-trip check reports 0.0000% normalized-length loss on evaluated text; this does not establish exact identity for arbitrary input.
- 🌐 Documentation Site: https://alinawazcode.github.io/sindhi-tokenizer/
- 🤗 Hugging Face Hub: alinawazmahar/sindhi-tokenizer
- 📄 Research Manuscript Draft (Not Yet Published):
docs/paper.md - 🔬 Technical Explainer:
docs/explainer.md - 📊 Benchmark Results:
results/README.md - 💻 Interactive Web Demo:
demo/app.py
- Project Overview
- Motivation and Problem Statement
- Project Objectives and Scope
- Key Features
- Technology Stack
- Repository Architecture
- Architecture and Workflow Diagrams
- Dataset Documentation
- Tokenizer Methodology
- Training Pipeline & Negative Results
- Installation and Usage
- Evaluation and Comprehensive Benchmarks
- Linguistic & Morpheme Analysis
- Downstream NLP Applications
- Limitations and Known Issues
- Complete Documentation Index
- References and Further Reading
- Team and Acknowledgments
- License and Citation
- Frequently Asked Questions (FAQ)
Sindhi is an Indo-Aryan language spoken by an estimated 25–30 million people, predominantly in the Sindh province of Pakistan and parts of India. Despite its rich literary heritage, Sindhi remains a severely under-resourced language in Natural Language Processing (NLP).
Existing modern Large Language Models (LLMs) and multilingual tokenizers (such as XLM-RoBERTa, LLaMA, Qwen, and BLOOM) under-represent Sindhi orthography, causing high token fertility, extreme sequence bloat, out-of-vocabulary (UNK) fallbacks, or destructive character drops.
This repository provides:
- Four Trained Models: Pretrained SentencePiece Unigram models at 32k, 64k (flagship recommended), 100k, and 150k vocabulary sizes.
- Corpus Engineering Pipeline: A 1.46M+ sentence corpus drawn from Sindhi and regional Wikipedia, mostly news text, a small Twitter component, and a 15% Urdu/Persian/Arabic anchor by character count.
- Full-Scale Benchmark Suite: Standardized evaluation of 18 tokenizers across 4 distinct registers (FLORES+, Wikipedia, Sentiment, Sindh Line News).
- Round-Trip Length Check: A normalized-length check that can flag text shortening; targeted examples were also inspected for deleted characters.
- Interactive UI: A Gradio web application for real-time subword tokenization inspection.
Sindhi uses a 52-letter extended Perso-Arabic alphabet containing implosive consonants (ٻ, ڄ, ڏ, ڳ), aspirated digraphs (ڀ, ٿ, ڇ, ٺ), retroflex consonants (including ڻ), and specialized diacritics. General Arabic or Urdu tokenizers frequently split these single graphemes into multi-byte fallback sequences, dramatically inflating token counts.
When a tokenizer splits a single word into 3–5 subword fragments, context window efficiency drops proportionally. A 2048-token context window accommodates only ~707 Sindhi words under LLaMA 3.2 (fertility 2.896) or ~1,354 words under XLM-RoBERTa (1.512), compared to ~1,786 Sindhi words under CustomSindhi_64k (1.147).
Standard tokenizer benchmarks often report token fertility and UNK rate without checking decoded output. During this project's audit, a normalized-length round-trip check, followed by inspection of affected examples, surfaced text-shortening defects in existing tokenizers:
TinyLlama_Sindhi: Drops native Sindhi letters such asU+06C1(HEH GOAL) and corrupts text encoded with Arabic Presentation Forms, destroying data in up to 24.9% of sentences.FahadQazi_BPE: Silently deletes Latin characters in code-switched technical text, corrupting up to 16.2% of sentences.- Our model: Shows 0.0000% normalized-length loss under this check on evaluated text with byte-level fallback; the metric is not proof of exact character identity.
- Deliver Flagship Tokenizer: Train and release a high-efficiency Sindhi subword tokenizer achieving the lowest fertility and highest single-token preservation across diverse registers.
- Measure Round-Trip Integrity: Use byte fallback and full observed-character coverage, then report the limits of the normalized-length check.
- Rigorous Empirical Comparison: Benchmark 14 baseline tokenizers under identical, unsampled test conditions.
- Transparent Documentation: Document negative results (what didn't work), corpus preparation heuristics, and evaluation caveats.
- Large Language Model (LLM) pretraining and fine-tuning for Sindhi.
- Sequence classification (Sentiment Analysis, Topic Modeling).
- Machine Translation (FLORES+ benchmarks).
- Named Entity Recognition (NER) and Information Retrieval.
- Speech recognition (ASR) acoustic unit modeling.
- Devnagari script Sindhi (this release targets Arabic-script Sindhi).
- SentencePiece Unigram Architecture: Statistically models subword probabilities using maximum likelihood estimation, proven superior to standard greedy BPE for morphologically rich languages.
- 100% Character Coverage + Byte Fallback: Zero unknown (
<unk>) token emissions. Unknown Unicode characters decompose cleanly into raw UTF-8 byte tokens without data corruption. - Flagship 64k Model: Balances vocabulary efficiency (1.147 fertility) with embedding table size, yielding an 88.6% single-token word retention rate.
- Round-Trip Check Result: 0.0000% normalized-length loss across the evaluated corpora; this does not establish exact character identity.
- Clean Subword Boundaries: Preserves leading whitespace symbols (
\u2581) for precise detokenization without text corruption.
| Layer | Technologies | Purpose |
|---|---|---|
| Core Engine | Python 3.9+, SentencePiece (C++ bindings) | Tokenizer training, Viterbi segmentation, byte fallback |
| NLP Framework | Hugging Face Transformers, huggingface_hub |
Adapter integration, model sharing and distribution |
| Datasets | Hugging Face Datasets, Pandas, NumPy | Corpus ingestion, normalization, filtering, metrics computation |
| Interactive Demo | Gradio | Real-time interactive subword visualization UI |
| Quality & CI/CD | Pytest, GitHub Actions | Automated regression tests across Python 3.9–3.12 |
| Documentation | GitHub Pages, Markdown, Mermaid.js | Static web docs, draft research manuscript, technical explainer |
sindhi-tokenizer/
├── .github/
│ └── workflows/
│ └── python-tests.yml # Automated CI workflow testing Python 3.9-3.12
├── .gitignore # Excludes models, artifacts, cache, and virtualenvs
├── CITATION.cff # Citation File Format (v1.2.0)
├── DATA_SOURCES.md # Complete provenance, volume, and licensing documentation
├── LICENSE # MIT Open Source License
├── README.md # Central documentation index and project guide
├── pyproject.toml # PEP 517/621 package metadata and dependency configuration
├── requirements.txt # Exact runtime dependencies
├── demo/
│ ├── README.md # Instructions for running and deploying the demo
│ └── app.py # Interactive Gradio comparison web app
├── docs/
│ ├── paper.md # Draft research manuscript, not yet published (14-baseline study)
│ └── explainer.md # 21-page technical companion log and negative results
├── evaluation/
│ ├── README.md # Evaluation metrics guide and CLI/API documentation
│ ├── __init__.py # Package initialization
│ └── metrics.py # Fertility, single-token, UNK, byte-fallback, and fidelity engine
├── notebooks/
│ ├── README.md # Reproducible notebook guide
│ ├── training.ipynb # Kaggle/Jupyter notebook: corpus download, split & training
│ └── benchmark.ipynb # Kaggle/Jupyter notebook: 18-tokenizer evaluation suite
├── results/
│ └── README.md # Benchmark tables and summary report across 4 registers
└── training/
├── README.md # Training recipes, regex splitters, and negative results
├── __init__.py # Package initialization
└── train_tokenizer.py # Standalone training script with optimal SentencePiece parameters
flowchart TD
subgraph Data_Pipeline["1. Corpus Assembly & Preprocessing"]
A1["Sindhi Wikipedia<br/>(6,000 articles)"] --> P["Sentence Extraction & Cleaning<br/>Regex Split: [۔.!\n؟]+<br/>Length Filter: 3–100 words"]
A2["Sindhi-NLP Dataset<br/>(Twitter + Kawish + Awami Awaz)"] --> P
A3["shaikhsalman Corpus<br/>(20k rows, >30% Latin filtered)"] --> P
A4["Multilingual Regional Anchor<br/>(Urdu, Persian, Arabic 15% mix)"] --> P
P --> C["Balanced Training Corpus<br/>(1.46M+ Sentences)"]
end
subgraph Training_Engine["2. Model Training (SentencePiece)"]
C --> SP["SentencePiece Unigram Trainer<br/>character_coverage=1.0<br/>byte_fallback=True<br/>split_by_whitespace=True"]
SP --> M1["Custom 32k Model"]
SP --> M2["Custom 64k Model (Flagship)"]
SP --> M3["Custom 100k Model"]
SP --> M4["Custom 150k Model"]
end
subgraph Evaluation_Suite["3. Rigorous Evaluation Engine"]
M2 --> E["evaluation/metrics.py"]
REG1["FLORES+ (snd_Arab)"] --> E
REG2["Wikipedia Formal (25.4k sent)"] --> E
REG3["Sentiment Informal (3.5k sent)"] --> E
REG4["Sindh Line News (87.1k sent)"] --> E
E --> MET["Metrics Computed:<br/>• Token Fertility<br/>• Single-Token Word %<br/>• UNK Rate (0.000%)<br/>• Byte Fallback Rate<br/>• Normalized-Length Loss (0.000%)"]
end
subgraph Downstream["4. Downstream Deployment"]
M2 --> HF["Hugging Face Hub<br/>alinawazmahar/sindhi-tokenizer"]
M2 --> GR["Interactive Gradio Demo<br/>demo/app.py"]
M2 --> LLM["Sindhi LLM Pretraining & NLP"]
end
sequenceDiagram
autonumber
participant Input as Raw Sindhi Text
participant Norm as NFKC & Formatting Normalizer
participant Tokenizer as SPMTokenizerAdapter
participant Tokens as Subword Tokens / Byte IDs
participant Output as Decoded Reconstructed Text
participant Audit as Normalized-Length Check
Input->>Norm: Normalize Unicode (NFKC) & remove zero-width formatting
Norm->>Tokenizer: Clean text string
Tokenizer->>Tokens: Encode via Unigram Viterbi path (out_type=int)
Tokens->>Output: Decode token IDs back to string
Output->>Audit: Compare Original vs Decoded
Note over Audit: Loss = max(0, input_length - decoded_length) / input_length<br/>Custom Model Result: 0.0000% normalized-length loss
Training and evaluation records are separated where known, but the sentiment evaluation set shares publisher-adjacent provenance with a training source. See DATA_SOURCES.md for source details and licensing status.
| Source Dataset | Origin / Provider | Volume Used | Preprocessing & Filtering Rules | License |
|---|---|---|---|---|
| Sindhi Wikipedia | wikimedia/wikipedia (20231101.sd) |
6,000 articles | Articles 2,001–8,000 (first 2,000 reserved for evaluation). Filtered to 3–100 words. | CC BY-SA 3.0 |
| Sindhi-NLP | GitHub: Sindhi-NLP/sindhi-NLP-dataset |
4 CSV collections | Primarily news (98.4% Awami Awaz and Daily Kawish articles) with 1.6% social tweets. | MIT |
| shaikhsalman Corpus | shaikhsalman/sindhi-llm-corpus |
20,000 rows | Seed 7 shuffle. Sentences with >30% Latin characters dropped (removed ~11% noise). | Project owner reports free use; exact license and conditions are not shown on the card (see DATA_SOURCES.md) |
| Multilingual Anchor | Urdu, Persian, Arabic Wikipedia | 1,500 articles each | Balanced to represent 15% of total corpus characters. Exposes model to shared Perso-Arabic roots. | CC BY-SA 3.0 |
| Register | Dataset Identifier | Sentences | Domain / Linguistic Style |
|---|---|---|---|
flores200_anchor |
openlanguagedata/flores_plus (snd_Arab) |
1,012 | Standardized translation benchmark; formal and modern register. |
wikipedia_formal |
wikimedia/wikipedia (20231101.sd) |
25,480 | Formal encyclopedic Sindhi prose (first 2,000 articles, disjoint from train). |
sentiment_informal (legacy ID) |
alinawazmahar/Sindhi_Sentiment_dataset |
3,536 | Newspaper text pseudo-labeled for sentiment; this is news, not social-media text. |
sindh_line_news |
Mozilla Data Collective (Sindh Line Publishers) |
87,158 | Contemporary, daily regional news coverage (CC BY-SA 4.0). |
Byte-Pair Encoding (BPE) merges the most frequent character pairs iteratively. In morphologically complex, agglutinative, or clitic-heavy languages like Sindhi, greedy merges frequently fuse grammatical suffixes onto root stems incorrectly.
The Unigram language model algorithm (Kudo, 2018) initializes an oversized vocabulary and optimizes subword segmentation using maximum likelihood:
This statistically discovers true morphological subwords while optimizing whole-sequence likelihood.
Consider the authentic Sindhi phrase:
- Input Normalization: NFKC normalization ensures canonical representation of Arabic script forms.
- Whitespace Prefixing: Words receive the boundary marker
\u2581). - Unigram Segmentation:
[' خداوند', ' ڪريم', ' کيس', ' اعليٰ', ' ذهانت', ' سان', ' نوازيو', ' هو'] - Token ID Mapping:
[12450, 4812, 1089, 7543, 14210, 521, 18920, 412] - Round-trip check: Evaluated text showed 0.0000% normalized-length loss. This length-based check can miss equal-length substitutions or reorderings.
python training/train_tokenizer.py \
--corpus corpus.txt \
--vocab_size 64000 \
--model_prefix models/sindhi_custom_64000model_type="unigram"character_coverage=1.0(Ensures every single observed Unicode character is covered)byte_fallback=True(Unknown characters fall back to raw byte tokens<0xXX>)split_by_whitespace=Trueunk_id=0,bos_id=1,eos_id=2,pad_id=-1
In accordance with rigorous research standards, two negative experimental results are preserved in docs/explainer.md:
- Forced Postposition Tokens: Forcing common postpositions (e.g.,
جو,جي,کي) and loanwords to remain single tokens degraded the training objective (final objective value rose from ≈9.6 unforced to ≈14.9 forced at 32k vocab). Given enough data, Unigram naturally discovers these tokens with superior probability allocations. - Pre-marking Clitic Boundaries: Pre-segmenting supposed fused clitics with whitespace proved harmful. Investigation revealed that suspected "fused forms" were simply coincidental orthographic overlaps with unrelated lexical roots.
- Python 3.9, 3.10, 3.11, or 3.12
sentencepiece,transformers,huggingface_hub
git clone https://github.com/alinawazcode/sindhi-tokenizer.git
cd sindhi-tokenizer
pip install -e .from huggingface_hub import hf_hub_download
import sentencepiece as spm
# Download the recommended 64k flagship model
model_path = hf_hub_download(
repo_id="alinawazmahar/sindhi-tokenizer",
filename="sindhi_custom_64000.model"
)
# Load into SentencePiece processor
sp = spm.SentencePieceProcessor(model_file=model_path)
text = "سنڌ جي تاريخ تمام قديم آهي"
tokens = sp.encode(text, out_type=str)
token_ids = sp.encode(text, out_type=int)
print("Tokens:", tokens)
# [' سنڌ', ' جي', ' تاريخ', ' تمام', ' قديم', ' آهي']
print("Token IDs:", token_ids)
print("Decoded:", sp.decode(token_ids))Caution
Do not use transformers.LlamaTokenizer to load raw .model files: Hugging Face's LlamaTokenizer can silently fail on custom SPM Unigram files and return a 3-token fallback vocabulary. Always load using sentencepiece.SentencePieceProcessor or the provided evaluation.metrics.SPMTokenizerAdapter.
All 18 benchmark entries were evaluated across the four evaluation registers without subsampling. SindhiLM_v2 is identical to Qwen2.5 for this evaluation; the entries therefore represent 17 distinct tokenizer behaviors.
| Tokenizer | Vocab Size | Fertility (tokens/word) ↓ | Single-Token Word % ↑ | UNK Rate % ↓ | Byte-Fallback % ↓ | Worst Normalized-Length Loss % ↓ |
|---|---|---|---|---|---|---|
| CustomSindhi_150k | 150,000 | 1.125 | 89.7% | 0.000% | 0.154% | 0.0000% |
| CustomSindhi_100k | 100,000 | 1.133 | 89.3% | 0.000% | 0.149% | 0.0000% |
| CustomSindhi_64k (Flagship) | 64,000 | 1.147 | 88.6% | 0.000% | 0.138% | 0.0000% |
| CustomSindhi_32k | 32,000 | 1.184 | 86.5% | 0.000% | 0.113% | 0.0000% |
| TinyLlama_Sindhi | 42,681 | 1.208 | 84.4% | 0.000%* | 0.000%* | 0.4147% (Corrupts 24.9% sent) |
| hellosindh_SindhiBERT_NER | 32,001 | 1.218 | 83.7% | 0.194% | 0.197% | 0.0751% |
| mBERT_Sindhi_Kashif786 | 119,547 | 1.278 | 78.4% | 0.143% | 0.000% | 0.0021% |
| XLM-RoBERTa | 250,002 | 1.512 | 65.4% | 0.034% | 1.402% | 0.0040% |
| SindhiLM_v3 | 32,000 | 1.730 | 55.8% | 3.605% | 0.000% | 0.0754% |
| FahadQazi_BPE | 13,031 | 1.919 | 39.1% | 0.000%* | 7.127%* | 0.8461% (Corrupts 16.2% sent) |
| hellosindh_SindhiBERT | 32,000 | 1.960 | 35.5% | 0.122% | 0.114% | 0.0089% |
| mT5 | 250,100 | 2.280 | 21.7% | 0.022% | 12.746% | 0.0040% |
| Gemma3 | 262,144 | 2.308 | 26.8% | 0.000% | 0.105% | 0.0000% |
| Qwen2.5 / SindhiLM_v2 | 151,643 | 2.580 | 23.8% | 0.000% | 8.665% | 0.0000% |
| BLOOM | 250,680 | 2.627 | 24.7% | 0.000% | 31.660% | 0.0000% |
| Llama3.2 | 128,000 | 2.896 | 18.2% | 0.000% | 28.472% | 0.0000% |
| SindhiLM_v1 | 43 | 4.306 | 0.1% | 14.017% | 0.000% | 0.0790% |
* TinyLlama_Sindhi and FahadQazi_BPE show apparent 0.000% UNK rates because both models silently delete unsupported characters rather than emitting a UNK token. FahadQazi_BPE shows real, severe character loss on round-trip fidelity testing alongside high byte-fallback. SindhiLM_v2 is numerically identical to Qwen2.5 across all metrics; both are listed in the table for completeness.
- Unconditional Superiority at ≥64k for Fertility & Single-Token Retention: CustomSindhi (64k, 100k, 150k) achieves the lowest fertility and highest single-token retention across all four registers simultaneously. At 32k, one narrow exception exists: TinyLlama_Sindhi beats CustomSindhi_32k on FLORES+ fertility (though it carries a critical character-deletion defect). On byte-fallback, CustomSindhi_64k (0.138%) achieves near-lowest fallback, with Gemma 3 slightly lower at 0.105%.
- Efficiency gains at 64k: Mean fertility improves by 24.1% over XLM-RoBERTa and 60.4% over LLaMA 3.2. This describes fertility and single-token retention, not every metric.
- Round-trip check results: Custom models show 0.0000% normalized-length loss under the stated check on all four registers. This length-based check does not prove exact character identity; TinyLlama_Sindhi and FahadQazi_BPE show measurable length loss.
To verify that vocabulary expansion did not simply overfit high-frequency substrings, all tokenizers were evaluated on a targeted 22-word linguistic test suite spanning four distinct categories:
- Genitive & Possessive Constructions (
دنيا جو,پاڻي جي,ماڻهن جو): All Custom models split these cleanly into base noun + postposition clitic. - Inflectional Suffixes (
ڪتابن,شهرن,توهانکي,منهنجو): Preserved as single tokens in Custom 64k models. - Sindhi Implosives & Aspirated Digraphs (
ٻار,ڄاڻ,ڏينهن,ڳوٺ,ڀاءُ,ٿورو,ڇوڪرو): Never broken into disconnected diacritics or fallback bytes. - Loanwords & Technical Terms (
ڪمپيوٽر,انٽرنيٽ,اسڪول,تعليم): Retained as intact single lexical units.
Note
Aggregate vs. Word-Level Distinction: Across the 22-word test suite, the Custom tokenizer ties the best competitor on 21 of 22 words and loses one (هنجي, to FahadQazi_BPE and mBERT_Sindhi jointly). The primary advantage of CustomSindhi is consistency across aggregate, full-scale text registers rather than hand-picked individual words. See docs/paper.md#appendix-b for the full 22-word breakdown.
Note: As documented in the paper (§6), the tokenizer is designed for textual NLP representation; downstream task-specific fine-tuning (e.g. BLEU/ROUGE benchmarks) represents intended future work.
| Intended Application | Hypothesized Architectural Benefit |
|---|---|
| Sindhi Large Language Models (LLMs) | Decreases token sequence length by 50–60% vs general multilingual LLMs (LLaMA, BLOOM), effectively doubling or tripling context length capacity during pretraining and inference. |
| Machine Translation (MT) | Preserves postpositions and implosive graphemes intact, intended to reduce subword fragmentation and hallucination during sequence-to-sequence translation. |
| Sentiment Analysis & Text Classification | Retains full grammatical morphemes, yielding richer dense embeddings for subword encoders. |
| Named Entity Recognition (NER) | Eliminates character drop bugs that corrupt proper nouns, locations, and transliterated person names. |
In accordance with rigorous academic transparency (see docs/paper.md §6), seven limitations are explicitly recorded:
- Absence of Classical / Literary Register: No classical or literary Sindhi register is included. Five independent candidate sources were investigated: a corrupted CSV export, an unrelated SQL dump, two unusable OCR scans (one incompatibly licensed), and a government-affiliated platform gated behind an institutional subscription.
- Partial Source Overlap in Evaluation: The Sentiment register and the Sindhi-NLP component of the training corpus share adjacent news-source provenance; full genre-level independence is not established for this pair.
- Metric-specific results: Against FLORES+ specifically, CustomSindhi_32k has one confirmed loss on fertility and single-token retention. The all-register lead at 64k and above refers to those two measures; byte-fallback is a separate trade-off.
- Qualitative Test Does Not Differentiate at Word Level: On the 22-word test list, the new tokenizer ties rather than beats the best competitor on 21 of 22 words; the demonstrated advantage is aggregate across full corpora, not example-level.
- Search for Existing Tokenizers May Be Incomplete: Four additional tokenizers were discovered after initial benchmarking; while subsequent searches found no further models, the open-source NLP space is active enough that this cannot be guaranteed exhaustive.
- Downstream Applications Undetermined: The tokenizer's target use is future Sindhi language modeling. It is not validated for Automatic Speech Recognition (ASR) or Text-to-Speech (TTS).
- Register Weighting Choice: The four evaluation registers differ substantially in size; an unweighted macro mean was used rather than a token-pooled mean, which would have been dominated by Sindh Line News (~2.5× the other three combined).
| Document | Path | Purpose |
|---|---|---|
| Root Guide & Navigation Hub | README.md |
Central repository overview, quickstart, and full benchmark summary |
| Research Manuscript Draft | docs/paper.md |
Draft for future publication; not yet published. Includes methodology and 18 benchmark rows |
| Technical Explainer Log | docs/explainer.md |
In-depth engineering explainer into empirical failures, metric corrections, and design rationale |
| Dataset Provenance Card | DATA_SOURCES.md |
Complete documentation of all training corpora and evaluation sets, licensing, and volumes |
| Evaluation Module Guide | evaluation/README.md |
Technical documentation of metrics algorithms, CLI arguments, and Python API |
| Training Pipeline Guide | training/README.md |
Exact SentencePiece training recipe, filtering thresholds, and negative findings |
| Benchmark Result Summaries | results/README.md |
Detailed register breakdowns across FLORES+, Wikipedia, Sentiment, and Sindh Line |
| Notebook Reproduction Guide | notebooks/README.md |
Instructions for Kaggle/Colab execution of training and benchmarking pipelines |
| Interactive Demo Guide | demo/README.md |
Local execution and Hugging Face Spaces deployment guide for Gradio UI |
| Citation Specification | CITATION.cff |
Machine-readable CFF citation metadata |
| Project Packaging | pyproject.toml |
Modern build-system and packaging metadata |
- Kudo, T. & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. EMNLP 2018 System Demonstrations. https://aclanthology.org/D18-2012/
- Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. ACL 2018. https://aclanthology.org/P18-1007/
- Conneau, A., et al. (2020). Unsupervised Cross-lingual Representation Learning at Scale (XLM-RoBERTa). ACL 2020. arXiv:1911.02116
- NLLB Team et al. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation (FLORES-200). arXiv:2207.04672
- Qwen Team (2024). Qwen2.5 Technical Report. arXiv:2412.15115
- Meta AI (2024). Llama 3.2 Model Card & Architecture Overview. https://huggingface.co/meta-llama
Note: For the complete list of 21 baseline model citations and references, please refer to docs/paper.md.
This research and tokenizer implementation were developed at the Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan, under the supervision of Dr. Attaullah Sahito.
- Ali Nawaz (Lead author and project lead): GitHub (@alinawazcode)
- Muqadas Fareed (author): LinkedIn Profile
- Soonh Sadaf (Sadaf Khaliq) (author): LinkedIn Profile
- Dr. Attaullah Sahito (co-author and academic supervisor): LinkedIn Profile, Department of Computer Science, Shaikh Ayaz University.
- Software Code: Licensed under the open-source MIT License.
- Trained Tokenizer Models: The Hugging Face listing states CC BY-SA 4.0. Source datasets are reported by the project lead to be free to use, mostly under MIT and other open licenses; exact terms and attribution conditions vary by source and are summarized in
DATA_SOURCES.md. - Corpus Texts: Refer to
DATA_SOURCES.mdfor original source licenses.
If you use this tokenizer, benchmark code, or findings in your academic research or applications, please cite:
@misc{nawaz2026sindhitokenizer,
author = {Nawaz, Ali and Fareed, Muqadas and Sadaf, Soonh and Sahito, Attaullah},
title = {A Corpus-Driven Sindhi Subword Tokenizer: Design, Empirical Validation, and Benchmark Comparison Against Fourteen Existing Tokenizers},
year = {2026},
howpublished = {\url{https://github.com/alinawazcode/sindhi-tokenizer}},
note = {Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan}
}1. Why choose the 64k model over 32k or 150k?
The 64,000 vocabulary size represents the empirical sweet spot between fertility efficiency (1.147 tokens/word) and embedding matrix parameter overhead. Scaling to 100k or 150k provides marginal fertility gains (1.133 and 1.125 respectively) but more than doubles the vocabulary table size, while the 32k model exhibits one minor loss on the standardized FLORES+ benchmark.2. What does the round-trip length check measure?
It applies Unicode NFKC, removes zero-width formatting characters and whitespace, then compares normalized input and decoded-output lengths. This detects shortening but can miss substitutions or reorderings that preserve length; 0.0000% is not proof of exact text identity.3. Why is there a 15% mix of Urdu, Persian, and Arabic in the training corpus?
Sindhi shares the Perso-Arabic alphabet and borrows significant religious, legal, and cultural vocabulary from Arabic and Persian, as well as shared regional terms with Urdu. This controlled 15% anchor was added by design to expose the subword vocabulary to shared Perso-Arabic root morphemes.4. Why should I avoid transformers.LlamaTokenizer for loading this model?
Certain versions of Hugging Face Transformers'LlamaTokenizer fail to load custom SentencePiece Unigram models correctly, silently substituting an empty or 3-token fallback vocabulary without throwing an exception. Always use sentencepiece.SentencePieceProcessor directly or via our verified SPMTokenizerAdapter.