Search results

Feature Request: Stochastic Tokenization (Dropout) via Vocabulary Masking

Status: Open.

#42 In alasdairforsythe/tokenmonster;

· Thomas-MMJ opened on Dec 22, 2025

Proper Method to Represent Special Characters?

Status: Open.

#41 In alasdairforsythe/tokenmonster;

· xTimeCrystal opened on Oct 26, 2025

Expose spans for mapping tokens to input characters

Status: Open.

#39 In alasdairforsythe/tokenmonster;

· iiLaurens opened on Jan 28, 2025

Preferred citation bibtex

Status: Open.

#35 In alasdairforsythe/tokenmonster;

· konstantinjdobler opened on May 8, 2024

Hangs with PyTorch data loaders when num_workers > 0

Status: Open.

#34 In alasdairforsythe/tokenmonster;

· ntoxeg opened on Mar 22, 2024

Update on multilingual

Status: Open.

#32 In alasdairforsythe/tokenmonster;

· kerighan opened on Feb 27, 2024

Inquiry on Extending Algorithm to Other Languages

Status: Open.

#30 In alasdairforsythe/tokenmonster;

· dsdanielpark opened on Dec 12, 2023

Special tokens not showing up correctly when tokenized.

Status: Open.

#29 In alasdairforsythe/tokenmonster;

· amazingvince opened on Nov 5, 2023

What is the difference between 50256-consistent-oneword and 50256-consistent?

Status: Open.

#24 In alasdairforsythe/tokenmonster;

· Calvinnncy97 opened on Sep 6, 2023

C implementation

Status: Open.

#17 In alasdairforsythe/tokenmonster;

· abb128 opened on Aug 3, 2023

Meaning of C and D

Status: Open.

#15 In alasdairforsythe/tokenmonster;

· Maxscha opened on Aug 3, 2023

HUggingface tokenizer coming soon?

Status: Open.

#4 In alasdairforsythe/tokenmonster;

· kyegomez opened on Jun 4, 2023