Picture for Koki Shibata

Koki Shibata

How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese

Add code
Jun 16, 2023
Viaarxiv icon