Picture for Juan Luis Gastaldi

Juan Luis Gastaldi

From Language Models over Tokens to Language Models over Characters

Add code
Dec 04, 2024
Viaarxiv icon

On the Proper Treatment of Tokenization in Psycholinguistics

Add code
Oct 03, 2024
Figure 1 for On the Proper Treatment of Tokenization in Psycholinguistics
Figure 2 for On the Proper Treatment of Tokenization in Psycholinguistics
Figure 3 for On the Proper Treatment of Tokenization in Psycholinguistics
Figure 4 for On the Proper Treatment of Tokenization in Psycholinguistics
Viaarxiv icon

The Foundations of Tokenization: Statistical and Computational Concerns

Add code
Jul 16, 2024
Viaarxiv icon

Tokenization and the Noiseless Channel

Add code
Jun 29, 2023
Viaarxiv icon

A Formal Perspective on Byte-Pair Encoding

Add code
Jun 29, 2023
Figure 1 for A Formal Perspective on Byte-Pair Encoding
Figure 2 for A Formal Perspective on Byte-Pair Encoding
Figure 3 for A Formal Perspective on Byte-Pair Encoding
Figure 4 for A Formal Perspective on Byte-Pair Encoding
Viaarxiv icon