Investigating the effect of sub-word segmentation on the performance of transformer language models

Add code
May 09, 2023
Jue Hou, Anisia Katinskaia, Anh-Duc Vu, Roman Yangarber
Figure 1 for Investigating the effect of sub-word segmentation on the performance of transformer language models
Figure 2 for Investigating the effect of sub-word segmentation on the performance of transformer language models
Figure 3 for Investigating the effect of sub-word segmentation on the performance of transformer language models
Figure 4 for Investigating the effect of sub-word segmentation on the performance of transformer language models

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

We would like to explore how morphemes can affect the performance of a language model. We trained GPT-2 and Bert model with StateMorph for both Finnish and Russian, which is a morpheme segmenting algorithm. As a comparison, we also trained a model with BPE and Morfessor. Our preliminary result shows that StateMorph can help the model to converge more efficiently and achieve a better validation score.

View paper onarxiv icon

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon