Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties

A language can have different varieties. These varieties can affect the performance of natural language processing (NLP) models, including large language models (LLMs), which are often trained on data from widely spoken varieties. This paper introduces a novel and cost-effective approach to benchmark model performance across language varieties. We argue that international online review platforms, such as Booking.com, can serve as effective data sources for constructing datasets that capture comments in different language varieties from similar real-world scenarios, like reviews for the same hotel with the same rating using the same language (e.g., Mandarin Chinese) but different language varieties (e.g., Taiwan Mandarin, Mainland Mandarin). To prove this concept, we constructed a contextually aligned dataset comprising reviews in Taiwan Mandarin and Mainland Mandarin and tested six LLMs in a sentiment analysis task. Our results show that LLMs consistently underperform in Taiwan Mandarin.

Data Assetization via Resources-decoupled Federated Learning

With the development of the digital economy, data is increasingly recognized as an essential resource for both work and life. However, due to privacy concerns, data owners tend to maximize the value of data through information flow rather than direct data transfer. Federated learning (FL) provides an effective approach to collaborative training models while preserving privacy. However, different data owners not only have variations in the quantity and quality of their data resources but also face mismatches between data and computing resources as model parameters and training data grow. These challenges hinder data owners' willingness to participate and reduce the effectiveness of data assetization. In this work, we first identify the resource-decoupled FL environment, which includes model owners, data owners, and computing centers. We design a Tripartite Stackelberg Model and theoretically analyze the Stackelberg-Nash Equilibrium (SNE) for participants to optimize global utility. We propose the Quality-aware Dynamic Resources-decoupled FL algorithm (QD-RDFL), in which we derive and solve the optimal strategies of all parties to achieve SHE using backward induction, and a dynamic optimization mechanism is designed to improve the optimal strategy profile by evaluating the contribution of data quality from data owners to the global model during real training. Our comprehensive experiments demonstrate that our method effectively encourages the linkage of the three parties involved, maximizing global utility and data asset value.

Learning to Write Rationally: How Information Is Distributed in Non-Native Speakers' Essays

People tend to distribute information evenly in language production for better and clearer communication. In this study, we compared essays written by second language learners with various native language (L1) backgrounds to investigate how they distribute information in their non-native language (L2) production. Analyses of surprisal and constancy of entropy rate indicated that writers with higher L2 proficiency can reduce the expected uncertainty of language production while still conveying informative content. However, the uniformity of information distribution showed less variability among different groups of L2 speakers, suggesting that this feature may be universal in L2 essay writing and less affected by L2 writers' variability in L1 background and L2 proficiency.

Like a bilingual baby: The advantage of visually grounding a bilingual language model

Unlike most neural language models, humans learn language in a rich, multi-sensory and, often, multi-lingual environment. Current language models typically fail to fully capture the complexities of multilingual language use. We train an LSTM language model on images and captions in English and Spanish from MS-COCO-ES. We find that the visual grounding improves the model's understanding of semantic similarity both within and across languages and improves perplexity. However, we find no significant advantage of visual grounding for abstract words. Our results provide additional evidence of the advantages of visually grounded language models and point to the need for more naturalistic language data from multilingual speakers and multilingual datasets with perceptual grounding.