Demographic Probing of Large Language Models Lacks Construct Validity

Demographic probing is widely used to study how large language models (LLMs) adapt their behavior to signaled demographic attributes. This approach typically uses a single demographic cue in isolation (e.g., a name or dialect) as a signal for group membership, implicitly assuming strong construct validity: that such cues are interchangeable operationalizations of the same underlying, demographically conditioned behavior. We test this assumption in realistic advice-seeking interactions, focusing on race and gender in a U.S. context. We find that cues intended to represent the same demographic group induce only partially overlapping changes in model behavior, while differentiation between groups within a given cue is weak and uneven. Consequently, estimated disparities are unstable, with both magnitude and direction varying across cues. We further show that these inconsistencies partly arise from variation in how strongly cues encode demographic attributes and from linguistic confounders that independently shape model behavior. Together, our findings suggest that demographic probing lacks construct validity: it does not yield a single, stable characterization of how LLMs condition on demographic information, which may reflect a misspecified or fragmented construct. We conclude by recommending the use of multiple, ecologically valid cues and explicit control of confounders to support more defensible claims about demographic effects in LLMs.

Mammography Assessment using Multi-Scale Deep Classifiers

Applying deep learning methods to mammography assessment has remained a challenging topic. Dense noise with sparse expressions, mega-pixel raw data resolution, lack of diverse examples have all been factors affecting performance. The lack of pixel-level ground truths have especially limited segmentation methods in pushing beyond approximately bounding regions. We propose a classification approach grounded in high performance tissue assessment as an alternative to all-in-one localization and assessment models that is also capable of pinpointing the causal pixels. First, the objective of the mammography assessment task is formalized in the context of local tissue classifiers. Then, the accuracy of a convolutional neural net is evaluated on classifying patches of tissue with suspicious findings at varying scales, where highest obtained AUC is above $0.9$. The local evaluations of one such expert tissue classifier is used to augment the results of a heatmap regression model and additionally recover the exact causal regions at high resolution as a saliency image suitable for clinical settings.