Generalizability analysis of deep learning predictions of human brain responses to augmented and semantically novel visual stimuli

The purpose of this work is to investigate the soundness and utility of a neural network-based approach as a framework for exploring the impact of image enhancement techniques on visual cortex activation. In a preliminary study, we prepare a set of state-of-the-art brain encoding models, selected among the top 10 methods that participated in The Algonauts Project 2023 Challenge [16]. We analyze their ability to make valid predictions about the effects of various image enhancement techniques on neural responses. Given the impossibility of acquiring the actual data due to the high costs associated with brain imaging procedures, our investigation builds up on a series of experiments. Specifically, we analyze the ability of brain encoders to estimate the cerebral reaction to various augmentations by evaluating the response to augmentations targeting objects (i.e., faces and words) with known impact on specific areas. Moreover, we study the predicted activation in response to objects unseen during training, exploring the impact of semantically out-of-distribution stimuli. We provide relevant evidence for the generalization ability of the models forming the proposed framework, which appears to be promising for the identification of the optimal visual augmentation filter for a given task, model-driven design strategies as well as for AR and VR applications.

Exploration and Comparison of Deep Learning Architectures to Predict Brain Response to Realistic Pictures

We present an exploration of machine learning architectures for predicting brain responses to realistic images on occasion of the Algonauts Challenge 2023. Our research involved extensive experimentation with various pretrained models. Initially, we employed simpler models to predict brain activity but gradually introduced more complex architectures utilizing available data and embeddings generated by large-scale pre-trained models. We encountered typical difficulties related to machine learning problems, e.g. regularization and overfitting, as well as issues specific to the challenge, such as difficulty in combining multiple input encodings, as well as the high dimensionality, unclear structure, and noisy nature of the output. To overcome these issues we tested single edge 3D position-based, multi-region of interest (ROI) and hemisphere predictor models, but we found that employing multiple simple models, each dedicated to a ROI in each hemisphere of the brain of each subject, yielded the best results - a single fully connected linear layer with image embeddings generated by CLIP as input. While we surpassed the challenge baseline, our results fell short of establishing a robust association with the data.

Calibrating Wayfinding Decisions in Pedestrian Simulation Models: The Entropy Map

This paper presents entropy maps, an approach to describing and visualising uncertainty among alternative potential movement intentions in pedestrian simulation models. In particular, entropy maps show the instantaneous level of randomness in decisions of a pedestrian agent situated in a specific point of the simulated environment with an heatmap approach. Experimental results highlighting the relevance of this tool supporting modelers are provided and discussed.

Cellular Automaton Based Simulation of Large Pedestrian Facilities - A Case Study on the Staten Island Ferry Terminals

Current metropolises largely depend on a functioning transport infrastructure and the increasing demand can only be satisfied by a well organized mass transit. One example for a crucial mass transit system is New York City's Staten Island Ferry, connecting the two boroughs of Staten Island and Manhattan with a regular passenger service. Today's demand already exceeds 2500 passengers for a single cycle during peek hours, and future projections suggest that it will further increase. One way to appraise how the system will cope with future demand is by simulation. This contribution proposes an integrated simulation approach to evaluate the system performance with respect to future demand. The simulation relies on a multiscale modeling approach where the terminal buildings are simulated by a microscopic and quantitatively valid cellular automata (CA) and the journeys of the ferries themselves are modeled by a mesoscopic queue simulation approach. Based on the simulation results recommendations with respect to the future demand are given.