TrustNavGPT: Modeling Uncertainty to Improve Trustworthiness of Audio-Guided LLM-Based Robot Navigation

While LLMs are proficient at processing text in human conversations, they often encounter difficulties with the nuances of verbal instructions and, thus, remain prone to hallucinate trust in human command. In this work, we present TrustNavGPT, an LLM based audio guided navigation agent that uses affective cues in spoken communication elements such as tone and inflection that convey meaning beyond words, allowing it to assess the trustworthiness of human commands and make effective, safe decisions. Our approach provides a lightweight yet effective approach that extends existing LLMs to model audio vocal features embedded in the voice command and model uncertainty for safe robotic navigation.

RU-AI: A Large Multimodal Dataset for Machine Generated Content Detection

The recent advancements in generative AI models, which can create realistic and human-like content, are significantly transforming how people communicate, create, and work. While the appropriate use of generative AI models can benefit the society, their misuse poses significant threats to data reliability and authentication. However, due to a lack of aligned multimodal datasets, effective and robust methods for detecting machine-generated content are still in the early stages of development. In this paper, we introduce RU-AI, a new large-scale multimodal dataset designed for the robust and efficient detection of machine-generated content in text, image, and voice. Our dataset is constructed from three large publicly available datasets: Flickr8K, COCO, and Places205, by combining the original datasets and their corresponding machine-generated pairs. Additionally, experimental results show that our proposed unified model, which incorporates a multimodal embedding module with a multilayer perceptron network, can effectively determine the origin of the data (i.e., original data samples or machine-generated ones) from RU-AI. However, future work is still required to address the remaining challenges posed by RU-AI. The source code and dataset are available at https://github.com/ZhihaoZhang97/RU-AI.

Game-MUG: Multimodal Oriented Game Situation Understanding and Commentary Generation Dataset

The dynamic nature of esports makes the situation relatively complicated for average viewers. Esports broadcasting involves game expert casters, but the caster-dependent game commentary is not enough to fully understand the game situation. It will be richer by including diverse multimodal esports information, including audiences' talks/emotions, game audio, and game match event information. This paper introduces GAME-MUG, a new multimodal game situation understanding and audience-engaged commentary generation dataset and its strong baseline. Our dataset is collected from 2020-2022 LOL game live streams from YouTube and Twitch, and includes multimodal esports game information, including text, audio, and time-series event logs, for detecting the game situation. In addition, we also propose a new audience conversation augmented commentary dataset by covering the game situation and audience conversation understanding, and introducing a robust joint multimodal dual learning model as a baseline. We examine the model's game situation/event understanding ability and commentary generation capability to show the effectiveness of the multimodal aspects coverage and the joint integration learning approach.

FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission

Communication overhead is a significant bottleneck in federated learning (FL), which has been exaggerated with the increasing size of AI models. In this paper, we propose FedRDMA, a communication-efficient cross-silo FL system that integrates RDMA into the FL communication protocol. To overcome the limitations of RDMA in wide-area networks (WANs), FedRDMA divides the updated model into chunks and designs a series of optimization techniques to improve the efficiency and robustness of RDMA-based communication. We implement FedRDMA atop the industrial federated learning framework and evaluate it on a real-world cross-silo FL scenario. The experimental results show that \sys can achieve up to 3.8$\times$ speedup in communication efficiency compared to traditional TCP/IP-based FL systems.

Brain-Inspired Modelling and Decision-making for Human-Like Autonomous Driving in Mixed Traffic Environment

In this paper, a human-like driving framework is designed for autonomous vehicles (AVs), which aims to make AVs better integrate into the transportation ecology of human driving and eliminate the misunderstanding and incompatibility of human drivers to autonomous driving. Based on the analysis of the real world INTERACTION dataset, a driving aggressiveness estimation model is established with the fuzzy inference approach. Then, a human-like driving model, which integrates the brain emotional learning circuit model (BELCM) with the two-point preview model, is designed. In the human-like lane-change decision-making algorithm, the cost function is designed comprehensively considering driving safety and travel efficiency. Based on the cost function and multi-constraint, the dynamic game algorithm is applied to modelling the interaction and decision making between AV and human driver. Additionally, to guarantee the lane-change safety of AVs, an artificial potential field model is built for collision risk assessment. Finally, the proposed algorithm is evaluated through human-in-the-loop experiments on a driving simulator, and the results demonstrated the feasibility and effectiveness of the proposed method.

A Marker-free Head Tracker Using Vision-based Head Pose Estimation with Adaptive Kalman Filter

The immersion and the interaction are the important features of the driving simulator. To improve these characteristics, this paper proposes a low-cost and mark-less driver head tracking framework based on the head pose estimation model, which makes the view of the simulator can automatically align with the driver's head pose. The proposed method only uses the RGB camera without the other hardware or marker. To handle the error of the head pose estimation model, this paper proposes an adaptive Kalman Filter. By analyzing the error distribution of the estimation model and user experience, the proposed Kalman Filter includes the adaptive observation noise coefficient and loop closure module, which can adaptive moderate the smoothness of the curve and keep the curve stable near the initial position. The experiments show that the proposed method is feasible, and it can be used with different head pose estimation models.