Overcoming Domain Limitations in Open-vocabulary Segmentation

Open-vocabulary segmentation (OVS) has gained attention for its ability to recognize a broader range of classes. However, OVS models show significant performance drops when applied to unseen domains beyond the previous training dataset. Fine-tuning these models on new datasets can improve performance, but often leads to the catastrophic forgetting of previously learned knowledge. To address this issue, we propose a method that allows OVS models to learn information from new domains while preserving prior knowledge. Our approach begins by evaluating the input sample's proximity to multiple domains, using precomputed multivariate normal distributions for each domain. Based on this prediction, we dynamically interpolate between the weights of the pre-trained decoder and the fine-tuned decoders. Extensive experiments demonstrate that this approach allows OVS models to adapt to new domains while maintaining performance on the previous training dataset. The source code is available at https://github.com/dongjunhwang/dwi.

Improving Response Time of Home IoT Services in Federated Learning

For intelligent home IoT services with sensors and machine learning, we need to upload IoT data to the cloud server which cannot share private data for training. A recent machine learning approach, called federated learning, keeps user data on the device in the distributed computing environment. Though federated learning is useful for protecting privacy, it experiences poor performance in terms of the end-to-end response time in home IoT services, because IoT devices are usually controlled by remote servers in the cloud. In addition, it is difficult to achieve the high accuracy of federated learning models due to insufficient data problems and model inversion attacks. In this paper, we propose a local IoT control method for a federated learning home service that recognizes the user behavior in the home network quickly and accurately. We present a federated learning client with transfer learning and differential privacy to solve data scarcity and data model inversion attack problems. From experiments, we show that the local control of home IoT devices for user authentication and control message transmission by the federated learning clients improves the response time to less than 1 second. Moreover, we demonstrate that federated learning with transfer learning achieves 97% of accuracy under 9,000 samples, which is only 2% of the difference from centralized learning.