Multimodal sensing refers to the concurrent collection and fusion of data from multiple heterogeneous sensor modalities — such as Wi-Fi channel state information (CSI), radio frequency signals, RGB cameras, depth sensors, or inertial measurement units — to jointly characterize human activities, gestures, or spatial context within an environment. By combining complementary signal types, multimodal sensing addresses the inherent limitations of any single modality, such as privacy concerns with vision-based systems or ambiguity in RF-based measurements, thereby improving recognition accuracy, robustness, and generalizability across diverse deployment conditions. Key variants include early fusion, where raw or feature-level data from each modality are combined prior to inference, and late fusion, where modality-specific predictions are integrated at the decision level; multimodal datasets such as OPERAnet also serve a critical role in benchmarking cross-modal generalization and enabling the development of sensing systems that remain reliable when transferred to new environments, users, or hardware configurations.
Source Papers
- A Survey on Wi-Fi Sensing Generalizability: Taxonomy, Techniques, Datasets, and Future Research Prospects ↗ — A Survey on Wi-Fi Sensing Generalizability: Taxonomy, Techni
- Deep Learning-Enhanced Human Sensing with Channel State Information: A Survey ↗ — Deep Learning-Enhanced Human Sensing with Channel State Info
- OPERAnet, a multimodal activity recognition dataset acquired from radio frequency and vision-based sensors ↗ — OPERAnet, a multimodal activity recognition dataset acquired
- RSSI-Assisted CSI-Based Passenger Counting with Multiple Wi-Fi Receivers ↗ — RSSI-Assisted CSI-Based Passenger Counting with Multiple Wi-
- Towards Environment Independent Device Free Human Activity Recognition ↗ — Towards Environment Independent Device Free Human Activity R