Peking University
Pose-Guided Kinematic Representation for Human Behavior Understanding
1Peking University 2ETH Zurich 3Institut Polytechnique de Paris
Existing tokenizers either collapse radar into 2D spectra, losing explicit 3D body structure, or quantize too early and wash out subtle motion variations.
Synchronized 3D pose is used only during training to shape the radar representation. The resulting continuous tokens preserve body configuration and temporal dynamics; at inference, radar alone connects directly to the LLM.
A six-minute qualitative walkthrough—from privacy-friendly radar sensing to fine-grained behavior descriptions and multi-turn reasoning.
Opens on YouTube for consistent HD playback. The video is not hosted by this website.
Large language model agents need to perceive human behavior in physical environments. Millimeter-wave radar provides a privacy-friendly and contactless sensing modality, but its sparse and noisy observations are difficult to align with language. The central bottleneck is efficient, high-fidelity tokenization: existing approaches can discard explicit 3D body structure or coarsen subtle motion through early quantization.
We present mmMind, a radar-language model that uses synchronized 3D pose as training-only supervision. A spatio-temporal radar encoder is pretrained to capture body configuration and motion dynamics. The pose head is then removed, so inference requires radar alone. The learned continuous tokens are aligned with an LLM for behavior captioning, spatio-temporal question answering, and multi-turn interaction.
Training-only pose supervision shapes continuous radar tokens to preserve body structure, global displacement, and local motion.
A large-scale, real-world radar-language benchmark with synchronized pose and fine-grained bilingual annotations.
One radar-language model supports captioning, QA, multi-turn dialogue, and zero-shot generalization to unseen actions.
A three-stage pipeline first learns human kinematics, then aligns radar tokens with language, and finally adapts the LLM for behavior reasoning.
Spatial and temporal Transformers encode sparse radar point clouds. A lightweight pose head supervises joint positions, root motion, local articulation, and skeletal consistency.
The pose head is discarded. A two-layer projector maps continuous 256-dimensional radar tokens into the LLM embedding space using caption supervision.
The projector and LoRA adapters are tuned on captions, single-turn QA, and multi-turn dialogues while the radar encoder stays frozen.
Real-world radar-language data collected across participants, environments, sessions, and commercial devices.
Each five-second clip contains concise and detailed captions in English and Chinese, 13 single-turn QA pairs, and four multi-turn dialogues. Annotations cover body state, limb motion, spatial relations, trajectory, speed, temporal order, and contextual reasoning.
Participant-, environment-, and session-disjoint splits evaluate generalization to unseen people and indoor settings.
If you find mmMind useful, please cite our arXiv preprint.
@misc{zhang2026mmMind,
title = {Teaching Foundation Models to Read mmWave: Pose-Guided Kinematic Representation for Human Behavior Understanding},
author = {Duo Zhang and Zhehui Yin and Zhiyun Yao and Haotong Qin and Xusheng Zhang and Hongliu Yang and Jianyu Sun and Junzhe Wang and Zizhou Fan and Michele Magno and Daqing Zhang},
year = {2026},
eprint = {2608.04127},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2608.04127}
}