选择语言
< 返回主菜单
zhaoxing_fuben.jpg

赵行

上海期智研究院PI(2020年7月-至今)
清华大学助理教授

个人简介

上海期智研究院PI,清华大学交叉信息研究院助理教授。

博士毕业于麻省理工学院MIT,后于谷歌无人车项目Waymo担任研究科学家。其研究领域涵盖多模态学习,自动驾驶和机器人。提出了自动驾驶中一系列框架型的工作,为行业大多数公司所使用或借鉴。曾获智能机器人顶会CoRL2023最佳系统论文提名Top3,ICCP最佳论文奖,福布斯中国30Under30。曾担任ICLR大会联席主席,NeuRIPS/ICCV领域主席。


个人荣誉:

福布斯中国30位30岁以下精英(科学榜,2020年)

Snap Research Fellowship 2019

ICCP最佳论文奖 2015

MIT Rohsenow Fellowship 2013

研究方向

多模态学习:贯通图像、文本、声音、视频等模态信号的多模态生成模型

自动驾驶:下一代以视觉为中心、数据驱动的自动驾驶技术

机器人学:视觉驱动的足式机器人导航和敏捷运动

亮点成果

成果4:视觉为中心的自动驾驶技术

       尽管自动驾驶技术在过去几年有着进展,但是高级别自动驾驶技术却一直难以落地。赵行团队指出了现有自动驾驶技术存在的泛化性问题,在于过分依赖激光雷达和高精度地图,并提出了以视觉为中心的自动驾驶框架。

在这个框架下,课题组发表了多篇代表性论文,改变了行业范式,包括首个视觉Transformer的三维物体检测模型DETR3D、跟踪模型MUTR3D、端到端运动预测模型ViP3D;用视觉神经先验网络实现在线的地图感知Neural Map Prior,代替了以往的手工地图标注方案;首个用于通用障碍物感知的三维占据网格数据集Occ3D等。

       该系列算法成果在多个国际竞赛上拿到冠军,为行业多数头部企业所使用或借鉴。合作企业理想汽车公司在多次产品发布会上提到我们的科研成果带来的价值。至今,理想汽车公司已经将该系列成果部署于超过60万台电动汽车的辅助驾驶系统中,为国产辅助驾驶方案装机量第一,在国际上仅次于Mobileye和Tesla,实现了巨大的产业价值。


2023赵行成果照片1.jpg


2023赵行成果照片2.jpg


2023赵行成果照片2.png

       成果研究论文:

[1] Xuan Xiong, Yicheng Liu, Tianyuan Yuan, Yilun Wang, Yue Wang, Hang Zhao, Neural Map Prior for Autonomous Driving, CVPR 2023 查看PDF

[2] Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, Hang Zhao. Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving. NeurIPS 2023 Dataset Track. 查看PDF


------------------------------------------------------------------------------------------------------------------------------


成果3:机器人跑酷学习

       近两年足式机器人的发展着重在复杂地形的移动能力,但是四足机器人的通过性始终没有超过传统的特种轮式机器人。赵行团队联合斯坦福大学,开发并开源了机器人跑酷学习(Robot Parkour Learning)项目,利用视觉和强化学习实现了四足机器人的高动态移动能力,包括:匍匐前进、跳上高台、跨越沟坎等。在这个项目中,我们从传统轨迹优化算法中寻找灵感,采用软动力学约束的方式成功让机器狗训练出特殊的步态以应对超过自身尺寸的障碍物,并且在微调阶段,让强化学习算法成功应用到了真实的机器狗上。Robot Parkour Learning项目得到的跑酷策略,还可以快速迁移到不同形态的机器狗上。Robot Parkour Learning的发表标志着四足机器人找到了它超过传统移动机器人的应用场景和机器学习算法实现。目前,Robot Parkour Learning项目已经开源了训练代码和强化学习模型,并成功在CoRL 2023会场实地展示和在同行的四足机器人上得到成功应用。Robot Parkour Learning项目在今年的机器人学习会议CoRL 2023上,入围了最佳系统论文奖(Best System Paper Award Finalist, Top3)


图片1.jpg

       

       研究领域:四足机器人的高动态移动                              

       主要完成人:庄子文

       项目网站:https://robot-parkour.github.io/ 

       研究论文:Zhuang, Ziwen, Zipeng Fu, Jianren Wang, Christopher Atkeson, Sören Schwertfeger, Chelsea Finn, and Hang Zhao. ‘Robot Parkour Learning’. In Conference on Robot Learning (CoRL), 2023. 查看PDF


------------------------------------------------------------------------------------------------------------------------------


成果2:自动驾驶行为仿真

       主流的自动驾驶方案依赖于大量的道路测试来衡量自动驾驶的水平,而把测试放入计算机仿真环境是未来规模化自动驾驶的重要路线。此外,仿真器还能被用于算法调试和训练数据生成。自动驾驶行为仿真器的构建面临了许多挑战,需要进行多智能体的意图和轨迹建模,同时需要考虑自车对环境和人的响应。赵行团队开发并且开源了首个基于机器学习的闭环自动驾驶行为仿真器InterSim。InterSim基于大规模真实数据集(Waymo Open Dataset)的车辆行为进行模型训练;在应用时,当轨迹规划器使用不同的策略时,仿真器能给出不同的、且逼真的行为反应。InterSim的发布是自动驾驶领域的重要里程碑,为自动驾驶规划算法提供了评测平台和训练数据。目前InterSim已经吸引了全世界几十个团队使用。


图片2.png


       

       研究领域:自动驾驶

       主要完成人:孙桥、赵行

       项目网站:https://tsinghua-mars-lab.github.io/InterSim/

       研究论文:Qiao Sun, Xin Huang, Brian C Williams, Hang Zhao, InterSim: Interactive Traffic Simulation via Explicit Relation Modeling, IROS 2022 查找PDF


------------------------------------------------------------------------------------------------------------------------------


成果1从射频信号中恢复高质量

       麦克风是人机交互和窃听领域中常见的设备,但在有干扰噪音和隔音材料的场景下,其性能会大幅下降。射频信号不受噪音和光照的影响并且可以穿过许多隔音以及不透明的障碍物。基于射频信号的这种性能,赵行团队提出了Radio2Speech,首个使用毫米波雷达信号来恢复高质量语音的系统。使用射频信号来恢复语音信号的原理是:声音产生于声源的震动,毫米波雷达向声源发射信号,通过对反射的雷达信号进行处理可以得到相应的震动信号,从而恢复原始的音频信号。Radio2Speech在安静环境下可以恢复与麦克风质量相当的语音,而在嘈杂环境和有隔音玻璃的环境下表现远优于传统的麦克风。


图片3.jpg

       

       研究领域:多传感器学习

       主要完成人:赵闰宁、于江涛、赵行

       项目网站:https://zhaorunning.github.io/Radio2Speech/

       研究论文:Running Zhao, Jiangtao Yu, Tingle Li, Hang Zhao*, Edith C.H. Ngai*, Radio2Speech: High Quality Speech Recovery from Radio Frequency Signals, Interspeech 2022 查看PDF

团队成员

论文发表

32. Uncertainty-Aware Decision Transformer for Stochastic Driving Environments, Zenan Li, Fan Nie, Qiao Sun, Fang Da, and Hang Zhao†, https://arxiv.org/abs/2309.16397, CoRL 2024.


31. Humanoid Parkour Learning, Ziwen Zhuang, Shenzhe Yao, Hang Zhao†, https://openreview.net/forum?id=fs7ia3FqUM, CoRL 2024.


30. PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors, Tianyuan Yuan, Yucheng Mao, Jiawei Yang, Yicheng Liu, Yue Wang, Hang Zhao†, https://arxiv.org/abs/2403.09079, ECCV 2024.


29. CVT-Occ: CVT-Occ: Cost Volume Temporal Fusion for 3D Occupancy Prediction, Zhangchen Ye*, Tao Jiang*, Chenfeng Xu, Yiming Li, Hang Zhao†, https://github.com/Tsinghua-MARS-Lab/CVT-Occ, ECCV 2024.


28. Zenan Li*, Fan Nie*, Qiao Sun, Fang Da, Hang Zhao, Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills, ICRA 2024


27. Simian Luo, Chuanhao Yan, Chenxu Hu, Hang Zhao, Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models, Conference on Neural Information Processing Systems (NeurIPS), 2023 查看PDF


26. Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher G Atkeson, Sören Schwertfeger, Chelsea Finn, Hang Zhao, Robot Parkour Learning, International Conference on Robots Learning (CORL), 2023 查看PDF


25. Liangtao Zheng, Yicheng Liu, Yue Wang, Hang Zhao, Cross-dataset Sensor Alignment: Making Visual 3D Object Detector Generalize, International Conference on Robots Learning (CORL), 2023 查看PDF


24. Tong Zhang, Yingdong Hu, Hanchen Cui, Hang Zhao, Yang Gao, A Universal Semantic-Geometric Representation for Robotic Manipulation, International Conference on Robots Learning (CORL), 2023 查看PDF


23. Qiao Sun, Xin Huang, Brian C. Williams, Hang Zhao, P4P: Conflict-Aware Motion Prediction for Planning in Autonomous Driving, International Conference on Intelligent Robots and Systems (IROS), 2023 查看PDF


22. Running Zhao, Jiangtao Yu, Hang Zhao, Edith C.H. Ngai, Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals, Ubicomp/ISWC, 2023 查看PDF


21. Xiaoyu Tian*, Tao Jiang*, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, Hang Zhao, Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving, Conference on Neural Information Processing Systems (NeurIPS), 2023 查看PDF


20. Yicheng Liu, Tianyuan Yuan, Yue Wang, Yilun Wang, Hang Zhao,  VectorMapNet: End-to-end Vectorized HD Map Learning, International Conference on Machine Learning (ICML), 2023查看PDF


19. Xuan Xiong, Yicheng Liu, Tianyuan Yuan, Yilun Wang, Yue Wang, Hang Zhao, Neural Map Prior for Autonomous Driving, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023查看PDF


18. Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, Song Han, SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023查看PDF


17. Junru Gu, Chenxu Hu, Tianyuan Zhang, Xuanyao Chen, Yilun Wang, Yue Wang, Hang Zhao, ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023查看PDF


16. Zitian Tang, Wenjie Ye, Wei-Chiu Ma, Hang Zhao, What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023 查看PDF


15. Renhao Wang, Jiayuan Mao, Joy Hsu, Hang Zhao, Jiajun Wu, Yang Gao, Programmatically Grounded, Compositionally Generalizable Robotic Manipulation, International Conference on Learning Representation(ICLR), 2023 查看PDF


14. Zihui Xue, Zhengqi Gao, Sucheng Ren, Hang Zhao, The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation, International Conference on Learning Representation(ICLR), 2023 查看PDF


13. Qiao Sun, Xin Huang, Brian C Williams, Hang Zhao,InterSim: Interactive Traffic Simulation via Explicit Relation Modeling, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022
 查看PDF


12. Renhao Wang, Hang Zhao, Yang Gao, CYBORGS: Contrastively Bootstrapping Object Representations by Grounding in Segmentation, European Conference on Computer Vision (ECCV), 2022 查看PDF


11. Tingle Li, Yichen Liu, Andrew Owens, Hang Zhao, Learning Visual Styles from Audio-Visual Associations, European Conference on Computer Vision (ECCV), 2022 查看PDF


10. Running Zhao, Jiangtao Yu, Tingle Li, Hang Zhao, Edith C.H. Ngai, Radio2Speech: High Quality Speech Recovery from Radio Frequency Signals, Interspeech 2022 查看PDF


9. Zui Chen, Yansen Jing, Shengcheng Yuan, Yifei Xu, Jian Wu, Hang Zhao, Sound2Synth: Interpreting Sound via FM Synthesizer Parameters Estimation, International Joint Conference on Artificial Intelligence(IJCAI), 2022 查看PDF


8. Sucheng Ren, Zhengqi Gao, Tianyu Hua, Zihui Xue, Yonglong Tian, Shengfeng He, Hang Zhao, Co-advise: Cross Inductive Bias Distillation, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022 查看PDF


7. Jianren Wang, Ziwen Zhuang, Hang Zhao, SEMI: Self-supervised Exploration via Multisensory Incongruity, IEEE International Conference on Robotics and Automation(ICRA), 2022 查看PDF


6. Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, Longbo Huang, What Makes Multi-Modal Learning Better than Single (Provably), Conference and Workshop on Neural Information Processing Systems(NeuRIPS), 2021 查看PDF


5. Chenxu Hu, Qiao Tian, Tingle Li, Yuping Wang, Yuxuan Wang, Hang Zhao, Neural Dubber: Dubbing for Videos According to Scripts, Conference and Workshop on Neural Information Processing Systems(NeuRIPS), 2021 查看PDF


4. Tingle Li, Yichen Liu, Chenxu Hu, Hang Zhao, CVC: Contrastive Learning for Non-parallel Voice Conversion, Interspeech 2021 查看PDF


3. Zihui Xue, Sucheng Ren, Zhengqi Gao, Hang Zhao, Multimodal Knowledge Expansion, IEEE International Conference on Computer Vision(ICCV), 2021 查看PDF


2.  Tianyu Hua, Wenxiao Wang, Zihui Xue, Yue Wang,Sucheng Ren, Hang Zhao, On Feature Decorrelation in Self-Supervised Learning, IEEE International Conference on Computer Vision(ICCV), 2021 查看PDF


1. Jianren Wang, Yujie Lu, Hang Zhao, CLOUD: Contrastive Learning of Unsupervised Dynamics, Conference on Robot Learning(CoRL), 2020 查看PDF