Jim Fan推文
触觉是机器人技术中最被忽视的模态
触觉是机器人技术中最被忽视的模态。想象戴着厚烤箱手套变戏法，这就是机器人现在的感觉。磁片卡入、纸杯剥离、USB插入，这些相机都看不见。学习感知必须全栈协同设计。我们开源了名为T-Rex的方法：1. 触觉作为模型的一等公民，混合Transformer异步运行两个时钟，慢速视觉运动专家规划动作，快速触觉专家实时修正。2. 开放数据，最大的触觉数据集，50小时约5500集。3. 训练方法，扩展EgoScale，人类视频预训练加触觉机器人数据。NVIDIA和伯克利合作。
译文
触觉是机器人技术中最被严重忽视的感知模态。想象一下戴着厚厚烤箱手套表演近景魔术——这正是当今机器人若有感知时的真实体验。磁吸件咔嗒归位、纸杯从叠堆中剥离、USB插头摸索着对准接口——这些过程摄像头全都看不见。学会“感知触觉”必须是一项全栈协同设计的工程。我们正在开源一套名为“T-Rex”的系统化方法论：1. 将触觉作为模型的一等公民。我们的混合Transformer异步运行两个时钟：一个慢速视觉运动专家负责规划动作，一个快速触觉专家以每视觉时钟4个“触觉滴答”的高频校正实时优化动作。力的变化比帧到达更快，因此架构也必须如此。2. 开放数据。据我们所知，这是迄今发布的最大规模触觉数据集：一个50小时（约5,500个回合）的高质量、精心同步的机器人操作语料，采集自具有22自由度的SOTA触觉手部硬件。今日已在HuggingFace上线！3. 训练方案：T-Rex扩展了我们此前的工作EgoScale。使用人类第一人称视频进行预训练，加入多样化的触觉机器人操作数据用于中期训练。我们的实验表明，这种方法能极好地弥合无接触预训练与高接触操作之间的鸿沟。像素廉价且无处不在，但在接触发生的瞬间便力不从心。触觉将承担“最后一公里”。下一轮扩展曲线将以触觉小时数来衡量。T-Rex是NVIDIA与Berkeley的精彩合作：🧵
The sense of touch is the most criminally under-explored modality in robotics. Imagine doing sleight of hand wearing thick oven mitts. That's exactly how a robot feels today if it were alive. A magnetic piece snapping into place, a paper cup peeling out of a stack, a USB negotiating its way into the port - all invisible to the camera. Learning how to feel must be a full-stack co-designed effort. We are open-sourcing a principled methodology called "T-Rex": 1. Tactile as first-class citizen of the model. Our mixture-of-transformer runs two clocks asynchronously: a slow visuomotor expert plans the motion, and a fast tactile expert refines it in real time with high-frequency corrections at 4 "touch ticks" per vision tick. Forces change faster than frames arrive, so the architecture had to as well. 2. Open data. The largest tactile dataset ever released to our knowledge: a 50-hour (~5,500 episodes) high-quality, carefully synchronized robot play corpus, collected on SOTA tactile hand hardware with 22 degrees of freedom. Available today on HuggingFace! 3. Training recipe: T-Rex extends our prior work, EgoScale. Human egocentric videos for pretraining, a diverse dose of tactile robot play for mid-training. Our experiments show this bridges contact-free pretraining to contact-rich manipulation remarkably well. Pixels are cheap and everywhere, but they run out of steam at the moment of contact. Tactile will carry the last mile. The next scaling curve will be measured in hours of touch. T-Rex is a great collaboration between NVIDIA and Berkeley: 🧵