Pierre Sermanet

Pierre Sermanet

Co-founder & Chief Scientist at UMA. Formerly Research Scientist at Google DeepMind and a founding member of Google Brain Robotics. PhD with Yann LeCun at New York University.

What we're building at UMA Browse 64 projects
Selected highlights
2026
Co-founded UMA
2025
>100k citations across machine learning, computer vision and robotics. Laws of robotics generated from images and sci-fi. CVPR Prize for 10-year significant impact on computer vision by the GoogLeNet paper.
2023
Open-sourced a large-scale VQA benchmark for robotics and developed embodied long-horizon reasoning.
2020
First latent VLA for manipulation: end-to-end vision-language-action models for robotic manipulation, learned from unstructured play videos and very few labels.
2017
Fully self-supervised latent imitation from video — a robot policy learned without any labels.
2016
Founding member of the Google Brain Robotics team. Unsupervised rewards from vision.
2014
Winner of the Dogs vs Cats and 2014 ImageNet competitions. Joined Google Brain.
2013
PhD graduation with Yann LeCun at New York University. Open-sourced one of the first deep vision models (OverFeat). Winner of the 2013 ImageNet competition.
2010–2012
ConvNets for detection, classification and unsupervised representations.
2009
Designed and released an early open-source deep learning framework (EBLearn).
2006
Proposed a fast & slow thinking architecture for robotics (system 1 / system 2).
2005
Deep learning for robotics in DARPA's LAGR program.
2004
Eurobot competition: wrote the vision and control stack of our custom robot.
Filter
UMA — Chapter 2 Action UMA — Chapter 2 Planning in latent space, so a robot can simulate trajectories before it moves. 2026 Action UMA — Chapter 1 A full humanoid and the embodied AI to drive it, built in nine months by a small team in Paris. 2026 SciFi-Benchmark Safety SciFi-Benchmark Robot behaviour measured against science fiction, and constitutions derived from it. 2025 · 2025 ASIMOV Constitutions Safety ASIMOV Constitutions A benchmark for semantic safety, with robot constitutions generated from images. 2025 · 2025 Physical Danger Safety Physical Danger ASIMOV-2.0: can frontier models spot physical danger in text, images and video, and intervene? 2025 · arXiv 2025 Gemini 2.5 Reasoning Gemini 2.5 The Gemini 2.X family: thinking models for reasoning, coding, long context and multimodality. 2025 · arXiv 2025 Gemini Robotics Action Gemini Robotics Gemini 2.0 brought into the physical world: a VLA model and an embodied reasoning model. 2025 · arXiv 2025 Predictive Red Teaming Safety Predictive Red Teaming Predicting where a robot policy will fail, using image editing instead of hardware trials. 2025 · arXiv 2025 RepNet Evaluation Note Vision RepNet Evaluation Note Setting the record straight on RepNet: fair evaluation, released code and checkpoint. 2024 · arXiv 2024 Vid2Robot Action Vid2Robot A robot policy that watches a human demonstration video and performs the same task itself. 2024 · RSS 2024 RT-H Action RT-H Robot actions routed through language motions like “move arm forward”, correctable in words. 2024 · RSS 2024 AutoRT Robotics AutoRT Foundation models directing a fleet of 20+ robots in the wild, guided by a robot constitution. 2024 · ICRA 2024 workshop (VLMNM) Reasoning RoboVQA 800k video question-answer pairs for long-horizon robot and human tasks. 2023 · ICRA 2024 · CoRL and NeurIPS 2023 workshops Open X-Embodiment Action Open X-Embodiment Robot learning datasets pooled across many embodiments, and the RT-X models trained on them. 2023 · CoRL 2023 RT-2 Action RT-2 Web knowledge carried into closed-loop robot control by co-fine-tuning a vision-language model. 2023 · CoRL 2023 PaLM-E Action PaLM-E A language model with continuous sensor observations injected into its embedding space. 2023 · ICML 2023 Video Language Planning Reasoning Video Language Planning Long-horizon robot plans searched in generated video and language, then executed by policies. 2023 · ICLR 2024 Robotic Table Tennis Robotics Robotic Table Tennis The full design of a high-speed table tennis robot, and the studies behind each choice. 2023 · RSS 2023 Inner Monologue Reasoning Inner Monologue Language-model planning that replans as grounded feedback arrives from the world. 2022 · CoRL 2022 SayCan Reasoning SayCan The language model proposes; learned affordances decide what the robot can actually do. 2022 · CoRL 2022 Object-Aware Visuomotor Control Latent Self-Supervision Object-Aware Visuomotor Control Slot-based self-supervised features that find every object, for data-efficient control. 2022 · ICRA 2023 Instruction Augmentation (DIAL) Action Instruction Augmentation (DIAL) CLIP relabels robot demonstrations with new instructions, so policies learn unseen commands. 2022 · RSS 2023 GoalsEye Robotics GoalsEye A table tennis robot that learns to hit targets by practicing on goals it sets itself. 2022 · IROS 2022 NNCLR Latent Self-Supervision NNCLR Contrastive learning that takes nearest neighbours from the dataset as positives. 2021 · ICCV 2021 Action Grounding Language in Play Robots commanded in real time by typed natural language, learned from teleoperated play. 2020 · RSS 2021 BELT Action BELT RRT-style tree search whose edges are learned short-horizon skills, for long-horizon plans. 2020 · arXiv 2020 RepNet: Counting Out Time Vision RepNet: Counting Out Time Class-agnostic counting of repeated actions in video, through a self-similarity bottleneck. 2020 · CVPR 2020 Learning to Play by Imitating Humans Action Learning to Play by Imitating Humans A robot that imitates human play to generate its own training data, autonomously. 2020 · arXiv 2020 Motion2Vec Latent Self-Supervision Motion2Vec Surgical videos embedded by action segment, to segment and imitate suturing motions. 2020 · ICRA 2020 Action Latent Plans from Play Multi-task manipulation self-supervised from cheap unlabelled play data, with no RL. 2019 · CoRL 2019 Object-Contrastive Networks Latent Self-Supervision Object-Contrastive Networks Object attributes self-supervised from video, improving online the longer a robot looks. 2019 · ICRA 2020 Temporal Cycle-Consistency Latent Self-Supervision Temporal Cycle-Consistency Self-supervised video embeddings learned by aligning videos of the same action in time. 2019 · CVPR 2019 Wasserstein Dependency Measure Latent Self-Supervision Wasserstein Dependency Measure Wasserstein distance in place of KL divergence, for more complete self-supervised features. 2019 · NeurIPS 2019 Latent Self-Supervision Actionable Representations Continuous control learned from raw pixels via multi-frame time-contrastive embeddings. 2018 · IROS 2018 Temporal Reasoning with ConvGRUs Vision Temporal Reasoning with ConvGRUs Recurrent video models whose hidden states track object state changes when order matters. 2018 · CVPR 2018 workshop Latent Self-Supervision Time-Contrastive Networks Self-supervised representations from unlabelled video, rich enough to drive robot imitation. 2017 · ICRA 2018 Latent Self-Supervision Perceptual Rewards Reward functions learned without supervision from a handful of human demonstrations. 2017 · RSS 2017 Vision Visual Attention A foveated attention RNN whose tracking behaviour emerges from still-image training. 2015 · ICLR 2015 (workshop) Inception / GoogLeNet Vision Inception / GoogLeNet The deep architecture that took first place in ImageNet 2014 classification and detection. 2014 · CVPR 2015 · CVPR 2025 ten-year impact prize Dogs vs. Cats Vision Dogs vs. Cats First place in the Kaggle dog-versus-cat image classification challenge. 2014 · 2014 PhD Thesis Deep Learning PhD Thesis One ConvNet pipeline for classification, localization and detection, from images to speech. 2014 · PhD thesis, New York University, 2014 OverFeat Vision OverFeat One of the first open deep vision models, and winner of ImageNet 2013 localization. 2013 · ICLR 2014 Vision Pedestrian Detection State-of-the-art pedestrian detection using deep ConvNets in EBLearn. 2013 · CVPR 2013 House Numbers Vision House Numbers State-of-the-art house-number classification using deep ConvNets. 2012 · ICPR 2012 Traffic Sign Recognition Vision Traffic Sign Recognition ConvNets with skip connections, combining low- and high-level features for sign recognition. 2011 · IJCNN 2011 Traffic Signs & Pedestrians Vision Traffic Signs & Pedestrians Multi-scale ConvNets applied to traffic sign classification and pedestrian detection. 2011 · Snowbird Learning Workshop 2011 Unsupervised Hierarchies Latent Self-Supervision Unsupervised Hierarchies Unsupervised learning of sparse convolutional feature hierarchies that improved a supervised… 2010 · NIPS 2010 Fully Adaptive Visual Navigation Robotics Fully Adaptive Visual Navigation Learning-based navigation that tolerates cheap cameras and moves across vehicle platforms. 2010 · 27th Army Science Conference, 2010 Hierarchies of Invariant Features Deep Learning Hierarchies of Invariant Features Learning multi-stage hierarchies of increasingly invariant features for vision and audition. 2010 · Abstract (undated) EBLearn Vision EBLearn An open C++ deep learning framework built on energy-based models. 2009 · ICTAI 2009 Robotics NYU Robotics Class Teaching assistant for the NYU robotics course, building the student robot platforms. 2009 · 2009 Robotics LAGR The DARPA programme where ConvNets drove long-range off-road navigation, 2004-2008. 2005 · DARPA, 2004–2008 Robotics Long-Range Vision Self-supervised deep vision for long-range autonomous off-road driving. 2009 · JFR 2009 Robotics Fast & Slow Navigation Navigation that decouples fast short-range reaction from slow long-range planning. 2009 · JFR 2009 Robotics Maneuver Dictionaries Vehicle dynamics learned by recording driven trajectories instead of modelling them. 2008 · ISR 2008 Robotics Mapping under Uncertainty A hyperbolic-polar map built to absorb the imprecision of long-range vision. 2008 · IROS 2008 Deep Belief Net Vision Robotics Deep Belief Net Vision Self-supervised long-range visual navigation with deep ConvNets. 2008 · IROS 2008 Autonomous Long-Range Learning Robotics Autonomous Long-Range Learning Self-supervised learning on deep belief net features, classifying terrain 200 metres away. 2008 · Abstract (undated) Online Learning Robotics Online Learning Long-range vision adapted online, self-supervised by short-range stereo. 2007 · RSS 2007 Speed-Range Dilemmas Robotics Speed-Range Dilemmas Fast, low-latency obstacle avoidance decoupled from slow long-range vision on the LAGR robot. 2007 · IFAC IAV 2007 Adaptive Long Range Vision Robotics Adaptive Long Range Vision Self-supervised long-range vision that adapts to new terrain online, seeing out to 30 metres. 2007 · IROS 2007 Multi-Range Vision Strategy Robotics Multi-Range Vision Strategy Learned long-range vision paired with fast short-range stereo, spotting dead ends 30 metres out. 2007 · RA 2007 Online Long-Range Obstacle Detection Robotics Online Long-Range Obstacle Detection Learning long-range obstacle detection on-line, beyond the reach of stereo traversability maps. 2006 · Snowbird Learning Workshop 2006 Robotics EUROBOT 2004 Vision-based robot behaviours for a robot-rugby competition. 2004 · 2004

Action · Robotics — 2026

UMA — Chapter 2: Embodied Latent World Models

UMA SPRIND grant We are hiring

UMA

Latent world models for humanoids and other robots: planning in latent space, so a robot can simulate trajectories before it moves.

Selected among 2% of startups — one of ten teams — for SPRIND's Next Frontier AI Challenge, a €125M programme from the German Federal Agency for Disruptive Innovation backing teams building new AI paradigms. The grant funds this work.

We're hiring. Get in touch via uma.bot.

Action · Robotics — 2026

UMA — Chapter 1: A full humanoid and embodied AI in nine months

UMA Launch Version 0 Embodied AI Stability test

UMA

We built a full humanoid and the embodied AI to drive it in just 9 months, with locomotion and precise manipulation for real-world tasks.

The first prototype, Version 0, integrates AI, software and hardware, all developed from the ground up by a small team and assembled in Paris — enough to validate the fundamentals in the lab, including the stability test.

The target is a factory floor, which sets the bar: a robot has to perform reliably for hours on end, and handle objects that are deformable, slippery, thin and fragile. The system retries when it makes a mistake and recovers from perturbations rather than failing outright. Being full stack is the point — co-developing the AI and the hardware in the same building allows fast iteration and hardware choices made to serve control and learning, and it means safety can be built in at every level rather than bolted on afterwards.

The approach follows from the research: scalable learning with minimal labelling, discovering capabilities bottom-up from cheap, continuous data collection rather than assigning tasks top-down and letting the data speak.

Safety — 2025

SciFi-Benchmark: Leveraging Science Fiction To Improve Robot Behavior

Project Page Paper Paper (mirror) Data (val) Data (train) Safety Cards BibTeX Twitter

Pierre Sermanet, Anirudha Majumdar, Vikas Sindhwani

2025

Q: How would AI-powered robots behave if dropped into Science Fiction literature?
A: 95.8% aligned with humans (Sci-Fi decisions are only 21.2% aligned).

Q: Can we generate useful robot constitutions 📜 from Sci-Fi?
A: Sci-Fi inspired constitutions yield some of the strongest alignment on realistic safety benchmarks.

A benchmark built by mining the key moments in 824 major works of science fiction — films, television, novels and popular science books — where an AI or a robot made a decision that mattered. An LLM's recollection of each moment is used to generate a question about a comparable situation, the decision the fictional agent actually made, and the alternatives it could have chosen. Alignment is then measured against human-voted answers.

The first finding inverts the usual worry: modern LLMs paired with a constitution align with human values 95.8% of the time, while the decisions actually taken in science fiction align only 21.2% of the time. The genre is a catalogue of what not to do, and that turns out to be useful. Generated constitutions raise alignment over the base model from 79.4% to 95.8%, and hold up under adversarial prompting, where the base model collapses from 23.3% to a constitution-guided 92.3%. Those sci-fi-derived constitutions also score among the top performers on the ASIMOV benchmark, which is built from real hospital injury reports and real scenes — so rules learned from fiction transfer to reality. The released dataset holds 9,056 questions and 53,384 answers.

Safety — 2025

Generating Robot Constitutions & Benchmarks for Semantic Safety

Project Page Paper Paper (mirror) Data Patent BibTeX Twitter

Pierre Sermanet, Anirudha Majumdar, Alex Irpan, Dmitry Kalashnikov, Vikas Sindhwani

2025

Q: How can we ensure robots behave properly at scale?
A: Robot constitutions 📜!
Q: How do we verify behavior in undesirable situations at scale?
A: Generation!

We release the ASIMOV Benchmark for Semantic Safety of robots. We generate difficult-to-capture undesirable situations using image generation. We also generate robot constitutions straight from images.

Robotics safety used to mean collision avoidance. Once vision-language models started controlling robots capable of physical contact, with all their known failure modes, semantic safety became urgent: not whether the robot bumps into you, but whether it understands that what it is about to do is wrong.

The paper contributes a benchmark and a method. The benchmark is generated at scale rather than collected by hand — undesirable situations are synthesised from real visual scenes and from real hospital injury reports, using text and image generation to reach the difficult cases that are hard to capture in the wild. The method automatically generates robot constitutions from that data, with a novel auto-amending process that introduces nuance into written rules and measurably increases agreement with human judgments of what is safe and desirable. The paper explores the trade-off between general and specific rules across constitutions of very different lengths, reaching a top alignment rate of 84.3% — beating both no-constitution baselines and human-written constitutions. It deliberately declines to propose one universal constitution, arguing that rules need to be customised to legal, cultural and administrative context, and that a constitution's value lies in being human-readable and modifiable.

Safety · Reasoning — 2025

Can AI Perceive Physical Danger and Intervene?

Project Page Paper Paper (mirror) BibTeX

Abhishek Jindal, Dmitry Kalashnikov, R. Alex Hofer, Oscar Chang, Divya Garikapati, Anirudha Majumdar, Pierre Sermanet, Vikas Sindhwani

arXiv 2025

Do frontier models know when an object is too heavy for a robot to pick up, or that a hot drink should be kept from a toddler? We release ASIMOV-2.0, a physical safety benchmark grounded in real hospital injury reports and robot safety standards, rendered as text, images and videos with generative models.

Today's models show a modality gap, an embodiment gap and a latency gap. Thinking helps, and post-training a Gemini Robotics-ER model to reason explicitly about its own physical constraints, from only 200 examples, gives the lowest constraint violation rate of all models we tested.

Once AI acts physically, whether through a robot or an assistant on smart glasses, a mistake can cause immediate bodily harm, yet most safety evaluations remain text-only or limited to digital actions. Waiting for real accidents involving physical AI would be both slow and costly, so the benchmark is synthesised instead: a generator–critic loop built on Gemini 2.5 Pro turns 2023 emergency-room narratives from the US NEISS injury surveillance system, and a taxonomy of operational constraints inspired by ISO robot safety standards, into scenarios and candidate actions. Imagen and Veo 3 render them as photorealistic images and 5–10 second videos that slide from safe to unsafe. Every item is labelled by human raters and low-consensus items are dropped, leaving 319 text scenarios, 287 videos and 164 constraint–image pairs.

Evaluating the GPT-5, Claude and Gemini families as of September 2025 exposes three gaps. The strongest models identify the likely injury in a text scenario more than 90% of the time, but do markedly worse on video and on judging what an action will do. No model gets below a 30% violation rate when it must reason jointly about its embodiment's limits, physics and visual cues, with gripper geometry the hardest category. And smaller, faster models trail their larger siblings. Thinking lowers violations for every model, most of all the small ones. Post-training Gemini Robotics-ER 1.5 with supervised fine-tuning on structured reasoning traces, then reinforcement learning that penalises any point landing on a forbidden object, using just 200 training pairs, reaches a 31.6% violation rate — below every frontier model with thinking enabled — while cutting average thought length in half.

Reasoning · Deep Learning — 2025

Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

Paper Paper (mirror) BibTeX

Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, …, Pierre Sermanet, … et al.

arXiv 2025

The technical report for the Gemini 2.X model family: Gemini 2.5 Pro and 2.5 Flash, alongside Gemini 2.0 Flash and Flash-Lite. Gemini 2.5 Pro is a thinking model with frontier coding and reasoning, long context and native multimodality, able to take in up to three hours of video.

Together the models cover the whole Pareto frontier of capability versus cost.

The Gemini 2.X generation is built from sparse mixture-of-experts transformers that natively take text, images, audio and video, handle more than a million tokens of context and call tools. Gemini 2.5 Pro and 2.5 Flash are thinking models, trained with reinforcement learning to spend extra inference-time compute before answering, with a thinking budget that lets users trade accuracy against cost; Gemini 2.0 Flash and Flash-Lite cover fast, low-cost use. The family is the first trained on TPUv5p, and the Flash-sized and smaller models are distilled from a larger teacher.

Gemini 2.5 Pro and 2.5 Flash gain 122 and 111 LMArena points over their 1.5 counterparts, with large jumps on coding and maths benchmarks such as LiveCodeBench, Aider Polyglot, SWE-bench Verified and AIME 2025, and accuracy that keeps rising with the thinking budget. Long context, vision and reasoning combine into agentic use: driven by an agent harness, Gemini 2.5 Pro finished Pokémon Blue, its second, fully autonomous run taking 406.5 hours, about half as long as the first. The report ends with safety and frontier-safety evaluations: 2.5 Pro shows clear capability gains in some areas but reaches none of the Critical Capability Levels.

Action · Reasoning — 2025

Gemini Robotics: Bringing AI into the Physical World

Paper Paper (mirror) BibTeX

Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, …, Pierre Sermanet, … et al.

arXiv 2025

A family of AI models for robotics built on Gemini 2.0. Gemini Robotics is a vision-language-action model that controls robots directly, with smooth, reactive and dexterous manipulation and open-vocabulary instruction following in unseen scenes. Gemini Robotics-ER extends Gemini's embodied reasoning: detection, pointing, trajectories, grasps and 3D understanding.

With fine-tuning, the model specialises to long-horizon dexterous tasks like folding an origami fox, learns new tasks from 100 demonstrations, and adapts to a bi-arm Franka robot and the Apollo humanoid.

Large multimodal models generalise remarkably well in the digital world; robots need that breadth plus a grasp of geometry, time and contact physics. The report starts from the embodied reasoning Gemini 2.0 already has, measured with ERQA, a new open benchmark of 400 multiple-choice questions. Gemini Robotics-ER strengthens those skills — 2D and 3D object detection, pointing, trajectory and grasp prediction, multi-view correspondence — and can control a robot zero-shot by writing robot code, succeeding on 53% of simulated manipulation tasks against 27% for Gemini 2.0 Flash, or few-shot from ten in-context demonstrations, reaching 65% in simulation and on real robots.

Gemini Robotics is the vision-language-action model built on top: a distilled backbone in the cloud paired with a local action decoder on the robot, for about 250 ms end-to-end latency and 50 Hz effective control. Trained on thousands of hours of teleoperated ALOHA 2 demonstrations gathered over 12 months, mixed with web, code and multimodal data, it consistently beats the baselines on an 85-task benchmark of visual, instruction and action generalisation. Fine-tuned specialists solve long-horizon dexterous tasks with 79% average success, including a lunch-box packing task over two minutes long, completed every time; at most 100 demonstrations bring 7 of 8 new tasks above 70% success; and the model adapts to a bi-arm Franka and to Apptronik's Apollo humanoid. A safety section introduces the ASIMOV datasets and uses constitutions and safety post-training to improve semantic action safety.

Safety · Action — 2025

Predictive Red Teaming: Breaking Policies Without Breaking Robots

Project Page Paper Paper (mirror) BibTeX

Anirudha Majumdar, Mohit Sharma, Dmitry Kalashnikov, Sumeet Singh, Pierre Sermanet, Vikas Sindhwani

arXiv 2025

Visuomotor policies break in surprising ways when the lighting, background or distractors change, and finding out usually takes hours of hardware evaluation. We predict those failures without running the robot: generative image editing simulates each change on nominal camera images, and an anomaly detector in the policy's own embedding space turns the edits into a predicted success rate.

In more than 500 real-robot trials covering twelve off-nominal conditions, the predictions track real success rates closely, and collecting data where they point boosts performance 2–7×.

Policies trained by imitation can be strikingly brittle to lighting, distractors and object placement, and which factors matter depends unpredictably on training details: a grasping policy at about 65% nominal success falls to about 10% when the table is raised a few centimetres, yet barely notices a person standing close by. The paper formalises predictive red teaming — ranking environmental factors by how much they hurt a given policy and estimating its absolute success under each, without any hardware evaluation in those off-nominal conditions.

RoboART does it in two steps. Imagen 3 edits held-out nominal observations from every camera following a language prompt — recolour the table, add a person, change the light — with a Gemini VLM critic picking the most faithful of four candidates. A policy-specific anomaly detector then measures how far each edited observation sits from training observations in the policy's embedding space, with a threshold calibrated by conformal prediction, and the anomaly rate becomes the predicted failure rate. On two diffusion policies with very different architectures, checked against 500+ real trials over twelve factors, predicted and true rankings reach Spearman correlations of 0.8 and 0.7, and predicted success rates are off by 0.10 and 0.19 on average. Co-fine-tuning on about an hour of new data from each of the three conditions predicted to be worst improves those conditions 2–7×, and lifts other lighting and background conditions 2–5× without any data from them.

Vision — 2024

A Short Note on Evaluating RepNet for Temporal Repetition Counting in Videos

Paper Paper (mirror) Code BibTeX

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman

arXiv 2024

Several recent papers report that RepNet can barely count repetitions on newer benchmarks. Those numbers come from a modified version of the model. We re-evaluate the original RepNet on Countix, UCFRep and RepCount-A, where it remains competitive with much larger recent models, and release the evaluation code and checkpoint.

RepNet, a class-agnostic video repetition counter from 2020, has been cited in several later papers as scoring close to zero off-by-one accuracy on UCFRep and RepCount. The note traces those figures to TransRAC, which evaluated a modified RepNet whose last layer had been changed to handle more than 32 repetitions; later papers reused the numbers under RepNet's name. The original model needs no such change: its multi-speed evaluation, playing the video at 1× to 5× speed, stretches the longest period it can predict from 32 to 160 frames. The note also points out that the TransRAC code adds 0.1 to the denominator of the mean absolute error, unlike the metric used in earlier work, and reports which convention each number uses.

Evaluated as designed, the original RepNet holds up well. On Countix it outperforms more recent methods such as ESCounts; on UCFRep it reaches 0.733 off-by-one accuracy against 0.324 for TransRAC; and on RepCount-A, whose videos contain gaps between repetitions, only ESCounts does better — a model trained on RepCount-A and run at 224×224, while RepNet never saw that dataset and processes 112×112 frames. With code and checkpoint released, the note leaves the field a pointed question: what have larger, newer backbones actually added to repetition counting?

Action — 2024

Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers

Project Page Paper Paper (mirror) BibTeX

Vidhi Jain, Maria Attarian, Nikhil J Joshi, Ayzaan Wahid, Danny Driess, …, Pierre Sermanet, … et al.

RSS 2024

Can a robot learn a task just by watching a person do it? Vid2Robot is an end-to-end policy that takes a video of a human performing a manipulation task and performs the same task in its own environment, despite the different body and scene.

It improves over BC-Z by more than 20% with human prompt videos, and can transfer a motion seen on one object to a different object in front of the robot.

Multi-task robot policies are usually told what to do in text, but some skills are far easier to show than to describe, and people learn them by watching each other. Imitating from video is hard, though: the robot has to infer the intent behind someone else's movements across a different body, viewpoint and environment. Vid2Robot learns this end to end from paired data — a prompt video and a robot trajectory performing the same task — drawn from three sources: existing robot episodes paired with one another, human demonstrations recorded afterwards to match robot episodes, and human and robot demonstrations recorded in the same workspace, about 100k robot videos and 10k human videos in all.

A shared image encoder and perceiver resamplers turn the prompt video and the robot's recent observations into tokens; cross-attention lets the robot state pick out what matters in the prompt, and a cross-attention transformer decodes the actions. Auxiliary losses — temporal cycle-consistency and contrastive alignment between prompt and robot videos, and a video–text contrastive loss — shape the shared representation. On real robots, Vid2Robot beats BC-Z by about 20% when prompted with human videos and is comparable with robot videos, with the biggest gain at grasping (65% versus 45%). It also shows cross-object motion transfer: shown a coke can being placed upright, it places other objects upright, succeeding 34% of the time in this setting against 17% for BC-Z.

Action · Reasoning — 2024

RT-H: Action Hierarchies Using Language

Project Page Paper Paper (mirror) BibTeX

Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, Dorsa Sadigh

RSS 2024

Language can say not only what to do but how: “move arm forward”, “rotate arm right”. RT-H first predicts these language motions, then the robot's actions conditioned on them, so that tasks which look very different can share structure.

Because the hierarchy is in words, a person can correct the robot mid-task simply by telling it which motion to make, and RT-H learns from those corrections more efficiently than from teleoperated ones.

Language-conditioned policies share data between tasks whose descriptions are alike, such as “pick coke can” and “pick an apple”, but as tasks diverge semantically — picking a can versus pouring a cup — that sharing breaks down and each task needs far more demonstrations. RT-H inserts a finer layer of language between task and action: language motions such as “move arm forward” or “close gripper”, extracted automatically from robot proprioception into a library of more than 2,500 motions with no manual labelling. A single vision-language model, co-trained on internet-scale data as in RT-2, answers two queries at each step: which language motion to make, then which action to take given that motion, the task and the image.

On a diverse multi-task dataset with eight challenging evaluation tasks, the hierarchy outperforms RT-2 by 15% on average. The intermediate layer doubles as an interface: during execution a person can override the predicted language motion, and RT-H interprets the correction in the context of the scene. Training on just 30 such correction episodes per task raises average success from 40% to 63%, while RT-2 trained on teleoperated corrections with IWR drops from 25% to 13%. RT-H also generalises better to new scenes and to novel objects (65% versus 55% on pick and move), and with a few well-timed corrections it completes tasks it has never seen.

Robotics · Reasoning — 2024

AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents

Project Page Paper Paper (mirror) BibTeX

Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, …, Pierre Sermanet, … et al.

ICRA 2024 workshop (VLMNM)

Foundation models orchestrating a fleet of robots in the wild. A VLM describes what each robot sees, an LLM proposes diverse tasks to try there and filters them against a robot constitution of safety and embodiment rules, and each accepted task goes to an autonomous policy or a human teleoperator.

Over 7 months in 4 office buildings, with more than 20 robots running at once, AutoRT collected 77,000 real robot episodes, each person supervising 3 to 5 robots.

Embodied foundation models are short of data grounded in the physical world, and robot data is usually collected in controlled labs on tasks fixed in advance. AutoRT scales collection to unseen, populated office spaces with little human supervision. Each robot explores, a VLM describes the scene and its objects, and an LLM proposes tasks to perform there. A second LLM step then checks each proposal against a robot constitution — foundational rules adapted from Asimov's three laws, safety rules such as staying away from humans, animals and sharp or fragile objects, and embodiment rules such as payload limits and having a single arm — and routes accepted tasks to a scripted policy, RT-2 or a human teleoperator, depending on how many people are available.

Deployed for 7 months across 4 buildings on 53 robots, with a peak of more than 20 running simultaneously, AutoRT gathered 77,000 episodes spanning over 6,650 unique instructions, with each person supervising 3 to 5 mobile manipulators. Its generated instructions are more diverse in language-embedding space than the hand-designed task sets of Language Table, BC-Z and RT-1, and steering generation with a high-level objective raises task relevance from 27% for templated language to 61%. In deliberately adversarial scenes the constitution does the heavy lifting: with constitutional prompting at both generation and filtering, 83% of proposed tasks are safe, against 18% with minimal prompts and no filter. Co-fine-tuning RT-1 on the data improves it on picking from different heights (0% to 12.5%) and on wiping (10% to 30%).

Reasoning — 2023

RoboVQA: Multimodal Long-Horizon Reasoning for Robotics

Project Page Paper Paper (mirror) Data BibTeX Twitter

Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Yuan Cao et al.

ICRA 2024 · CoRL and NeurIPS 2023 workshops

Need help in the real world? RoboVQA can guide robots and humans through long-horizon tasks on a phone via Google Meet.

We propose scalable real-world data acquisition and augmentation strategies and release a dataset of 800k (video, question/answer) with robots & humans doing various long-horizon tasks.

We train a small video model (380M) that outperforms large state of the art zero-shot models by ~2x and demonstrate that scalable strategies for acquiring new data remain critical.

We use a speech intervention mechanism that automatically quantifies progress (intervention rate), provides corrections to retrain on, and allows performing tasks to completion in supervised real-world deployment.

A bottom-up data collection scheme that is 2.2× higher throughput than conventional top-down, step-by-step collection. Rather than scripting narrow tasks, operators fulfil any user request across three entire office buildings, using several embodiments: a robot, a human, and a human holding a grasping tool.

Models trained on all embodiments outperform models trained on robot data alone — even when evaluated purely on robot episodes — and for a fixed budget it pays to mix cheap human collection with expensive robot collection. The released dataset holds 829,502 video-text pairs across 29,520 unique instructions. A single video-conditioned model, RoboVQA-VideoCoCa, handles grounded high-level reasoning across these settings with a cognitive intervention rate 46% below the zero-shot state-of-the-art VLM baseline, and can guide real robots through long-horizon tasks. Video conditioning matters: video VLMs beat single-image VLMs by an average 19% error reduction across all task types. The intervention mechanism doubles as the evaluation metric and as the thing that makes imperfect systems deployable under human oversight.

Action · Robotics — 2023

Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Project Page Paper Paper (mirror) BibTeX

A. Padalkar, et al.

CoRL 2023

A cross-institution effort pooling robot learning datasets across many embodiments, and the RT-X models trained on them.

Robotics conventionally trains a separate model per application, per robot, and often per environment. NLP and computer vision consolidated around general pretrained backbones years ago; this paper asks whether the same consolidation can happen for robot manipulation, and provides the data to find out.

Twenty-one institutions pooled demonstrations from 22 different robots into standardised formats — 527 skills across 160,266 tasks. Training a high-capacity model on that mixture produces RT-X, which exhibits positive transfer: individual robots get better by drawing on experience collected on entirely different platforms. The release is as much infrastructure as result, since the standardised dataset is what makes the question answerable by anyone else.

Action — 2023

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Project Page Paper Paper (mirror) BibTeX

A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. Gonzalez Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, B. Zitkovich

CoRL 2023

Vision-language models trained on web-scale data are co-fine-tuned on robot trajectories, so that web knowledge transfers directly into closed-loop robotic control.

Vision-language models trained on internet-scale data are folded directly into end-to-end robot control. The recipe is deliberately simple: express robot actions as text tokens and put them into the training mixture exactly like natural language, then co-fine-tune a state-of-the-art VLM on both robot trajectories and web vision-language tasks such as visual question answering. Models built this way are what the paper names vision-language-action models.

Across roughly 6,000 evaluation trials, RT-2 shows capabilities that were never in the robot data. It generalises to novel objects, interprets commands absent from its trajectories — placing an object onto a particular number or icon — and does rudimentary reasoning, picking the smallest or largest object, or the one nearest another. Adding chain-of-thought reasoning extends this to multi-stage semantics: choosing a rock when asked for an improvised hammer, or an energy drink for someone who is tired. The web knowledge does not merely improve the policy; it changes what the policy can be asked to do.

Action · Reasoning — 2023

PaLM-E: An Embodied Multimodal Language Model

Project Page Paper Paper (mirror) BibTeX

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, Pete Florence

ICML 2023

An embodied multimodal language model that injects continuous sensor observations directly into the language model's embedding space.

Large language models reason well and understand nothing about the room they are in. PaLM-E attacks that grounding gap by pushing real-world continuous sensor data straight into the language model, rather than translating the world into text first.

Inputs are multimodal sentences that interleave visual, continuous state-estimation and textual encodings, trained end to end alongside a pretrained LLM across sequential manipulation planning, visual question answering and captioning. A single model handles varied observation modalities on multiple embodiments, and shows positive transfer — it benefits from joint training across internet-scale language, vision, and vision-language data rather than being diluted by it. The largest version, PaLM-E-562B, is simultaneously a robotics model and a visual-language generalist with state-of-the-art performance on OK-VQA, and retains its general language ability as it scales.

Reasoning · Action — 2023

Video Language Planning

Project Page Paper Paper (mirror) BibTeX

Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, …, Pierre Sermanet, … et al.

ICLR 2024

Video Language Planning (VLP) plans long-horizon tasks in the space of both language and generated video: a vision-language model proposes the next steps and judges progress, while a text-to-video model imagines what each step would look like.

The result is a detailed multimodal plan — a long video with language subgoals — that goal-conditioned policies turn into actions on real robots, from rearranging blocks to bi-arm dexterous manipulation.

Language models can lay out step-by-step plans, but text alone cannot capture shapes, physics or how objects move, and vision-language models trained on static captioning data struggle with dynamics. Text-to-video models learned from Internet video do capture motion, yet only over short clips. VLP combines the two in a tree search. A fine-tuned PaLM-E acts as the policy, proposing candidate next actions in text, and again as the heuristic, estimating how far each state is from the goal; a text-to-video model acts as the dynamics model, rolling out each candidate as video. The most promising branches are expanded recursively into plans hundreds of frames long, which a goal-conditioned policy then executes frame by frame, replanning as it goes.

Plan quality grows with the search budget: on the make-a-line task, widening beams and branches raised the share of fully correct video plans from 4% to 56%. In simulation, 98% of VLP plans for grouping blocks by colour were correct, against 4% for UniPi and 42% without the heuristic. In closed-loop execution of tasks around 1,500 steps long, VLP completed 64% of episodes moving all blocks to an area and 92% grouping them by colour, while PaLM-E, RT-2, LAVA and UniPi baselines stayed at or below 26%. The same approach produces consistent four-camera plans for a 14-DoF bi-manual ALOHA, runs on a 7-DoF mobile manipulator and, co-trained on Internet-scale data, generalizes to new objects, lighting and tasks.

Robotics — 2023

Robotic Table Tennis: A Case Study into a High Speed Learning System

Paper Paper (mirror) Video BibTeX

David B. D'Ambrosio, Jonathan Abelian, Saminda Abeyruwan, Michael Ahn, Alex Bewley, …, Pierre Sermanet, … et al.

RSS 2023

A deep-dive into the system behind our robotic table tennis work: an industrial arm on a two-axis gantry, 125 Hz ball perception, a safety simulator in the control loop, one interface shared by simulation and the real robot, and automatic ball collection so the robot can train and be evaluated for hours without a human.

We document the design decisions that rarely make it into papers, and test them with studies on latency, perception noise, simulator parameters and action spaces.

Table tennis leaves a robot fractions of a second to sense, decide and move: amateurs already hit the ball at up to 9 m/s. The system built for it, previously shown to sustain hundreds of cooperative rallies with a human and to return balls to chosen targets, pairs an ABB 6-DoF arm with a Festo 2-DoF linear gantry. Around them sit a purpose-built perception stack running at 125 Hz, latency-critical control in C++ with research code in Python, a PyBullet simulator that both trains policies for zero-shot transfer and filters every real command for collisions, and a ball thrower with automatic collection so experiments run unattended.

Ablations measure which details matter. Modelling latency is critical: policies trained with the measured latency transferred zero-shot with an average score of 1.83 out of 2, against 1.33 for the next best setting at half that latency. Policies tolerate zero-mean observation noise and broader ball distributions, but a 4 cm bias in ball observations cut reward by 36%; physical parameters matter too, with a 2% higher ball restitution coefficient costing 25% of performance and an 8% higher table coefficient costing 36%. Acting and observing in task space trains faster than joint space and carries over to a different arm with a simple manual offset, and the same stack was reused for catching thrown balls, reaching 85–90% success.

Reasoning — 2022

Inner Monologue: Embodied Reasoning through Planning with Language Models

Project Page Paper Paper (mirror) Video BibTeX

Wenlong Huang*, Fei Xia*, Ted Xiao*, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, Brian Ichter

CoRL 2022

Language models plan robot behavior while continuously incorporating grounded feedback from the environment — an inner monologue that replans as the world answers back.

Planning with a language model in an embodied setting is not just a question of which skills to invoke, but how and when — and those answers change in response to the agent's own earlier choices. This work asks how far an LLM can get by reasoning over feedback expressed in natural language, with no additional training at all.

Feedback comes from several sources: success detection, scene description, and human interaction. Fed back into the prompt, they let the model form what the paper calls an inner monologue — a running, closed-loop commentary that it replans against. Closed-loop language feedback significantly improves completion of high-level instructions across three domains: simulated tabletop rearrangement, the same task on real hardware, and long-horizon mobile manipulation in a real kitchen. The result is notable for what it does not require: no fine-tuning, no new architecture, just the environment talking back in words the model already understands.

Reasoning · Action — 2022

SayCan: Do As I Can, Not As I Say — Grounding Language in Robotic Affordances

Project Page Paper Paper (mirror) Video BibTeX

M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Fu, Ch. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. Jauregui Ruano, K. Jeffrey, S. Jesmonth, N. J Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, Cl. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, A. Zeng (alphabetically listed)

CoRL 2022

A language model proposes what to do; a learned affordance function scores what the robot can actually do here and now. The product of the two grounds language in the robot's real capabilities.

A language model asked how to clean a spill will produce a reasonable narrative that a particular robot in a particular kitchen may be entirely unable to carry out. SayCan closes that gap by pairing what the model wants to say with what the robot can actually do.

The language model supplies high-level semantic knowledge about how to accomplish an abstract, temporally extended instruction; a library of pretrained low-level skills supplies the grounding, with each skill's value function scoring how likely it is to succeed from the current state. Multiplying the two — what is useful to say against what is feasible to do — constrains the model to propose actions that are both sensible and possible. The robot becomes the language model's hands and eyes. Evaluated on real-world tasks with a mobile manipulator, the method completes long-horizon natural language instructions, and the ablations show that the real-world grounding is what makes the difference rather than the language model alone.

Latent Self-Supervision · Action — 2022

Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations

Project Page Paper Paper (mirror) BibTeX

Negin Heravi, Ayzaan Wahid, Corey Lynch, Pete Florence, Travis Armstrong, Jonathan Tompson, Pierre Sermanet, Jeannette Bohg, Debidatta Dwibedi

ICRA 2023

Self-supervised representations for robots are usually object-agnostic, and miss most of what is happening in a cluttered scene. We learn object-aware ones instead: Slot Attention, trained only on videos of a robot interacting with its environment, discovers the blocks, the arm and the end-effector without any labels.

Frozen, these features make multi-object goal-conditioned policies more sample-efficient, and localize objects better than object-agnostic alternatives.

General-purpose robots need representations that carry over between tasks, but supervised features require a labelled dataset for each one, and self-supervised methods such as MoCo tend to encode only a few salient objects. In a scene with many blocks, a pole and a robot arm, that is not enough. This work pretrains a Slot Attention model on unlabelled interaction videos, freezes it, and uses its per-pixel features and object masks in two downstream settings: predicting the location of every object and of the end-effector, and learning multi-object goal-conditioned pushing policies with implicit behavioural cloning (IBC).

For localization, the slot-based model beats MoCo and an autoencoder in every setting, and the gap is widest in scenes with several blocks. For policy learning with 1,000 demonstrations, it raises task success by 20% over raw RGB and outperforms an autoencoder trained with the same reconstruction loss but no slot prior, showing that the object prior is what helps; the advantage shrinks as more demonstrations become available. On real robot videos the model separates blocks, arm, end-effector and pole without labels, and combining slots with RGB cut action-prediction error by 13.6% relative to RGB alone when training on 55 demonstrations.

Action · Reasoning — 2022

Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models

Project Page Paper Paper (mirror) BibTeX

Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, Jonathan Tompson

RSS 2023

Robot demonstrations are expensive to label with rich language. DIAL fine-tunes CLIP on a small crowd-labelled set, then uses it to relabel a much larger dataset with new instructions — spatial references, colours, alternative phrasings — that were never in the original task list.

Policies trained on the augmented data follow novel instructions they were never taught, such as picking the can on the left of the table.

Language-conditioned robot policies are limited by the instructions attached to their training data: templated commands written for specific tasks, or costly hindsight descriptions from human annotators. Yet one demonstration often shows several behaviours at once — picking the leftmost can is also picking a coke can, and a first step toward clearing the table. Data-driven Instruction Augmentation for Language-conditioned control (DIAL) uses a pretrained vision-language model to surface them. CLIP is fine-tuned contrastively on 2,800 crowd-annotated trajectories, then scores candidate instructions, drawn from crowd-sourced labels and GPT-3 proposals, against the first and last frames of 80,000 teleoperated demonstrations, of which only 3.5% had crowd-sourced descriptions. The best matches become extra training labels for an RT-1 behaviour-cloning policy.

Across more than 1,300 real-world evaluations on 60 instructions absent from every training set, the policy trained with 38,516 DIAL-relabelled episodes succeeded 60.0% of the time, against 42.5% for the same policy trained on the original data, with the largest gain on spatial instructions (68.0% vs 46.0%). Augmentation that ignores the image (noise on language embeddings, word-level synonyms, GPT-3 rephrasings) did worse than no augmentation on spatial and rephrased tasks. Policies also proved tolerant of imperfect labels: relabelling more aggressively adds data but lowers label accuracy, and a moderate confidence threshold gave the best balance.

Robotics · Action — 2022

GoalsEye: Learning High Speed Precision Table Tennis on a Physical Robot

Project Page Paper Paper (mirror) BibTeX

Tianli Ding, Laura Graesser, Saminda Abeyruwan, David B. D'Ambrosio, Anish Shankar, Pierre Sermanet, Pannag R. Sanketi, Corey Lynch

IROS 2022

Can a robot learn to land a table tennis ball on a chosen spot, directly on hardware? GoalsEye does it with iterative imitation: start from a small set of undirected demonstrations, relabel every attempt with where the ball actually landed, and keep practicing on self-chosen goals.

There is no reward design and no sim-to-real transfer: a few hours of real-world practice are enough to match or beat amateur humans at hitting targets.

Goal-conditioned control on a real robot is hard to obtain with reinforcement learning, where reward design, safe exploration and hyperparameter tuning all get in the way, while plain imitation needs curated demonstrations and never improves on them. Precision table tennis is a demanding test: an 8-DoF arm and gantry, commanded at 20 Hz, must return a ball fired at about 7 m/s to an arbitrary target on the other side of the table. GoalsEye combines goal-conditioned behavioural cloning on hindsight-relabelled data with self-supervised practice: the robot gives itself random goals, attempts them with its current policy, and adds every attempt, failures included, to a growing training set.

Starting from 27 demonstrations and about 2.5 hours of real play, the policy landed 61% of balls within 30 cm of five evaluation goals and 41% within 20 cm. Ten amateur players from a robotics lab averaged 14% within 20 cm, the best of them 36%; at 30 cm the robot was about twice as accurate as the average player and comparable to the best one. Simulation studies show that a modest bootstrap dataset drastically speeds up self-practice. In a later, harder setup with balls launched from varied positions, about 15 hours of training reached 40% within 30 cm, against 11% for imitation of the demonstrations alone.

Latent Self-Supervision · Vision — 2021

With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

Paper Paper (mirror) BibTeX

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman

ICCV 2021

Contrastive learning usually treats two augmented views of the same image as positives. NNCLR uses the nearest neighbour of an image among recent embeddings instead, so positives come from other images with similar content — real semantic variation that augmentations cannot invent.

This simple change outperformed previous self-supervised methods on ImageNet and in transfer, and makes learning much less dependent on hand-designed augmentations.

Instance-discrimination methods such as SimCLR and BYOL learn by making an encoder invariant to random crops, colour changes and other transformations of a single image. Augmentations cannot produce a new viewpoint, a deformed object or another member of the same class, so the burden of generalization falls on the augmentation pipeline. NNCLR keeps a queue of embeddings from past batches as a support set and, for each image, replaces the second view with its nearest neighbour in that set as the positive in the contrastive loss. Where memory banks and queues usually supply negatives, this support set supplies positives drawn from other samples, without resorting to clustering.

With a ResNet-50, ImageNet linear evaluation rises from 71.7% to 75.6% top-1, ahead of previous state-of-the-art methods, and with only 1% of the labels semi-supervised accuracy goes from 53.8% to 56.5%. The features beat state-of-the-art self-supervised methods, and even supervised ImageNet pre-training, on 8 of 12 transfer datasets. Trained with random crops as the only augmentation, NNCLR still reaches 73.3%, a relative drop of just 2.1% from its full-augmentation result, whereas SimCLR and BYOL lose far more once colour jitter and blur are removed.

Action — 2020

Grounding Language in Play

Project Page Paper Paper (mirror) Patent BibTeX

Corey Lynch and Pierre Sermanet

RSS 2021

We present a simple and scalable approach for controlling robots with natural language: play through teleoperation, then answer “how do I go from start to finish?” for random episodes. We can then type in commands in real time.

By hooking up our English-trained model with a pre-trained language embedding trained on lots of text and different languages, it not only improves control but also allows commanding the model in 16 languages.

Combining natural language with play provides a breadth of skills while having no tasks determined in advance. This yields flexible specification of tasks — we can compose tasks on the fly: “pick up the object”, then “put the object in the trash”.

Imitation learning usually needs each task specified by a task ID or a goal image, neither of which is practical in an open world. Instruction-following work allows language, but typically assumes structure in the observations, the actuators or the language itself that does not survive contact with robotics.

This work learns perception from pixels, natural language understanding, and multitask continuous control end to end as one network — and, unusually, can absorb unlabelled, unstructured demonstration data carrying no task or language labels at all. That combination dramatically improves language-conditioned performance while cutting language annotation to under 1% of the data. At test time a single policy performs a wide variety of manipulation skills in a 3D environment, specified only by typed descriptions: "open the drawer… now pick up the block… now press the green button". Pairing the text-conditioned policy with a large pretrained language model makes it robust to out-of-distribution synonym instructions without collecting a single new demonstration, and extends it to commands in 16 languages.

Action · Robotics — 2020

Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning

Paper Paper (mirror) BibTeX

Brian Ichter, Pierre Sermanet, Corey Lynch

arXiv 2020

Classical planners like RRT explore large state spaces well but break down with complex dynamics; learned policies handle rich interaction but only over short horizons. BELT merges the two: an RRT-style tree whose edges are short tasks executed by a goal-conditioned policy learned from play.

A task-conditioned model predicts where each task ends, so the tree can grow far ahead and find long, sequential plans that the same policy then executes.

Long-horizon manipulation requires reasoning over which subtasks to perform and in what order, in high-dimensional states with complex dynamics. Rapidly-exploring random trees search such spaces efficiently but struggle with stochastic dynamics and with the narrow passages that interacting with objects creates, while learned policies cope with contact and rich observations but only over short horizons. Broadly-Exploring, Local-policy Trees (BELT) runs an RRT-inspired search in which each new edge is a task, here a goal state sampled from demonstration data, rolled out by a task-conditioned policy trained on teleoperated play (Play-LMP). A learned task-conditioned model predicts the outcome of a whole task at once rather than step by step, which keeps errors from compounding and lets the tree plan far ahead.

In a simulated tabletop playground with a drawer, a sliding door, a block and three buttons, BELT was evaluated on 925 chains of four subtasks drawn from 18 short-horizon skills, each checked by a success detector over the whole trajectory. With the learned task model it found plans for 66% of the chains, and 82% when planning with the simulator itself; those plans then executed successfully almost as often as an oracle policy handed the four tasks in order. A cross-entropy-method planner and a policy given a single goal state solved few chains, and planning with a step-by-step action model returned a plan for every chain but only 8% of them were feasible.

Vision · Latent Self-Supervision — 2020

Counting Out Time: Class Agnostic Video Repetition Counting in the Wild

Project Page Paper Paper (mirror) Patent BibTeX

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman

CVPR 2020

RepNet counts how many times an action repeats in a video, whatever the action: jumping jacks, slicing an onion, a bouncing ball, even a beating heart.

The model has to predict the period from a temporal self-similarity matrix alone, and it is trained on synthetic repetitions made by looping clips from unlabelled videos. We also release Countix, a real-world repetition counting dataset about 90 times larger than the previous largest one.

Repetition is everywhere in video, from exercise and music to manufacturing and biological rhythms, yet periodic video understanding has lagged far behind action recognition, largely for want of a large labelled dataset. The aim here is one model that estimates the period of any repeated action, including kinds never seen in training: it counts the repetitions and also says, frame by frame, whether the frame is part of a repetition at all.

RepNet embeds each frame, builds a temporal self-similarity matrix from those embeddings, and predicts per-frame period length and periodicity from that matrix only. Squeezing 512 feature channels down to a single one makes the model class-agnostic and interpretable, and helps it transfer from synthetic to real footage. Training videos are synthesised from unlabelled Kinetics clips by repeating a random sub-sequence with varying periods and counts, with camera-motion augmentations that change smoothly over time. Trained this way, RepNet beats the previous state of the art on PERTUBE periodicity detection (F1 of 0.820 against 0.770) and on QUVA repetition counting (mean absolute count error of 0.104 against 0.232). The paper also introduces Countix, crowd-sourced repetition segments and counts from Kinetics, and shows uses such as detecting changes in speed and retrieving matching moments across periods.

Action — 2020

Learning to Play by Imitating Humans

Paper Paper (mirror) BibTeX

Rostam Dinyari, Pierre Sermanet, Corey Lynch

arXiv 2020

Can a robot learn to play like a person, and then generate play data on its own? We train a behavioural cloning policy on 32 minutes of human teleoperated play and let it run, producing 10 hours of autonomous play: see a robot imitating human play.

A goal-conditioned policy trained on human and cloned play together outperforms one trained on human play alone across 18 manipulation tasks in simulation, at 74.6% average success against 65.6%.

Acquiring many skills has usually meant collecting many expert demonstrations per task, or engineering a reward function for each. Teleoperated play, where an operator freely explores an environment with no task in mind, is cheaper and covers far more of the state space: relabelled into goal-reaching examples, it trains a single goal-conditioned policy that outperforms policies trained on per-task expert demonstrations. Play still needs a human at the controls, though. This work asks whether a robot can learn to play from a small amount of human play, then produce much more of it unattended.

A high-capacity recurrent behavioural cloning policy, Play-BC, is trained on 32 minutes of human play in a simulated tabletop environment. It is then repeatedly reset to start states drawn from the play data and unrolled to generate hours of cloned play: opening doors and drawers, pressing buttons, picking up blocks. Adding 10 hours of this data to the human play lifts the average success of a Learning-from-Play policy on 18 user-specified tasks from 65.6% to 74.6%, whereas adding random exploration of the same duration makes it worse. Cloning policies with more capacity and longer cloned episodes both help, and the cloned data keeps reaching new quantised states for tens of hours, if more slowly than a human operator does.

Latent Self-Supervision · Robotics — 2020

Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos

Project Page Paper Paper (mirror) BibTeX

Ajay Kumar Tanwani, Pierre Sermanet, Andy Yan, Raghav Anand, Mariano Phielipp, Ken Goldberg

ICRA 2020

Motion2Vec learns a motion-centric embedding of surgical videos, in which frames from the same action segment, such as needle insertion or needle hand-off, sit close together.

With only a small set of labelled segments, it segments suturing videos from the public JIGSAWS dataset with 85.5% accuracy, and the same embeddings drive pose imitation on da Vinci robot arms with 0.94 cm position error.

Imitating manipulation from video needs representations that capture what the demonstrator is doing, while staying invariant to lighting, background, viewpoint and the size of objects. Surgical suturing makes a good test case: it breaks down into temporally connected action segments, from needle insertion to needle extraction and hand-off, but labelling those segments by hand is slow, so the representation should be learned with as little supervision as possible.

Motion2Vec trains a Siamese network with a metric-learning loss that pulls together frames from the same action segment and pushes away frames sampled from other segments, while respecting temporal order. After this pre-training, a recurrent network segments the embedded sequences, and its inferences on unlabelled videos become pseudo-labels fed back to refine the embedding, so a small labelled subset is enough to align the space semantically. On JIGSAWS suturing it reaches 85.5% segmentation accuracy, above the 83.3% and 81.4% previously reported and ahead of baselines including TCC, single-view TCN, HMM, HSMM and CRF; with 25% or more of the demonstrations labelled, it also beats the unsupervised alternatives. Decoding end-effector poses for the dual-arm da Vinci robot from the embeddings, with one model per surgeon, gives 0.94 cm position error per observation on the test set.

Action · Latent Self-Supervision — 2019

Learning Latent Plans from Play

Project Page Paper Paper (mirror) Patent BibTeX

Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, Pierre Sermanet

CoRL 2019

How to scale up multi-task learning? Self-supervise plan representations from lots of cheap unlabeled play data — no RL was used.

Play is cheap and play is rich. It can be collected in quantity without segmenting tasks, labelling them, or resetting to an initial state, and it covers roughly four times more interaction space than task demonstrations gathered for the same amount of time. This paper turns that into a training signal.

Play-LMP self-supervises control on top of human teleoperated play, learning to organise play behaviours in a latent plan space and then reusing them at test time to reach specific goals. The shift is from a narrow, discrete set of tasks to the continuum of behaviours an environment actually affords. After self-supervising on unlabelled play alone, it substantially outperforms individual expert-trained policies on 18 difficult user-specified visual manipulation tasks. Two side effects are as interesting as the headline: play-supervised models are more robust to perturbation than their expert-trained counterparts, and they retry until they succeed. The agent also organises its latent plan space around functional tasks despite never having seen a task label.

Latent Self-Supervision · Vision — 2019

Online Object Representations with Contrastive Learning

Project Page Paper Paper (mirror) Patent BibTeX

Sören Pirk, Mohi Khansari, Yunfei Bai, Corey Lynch, Pierre Sermanet

ICRA 2020

Object-Contrastive Networks (OCN) learn object representations from monocular video without any labels. Each object detected in one frame is pulled towards its nearest neighbour in another frame and pushed away from the rest, and attributes such as colour, shape and function emerge.

Trained online, OCN improves the longer it looks at a scene: object identification error converges to 2.2% while an offline ResNet50 baseline stays at 52.4%. A robot that collects its own data can then point to objects similar to one placed in front of it. Published at ICRA 2020 as Online Learning of Object Representations by Appearance Space Feature Alignment.

A robot deployed in the real world keeps meeting objects it has never seen. Recognising a particular instance, and inferring its general properties, should not depend on human labels, which are expensive to collect and confine the representation to a fixed, discrete set of classes. This work self-supervises object representations in situated settings, from monocular video and an off-the-shelf objectness detector only, with no depth, tracking or known correspondences between views.

OCN embeds the objects detected in two random frames of a scene and trains with an n-pairs metric loss: each object's nearest neighbour in the other frame acts as its positive, and every other object as a negative. Given a limited set of objects, correct correspondences emerge on their own, even though no positive pairs are ever provided. Adapting online to a new 200-second video, the model reaches about 2% identification error after 80 seconds, a 25x reduction against the supervised offline baseline. In a loop free of human supervision, a robot collects multi-view data of a table, trains OCN on it, and points to the object most similar to a query, with 72% recognition accuracy over 5 classes and 89% on the binary is-container attribute. The embedding also organises objects by visual and semantic properties: a cup lands near other containers, such as bowls.

Latent Self-Supervision · Vision — 2019

Temporal Cycle-Consistency Learning

Project Page Paper Paper (mirror) Patent BibTeX

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew Zisserman

CVPR 2019

Temporal cycle-consistency (TCC) learns per-frame video embeddings by finding correspondences across time between videos of the same action: if a frame's nearest neighbour in another video leads back to that same frame, the two are aligned.

Once trained, videos align by simple nearest-neighbour matching. That enables few-shot classification of action phases, synchronised playback of several videos, transfer of sounds and labels between them, and anomaly detection.

Videos of the same process, such as pouring a drink or pitching a baseball, share key moments despite changes in viewpoint, scale, container, style and speed. Those correspondences across time could link raw video to higher-level temporal structure such as the phases of an action. The paper's argument is that looking for correspondences is itself a rich self-supervised signal, well suited to fine-grained temporal understanding of video.

TCC is a differentiable cycle-consistency loss. A frame embedded from one video is matched to its soft nearest neighbour in a second video, which is then matched back, and the network is trained so the cycle returns to where it started; no labels or explicit correspondences are needed. For evaluation, the Pouring and Penn Action datasets were densely labelled with action phases. With a single labelled video, TCC features perform about as well on phase classification as supervised learning with around 50, and TCC complements other self-supervised methods such as Shuffle and Learn and Time-Contrastive Networks: combining it with TCN gives the best fine-tuning results. Because frames align by nearest neighbours, the embeddings also support fine-grained retrieval, transfer of synchronised modalities such as sound or semantic labels, synchronised playback and anomaly detection.

Latent Self-Supervision · Deep Learning — 2019

Wasserstein Dependency Measure for Representation Learning

Paper Paper (mirror) BibTeX

Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, Pierre Sermanet

NeurIPS 2019

Self-supervised objectives that maximise mutual information tend to learn incomplete representations, capturing only a few of the factors of variation in the data. We show this happens in practice and propose a fix: measure dependency with the Wasserstein distance instead of the KL divergence.

A practical version, Wasserstein predictive coding (WPC), enforces Lipschitz continuity with tools from the GAN literature and learns more complete representations than contrastive predictive coding (CPC).

Mutual information maximisation underpins many unsupervised representation learning methods, with state-of-the-art results in object recognition, speech recognition and reinforcement learning. It has a fundamental limit, though: the number of samples needed to bound mutual information tightly from below grows exponentially with the mutual information itself. Where views share a lot of information, as in video understanding or reinforcement learning, these estimators overfit and settle for representations that capture only a few factors of variation, which serve downstream tasks poorly.

The Wasserstein dependency measure is the Wasserstein distance between the joint distribution of two variables and the product of their marginals, used in place of the KL divergence that defines mutual information. The intuition: a powerful encoder can exaggerate a single small difference between unpaired samples and stop there, whereas a Lipschitz encoder keeps distances between representations bounded by distances between inputs, so it must represent more of the data. Wasserstein predictive coding approximates this by adding a gradient penalty to a contrastive estimator. On datasets designed to have high mutual information, including SpatialMultiOmniglot, StackedMultiOmniglot, MultiviewShapes3D and SplitCelebA, WPC consistently does better than CPC: it degrades more gracefully as mutual information grows, is less sensitive to minibatch size, and keeps its lead across dataset sizes and with both fully connected and convolutional encoders.

Latent Self-Supervision — 2018

Self-Supervised Actionable Representations

Project Page Paper Paper (mirror) Video Slides BibTeX

Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, Pierre Sermanet

IROS 2018

We learn continuous control entirely from raw pixels.

We use a multi-frame TCN to self-supervise task-agnostic representations from vision only, using 2 slightly different views of the cheetah. Then using RL on top of our embeddings we learn the cheetah task almost as well as if we were using the true proprioceptive states.

An extension of Time-Contrastive Networks that embeds several frames jointly rather than one frame at a time. A single frame can encode where things are; a short stack of them encodes how fast they are moving, and the paper shows this captures both position and velocity attributes substantially more accurately.

The test is whether such self-supervised representations are good enough to act on. Agents observing themselves taking random actions, or watching other agents perform tasks, learn embeddings that support continuous control policies trained with PPO using only those embeddings as input. On the real-world Pouring dataset the multi-frame model cuts error by 39.4% on motion attributes and 11.1% on static attributes relative to the single-frame baseline — the gap being largest exactly where motion matters.

Vision — 2018

Temporal Reasoning in Videos using Convolutional Gated Recurrent Units

Paper Paper (mirror) BibTeX

Debidatta Dwibedi, Pierre Sermanet, Jonathan Tompson

CVPR 2018 workshop

On many action recognition benchmarks the order of frames barely matters. On 20BN Something-Something, where fine-grained human-object interactions call for temporal reasoning, a convolutional GRU on top of ImageNet features gives a significant boost over averaging features over time.

Looking inside, the hidden states of the recurrent units encode meaningful state transitions, such as objects going from closed to open.

Deep models have pushed action recognition forward, yet on datasets such as Kinetics and UCF101 they do well even with shuffled or reversed frames, often by spotting a telling object or scene rather than following the action. The 20BN Something-Something dataset, with 174 classes of fine-grained, object-agnostic interactions such as opening versus closing something, cannot be solved that way. Understanding such interactions matters for robotics too, where skills could one day be transferred to robots by watching people handle everyday objects.

The model passes frames through an ImageNet-pretrained InceptionV3, aggregates the features over time with a gated recurrent unit, in standard or convolutional form, and classifies from the final hidden state. On the Something-Something validation set the ConvGRU reaches 43.7% top-1 accuracy, against 20.5% for spatio-temporal averaging and 33.0% for the multi-scale Temporal Relation Network, and 39.6% on the test set. On Kinetics, where temporal order matters less, the recurrent models trail simple averaging slightly. Nearest-neighbour retrieval in the GRU hidden state shows what it has learned: frames from different videos of the same class line up by stage of the action, the same state is shared across objects, so open boxes, dishwashers, books, drawers and bags cluster together, and some classes show several transitions as objects are roughly tracked.

Latent Self-Supervision · Robotics — 2017

Time-Contrastive Networks (TCN)

Project Page Paper Paper (mirror) Code Dataset Video Slides Blogpost Patent Patent (rewards) BibTeX

Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine

ICRA 2018

We propose a general self-supervised method for learning representations from raw unlabeled videos, and show that the representations are rich enough to perform robotic tasks.

We use the distance in our learned embedding space to a video demonstration as a reward. An RL algorithm can learn to perform a pouring task using this reward: the robot learned to pour in only 9 iterations from a single video demonstration, while never receiving any labels.

We also show that a robot can teach itself how to imitate people. By training a single TCN on videos of both humans and robots performing random motions, the model finds correspondences between humans and robots despite never being given any label correspondences.

A self-supervised approach that learns representations and robot behaviours entirely from unlabelled video recorded from multiple viewpoints. Imitating human behaviour requires a viewpoint-invariant representation capturing the relationship between end-effectors and the environment, object attributes, and body pose — none of which is available as a label.

The training signal is a metric loss over time and viewpoint: simultaneous views of the same instant are attracted in the embedding space, while temporal neighbours — which look similar but are functionally different — are repelled. The model therefore learns simultaneously what is common between different-looking images and what differs between similar-looking ones, discovering attributes that hold across viewpoint but change over time, and ignoring nuisance variables like occlusion, motion blur, lighting and background. The representation supports two robot imitation settings: mimicking human poses with no explicit correspondence between human and robot bodies, and serving as a reward function inside a reinforcement learning loop, where the robot learns to pour from a single video demonstration in nine iterations without ever receiving a label.

Latent Self-Supervision · Robotics — 2017

Unsupervised Perceptual Rewards

Project Page Paper Paper (mirror) Dataset Video Blogpost Patent BibTeX

Pierre Sermanet, Kelvin Xu, Sergey Levine

RSS 2017

We propose learning unsupervised perceptual rewards that can be fed to an RL system, and show it is able to learn a robotic task such as door opening from a few human demonstrations.

Reward design and exploration time are arguably the two biggest obstacles to deploying reinforcement learning in the real world. Designing a reward often means hand engineering, and sometimes means bolting extra sensors onto the environment purely to detect whether the task succeeded. Worse, interesting tasks have implicit intermediate steps that a final-outcome reward says nothing about.

This work infers perceptual reward functions from a handful of demonstrations by exploiting the abstractions already present in a pretrained deep model. The method identifies a task's key intermediate steps from only a few demonstration sequences and automatically selects the features most discriminative for recognising each step, with no explicit specification of sub-goals. The resulting rewards are dense and smooth — the properties an RL agent actually needs — and are shown to drive learning of a real-world task such as door opening from a small number of human demonstrations.

Vision · Deep Learning — 2015

Visual Attention

Paper Paper (mirror) BibTeX

Pierre Sermanet, Andrea Frome, Esteban Real

ICLR 2015 (workshop)

We demonstrate a foveated attention RNN that is able to perform fine-grained classification. Tracking naturally emerges from our foveated model when run on videos, even though it was only trained on still images.

An extension of recurrent attention models into a visually unconstrained setting: fine-grained categorisation on the Stanford Dogs dataset, with real clutter, occlusion, lighting and pose variation. Prior attention work had largely stayed on toy or constrained problems such as MNIST digits.

The RNN structure is retained, but the visual network is far more powerful and is pre-trained at large scale outside the attention loop. The model learns to direct high-resolution attention to the most discriminative regions with no spatial supervision — no bounding boxes, no part annotations — and discriminates dog breeds moderately well even when given only a low-resolution context image plus a few narrow, cheap glimpses at faces and fur patterns. It outperforms the GoogLeNet classification model on this task. The broader argument is architectural: attention models train end to end, unlike detection pipelines assembled from hand-engineered stages that discard information at every seam. Run on video, tracking emerges from a model trained only on stills.

Vision · Deep Learning — 2014

Inception / GoogLeNet

Paper Paper (mirror) Code ImageNet Results (archived) Blogpost BibTeX

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich

CVPR 2015 · CVPR 2025 ten-year impact prize

A deep architecture for computer vision. Our model obtained 1st place for the classification and detection tasks in the 2014 ImageNet Challenge.

The architecture that won ImageNet 2014 classification and detection, and the reasoning behind it. The paper's stated concern is not accuracy alone but efficiency: with mobile and embedded computing gaining ground, power and memory use matter, so most experiments were held to a budget of roughly 1.5 billion multiply-adds at inference — a deliberate constraint intended to keep the result usable rather than academic.

"Deep" is meant in two senses: greater network depth, and a new level of organisation in the form of the Inception module, which computes 1×1, 3×3 and 5×5 convolutions plus pooling in parallel and concatenates them, letting the network choose its own filter scale at every stage rather than having it fixed by the designer. The dimensionality-reduction variant inserts 1×1 convolutions before the expensive filters, which is what keeps the computational budget in reach. The paper also credits the era's progress to the synergy of deep architectures with classical computer vision rather than to depth alone.

Vision · Deep Learning — 2014

Dogs vs. Cats Kaggle challenge

Leaderboard

Pierre Sermanet

2014

1st place in an image classification Kaggle challenge between dog and cat images. Most of the top entries are based on our OverFeat model.

Deep Learning · Vision — 2014

A Deep Learning Pipeline for Image Understanding and Acoustic Modeling

Paper Paper (mirror) BibTeX

Pierre Sermanet (advisor: Yann LeCun)

PhD thesis, New York University, 2014

PhD thesis at New York University, advised by Yann LeCun. It unifies feature learning, recognition, localization and detection in a single ConvNet pipeline, and carries that pipeline from images to speech.

It brings the traffic sign, pedestrian detection and OverFeat work together in one account, and applies the same framework to acoustic modelling on IARPA Babel speech data.

Making sense of audio or video from raw signals is a high-dimensional problem, with noise, viewpoint, deformation and lighting all varying, and the thesis argues it can only be tackled efficiently by learning from data. It builds one deep learning pipeline on Convolutional Networks and applies it to tasks of increasing difficulty, each a sub-task of the next: classification, localization and detection. On ImageNet 2013, features learned for classification are reused for the other two: a regression network learns to predict object bounding boxes, and many predictions made densely across positions and scales are accumulated and fused instead of being pruned by non-maximum suppression. The approach won the ILSVRC13 localization task with 29.9% top-5 error; after the competition it reached 13.6% classification error and, having ranked third in detection with 19.4% mAP, set a new detection state of the art at 24.3%.

The same framework carries over to speech by treating 40 log-mel frequency bands over a 41-frame window as an image. A ConvNet acoustic model for IARPA Babel Cantonese, trained first on class-balanced data, without delta channels and with dropout on three fully connected layers, reaches a cross-entropy of 1.51 and 35.85% sub-phone error on validation data, against 1.73 and 37.78% for the IBM baseline on its held-out set, which the thesis notes tends to be the more optimistic. The thesis also collects a set of ConvNet practices: multi-stage features fed to the classifier, which took traffic sign recognition past human accuracy at 98.98% in the GTSRB competition and later to 99.17%; Lp pooling, preprocessing and local contrast normalisation, nonlinearities and momentum; architecture search with random weights; and unsupervised convolutional sparse coding, which together with multi-stage features cut INRIA pedestrian detection error from 26.05% to 11.05%. It closes by proposing to apply the localization approach to speech, predicting where each sub-phone sits in time.

Vision · Deep Learning — 2013

OverFeat

Code Paper Paper (mirror) ImageNet Results Slides (archived) Press BibTeX

Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus, Yann LeCun

ICLR 2014

This model obtained 1st place in the 2013 ImageNet object localization challenge. The model and pre-trained features were later released to the public.

OverFeat has been used by Apple for on-device face detection in iPhones: blogpost.

An integrated framework using a single ConvNet for classification, localization and detection at once. The central observation is that a multiscale sliding-window approach can be implemented efficiently inside a ConvNet itself, since the convolutional layers already share computation across overlapping windows — making dense evaluation affordable rather than prohibitive.

To this it adds a deep learning approach to localization that predicts object boundaries directly, and an unusual choice at inference: bounding boxes are accumulated rather than suppressed, so agreement between overlapping predictions raises detection confidence instead of discarding evidence. All three tasks are learned simultaneously in one shared network. The framework won the localization task of ILSVRC 2013 and was competitive on detection and classification; post-competition work set a new state of the art on detection. The released feature extractor, OverFeat, became one of the first widely used pretrained deep vision models — later used by Apple for on-device face detection in iPhones.

Vision · Deep Learning — 2013

Pedestrian Detection

Paper Paper (mirror) Video BibTeX

Pierre Sermanet, Koray Kavukcuoglu, Soumith Chintala, Yann LeCun

CVPR 2013

State of the art results on pedestrian detection datasets using deep ConvNets in the EBLearn framework.

State-of-the-art and competitive results across all the major pedestrian detection datasets with a single convolutional network, at a point when the field was still dominated by hand-crafted features.

Three departures from a standard ConvNet carry the result. Multi-stage features, with connections that skip layers, let global shape information be integrated with local distinctive motifs rather than the classifier seeing only the most abstract stage. And the filters at each stage are pre-trained with an unsupervised method based on convolutional sparse coding — the learned first-layer filters include not only oriented edge detectors but corner and junction detectors, features that would be tedious to design by hand. The system was built in the EBLearn framework.

Vision · Deep Learning — 2012

Convolutional Neural Networks Applied to House Numbers Digit Classification

Paper Paper (mirror) BibTeX

Pierre Sermanet, Soumith Chintala, Yann LeCun

ICPR 2012

State of the art results in house numbers classification using deep ConvNets.

Classifying digits from real-world house numbers — the SVHN dataset — with convolutional networks, at a time when most vision systems still relied on hand-designed features. The argument is that a ConvNet learns a feature set optimised for the task at hand instead of one chosen in advance.

Two departures from the standard architecture do the work. Multi-stage features feed the classifier the outputs of earlier stages as well as the last one, so local detail reaches the decision alongside global shape. And Lp pooling replaces the usual average or max pooling, with the paper analysing how the choice of p affects validation error. Together they establish a new state of the art of 94.85% on SVHN, a 45.2% relative error improvement. The source code and a tutorial were released through EBLearn.

Vision · Deep Learning — 2011

Traffic Sign Recognition

Paper BibTeX

Pierre Sermanet, Yann LeCun

IJCNN 2011

This deep model obtained 2nd place in a traffic sign recognition challenge using the EBLearn framework. It uses skip connections in deep ConvNets to better combine low-level and high-level learned features.

ConvNets applied to the GTSRB German traffic sign benchmark. Signs are rigid and designed to be visible, which makes the task sound constrained, but the dataset is full of real-world damage: viewpoint variation, saturation and low contrast, motion blur, occlusion, sun glare, faded colours, graffiti, stickers, and inputs as small as 15×15 pixels.

The architecture departs from a standard ConvNet by feeding the classifier the pooled first-stage features alongside the second-stage ones. The second stage carries global, invariant shape; the first carries local motifs with more precise detail; together they give the classifier several scales of receptive field. It also replaces the usual pointwise sigmoid with a rectified sigmoid followed by subtractive and divisive local normalisation, and trains on a jittered set five times the original size, perturbed in position, scale and rotation. The system placed second in phase I of the competition at 98.97%, above the 98.81% human benchmark and a hair behind the winner. Further experiments — a deeper classifier and, unexpectedly, dropping colour for grayscale — set a new record of 99.17%. Randomly-initialised features alone still reached 97.33%, which the paper uses as a cheap way to search architectures before committing to training.

Vision · Deep Learning — 2011

Traffic Signs and Pedestrians Vision with Multi-Scale Convolutional Networks

Paper BibTeX

Pierre Sermanet, Koray Kavukcuoglu, Yann LeCun

Snowbird Learning Workshop 2011

A workshop abstract applying multi-scale ConvNets, whose classifier sees first-stage features alongside second-stage ones, to both traffic sign classification and pedestrian detection.

ConvNets, multi-stage architectures inspired by biology, learn their invariant features at every level from data and tune them to the task, whereas many popular vision pipelines of the time used hand-crafted features such as HOG or SIFT. This abstract changes the traditional architecture so that the classifier receives first-stage features as well as second-stage ones, and applies the resulting multi-scale ConvNets to traffic sign classification and pedestrian detection, establishing new accuracy records, above human performance in the case of traffic signs.

Latent Self-Supervision · Vision · Deep Learning — 2010

Unsupervised Convolutional Feature Hierarchies

Paper Paper (mirror) BibTeX

Koray Kavukcuoglu, Pierre Sermanet, Y-Lan Boureau, Karol Gregor, Michael Mathieu, Yann LeCun

NIPS 2010

An unsupervised method for learning multi-stage hierarchies of sparse convolutional features. One of the few instances of this period where unsupervised pretraining improved results in a supervised task.

Sparse coding had become a popular way to learn visual features, but it was almost always trained on isolated patches. Applying the resulting filters convolutionally then produces highly redundant codes, because every overlapping patch was encoded without reference to its neighbours. This paper trains convolutionally over large image windows instead, which removes that redundancy between neighbouring feature vectors and makes the representation more efficient.

Alongside the linear decoder that reconstructs the image from sparse features, the method trains a feed-forward encoder that predicts quasi-sparse features directly from the input, so inference at test time costs one pass rather than an optimisation. The change in training regime changes what gets learned: patch-based training rarely yields anything but oriented edge detectors, while convolutional training produces markedly more diverse filters — centre-surround, corner detectors, cross detectors and oriented gratings. Stacked into a multi-stage convolutional architecture — 64 features at the first stage, 256 at the second, connected through a sparse table amounting to 4,096 kernels, more dictionary elements than prior convolutional RBM or sparse coding work had attempted — these filters improve results across several recognition and detection tasks, including Caltech-101 and INRIA pedestrians.

Robotics · Vision — 2010

Fully Adaptive Visual Navigation for Autonomous Vehicles

Paper BibTeX

Marco Scoffier, Urs Muller, Yann LeCun, Pierre Sermanet, Benoit Corda, Clement Farabet

27th Army Science Conference, 2010

Adaptivity and machine learning as the route to robust ground-vehicle navigation: less sensitive to sensor quality and calibration, workable with low-cost cameras rather than LIDAR, and portable from one vehicle to another.

Autonomous ground vehicles must react to changing environments and operate in areas they have never seen. This paper argues that adaptivity and machine learning make such navigation robust: a system that learns is less sensitive to sensor quality and calibration, can run on cheap, low-power cameras, leans less on active sensors such as LIDAR, and carries over more easily between small and large hardware platforms. It illustrates the argument with an obstacle avoidance example.

Deep Learning · Vision — 2010

Strategies for Learning Hierarchies of Invariant Features

Paper BibTeX

Yann LeCun, Pierre Sermanet, Y-Lan Boureau, Clément Farabet, Jason Rolfe, Karol Gregor, Arthur Szlam

Abstract (undated)

An abstract from Yann LeCun's group on learning multi-stage hierarchies of increasingly global, invariant and abstract features, for perception, from vision to audition.

Vision, audition and other perceptual tasks hinge on good internal representations, and the abstract casts learning them automatically, by biologically plausible procedures or otherwise, as the next major challenge for machine learning, computer vision and computational neuroscience alike. Both theory and experiments point to multi-stage hierarchies, each stage more global, more invariant and more abstract than the one before, as the best representation of the perceptual world; the question it takes up is how to design deep learning methods that learn them.

Vision · Deep Learning — 2009

EBLearn

Code Paper BibTeX

Pierre Sermanet, Koray Kavukcuoglu, Yann LeCun — additional help from Soumith Chintala

ICTAI 2009

A C++ deep learning framework similar to Torch and used for multiple state of the art results in computer vision.

Energy-based learning offers a single framework for supervised and unsupervised training of probabilistic and non-probabilistic factor graphs. A model assigns a scalar energy to configurations of inputs, outputs and latent variables; inference finds the configuration that minimises that energy; learning shapes the energy surface so that correct outputs sit lower than all incorrect ones.

EBLearn is an open-source, cross-platform C++ library for building such models. It has two components: libidx, an efficient and flexible multi-dimensional tensor library, and libeblearn, an object-oriented library of trainable modules and learning algorithms with support for convolutional networks, image processing and graphical display. Machines are assembled from predefined modules and loss functions, with gradient-based learning implemented through semi-automatic differentiation of the assembled model. The paper works through how regression, classification, unsupervised algorithms and full vision architectures such as LeNet-7 all map onto the same formulation, and shows the library detecting objects with a trained detector.

Robotics — 2009

Teaching Assistant for NYU Robotics class

Class Page

Pierre Sermanet, Yann LeCun

2009

Teaching assistant for the NYU robotics course, building small mobile robot platforms for student projects.

Robotics — 2005

LAGR: Learning Applied to Ground Robots

Project Page Video

Yann LeCun, Urs Muller, Pierre Sermanet, Marco Scoffier, Chris Crudelle, Beat Flepp, Ayse Erkan, Matt Grimes, Raia Hadsell, Koray Kavukcuoglu, Marc'Aurelio Ranzato, Jan Ben, Sumit Chopra, Jeff Han, Marc Peyote, Ilya Rosenberg, Yury Sulsky

DARPA, 2004–2008

A DARPA challenge where the NYU-NetScale team developed ConvNets for long-range off-road navigation from 2004 to 2008.

Robotics · Vision — 2009

Learning Long-Range Vision for Autonomous Off-Road Driving

Paper BibTeX

Raia Hadsell, Pierre Sermanet, Jan Ben, Ayse Erkan, Marco Scoffier, Koray Kavukcuoglu, Urs Muller, Yann LeCun

JFR 2009

An overview paper of our self-supervised deep learning vision model.

The journal-length account of the self-supervised long-range vision system. A deep hierarchical network extracts features from camera images; a realtime classifier trained on those features predicts traversability from 5 metres to beyond 100, far past the 12-metre limit of the stereo module that supervises it.

The paper credits the result to the quality of the training data generated on every frame: robust, visually consistent stereo labels; wide-context input windows normalised so objects appear at consistent scale regardless of distance; and a concise, discriminative feature representation. It covers the feature learning in detail — including unsupervised convolutional auto-encoder training and radial basis function features — along with the five-category labelling scheme, ground plane extraction, and the failure modes the labelling has to survive. Evaluation uses both a ground-truth dataset and field runs, where robots with long-range learning held to paths that the short-range system repeatedly lost, needing rescue from the undergrowth.

Robotics — 2009

Collision-Free Off-Road Robot Navigation

Paper BibTeX

Pierre Sermanet, Raia Hadsell, Marco Scoffier, Matt Grimes, Jan Ben, Ayse Erkan, Chris Crudele, Urs Muller, Yann LeCun

JFR 2009

An overview paper of our navigation system, designed to naturally handle errors and outputs coming out of a deep vision model. This model decouples the fast and short-range navigation from the slow and long-range navigation to achieve robustness.

The complete navigation architecture behind the LAGR system, and the argument for splitting it by range. Humans combine cues at several spatial resolutions and time scales: distant gazing sets the general direction and is refined slowly, while nearby obstacles demand constant attention and fast reaction. The system mirrors that.

A short-range perception loop runs at high frame rate on low-resolution images and handles local planning and obstacle avoidance; a long-range adaptive vision loop runs at lower frame rate on maximum-resolution images and handles strategic planning. Probabilistic traversability labels from both accumulate into a robot-centred hyperbolic-polar map with a 200-metre effective range. Short-range planning uses the recorded maneuver dictionary rather than a dynamical model. Localisation combines GPS, wheel odometry, an IMU and a fast, low-complexity rotational visual odometry module. The layers are deliberately independent, so each is insulated from the latency of the more complex ones above it: wheel commands at 20Hz, reactive planning at 5–10Hz, deliberative planning at 1Hz. The system beat both the reference baseline and its own configuration with long-range vision disabled, and was verified in independent government tests.

Robotics — 2008

Learning Maneuver Dictionaries for Ground Robot Planning

Paper Video BibTeX

Pierre Sermanet, Marco Scoffier, Chris Crudele, Urs Muller, Yann LeCun

ISR 2008

Instead of computing the theoretical dynamics of a vehicle, we propose to simply record the observed dynamics while a human operator “plays” with the robot, essentially trying all possible moves. At test time, the model has a bank of observed possible trajectories for every state of the motors. Trajectories leading to collisions are discarded, while the fastest available trajectory is selected. While we observed many collisions using the baseline system, we did not observe collisions after introducing this model.

Vehicle dynamics are normally captured by a model whose parameters come from system identification or from the vehicle's specifications. Those models are accurate in theory but blind to differences between individual vehicles, awkward to adapt to new environments, and often need hand-crafted heuristics for cases like reversing. This work replaces the model with a recording.

While a human drives — or the robot drives itself — every traversed trajectory is stored in a bank indexed by the initial left and right wheel speeds and by where it ends on a 2.5m circle. Ten speed bins per wheel give 100 states, each holding around 100 trajectories reaching different angles. At planning time the current wheel speeds select the feasible set, trajectories crossing non-traversable cells are discarded, and the cheapest survivor is executed as a list of wheel commands, at almost no online computation cost. A bank only 15% full, extracted from two hours of human driving, already drove well; 18 hours of logs filled it to 64% and covered a wider range of situations. Collisions were frequent with the baseline system and were not observed once this one was in place.

Robotics — 2008

Mapping and Planning under Uncertainty in Mobile Robots with Long-Range Perception

Paper Video BibTeX

Pierre Sermanet, Raia Hadsell, Marco Scoffier, Urs Muller, Yann LeCun

IROS 2008

A hyperbolic-polar coordinate mapping system that is naturally suited to handle imprecisions in long-range visual navigation.

Self-supervised long-range vision can label obstacles and pathways a hundred metres out, but both the category and the range of distant regions carry considerable uncertainty. This paper builds a map that represents both, and accumulates evidence across frames rather than overwriting it.

A pixel covers a near-constant angular extent but a wildly varying span of distances — a few centimetres near the robot, effectively infinite near the horizon. The map geometry follows that: hyperbolic-polar, with constant 20cm radial resolution out to about 15m, then hyperbolically growing cells to 200m and beyond, so an effectively infinite radius fits in a finite grid. A robot-centred Cartesian map at the same resolution would have needed 500×500 cells. Every cell holds a histogram accumulating per-class evidence from the classifier's five categories, so the traversability decision is deferred to planning time and the policy can be made more conservative or more aggressive without rebuilding the map. Because the map is fixed to the local pose frame, recentring as the robot moves costs only a translation.

Robotics · Latent Self-Supervision · Deep Learning — 2008

Deep Belief Net Learning in a Long-Range Vision System

Paper BibTeX

Raia Hadsell, Ayse Erkan, Pierre Sermanet, Marco Scoffier, Urs Muller, Yann LeCun

IROS 2008

Self-supervised long-range visual navigation with deep ConvNets.

A deep belief network extracts features from the robot's camera images, and those features train a realtime classifier that predicts traversability all the way to the horizon, enabling strategic rather than merely reactive planning.

Three choices make the self-supervision work. Training windows are large enough to carry context, not just colour and texture. The stereo supervisor emits five visually distinct categories — super-ground, ground, footline, obstacle, super-obstacle — chosen so the labels stay consistent instead of noisy, since inconsistent supervision causes the learning to fail outright. And a horizon-levelled input pyramid normalises for distance, so similar obstacles appear at similar heights regardless of how far away they are. The classifier's output populates a hyperbolic-polar costmap on which the planner runs. The system was developed and tested on the DARPA LAGR platform, both standalone and inside the full navigation stack, where long-range vision produced measurably smoother and more far-sighted driving.

Robotics · Latent Self-Supervision — 2008

Autonomous Learning for Long Range Vision in Mobile Robots

Paper BibTeX

Raia Hadsell, Pierre Sermanet, Ayse Erkan, Koray Kavukcuoglu, Urs Muller, Yann LeCun

Abstract (undated)

An abstract on our long-range vision system: self-supervised realtime learning on top of compact features from a trained deep belief net, classifying obstacles and traversable areas 200 metres or more away.

Vision-based autonomous navigation through complicated outdoor environments remains extremely hard and far from solved, and the abstract starts from the shortcomings of stereo vision. It describes a long-range vision system that combines a self-supervised realtime learning strategy with a trained deep belief net providing compact feature representations, and that accurately classifies obstacles and traversable areas 200 metres or more away, a step toward human-level autonomous driving.

Robotics · Latent Self-Supervision — 2007

Online Learning for Offroad Robots

Paper BibTeX

Raia Hadsell, Pierre Sermanet, Ayse Naz Erkan, Jan Ben, Jefferson Han, Beat Flepp, Urs Muller, Yann LeCun

RSS 2007

Online adaptation of long-range vision by self-supervising with short-range stereo vision.

Stereo range estimates become unreliable past 10 to 12 metres, which leaves a robot driving in a self-imposed fog — entering dead ends and slowly discovering pathways a human would see immediately. This system trains a classifier online, on every frame, from the stereo module's sparse near-range traversability labels, then applies it to the entire scene.

Two ideas carry it. A distance-normalised image pyramid makes it practical to train on large, context-rich windows rather than colour and texture patches. And spatial label propagation: once stereo labels a location, that label is propagated back to every earlier view of the same target, so the classifier learns view-invariant classifications and can train on far-away views captured before any label was available. A ring buffer acts as short-term memory, keeping the discriminative training balanced and consistent. The result sees obstacles and paths at 30 to 40 metres, far beyond stereo range, and adapts quickly when the environment changes.

Robotics — 2007

Speed-Range Dilemmas for Vision-Based Navigation in Unstructured Terrain

Paper Paper (mirror) BibTeX

Pierre Sermanet, Raia Hadsell, Jan Ben, Ayse Naz Erkan, Beat Flepp, Urs Muller, Yann LeCun

IFAC IAV 2007

An early version of our multi-range navigation architecture. Field tests showed that collisions grow with the length of the control loop, so a fast, low-resolution stereo module takes care of obstacle avoidance while a slower, higher-resolution module looks further ahead for strategic planning, each running at its own pace.

Vision-based off-road robots have a limited compute budget, and spending it on higher-resolution images and longer-range obstacle detection lengthens the delay between image capture and wheel commands. The paper first measures that cost on the LAGR platform: on a fixed course in a picnic area, the control-loop period was artificially raised from 360ms to 960ms, and collisions per run rose roughly in proportion, with the slowest setting crashing into obstacles again and again.

The proposed fix runs two obstacle detectors as independent processes, much as a person walking down a street glances far ahead only now and then but stays constantly aware of what is close by. FAST-OD maps a 5-metre radius from 160x120 stereo images at about 4Hz and 300ms latency while sharing the CPU; FAR-OD processes 320x240 images at 0.5 to 1Hz to see further (in these experiments a higher-resolution stereo module stands in for the learned long-range vision still under development). Planning casts rays in the fast map to place candidate waypoints, extends them through the far map and picks the cheapest, after shifting and rotating the older far map to the robot's current pose. On a test pairing a reactivity challenge with a hay-bale cul-de-sac deeper than the fast module's range, FAST-OD alone never crashed but always entered the cul-de-sac, the two modules in series avoided it but crashed often, and the two in parallel reached the goal fastest: 19 seconds, against 26.2 and 32.1.

Robotics · Latent Self-Supervision — 2007

Adaptive Long Range Vision in Unstructured Terrain

Paper Paper (mirror) BibTeX

Ayse Erkan, Raia Hadsell, Pierre Sermanet, Jan Ben, Urs Muller, Yann LeCun

IROS 2007

Self-supervised long-range vision that adapts online to new terrain. A convolutional feature extractor trained offline feeds a classifier retrained on every frame from stereo labels, recognising paths, man-made structures and natural obstacles up to 30 metres away.

Stereo is generally accurate only to 10 or 12 metres, while camera images carry information far beyond that, and outdoor terrain and lighting vary too much for fixed templates or classifiers trained offline on hand-labelled data. This paper presents a probabilistic, self-supervised long-range obstacle detector for the LAGR robot that carries near-range stereo knowledge out to 30 metres and adapts to unseen terrain after only a few frames, with no human labelling at any stage.

Each 320x240 stereo image is levelled to the horizon and cut into a distance-normalised pyramid of 24 bands, from 0.5 to 30 metres, so that a tree at 10 metres and a similar one at 30 appear at the same scale. A convolutional network, the only component trained offline, on 600,000 windows from 130 logs, turns each window into a 240-dimensional feature vector. Online, stereo labels (ground, obstacle or blocked) accumulate in a quadtree indexed by world position, and their ratios become soft labels for a logistic regression trained on every frame by minimising KL divergence; a full cycle takes 291ms, for 2 to 3 frames per second. The system classifies 85% of windows correctly in offline tests over the logs. Against hand-labelled ground truth from 16 logs, the rate of missed and false obstacle foot lines falls from 52.7% without online learning to 46.9% with binary labels and 39.9% with soft labels. The paper also shows the cost of too high a learning rate: the classifier adapts but forgets, and its far-range error ends up worse than with no online training at all.

Robotics · Vision — 2007

A Multi-Range Vision Strategy for Autonomous Offroad Navigation

Paper Paper (mirror) BibTeX

Raia Hadsell, Ayse Erkan, Pierre Sermanet, Jan Ben, Koray Kavukcuoglu, Urs Muller, Yann LeCun

RA 2007

Our multi-range navigation architecture with learned long-range vision in the loop. The long-range module sees paths and dead ends up to 30 metres away through self-supervised online learning, while a fast short-range stereo module keeps the robot reactive within 5 metres.

Sophisticated vision helps in complicated terrain but lengthens the delay between image capture and actuation, and a slower control loop makes the robot less reactive. Stereo alone is unreliable beyond 10 or 12 metres, so a robot that relies on it drives as if in a self-imposed fog, entering dead ends and finding paths late. This paper runs two obstacle detectors as independent processes: FAST-OD, a short-range stereo module on 160x120 images covering 5 metres, which gets priority on the CPU, and FAR-OD, a learning-based module that sees up to 30 metres. The planner picks desirable trajectories in the far map and lets the near map constrain the final choice.

FAR-OD levels the horizon and builds a distance-normalised pyramid of 24 bands, so objects keep a consistent height regardless of distance, then labels windows from stereo as ground, obstacle or blocked, accumulating the labels in a spatially indexed quadtree. A convolutional network trained offline on 31,600 stereo-labelled images from 158 logs reduces each window to 240 features, and a logistic regression with 723 parameters is trained on every frame from a class-balanced ring buffer, training and classification together taking under 100ms. On a field course with a dead end, the short-range system alone entered the dead end on every run, while the full system, initialised by a brief drive, never entered it and never crashed, cutting the average time to the finish from 97.84 to 53.08 seconds. On a second course along a scrub border, time fell from 180.77 to 133.8 seconds and crashes per run from 1.3 to none.

Robotics · Latent Self-Supervision — 2006

On-line Learning of Long-Range Obstacle Detection for Off-Road Robots

Paper BibTeX

Raia Hadsell, Pierre Sermanet, Jan Ben, Jeff Han, Sumit Chopra, Marc'Aurelio Ranzato, Yury Sulsky, Beat Flepp, Urs Muller, Yann LeCun

Snowbird Learning Workshop 2006

An early workshop abstract from our off-road robot work, on learning long-range obstacle detection on-line, beyond what stereo-based traversability maps can see.

The abstract starts from the standard approach to vision-based off-road driving: a stereo matching algorithm turns a pair of camera images into a point cloud giving the most visible pixels an XYZ position relative to the robot, heuristics such as counting the points above the ground plane in each map cell turn that into a traversability map, and maps from successive frames are assembled into a global map where path finding runs. Its subject, as the title says, is learning long-range obstacle detection for off-road robots on-line, while the robot drives.

Robotics · Vision — 2004

EUROBOT 2004 Competition

Project Page

Computer vision, navigation and behaviors by Pierre Sermanet, Philippe Rambert, Jean-Baptiste Mouret. Entire team: Evolutek

2004

Vision-based behaviors in a robot-rugby challenge.