Pierre Sermanet

I am a Research Scientist at Google DeepMind, working at the intersection of language, vision and actions with applications to robotics and embodied AI. My research interests span from multimodal learning, computer vision, self-supervised learning, learning from play, scalable data acquisition to embodied reasoning and AGI.

I obtained a PhD at New York University under the supervision of Yann LeCun in 2013 during which I published on various topics in computer vision (object detection, the “Overfeat” algorithm, pedestrian detection, visual attention, etc) and machine learning, e.g on unsupervised feature learning.

Research: After joining Google Brain in 2014 (now Google DeepMind), I continued working on self-supervised learning, computer vision, robotics, visual language models (VLM) and embodied AI. Because data is expensive to acquire for robotics and arguably its biggest bottleneck, I have developed self-supervision and contrastive learning techniques to learn from cheap, abundant and embodiment-agnostic (robot, human) unstructured video data (Time-Contrastive networks, Unsupervised Perceptual Rewards), as well as natural and information-rich multimodal data acquisition techniques for embodied AI (Learning from Play, RoboVQA) and training integrated Vision-Language-Action models (Grounding Language in Play, Robotics Transformers 2).

Competitions: OverFeat ranked 1st place in both the 2013 ImageNet object localization challenge and the 2014 Dogs vs Cats Kaggle competition (profile), GoogLeNet ranked 1st in the 2014 ImageNet challenge for both Image Classification and Object detection challenges and 2nd in the Object Localization challenge. In 2011, the deep learning framework I developed “EBLearn” ranked 2nd in the traffic sign recognition challenge.