10x Autonomy GmbH

Automated generation of simulation assets and realistic pedestrian interactions for LiDAR-based AI data.

Description of the company

10x Autonomy is a Hamburg based StartUp that accelerates the development of autonomous systems across different verticals – from Advanced Driver Assistance Systems (ADAS) and Autonomous Driving (AD) to logistics, robotics and industrial applications.

Every autonomous system relies on AI-models that have to be trained and validated with large amounts of accurately labeled sensor data. Today, labeling this data and evaluating how a system’s performance has changed after every development step are still processes built around human assessment. They take weeks, are expensive and slow down the entire development cycle. We are changing that.

We are developing a Physical AI that automatically labels sensor data such as LiDAR point clouds. Our customers use these labels to train and validate their own AI-models. On top of that, it evaluates the performance of a system overnight and directly gives the engineers feedback on what the system needs to improve on. This enables faster development cycles, and early detection of ffunctional weaknesses, at scale and across verticals.

By joining this project, you will not just be building academic prototypes; the tools you develop will directly extend the simulation pipeline that produces the data for the core AI of an early-stage deeptech startup.

Situation

Modern AI-models need to be trained on a large and representative amount of data. Since collecting real world data is time consuming, expensive and edge case scenarios cannot be controlled. To be able to test the software’s performance, data is generated in simulation.

The quality of this data is limited by what the simulator can represent: the variety of 3D objects (props) available in the scene, and how realistically dynamic actors such as pedestrians behave. Today, every new object has to be modeled, scaled, configured and imported by hand, and pedestrians mostly walk around as isolated actors that do not interact with their surroundings. Realworld scenes, however, are full of diverse objects and of people who interact with each other and with these objects.

As a student team, you will work directly with the founding team, gaining firsthand mentorship in cutting-edge AI and simulation development.

Problem

Scaling up the diversity and realism of simulated scenes is currently a manual bottleneck. Adding a single new object type or a new pedestrian behavior requires hours of work by a 3D/simulation expert. To make simulated data a scalable and realistic source for training and validating AI-models, the following aspects have to be automated and physically plausible:

• Asset creation: Generating a 3D model of an object from a simple image instead of modeling it by hand
• Geometry & scale: Correct real-world dimensions and a clean mesh, so that the LiDAR returns and bounding boxes are realistic
• Materials: Surface properties that influence the LiDAR intensity - Behavior & physics: Imported objects that automatically behave like comparable known objects, e.g. a tuk-tuk that drives like a bike
• Actor interactions: Pedestrians that interact with each other and with objects (e.g. holding hands, picking up and putting down objects, opening a car door)
• Ground truth: Consistent labels (class, bounding box, attached/carried objects) for all newly generated assets and interactions

Automating these steps would drastically increase the variety of scenarios we can simulate and lead to more representative training data and better AI-models.

Aims of the project

The project consists of two tracks. Depending on team size and interests, the team can work on both tracks in parallel or focus on one of them.

Project 1a: Image-to-Simulation – automatic 3D asset generation from images 
Develop a pipeline that takes a single image (or a few images) of an object and automatically brings a 3D version of it into the simulator. This includes generating the 3D model (e.g. with state-of-the-art image-to-3D methods), estimating its real-world scale, cleaning and optimizing the mesh, assigning materials and semantic class, and packaging it as a usable prop in the simulation, without manual steps.

Project 1b: Behavior transfer for imported objects
Develop a method that takes the file of an object (e.g. a tuk-tuk) and automatically makes it behave like a similar, already supported object class. A tuk-tuk, for example, moves mainly like a bike, so it should be identified as vehicle-like, its movable parts (e.g. wheels, tracks, turret) should be detected, and it should be configured with vehicle physics and control, so that it can drive through the simulated scene like any other vehicle. The whole process should run fully automatically.

Project 2: Pedestrian interactions
Extend the pedestrians in the simulator with realistic interactions, both between pedestrians and between pedestrians and objects. Examples are two pedestrians holding hands, a pedestrian picking up and putting down a static object, or opening a car door. This includes the animation and synchronization of the actors, attaching and detaching objects, the integration into scenario descriptions, and the correct ground truth labels in the resulting LiDAR data.

Scopes

- Research the State of the Art: Investigate current methods for image-to-3D generation, automatic rigging and behavior assignment, and multi-actor/human-object interaction animation.
- Engineer Automated Pipelines: Research, design, and implement tools that bring new assets and behaviors into the simulator without manual work.
- Validate in Simulation: Put your pipelines to the test by generating new scenarios, rendering LiDAR data and evaluating the quality and realism of the results, e.g. by their influence on the performance of our AI-models.

Target group (students)

We are looking for students:

• Computer Science
• Informatik-Ingenieurwesen
• Data Science
• Robotics
• Mechatronics
• Interdisciplinary Mathematics
• Technomathematik

Following prior experience, knowledge and/or skills are beneficial:

• Strong python skills
• Experience with git
• Experience with 3D tools or game engines (e.g. Blender, Unreal Engine, Unity) is a plus, but not required
• Experience with C++ is a plus, but not required
• Experience with LiDAR data or robotics/driving simulators is a plus, but not required

Dates
Please save these dates: Fishing for Experience Termine

Registration
You can apply for Fishing for Experience online.