Deep Learning Based Multi-modal Perception and Semi-automatic Labelling Algorithms for Automotive Sensor Data

Automated driving has great potential in solving the mobility needs of the future and reducing the amount of traffic accidents. Advances in the environment perception for automated driving are to a large extent driven by deep learning and huge amounts of sensor data. Multiple-sensors are used in automated driving to make use of complementary information to increase performance and reliability of the system. Many different artificial neural network architectures have been proposed, increasing the recognition accuracy. In this work it is shown how deep learning based object detection can be used to fuse measurements from several different kinds of sensors. Multiple methods combining, e.g., lidar and camera are compared and existing research datasets are introduced. Differences between the approaches are discussed and recommendations derived. From those a flexible deep-learning based fusion algorithm is developed and evaluated on real-world data. Even more important than the specific architectural details are the amount and quality of available data needed to properly represent the joint distribution of potential inputs and targets. This thesis thoroughly studies the effect of differences in the labelling quality on deep learning based classification and deep learning based detection. From the results a novel quality estimation algorithm is derived and a method for estimating the maximum reachable accuracy with a given network and dataset as well as the trade-off between increasing labelling effort and reducing gains as approaching this threshold is developed. While hand-labelling vast amounts of real-world data becomes quickly infeasible, semi-automated methods have great potential of enabling labelling relevant datasets at scale. This thesis develops and evaluates means to handle partially labelled and unlabelled data with the aim of providing sufficient data to generalize to real driving scenarios. To this end, algorithms for semi-supervised learning are developed and improved, as well as iterative algorithms to increase the labelling quality of a given labelled dataset.