Enabling the full potential of automotive 3D ToF imaging
By Gualtiero Bagnuoli, Melexis
The automobile industry is placing today great emphasis on protecting vehicle occupants and pedestrians from harm and ensuring that elevated safety levels are realized. At the same time, customers expect an intuitive user experience while driving - interacting with infotainment and connectivity functions in their car. Meanwhile, gradually higher degrees of driving automation are expected to be incorporated into vehicle designs.
There is, however, still quite some way to go to transition from the capabilities of current advanced driver assistance system (ADAS) implementations to what could be considered a truly autonomous operation. An important part of this evolutionary process will be how seamlessly the switch from autonomous mode to manual mode can be carried out in specific situations. If it is to be fully effective, this change needs to be done in a straightforward and user-friendly manner. The vehicle’s ADAS will require access to accurate, highly detailed information of the driver’s bio-mechanical and cognitive state on an ongoing basis.
Figures recently published by the World Health Organization (WHO) estimate that there are approximately 1.25 million fatalities every year through road accidents.
The WHO also identifies driver distraction as one of the major causes of such accidents - with calling/texting on mobile phones, checking wing/overhead mirrors, talking to passengers, eating/drinking, reading instrumentation dials (speedometer, fuel gauge, etc.), dealing with children and operating infotainment/navigation systems all being possible sources of risk.
With driver distraction often proving to be the reason (or at least contributing) to road accidents, the value of ADAS systems like automatic emergency braking and collision avoidance maneuvering is clear. However, when the driver is aware and bio-mechanically able to react to a hazardous situation, an overly abrupt intervention of the ADAS system can be annoying or even more dangerous. This implies that if the ADAS is able to ascertain the full driver’s state, it can then more intelligently judge whether or not it needs to take over from the driver in response to a hazardous situation - such as an imminent collision. Real-time monitoring of the driver’s position and movements is crucial if this is to be accomplished.
Through the employment of 3D imaging data, the driver’s body pose, their head position and hand positioning can all be accurately ascertained. It can be confirmed, for example, that the driver’s attention is on the road and their hands placed on the steering wheel. The 3D information can be used to estimate the bio-mechanical reaction time of the driver to re-engage, and compared with the event horizon that the vehicle has calculated to gauge the safety margin. Then if the driver is not adequately engaged to react quickly enough, the ADAS is aware that, should a potentially dangerous situation arise, it may be required to step in.
Until now, unfortunately, optical 3D imaging technologies have been difficult to implement within automotive settings. The main reason for this is that they struggle to cope with large and sudden variations in ambient light levels that occur in common driving scenarios. In addition, the supporting electronics must be suited to harsh operational conditions and support the ASIL B/C functional safety standards - something that few suppliers have expertise in. There are also issues relating to integration within the vehicle that need to be addressed. Often optical sensor implementations are very specific to the vehicle, with any differences in the size of the cabin or the orientation of the central console/seating having serious implications for the sensor system’s effectiveness.
Time-of-Flight offering for the automotive industry
Melexis has taken the optical Time-of-Flight (ToF) technology that has been used for several years for 3D imaging in applications like gaming machines and applied it to the automobile industry. Here an infrared light source emits a wide angle beam which upon striking obstacles is reflected back towards the ToF sensor. The sensor then detects the reflected infrared signal and compares it to a reference signal. By determining the phase shift that has occurred, the distance to a particular obstacle can be calculated and detailed 3D images constructed.
Melexis’ MLX75023 The ToF technology has gained momentum because of new active safety standards (e.g. NCAP) and features required by level 4 and 5 autonomous vehicles, going beyond in-cabin use. ToF technology is now evaluated for exterior use cases like cocooning because it can combine high resolution with accurate depth information at short range, complementing long-range sensors like FWC camera, Radar or Lidar.
It is clear that the variety of use cases implies the availability of sensor characteristics to enable the industry to choose the most appropriate solution and optimize performance/cost ratio. More than ever, the industry needs a complete portfolio.
Gesture recognition, for example, is usually limited to a restricted interaction area, usually around the central console. The field of view is quite narrow, the distance range is limited and depth precision is very crucial. So usually the QVGA resolution (320 x 240) is enough and cost-effective LED illumination can deliver the required performances. With the same resolution, it is even possible to use a wider field of view in order to cover both front seats. Tracking of an arm, head and in general larger object will still be possible.
Similar considerations are relevant for in-cabin monitoring, such as people counting and the measurement of the position of people in the vehicle.
When it comes to the collection of detailed information of the driver’s biomechanical and cognitive state, however, higher resolution and depth precision become mandatory. So cost-effective LEDs have to be replaced by VCSELs, which can operate at modulation frequency well beyond the typical maximum operating frequency of LEDs (typ. 30-40 MHz) and the sensor must be able to operate without a significant drop of its characteristics ( e.g. demodulation contrast).
The Melexis ToF portfolio for the automotive industry is today the most complete offering, allowing customers to select the optimal sensor for their system in terms of precision, resolution and integration level.
Melexis’ MLX75023 AEC-Q100 qualified, high dynamic range image sensor is on the road since 2015, focusing on human-machine interfaces (HMIs) and providing capacity for gesture recognition.
In 2018, the second generation of the QVGA chipset was released (MLX75024 + MLX75123), focusing on in-cabin monitoring. The MLX75024 ToF sensor has the same resolution (320 x 240 px) and maintains ambient light robustness while offering double the sensitivity of the previous generation, allowing operation in lower light levels, or requiring 30% less illumination power. Supporting the ToF sensor is the new MLX75123BA ToF companion chip with three times lower front-end noise than its predecessor.
In 2019, the third generation VGA ToF sensor MLX75027 was launched. Key features of this generation include support for modulation frequencies up to 100MHz, which allows manufacturers to take advantage of the full potential of VCSEL to achieve high distance accuracy. The MLX75027 uses Sony Semiconductor Solutions Corporation’s Back Illumination technology and includes advanced features, such as programmable region of interest, continuous or triggered operation modes and horizontal/vertical flip image modes.
Table: Melexis Time-of-Flight (ToF) sensor portfolio
A common footprint across the generations is an advantage for existing customers who can easily switch while a comprehensive set of evaluation kits reduces time-to-market for designers developing solutions for new projects and enables significant cost savings.
Table: Melexis Time-of-Flight (Tof) evaluation kits
Melexis is leading the way forward for automotive ToF sensing. Through the implementation of proprietary technology, and its continuous interest in new forging strategic partnerships, the engineering team is pushing the technology envelope, so that performance levels are heightened and cost-effectiveness increased. This will ensure that the full scope of ToF within the automobile application environment is realized.