Part 1: Sensor Fusion of different sensor types for reliable 3D Collision Avoidance in AGVs
Reading Time: 5 Miutes approx. – 18/11/2021
“Any sufficiently advanced technology is indistinguishable from magic.” – Arthur C. Clarke, British science-fiction author and physicist once stated. It is a truth universally acknowledged that today’s world has seen technological magic happen on an unprecedented scale. innovative products and processes have shaped the industrial landscape as we know it today.
Our last blog article provided an introduction to automated guided vehicles (short AGVs), which have been implemented in factory settings to improve manufacturing and material handling processes, trailblazing the path to Industry 4.0 and Intralogistics 4.0. Besides, it touched upon the various sensor types used to help AGVs and other robots perceive their environment for navigation purposes and to avoid collisions with machinery and humans.
In this article, we now want to take a closer look at the different types os perception sensors used, comparing and contrasting them to find out which types of sensors are best suited for 3D collision avoidance in robotic and autonomous applications.
AGV – what?
As a recap, let us briefly clarify the term “AGV”:
An automated guided vehicle or automatic guided vehicle is an electric, floor-bound, driverless, mobile robot which moves goods from A to B without human intervention, being guided by various types of navigating systems. They are most often used in industrial applications to transport heavy materials around a large industrial building, such as a factory or warehouse.
The first AGVs where brought into use in the 1950s in the shape of tow trucks following a wire path integrated into the floor. They have since been optimised to navigate with the means of a complex navigation systems called “Automated Guidance System”, comprising of path planning, navigation and control. We will dive deeper into these phases in our upcoming blog articles.
Classification of sensor systems
By and large, the kind of guidance technology implemented in AGVs and other robots as well as in autonomous driving, can be categorised into sensing and non-sensing.
The non-sensing group comprises of but is not limited to:
- contact bumpers, emergency stop buttons, safety PLCs (active non-sensing safety devices)
- warning lights, alarm signals, signs on the vehicle (passive non-sensing safety devices)
In the context of this article, however, we shall be focusing on sensing safety devices / sensor types – both active and passive.
The ABC of sensor technology
There are three different types of sensors:
- Location sensors: They determine a robot’s own location in the environment.
- Proprioceptive sensors: Those determine a robot’s own motion.
- Range-finding sensors: These determine the location of objects within the environment of a robot.
For the purpose of this article, we shall concentrate on range-finding sensor types only, as this is what we focus our technology on to offer next-generation 3D collision avoidance solutions for the robotics and automotive industries.
Perception sensors, or range-finding sensors, are the perceptual interface between an AGV or any other robot or autonomous vehicle and its environment. For one of a better word, they are the “eyes and ears” of aforementioned devises which could not perceive and thus navigate their environment without them. Common environment perception sensors are based on the detection of either radio waves (RADAR), acoustic waves (Ultrasound) or light (Cameras and LiDAR).
They can further be categorised into passive and active.
In layman’s terms, active sensors speak and listen to the environment, whilst passive sensors only read the environment. The latter are labelled the true observers of the environment as they do not disturb it by emitted signals but quietly and constantly “read” instead.
Comparison of active and passive sensor types
Passive sensor types
So let us talk about passive sensor types fist: We have learned that they depend on external stimuli meaning they receive signals from their environment which they subsequently process to generate data. Whilst there are various passive sensors, the most common ones are cameras, which are described in the following.
In contrast to active sensors, which actively observe and analyze their environment using the principle of time-of-flight (short ToF), passive sensors, such as certain types of cameras, process data by receiving information from their environment in form of light. They detect light within the visible or infrared spectrum. Different types of cameras use different approaches to extract range information from this data.
Mono and wide-angle cameras
These cameras typically provide an image of the surrounding environment. When implemented with a moving object, a mono camera can also create a 3D image or even a 3D map of multiple photos from different angles using photogrammetric range immaging techniques (“Structure from motion”-principle).
Stereo cameras
They capture the same scene with two or more cameras, measuring distances and generating 3D data through triangulation, which probably makes them the most precise sensors when it comes to measuring 3D geometries. Newer cameras can offer on-chip processing and are capable of edge-detection. Their output is an image that consists of edges instead of color information. This simplifies the 3D image reproduction for stereo cameras or structure by motion algorithms.
Active sensor types
As introduced above, active sensors send out a signal into the environment. In so doing, they use energy obtained by an external power called excitation signal and are dependent on the emitted signal, or part of it, being reflected back to them in order to measure what is known as Time-of-Flight (ToF). This is calculated by measuring the time it takes from emitting a signal to receiving (“hearing”) the reflected signal. As this measured time is proportional to the distance of the reflecting object, the range can be measured as follows:
r = range/distance to reflecting object
v signal = velocity of the signal, for example:
→ speed of light: 299 792 458 m / s
→ speed of sound: 343 m / s
t = measured ToF
Depending on which sensor technology is used, the velocity of the signal may differ. Active sensor types can provide more information than passive sensors due to being able to control when and how much energy is emitted, however they need more power to do so and run the risk of interfering with other active sensors (called cross-talk or noise). Cross-talk can lead to deadlocks (as described in our last blog) and false positives. The latter describes what happens when a sensor falsely perceives a non-existing object in its environment and subsequently orders an AGV to slow down or worse still, come to a halt. This would negatively impact production, thus resulting in a decreased productivity and revenue.
So-called false negatives (a sensor not perceiving a physical object in its environment even though it is actually present) could happen, too, which would have even more serious consequences, as besides potentially causing damage to equipment, they could lead to personal injury, or in the worst case, death.
Common active sensors include:
- LiDAR, based on light rays
- ToF – and Structured Light Cameras, based on imaging
- RADAR, based on radio waves
- Ultrasound, based on sound waves / acoustic waves
RADAR Sensors (Radio Detection and Ranging)
For automotive and robotic applications, Radar sensors typically operate as FMCW (= Frequency-Modulated Continuous Wave). Using the Doppler Shift, velocity and distance of objects in the environment can be measured.
There are different types of RADAR Sensors for different applications: Whilst short-range RADARs offer a signalling range of only up to 30 m, long-range RADARs have a limited FoV but longer range. FMCW RADARs are special in that they can change their operating frequency during the measurement, in contrast to usual CW (= Continuous Wave) sensors. Automotive RADARs for instance typically use either the 24 GHz or the 77 GHz frequency band for their applications. Owing to the nature of sensing, RADARs are mostly in competition with LiDARs – their generated data resolution is oftentimes a lot worse than that of LiDARs, yet they are much cheaper.
Whilst RADAR technology is a great collision avoidance technology in automotive applications, it does not have a high market acceptance in the AGV market. This is largely owing to limited reliability in dense indoor industrial environments typically filled with many metal reflectors causing possible interference.
Camera Sensors
Structured Light Cameras
They actively emit signals into the surrounding environment to measure the return signals. One of the camera techniques used is Structured Light Cameras that project a light pattern into the environment (the structured light).
From the distortion of this pattern onto an arbitrarily shaped surface, the 3D geometry of this surface is produced.
ToF Cameras
They can capture an entire scene by emitting a light pulse via an active modulated light source to illuminate the scene and measuring the ToF of the return signals for each pixel.
A solid-state Laser or LED is typically used for illumination in the near-infrared spectrum (around 800nm)
Thermal cameras
Cameras are not only limited to visible light. Thermal cameras use infrared light to produce a thermal image. They work similarly well during day and night and offer a good resolution of the surrounding environment. Images produced in this way, however, are not comparable to standard images. Due to their limited object detection capabilities, they are currently not widely used in the AGV market.
→ Out of all described sensors, cameras are the best for semantic segmentation and classification of obstacles.
The downside of cameras in general is that their functionality depends largely on weather conditions, which reduces their reliability. Besides, they are quite expensive whilst requiring a high rate of data processing. Rain, as well as fog or direct sunlight, can seriously reduce the data quality. For stereo cameras, the opening angle is often limited and the technology itself is quite expensive. For thermal cameras, precise distance measurements are difficult, and ranges can only be roughly determined.
AGVs guided by cameras are typically called “Vision Guided Vehicles” (“VGVs”).
LiDAR Sensors (Light Detection and Ranging or Laser Imaging Detection and Ranging)
LiDAR sensor technology uses laser beams in the (near-) infrared, ultraviolet or visible spectrum. By measuring the ToF of these laser beams, LiDARs can create a high-resolution 2D or 3D map of the environment. Most LiDAR technologies use pulsed laser beams to illuminate the surrounding environment, some others use continuous-wave signals. As the prices for LiDAR systems have dropped significantly due to the reduced amount of moving and rotating parts, the technology has become more competitive recently, yet remains the most expensive sensing system. There are spinning LiDARs which have issues with long lifetime because of the moving parts and solid-state LiDARs which are more robust as all components are practically static. For more information on these two types as well as all other sensors described in this blog article, please see our Range-Finding Sensor Compendium.
These days, every AGV should have a 2D safety LiDAR fitted to operate safely. The data obtained is used for reliable people detection and sometimes also for generating navigation data. One of the big shortcomings of 2D LiDARs is their missing capability for reliable 3D space surveillance. Hence, additional 3D sensors for ultra-short-range 3D collision avoidance, like the ECHO ONE DK by Toposens, can bridge the gap.
AGVs using LiDAR sensors can be referred to as “Laser Guided Vehicles” (“LGVs”).
Ultrasonic Sensors – traditional 1D
Ultrasonic sensors measure an object’s proximity by transmitting short bursts of sound waves (ultrasonic pulses) into the environment. They then measure the time it takes for the waves to travel to a target object and for the reflected echo to arrive back at the receiver (which can be the transducer itself or a separate microphone). The high-frequency sound waves (typically around 40kHz) produce echo patterns which are then processed by algorithms. Whilst inaudible to humans, they can be heard by bats, dolphins and other specific types of animals capable of hearing in the ultrasonic sound range.
Conventional 1D ultrasonic sensors have certain limitations:
- They can only detect objects in one-dimensional space for distance measurement
- They can typically only detect the closest object to the sensor due to the very limited operating range compared to the 3D ultrasonic sensor offered by Toposens
- They have a limited opening angle
Although having been a cheap parking assistant solution in ADAS applications for decades, they lack in data quality and sufficient reliability for complete autonomous navigation. Especially in terms of 3D collision avoidance as they do not yield 3D data.
3D Ultrasonic Sensors
Compared to standard 1D Ultrasonic Sensors, the 3D Ultrasonic Sensor “ECHO ONE DK” manufactured by Toposens yields 3D data of the environment. For this, it receives the reflected pulse by an array of microphones on the sensor which then allow for the positions of the echoes to be calculated as 3D coordinates, based on the different times when those echoes arrive at the individual microphones where they are sent to the processor as electrical signals. 3D coordinates are then processed at the end of each measurement cycle and evaluated in the post-processing phase.
The ECHO ONE DK features the worlds-first and only 3D ultrasonic sensing technology, using a combination of unique hardware setup including multiple microphones and sophisticated, patented ToF software algorithms. The generated data is subsequently converted and displayed for further processing in a 3D point cloud output.
Having a much bigger opening angle of up to 180 degrees in ultra-short-range, the sensor can detect multiple objects accurately and within close range, utilising the principle of echolocation whilst delivering robust data in an energy-efficient way.
The fact that it can detect even the smallest and most complex objects such as screws, cables and especially forklift forks – all of which potentially causing costly damages in AGV operations – makes 3D Ultrasonic Sensing a great solution for reliable machine and object protection.
The ECHO ONE DK will help AGVs detect protruding objects like forklift forks, suddenly appearing persons and all objects at close range in 3D space, regardless of dust, dirt and varying light conditions.
Consequently, 3D ultrasonic technology, on its own or in combination with, for example, LiDAR, will become an increasingly crucial combination.
Conclusion – one for all or all for one?
Sensors have been used for decades in a bid to revolutionize the industry by increasing efficiency and safety standards in everyday processes. In today’s fast-paced world, marked by exponential technologies and constantly evolving automated and autonomous processes taking over traditional manufacturing practices, a huge amount of expectations and responsibilities are placed upon sensor technologies.
→ The conclusion which can be drawn from the above comparison is that every sensor type, be it passive or active, has its advantages as well as shortcomings:
Due to their preciseness, 2D LiDARs are widely used as standard sensors, especially for person protection and should be fitted to every AGV by default.
RADAR, whilst being a lower cost solution applied to a wide variety of applications, fails to deliver precise enough data, just like 1D Ultrasonic Sensors, and is thus not very widely used in AGV applications.
Cameras, whilst providing a human-like observation of the environment, yield data quality which is very much subject to environmental and weather conditions (dark nights or foggy weather to name but a few), which in turn can reduce their reliability. With the emerging trend of 3D ToF technologies taking hold in industrial processes, there is an increased need for more reliable sensor systems in 3D space, especially in connection with 3D collision avoidance. This is where ECHO ONE DK by Toposens bridges the gap between existing sensor technologies, offering reliable 3D collision avoidance with multi-object detection especially in ultra-short-range. This in turn makes AGVs or other robots achieve next-level robotic safety, increasing the associated ROIs and contributing toward safe and efficient factory settings.
In summary, all above-described sensor types play an essential role in the development of technologies now and in future. Consequently, the most efficient, effective and safe performance will be achieved by having different types of sensors work together on one application, combining their collective strengths whilst diminishing their individual weaknesses.
This is called sensor fusion – a system which, to take up Arthur C. Clarke’s words, certainly is “indistinguishable from magic”.
Whilst this article provides an overview of different sensor types,
we will dive deeper into the subject matter of AGVs in our next blog article about
damages caused by forklift forks in mixed operations in AGVs.
For more information on our state-of-the-art sensor solutions, do get in touch via sales@toposens.com.
