Beyond the frame rate
Cameras with event-based sensor technology
The precise detection and evaluation of fast movements is a challenge. Highspeed cameras with high sensor resolution and high frame rates provide detailed information, but generate enormous amounts of data that need to be stored, transmitted and processed. They also contain large amounts of irrelevant static information that has to be filtered at great expense. Lower frame rates reduce the amount of data, but lead to motion blur or a lack of detail. Solutions with classic (image-based) cameras are therefore often affected by compromises. However, movement information can also be recorded in a completely different way - more efficiently and with significantly less data!
A new sensor technology, inspired by nature and the human brain in particular, pursues precisely this approach. Instead of generating a flood of redundant image information, this technology concentrates exclusively on what is really changing. Result: High-precision movement information with a drastically reduced amount of data. This opens up completely new possibilities - especially in applications where speed and efficiency are important. The exact name of this groundbreaking revolution is "event-based vision sensor", or EVS for short.
Groundbreaking sensor technology
Event-based imaging is also referred to as neuromorphic sensor technology - sensors with the ability to process information in a similar way to the human neural system. To do this, we need to realize that the evolutionary advantage of our brain is not just that it can efficiently process enormous amounts of data from continuous light stimuli from the photoreceptors in our eyes. The ability to react to changes such as differences in brightness, contrasts and movements is crucial, while uniform stimuli are largely ignored. This means that we focus primarily on movements in a scene instead of constantly re-capturing every static detail of our surroundings. This allows our brain to process relevant information quickly without being flooded with unnecessary data.
To replicate this ability, Prophesee has developed special pixel electronics in cooperation with Sony, whose main task is to detect and record only changes. If the contrast of a pixel value changes over a certain threshold value, a so-called "change event" is triggered. Each pixel acts independently of its pixel neighbors and in real time. This means that it is not bound to a fixed time interval (cf. frame rate). The minimum time span between two pixel events is an important property of this sensor and is referred to as "temporal resolution". Sony specifies this as 1 microsecond for the IMX636 sensor. This enables ultra-fast and almost "gapless" scanning of movements. To realize such a fast rate of change with image-based sensors, this would correspond to frame rates of more than 10,000 images per second!
No pictures! Changes only
While image-based sensors always transmit the complete amount of data from the entire sensor surface at regular intervals, an event-based camera often only generates a very small amount of data in the same period of time. This means that application developers do not have to compromise between high frame rates and large amounts of redundant data to accurately capture fast events. This is because the amount of data generated by EVS cameras depends on the activity in the field of view and adapts automatically if the scene conditions change. In contrast to image-based sensors with a fixed frame rate, EVS pixels only transmit information and generate data traffic when changes occur in the field of view.
From motion blur to motion sharpness
Due to their technology, classic sensors can exhibit motion blur during fast movements. It occurs when contrast boundaries (e.g. due to object edges) move across several neighboring pixels while they are exposed. Each pixel picks up light from different positions of the moving object. The faster the movement or the longer the exposure time, the more difficult it is to obtain a clear image without distortion. EVS pixels, on the other hand, continuously evaluate the incident light and only register the increase or decrease in the amount of light in a comparator. If they exceed the set threshold values, they generate ON or OFF change events with a time accuracy of around one microsecond. Even the fastest movements are scanned pixel by pixel using EVS technology. The result is a high-resolution sequence (stream) of independent pixel events that describe a motion path without any motion blur.
Comparison of data from image-based and event-based cameras
Image-based camera |
Event-based camera |
|
|---|---|---|
Data |
|
|
Information |
|
|
Amount of data |
|
|
Quality |
|
|
Less data - more efficient information
The native output data of EVS cameras (pixel position X/Y, ON/OFF polarity of the event, timestamp T), i.e. the information content of the stream of change events, is extremely compact and efficient, but does not provide classic images. This makes them ideally suited for processing by machines and algorithms, but less intuitive or directly usable for humans. If we still want to visualize the result stream in images, these are reminiscent of a 2D camera image after edge detection. The reason for this is that changes in contrast during movement are less noticeable on evenly illuminated surfaces and more noticeable on object edges.
Since only relevant and therefore significantly less data is recorded, storage requirements and processing effort are considerably reduced. In addition, the event information already supports the recognition of movement patterns and directions. The time gaps between the recorded events can also be used to directly calculate how fast a pixel or object is moving without having to process a large number of images, for example to separate relevant information from unnecessary static background data.
Due to the small amount of data, many processes can be analyzed almost in real time. Multi-camera systems are also much easier to implement, as they require significantly less technical effort. Both the image processing performance of the host PCs and the peripherals consisting of cabling, power supply and the like can be dimensioned smaller and more cost-effectively.
Time as information
Based on the microsecond-precise time stamp and the position of each individual pixel event, completely new application possibilities arise. The change events already contain valuable information from which further important information can be derived. Conventional cameras with a fixed frame rate cannot capture these due to their constant sampling rate or they are lost in a large amount of redundant data due to the type of output.
The event data for creating slow-motion recordings offers an exciting analysis option. By accumulating the captured pixel events in a temporal grid and generating complete sensor images from them, slow-motion videos with a variable "exposure time" are created. The playback speed remains variable due to the selected accumulation time and display frame rate. It ranges from real time (super slow motion with one frame per event) to actual movement speed (at approx. 1 frame per 33 ms) to a still image. If all recorded events are combined (in terms of time), the complete movement history becomes visible.
Speed and direction information can also be extracted for precise, numerical analysis of object movements. No more complex image processing is required for this. If, on the other hand, the location and time of several pixel events are accumulated over a certain time range in a 3D visualization, the result is a qualitative representation of the movement path. This in turn helps to understand how and on which paths objects move in a (time) domain. This procedure is used, for example, in flow analysis for high-precision detection of the movement of liquids and gases.
"For applications that require exceptional frame rates of 1000 Hertz, such as in flow visualization, the implementation with image-based cameras is often very complex and expensive. With the help of event-based camera technology, we achieve comparable frame rates of 10,000 frames per second and more. To transfer the significantly reduced amount of data, however, we only need standard PC interfaces such as USB. This makes this innovative camera technology particularly interesting for smaller teaching and research facilities."
You can read more about event-based flow visualization in our application story: To the case study "only the changes count"
New data - new processing approaches
However, in order to be able to use this new sensor information, developers need to find a different way to handle the previously cyclical image-based processing sequences. Of course, several event data can be combined in classic frames in order to even process them like conventional images with a constant frame rate. However, this method is not necessarily optimal, as it leaves the advantages of data dynamics unused. For example, the high temporal precision for fast movements and the efficient processing of less data at once, which can also reduce energy consumption. Only with the appropriate functions, tools and algorithms can patterns, movements, times and structures be extracted and processed quickly and efficiently from the event data. Today, this is not found in any of the known (image-based) standard vision frameworks.
However, Prophesee and Sony, the manufacturers of the new sensor technology, have already developed corresponding processing methods and made helpful functions available in a software development kit, the Metavision SDK, together with detailed documentation and numerous samples. This means that users can start using the new possibilities of this innovative technology immediately.
To operate the IDS EVS camera "uEye XCP-E", only the IDS HAL plug-in needs to be installed on the host PC. The camera is then immediately ready for use in Prophesee's Metavision SDK. Watch the how-to video now:
High precision in real time - EVS in quality assurance?
The capabilities of neuromorphic sensors can also play an important role in quality assurance and improvement. Especially in applications where accuracy, speed and efficiency in fault detection are required. The added value of being able to record the smallest object and material changes in pixel size and in real time is evident, for example, in the monitoring of machines and processes. Thanks to the high temporal resolution, which extends into the low microsecond range, even high-frequency movements such as vibrations or acoustic signals can be visualized. Analyses reveal unusual patterns (e.g. due to wear, malfunctions) at an early stage, which can lead to damage or production downtime.
As they only perceive movements or contrasts, neuromorphic sensors are much less sensitive to changes in light, which makes them far superior to conventional image processing systems in strongly varying lighting conditions (e.g. reflections, shadows). When it comes to rapid fault detection, process monitoring or inspections under difficult conditions, quality assurance processes can only benefit from the capabilities of neuromorphic sensors.
EVS - trend or must-have?
Event-based sensors do not capture complete images, but only pixel changes over time. However, these can be used to dynamically compile very different visualizations, providing applications with significantly more movement information than cameras with conventional image sensors alone could. Therefore, the technologies do not work in competition! Event-based sensors are therefore not a general replacement for classic image-based cameras or even AI-based image processing, but rather a complementary technology. It opens up new additional possibilities when it comes to recording movements. In various applications, a single sensor type or type of result data is not sufficient. A combination of different information and therefore different camera categories is often necessary to provide an optimum solution to a customer requirement. Event-based cameras are therefore interesting and worthwhile components for fast motion analysis, industrial quality assurance tasks, robotics and autonomous systems in general.
Note from IDS
"We already have a number of concrete application examples for analyzing gases, liquids and vibrations for which this technology is ideally suited for the start of series production of the uEye EVS camera."
Further information
- In this presentation, we will explain how the uEye EVS cameras work and their areas of application: To the video
- This allows you to commission your event-based camera quickly and efficiently: Watch the video
- See how uEye EVS cameras are used using the example of event-based flow visualization: Read the case study
- Find out more about the uEye EVS cameras: Camera family
- Technical details of the camera: Select in the camera finder