Polarization image sensor technology that
captures characteristics of light invisible to humans

Polarization image sensors are able to visualize events that have never been seen by humans before. They do this by capturing polarization, the third element of light that cannot be seen by human eyes. This demo introduced an example in which scratches on a model car, which are virtually invisible to the human eye, can be clearly seen by passing it through a polarization sensor, as well as some examples of distortion detection in transparent objects such as the lenses of glasses.

Sony developed a sensor equipped with Polarsens,* the world's first technology capable of capturing polarized incident light with extremely high precision thanks to a microfine polarizer incorporated in each pixel using a sophisticated semiconductor process. By integrating this sensor with AI-powered signal processing, Sony is creating new value in various markets.

* Polarsens is a trademark of Sony Corporation.

Automotive CMOS image sensor and sensor fusion technology

Sony’s automotive CMOS image sensors and its Sensor Fusion technology quickly detect changes in the surrounding environment and support prompt risk-avoidance behavior. This demo introduced how pedestrians and road signs can be seen with a camera equipped with an automotive CMOS image sensor even when illumination is so low that objects are not visible to human eyes, and how LED traffic signs and signal lights can be recognized without flickering.

Using the state-of-the-art CMOS image sensor technology cultivated for digital cameras and smartphones, Sony has further developed it to meet the automotive quality requirements and challenges to address difficult driving conditions, such as detecting objects under low illumination, detecting distant objects while driving at high speed, and LED flickering.

Furthermore, Sony’s Sensor Fusion technology, which combines CMOS image sensors and radar, provides highly accurate object recognition even in scenes of backlight and fog where the object shape cannot be captured by the camera alone, and in tunnels and on iron bridges where noise occurs due to the diffused reflection of radar signals. In conventional methods, object recognition is unified or associated the recognition result of individual sensors. In Sensor Fusion, raw data (data before processing) derived from the camera and radar are used to extract feature quantities of objects and integrate them, thereby enabling highly precise object recognition—technology that Sony, the sensor manufacturer, is uniquely positioned to make possible.

Low power vision sensing processor and
real-time 3D sensing technology

2D/3D spatial sensing systems that process a large number of image sensor inputs at once with a vision sensing processor on an edge device play an important role for autonomously driven robots such as aibo and drones. Self-position estimation, image recognition, and obstacle detection are performed using various sensors; meanwhile, safety, high speed, stability, and power saving are some of the challenges.

The newly developed vision sensing processor can process up to 12 image sensors, and gyroscopes, accelerometer sensors, and so forth and can measure the surrounding situation to estimate its own position and create a map of the environment. Since this vision sensing processor incorporates the inference function of Sony's open source Neural Network Library, it can also operate its own custom recognizers and inference devices created with the Neural Network Console.

Precision bilateral control that makes people more dexterous

In order to create robots capable of working in close cooperation with people, there are three critical technologies: force control, which allows the robot to interact with people and the environment; precise operational control, in which robots excel compared to humans; and intelligence.
This demo introduced how to achieve an interaction in a one-tenth scale world using force control and precise operation.

The operations carried out by the human are transmitted to the precision control robot on a one-tenth scale, and the feedback of the force from the robot is expanded 10 times to the human operator, enabling the operator to sense even a very small object as if touching it directly.
The core technologies that enable precise operation beyond the limits of human hands while feeling a small change in force are high-sensitivity force sensing, small tip mechanical structure, and precision control of position and force. In high-sensitivity force sensing, eight optical fibers are attached to the metal at the tip of a precision control robot, and the direction and magnitude of the applied force are determined by measuring the wavelength of light that changes due to metal deflection. The realization of bilateral control using this technology is a world's first*. *based on our research