Reinforcement Learning Platform for Sony AI
Leon Barrett and the Sony AI team built the reinforcement learning platform that trained Gran Turismo Sophy, a superhuman Gran Turismo® Sport AI agent. This platform connected researchers to the large numbers of PlayStation® 4 consoles and GPU-powered computers they needed, allowing them to easily train and test AI driving agents. These AI agents beat some of the top human Gran Turismo Sport players, as described in our Nature paper.
Development of a new fast pixel readout architecture for flagship mirrorless cameras
The Sony Alpha 1 camera was introduced in early 2021. It has leaped the performance of its predecessor, the 24Mpix Alpha 9, in both pixel count and readout time. The Alpha One1 reads a 50.1Mpix frame at 4 milliseconds while keeping 14bit resolution. Noam Eshel was the key inventor and analog developer of a new pixel readout architecture, combining Sigma Delta ADC with a low noise sampling circuit. That fast readout is among the key technologies which enabled the success of the Alpha 1 and of other products having that technology in their sensor. This achievement has been announced at the 2021 International Solid-State Circuits Conference (ISSCC 2021) and in the IEEE Journal of Solid-State Circuits.
Development of the immersive audio technology Monopole Synthesis (MPS)
Monopole Synthesis is originally a sound source radiation model of Franck Giron’ PhD thesis. At Sony, he studied and patented its application for Spatial Sound rendering. He further developed it with his team in different real-time demonstrators for gaming and music. Its low computational requirements and speaker’s layout flexibility, including the use of wall reflections, facilitated its integration into different product categories. It is the Spatial Sound rendering component of Sony’s Home Theater products using 360 Spatial Sound Mapping (e.g., HT-A9 or HT-A7000), and of the Sound Building Blocks technology used in automotive (e.g., 2021 MAHINDRA XUV700), which were both developed in close collaboration with Sony Corporation team members.
Development of Legged Robots Enabling Robust and Highly Efficient Locomotion on Uneven Terrain with High Payload
Developed “Series-Parallel Elastic Actuator” which makes the quadruped robot climbs stairs with 20 kg payload and “Six-Legged Wheeled Robot” for further stability and energy efficiency. The results and technologies developed here are expected to be used to enable robots to safely and efficiently transport objects even in environments with uneven terrain, and to be applied to a variety of industries, from the entertainment field to the construction industry and last mile logistics. Part of this technology is being used in demonstration experiments at construction sites with Shimizu Corporation and was presented at IROS 2021 which is one of the top robotics conferences.
SLAM system development and adoption
Tatsuki Kashitani lead a team of researchers to develop visual and inertial sensor fusion technology called SLAM (simultaneous localization and mapping). It estimates the motion of a device and the 3D structure of the environment it is in. The technology serves as a basis for other vision and robotics algorithms. His contribution covers wide range of work from mechanical design of devices to software architecture design of the processing pipeline. Thanks to many collaborators in R&D, SSS, and SIE, it is ready for commercial use now. Professional-Use Drone Airpeak S1 is the first to adopt this technology to realize various intelligent user-friendly features such as wind-resistant hovering, obstacle avoidance, and return-to-home.
Development of 4K OLED Microdisplay for VR application
In the field of VR head-mounted display, we have developed a 4K OLED microdisplay (M-OLED) that achieves over 4 times higher resolution of a conventional panel with superior image quality. A novel device structure is applied in order to reduce its power consumption for practical use in wearable devices. As a result, it has contributed to the commercialization of M-OLED for VR.
Development of 3D autonomous robot software and commercialization of practical autopilot drone for aerial shooting
Dai Kobayashi developed a 3D autonomous flight software for the Airpeak S1 which is a drone developed primarily for aerial shooting use cases. The autonomous flight software, which is based on the Robot Operating System platform, is designed to be both safety and flight stability to achieve the concept of "Automated Flight Missions for Workflow Efficiency”. The airborne trajectories tracing and repeat flight introduced by the software enables to provide a new experience to professional creators for whom conventional autopilot was not suitable for their applications.
Development and commercialization of HT-A9 with revolutionary 360 Spatial Sound Mapping technology that delivers ultra-wide spatial sound in the living room
Natsuko Maeda led acoustic development and commercialization of HT-A9, a new concept four-speaker home theater system that allows users to experience ultra-wide spatial sound in their living room. This experience has been achieved by combining Sony’s unique spatial sound signal processing technology 360 Spatial Sound Mapping with acoustic transducers and mechanical structures that enable this technology to work ideally.
Contribution to higher image-quality for image sensors by establishing generation model of noise sources
Hideki Minari constructed a generation model of noise sources, and conducted early detection of a degradation risk of image quality and planning of countermeasures in the R&D phase. By analyzing noise signals with several types of semiconductor simulations, he identified generation mechanisms of noise sources. He also developed unique techniques to suppress the generation of noise sources jointly with Sony Semiconductor Manufacturing. This achievement has contributed to the improvement of the image quality of Sony’s image sensors.
Development of 360 Virtual Mixing Environment for remote sound production and contribution to filmmaking
Developed a technology (360 Virtual Mixing Environment) that accurately reproduces theater sound environment of a film production with over 300 seats using only headphones. It enables remote sound production and contributed to Sony Pictures Entertainment filmmaking, including “Ghostbusters: Afterlife” and “Venom: Let There Be Carnage”.
Reduction of design period and design cost for HDC series professional video production system cameras by designing new architecture
Shingo Nakano newly developed architecture of System Camera products which are used for sports, live music, studio recording, etc. and realized lineup on a single platform. This structure achieved a shortening of the commercialization period and cost reduction by reconstructing a huge amount of data flow and reflecting it to the products. Also, this structure improve separability of each functional block and include additional various sensors to serve the new needs of customer. These products have impressed people all over the world by shooting in various events such as large sport events.
Development of tactile sensors and adaptive grasp force control algorithm for a robot hand that gently handles a variety of unknown objects
Tetsuya Narita developed the world’s first(*) mathematical models for slip prediction in any direction for robot hands, and developed original tactile sensors and grasp force control algorithm. With this technology, the minimum necessary grasp force can be calculated autonomously even when physical information about objects is not available. Finally, he contributed to the realization of a manipulator that can grasp various unknown objects, including fragile objects (e.g., flowers and cakes), as gently as a human hand without prior learning. Unknow object grasping is essential for acquiring autonomy in unstructured environment. In addition to demonstrating the possibility of social implementation, his research contributed to enhancing Sony's presence in the robotics field through publication of papers.
*Based on our own research as of December, 2021.
Development of Stacked Event-based Vision Sensor (EVS) with the World’s Smallest Pixels
Atsumi Niwa has developed an EVS which detects scene changes with very low delay and low power consumption. By combining the low-noise device technology, wafer stacking technology, and high-performance circuit design technology cultivated in the development of image sensors, he has succeeded in achieving the world’s smallest 4.86 μm pixels and the world’s highest level of performance, and has commercialized the IMX636 and IMX637 for industrial equipment.
The Development of a 50M-pixel 14-bit 250-fps Back-Illuminated Stacked CMOS Image Sensor with Column-Parallel kT/C-Cancelling S&H and ΔΣADC and contribution for 120-frame/s high-speed “Black Out Free” shooting and Alpha 1 mass production
We developed a 50-Mpixel 14-bit 250-frames/s back-illuminated stacked CMOS image sensor on 35-mm optical format exhibiting 1.18-e−rms random noise at 0 dB, which achieves a high-performance characteristics extremely on both resolution and speed at the same time. New developing system based on the architecture can achieve 120-frame/s high speed “Black Out Free” shooting employed by Alpha-1. And these technologies reported on ISSCC2021 and JSSC were highly assessed internationally.
Realization of Dynamic Spectrum Access System Boosting New Generation Mobile (5G/6G, Wi-Fi, etc.) evolutions
Early attention is paid to the problem of the shortage of finite frequency (radio) resources, which is a fundamental issue in securing the convenience of wireless communications, he contributed to the legislation, standardization, and the social implementation of new ICT infrastructure technologies, dynamic spectrum access systems in Japan, the United States, and Europe. Continuing to work tirelessly on the three axes of legislation, standardization, and technology development, the concept of spectrum sharing, which had been said to be impossible, has been developed globally in areas with strong legal regulations and vested interests. In recent years, this achievement has received a lot of attention in the Japanese and overseas media, and contributed to greatly enhancing the reputation of Sony Group in the field of wireless communication technology. Secured a top patent position in the industry.
SkeleTRACK - A real-time skeletal tracking system for sports
James led the development of SkeleTRACK, a camera based real-time skeletal tracking system. The software uses fixed cameras installed around a sports stadium, and supplements them with unsynchronised and highly dynamic broadcast cameras, which the software can successfully utilize to improve redundancy and accuracy.
The software features a general player tracking framework that can use a form of multi-hypothesis tracking to improve past data as new information becomes available. It is also able to fuse information from a variety of sources including kit colours, player appearances and jersey numbers to automatically identify players on the pitch.
This system makes it possible automatically identify and track all players in a sports game and then use that data for officiating, statistics, augmented reality or full live virtual recreations.
World No.1 Sound Event Localization and Detection
Kazuki Shimada and his team have won the first place in sound event localization and detection (SELD) task of IEEE AASP Challenge on DCASE 2021, the world's largest competition for detection and classification of acoustic scenes and events (DCASE). To estimate accurately activity and direction of arrival of sound events such as human voices, footsteps and dog barks, they addressed new representation methods and data augmentation methods. As part of task organizers for DCASE2022, they published STARSS22 dataset with Tampere University. The dataset enables evaluation in real-world conditions. This contributes to DCASE community.
Contribution to de facto standardization of high-resolution audio wireless codec "LDAC"
Shiro Suzuki has standardized LDAC, which enables high-resolution audio transmission via Bluetooth, developed technical packages, and built a license framework. LDAC is licensed not only to Sony products but also to domestic and foreign audio makers and major automobile makers. The number of licensees exceeds 100, and it is becoming the de facto standard. LDAC has been certified as "Hi-Res Audio Wireless" by the Japan Audio Society. Recently, We have realized the installation of LDAC on WF-1000XM4 truly wireless headphones, and continues to contribute to improving the product appeal of Sony products.
Development of world “fastest” and “smallest” full-frame focal plane shutter by original mechanism.
Developed the new shutter actuator by combining an electromagnetic actuator and a spring. This dual-drive shutter was mounted on the flagship full-frame mirrorless camera Alpha 1, with the world‘s first flash sync speed of 1/400 sec.
And developed a new structure for an electromagnetic actuator. This world's first full-size electromagnetic drive shutter was mounted on the Alpha 7C, realizing a full-size mirrorless camera with a flat-top design.
Development of "XR OLED Contrast Pro"
Development of "XR OLED Contrast Pro," which achieves even higher brightness while utilizing the contrast characteristic of OLEDs through simultaneous emitting of all colors in WRGB.
Enabled extreme emitting by achieving precise control of each pixel while suppressing rapid temperature rise due to current concentration. As a result, it contributed to the enhancement of BRAVIA's product appeal.