Product development of the video AF(autofocus) system using phase detection image sensor that provided a new shooting experience for creators
Yuki Asukabe led the development of a high-speed and highly stable video AF system by appropriately converting the advantage of the focal plane phase detection sensor to the commercial value of cameras. When his project started, manual focus had been frequently used in professional video shooting, but Yuki Asukabe and his colleagues discovered potential customers’ needs for using AF and worked on the development by gaining a deeper understanding of video creators on sites. Now, Sony’s AFis highly supported by many creators. His team contributed to providing new video shooting experience and Sony’s business growth as the video content industry is expanding for advanced visual expression.
Development of Σ (Sigma) graphite heat dissipation structure that realizes both Anti-shake performance and longtime video recording for alfa-body
Hiroshi Abe developed a new heat dissipation structure, achieving both high-efficiency heat exhaust from image sensor and low-load drive for in-body image stabilization. By thoroughly reviewing the heat dissipation structure from the conventional method, he mechanically connected the heat dissipation structure to the image sensor driving inside the camera with low mechanical load, and developed Σ (sigma) graphite heat dissipation structure, which is about 5 times higher in heat dissipation performance,(*) to apply to this structure. This new mechanism greatly expanded the recording time of 4K/8K video and achieved 5.5-step in-body image stabilization performance at the same time. This technology is integrated into the full-frame mirrorless cameras Alpha 1 and Alpha 7S III, and the Cinema Line camera FX3. It has contributed to strengthening the competitiveness of Sony products.
*Based on Sony’s testing
Development of system architecture for camera and commercial realization to camera products
Satoshi Ikuta developed a new system architecture for the Alpha series cameras. The new system enables up to about 8 times* the processing capacity compared to the past models. The load segmentation of "real-time" processing for AF and image recognition and "best-effort" processing for User Interface and Network is optimized to maintain fast control response even when the real-time processing is heavily loaded. This system architecture has been introduced to the full-frame mirrorless camera Alpha 7S III and the Cinema Line camera FX6, and it has contributed to enhance the brand power of the Alpha series and Cinema Line cameras.
*Based on Sony’s testing
World's First Intelligent Vision Sensors with AI Processing Functionality
In recent years, the vision AI market has been expanding and much attention has been paid to the possibility of in-camera AI processing. Under such a situation, Ryoji Eki and his team developed a processor dedicated for convolutional neural network (CNN) and commercialized an intelligent vision sensor “IMX500” and “IMX501” as a single chip by combining the image sensor stacked configuration and the Cu-Cu connections. This configuration has allowed smaller size cameras. Also metadata output allowed lower latency, while at the same time addressing privacy concerns. It promises to be one of core devices for Sony’s imaging & sensing solution business in the future, and it can be expected to contribute to power consumption reduction due to reduced data size transmitted to the cloud.
Development and launch the DualSense wireless controller for PlayStation®5
Kazuyoshi Enomoto developed and launched the DualSense wireless controller for PlayStation®5 (PS5). He developed new features, such as haptic feedback with precise expression, adaptive triggers that generate reaction force on trigger buttons, and built-in microphones for easy communication, as well as hardware architecture and dedicated semiconductors in order to implement these features at low cost. The next-generation gaming experience with the DualSense wireless controller has been highly valued, not only by users, but also by game creators, and has contributed to increasing the value of the PlayStation® brand.
Cooling design for PlayStation®5
For PlayStation®5 (PS5), Yasuhiro Ootori and his team developed an unprecedentedly powerful cooling system that can accommodate the heat-generating SoC, high thermal density due to the small die size (IC chip size), and constant TDP (thermal design power) operation while gaming. By adopting liquid metal and improving heat-receiving capacity with rectangular heat pipes, his team was able to mass-produce a cooling system that satisfied these requirements. Their achievement significantly contributed to the low-cost design of PS5, accommodating production volume, and enhancing the product's appeal with low noise design.
4K120p/8K output development for PlayStation®5 by HDMI2.1 FRL
Taichi Okabayashi led the development to enable PlayStation® 5 (PS5) to output 4K120p/8K video using Fixed Rate Link (FRL) signaling technology. This technology is newly defined in the HDMI 2.1 standard, which can transmit audio, video, and control signals together with a single cable. Thanks to his effort, PS5 became the first HDMI 2.1 FRL source device to be certified in the industry. He accelerated the development of the HDMI 2.1 FRL compliance test specification and conducted verification to improve connectivity. These activities have greatly contributed to the launch of PS5 with 4K 120p/8K output.
Signature series/Near Field Powered Speaker System SA-Z1
Yoshiaki Kaku developed the near-field powered speaker SA-Z1, which realizes an unprecedented innovative music experience in personal space such as desktop. Generally, listening to speakers can give the user the feeling of being in a real concert hall, but in order to enjoy it with truly good sound, a dedicated listening room with well-tuned tones is necessary. Listening with headphones, on the other hand, can deliver high-resolution sound, but has limitations in reproducing the feeling of being in a concert hall. To overcome these problems, Kaku worked on developing the near-field listening technology. This device can be simply connected to the music source device with a USB cable and achieve both the immersive feeling of being in a live concert and the high-resolution sound as if the users were listening with headphones.
Machine Learning based Game AI control for non-player character (NPC)
Kenta Kawamoto initiated and led the project to develop an in-game autonomous agent (NPC) that can outperform human expert players by applying machine learning technology to its real-time motion control in nonlinear dynamics environment. This remarkable achievement using reinforcement learning as a core technology is expected to be utilized to acquire superhuman control skills, even beyond games, for real-time systems with real-world complexities in the future.
Development of New Architecture for SPAD Depth Sensor used in Automotive LiDAR and Contribution to Commercialization
Oichi Kumagai led the development of a stacked direct Time-of-Flight (dToF) depth sensor for automotive LiDAR using SPAD (Single Photon Avalanche Diode) pixels. By utilizing back-illuminated pixel structure, stacked configurations, and Cu-Cu connections that Sony has cultivated through its CMOS image sensor development, he combined SPAD pixels and distance-measurement processing circuit into one chip, which is compact yet ensures high resolution. As a result, it achieves high-precision and high-speed measurement for a distance of up to 300 m at 15 cm intervals. This result was announced at the 2021 International Solid-State Circuits Conference (ISSCC 2021). It is greatly expected that this sensor will be commercialized and contribute to new automotive business in the future.
Establishing binary level backward compatible GPU hardware for PlayStation®5
Junichi Sakamoto and his team developed a backward compatible GPU for PlayStation®5 (PS5), establishing next-generation performance and features by implementing Ray Tracing hardware and high computational capabilities to perform about 10TFlops, 5.6 times faster than PlayStation®4 (PS4). Its backward compatibility allowed PS5 users to enjoy PS4 game titles from when PS5 launched. In addition, his team achieved higher performance with most of the titles at almost double speed, called Boost Mode. Their accomplishment has enabled PS5 users to enjoy higher frame rates and resolution as a new game experience.
Development of Universal Audio Source Separation Using Deep Neural Networks (DNNs)
Stefan Uhlich and his team developed Universal audio source separation to help create new customer experiences that would have been impossible without it.
This source separation technology is the task of separating audio into its constituent sound sources, e.g., for music into individual instruments or for movie audio into individual foleys/effects, dialogues and music.
For example, using the movie separation, an upmixing of the movie audio in Dolby Atmos creates a new, revived version of the original movie audio. This was used by Sony Picture Entertainments to create the Columbia Classics 4K Ultra HD Collection. Another example is a piano extraction of a performance by Glenn Gould. Using the extracted piano track and mixing it with a new Japanese speaker allowed Sony Music Entertainment Japan to release a new CD as if Glenn Gould would still be alive.
Development of X-Balanced-Speaker Unit that realize high sound quality
Takahisa Tagami newly developed X-Balanced Speaker, which realizes high sound pressure and high sound quality within size constraints. It has been adopted for Sony’s wireless portable speakers (SRS-XB43, SRS-XB33, and SRS-XB23) and BRAVIA TV series (X9500H, X9000H, and X8000H). This device employs a new-shaped diaphragm that reduces sound distortion to achieve performance. It contributes to improving the product appeal of audio equipment with impressive bass power and high clarity sound.
Standardization of Video-based Point Cloud Compression
Point Cloud is a 3D media data format that enables the production of high-quality visual content, and it requires a large amount of volumetric data. Point clouds can be compressed by being segmented and packed into 2D images using orthographic projections, which will then be compressed with a 2D encoder. Danillo Bracco Graziosi and his team proposed a new packing strategy, which was incorporated into the first international standard for compression of point clouds (the Video-Based Point Cloud Compression, V-PCC). The adopted packing strategy rotates the projected segments using 8 distinct orientations, providing computational efficiency, memory savings, and higher compression. The adopted proposal is a part of a large set of tools in the new international standard, which allows existing 2D systems to create, distribute and consume 3D assets.
Fyber - A standalone hair creation and grooming software for 3D characters
Daniela Hasenbring, alongside key production team members, developed a new hair and fur grooming software for 3D characters called Fyber. It is in active development and is being used on all animated feature and visual effects shows at Sony Pictures Imageworks. As a node-based, standalone software, Fyber addresses the need for a faster, more interactive and more artist-friendly tool to generate any type of hair, ranging from a character's head hair to fully furred animals, with the goal to significantly lower grooming times and create an improved learning experience for new artists.
Fyber offers great flexibility to achieve any look an artist might desire while also providing fast visual feedback thanks to its highly multi-threaded computation graph and integrated OpenGL and Arnold viewports.
Fyber's underlying engine is fully separated from its user interface which allows for a seamless and easy integration into third party applications like Maya and Katana where it can be used to compute hair on the fly or to render it to their respective viewports.
Development of new clock generation architecture suitable for advanced CMOS process
Masahisa Tamura developed a new clock generation architecture, which is well-suited for integration with advanced CMOS nodes and therefore contributes to its commercialization. All-Digital Phase-Locked Loops (ADPLLs) show a high performance at a low power consumption, low voltage operation, and with small space requirements—features that were hardly ever achieved with conventional analog phase-locked loops (PLLs). This technology has already been adopted for several LSI products, and it is positioned as one of the important core technologies for Sony’s imaging and sensing solution business. He also designed this ADPLL for Bluetooth® Low Energy, which operates with a low power supply of 0.5V and consumes just 1mW, resulting in the RF transceiver functionality with ultra-low power consumption of 1.9mW. This achievement has been announced at the 2020 International Solid-State Circuits Conference (ISSCC 2020) and in the IEEE Journal of Solid-State Circuits.
Standardization of Geometry-based Point Cloud Compression
As an editor in the MPEG standardization activities, Ohji Nakagami worked on the development of the international standard for point cloud video data compression and transmission technology, and contributed to releasing ISO/IEC 23090-9 (Geometry-based Point Cloud Compression, or G-PCC), which is one of the world’s first two international standards (ISO/IEC 23090-5 and -9) for the point cloud compression technology. Since G-PCC has high compression performance, it is expected to be applied to virtual production and volumetric video capture in 3D sensing where huge data size is an issue to resolve.
Development of Vibrotactile & Force Feedback Technology and Contribution to DualSense Wireless Controller
Yusuke Nakagawa and his team developed the vibrotactile feedback technology that can provide users more realistic vibration and the force feedback technology that can provide different types of feelings to fingers through the trigger buttons. These technologies are used for haptic feedback and adaptive triggers in DualSense wireless controller for PlayStation®5. Nakagawa was in charge of a wide range of activities, from developing elemental technologies such as vibration actuator control methods and vibration waveform design tools, as well as force feedback device structure and its control methods, to creating new user experiences with game audio and video combined.
Development and Commercialization of ID7000, a Flagship Spectral Cell Analyzer that achieves high-plex multicolor analysis
Koji Futamura led the project to develop the ID7000, the flagship model of spectral cell analyzer, devoting himself entirely from product planning to commercialization. Bringing together Sony's own state-of-the-art spectral technologies and utilizing up to 7 lasers, a total of 186 channels of detectors, and analysis algorithms based on the weighted least squares method, the ID7000 has realized high-plex multicolor analysis, in response to the strong demand from cutting-edge life science, medical, and drug discovery research for a comprehensive study of all characteristics and states of cells at once. Since its launch, the system has been adopted by the world's most advanced research institutions and companies. It is contributing to Sony’s life science business as a major product.
Development of AI powered driving behavior-based car insurance “GOOD DRIVE”
Ryosuke Furukawa developed a new type of driving behavior-based car insurance called “GOOD DRIVE,” utilizing an AI incorporated into the smartphone app, which measures driving characteristic data dynamically changing. It is a very unique financial product that uses Sony’s technology as a core and offers up to 30% of premiums as a cash back to drivers with low accident risk. In order to achieve this system, he gathered the Sony Group’s various assets, such as AI/cloud data science knowledge and sensor signal processing technology. This achievement allowed Sony to show its excellent synergy with diverse portfolio including financial services business.
Development of the PlayStation®5 Ray Tracing Technology
Benjamin Segovia, along with his teammates, developed a flexible and powerful ray tracing software stack for PlayStation®5. It includes state-of-the-art and innovative routines for ray tracing data structure management and traversal, rendering strategies, and major new shader compiler techniques to make efficient use of the hardware. It was used in several major launch titles, with competitive performance, and showing key “next generation” differentiation.
Real-time data platform to unlock growing needs for low latency within the music industry
Complex rights models have made scattered data a long standing issue within the music industry. Myron Machado and his team built a reliable, scalable and cost-effective data platform to power low-latency front-end and mobile applications. A real-time event streaming architecture combined with a graph-based data model, made it possible to enrich artist personas while allowing for a massive improvement in data quality and freshness. This platform makes it possible for independent music artists and record labels to get a comprehensive view of music consumption and social media data on the go, making it easier for them to better analyze and market.
Development and Mass Production of Highdefinition SWIR Image Sensor using Cu-Cu connections
Shuji Manda has developed an innovative manufacturing architecture and achieved its commercialization in the field of SWIR (Short-Wavelength Infrared) image sensors that uses a compound semiconductor as a photoelectric conversion part layer. This new architecture fuses compound and CMOS image sensor technology by dicing small compound wafer into chips and stacking them on a large silicon wafer. It also adopts the world’s first technology stacking dissimilar materials compound and silicon using Cu-Cu connections. This technology created the new high-definition SWIR imaging market with overwhelmingly higher image quality, compact size, and high sensitivity in a wide range.
Contribution to commercialization of Spatial Reality Display ELF-SR1 with development of lenticular lens design suitable for eye tracking system
By developing a lenticular lens design technology suitable for eye sensing, Tomoya Yano and his team have contributed to the development and commercialization of the Spatial Reality Display ELF-SR1, which can reproduce three-dimensional spatial images as if they were really there. The Spatial Reality Display detects the positions of the viewer's eyes and displays images so that the objects would be visible where they should be from the viewer's position. It can naturally express the depth of objects and provide the highest reality video experience ever for viewers.
Realization of PlayStation®4 BC using GPU analysis technology in PlayStation®5
Kenichiro Yokota and his team have been engaged in building a GPU problem analysis system since the early phase of PlayStation®4 (PS4) development and succeeded in solving incompatibility issues by applying this system in PlayStation®5 (PS5) development. This achievement enhanced the backward compatibility of PS5 while ensuring high quality of the GPU and made it possible to play PS4 titles on PS5. It has greatly contributed to creating an enjoyable gaming environment where users can play both PS4 and PS5 titles using a single PS5 console.
Contribution to the commercialization of Spatial Reality Display with novel technology for better binocular fusion
Kazuki Yokoyama developed a prototype of the Spatial Reality Display ELF-SR1 and supported to bring this prototype into the final product for commercialization. This display system detects and calculates the positions of the viewer’s eyes, and displays a real-time image on the display panel as if it appeared right in front of the viewer. He devised the device’s form, including the positioning of the panel and arrangement of objects as comparative clues, and established a display device that helps anyone easily see 3D images.