Dec 3, 2024
Insights from a PlayStation Developer: The Past, Present, and Future of Real-Time Interactive Technology (1/2)
Teiji Yutaka, a contributor in the development of the original PlayStation®, has been deeply involved in the gaming sector for years. Recently, he has been at the forefront of research on cutting-edge technology for interactive entertainment, including games. His work spans a wide array of advanced technologies, such as photorealistic CG rendering using real-time ray tracing and UI technology that incorporates cognition-oriented intelligent information processing. We sat down with him to discuss his career and his passion for technology.
-
Teiji Yutaka
Distinguished Fellow
Sony Interactive Entertainment Inc.
From AI Research to PlayStation Development
──To start, could you tell us what led you to join Sony? What did you major in as a student, and what were your first responsibilities upon joining the company?
As a student, I majored in computer science, focusing primarily on AI (artificial intelligence). My research centered around the ambitious hypothesis that, if computational speed and memory capacity were unlimited, we could create an AI indistinguishable from a human. To test this idea, I worked on applications in automatic translation. I published a paper on this approach, which caught the attention of my professor, who encouraged me to explore neural networks. As I began to develop an interest in systems where computers learn autonomously, the time came for me to enter the workforce.
A senior from my university was researching AI at Sony's laboratory, which led me to join Sony as well. My initial role was in the contract development of sound chips for other companies' gaming consoles, utilizing Sony’s digital audio technology. The work involved sampling the sounds of instruments like violins and human voices to enable playback at various pitches—a revolutionary capability at the time. I developed an input system for composers to enter musical score data. Although I initially joined with an interest in AI, I was also passionate about rock music since junior high school. I was fascinated by Sony's audio technology, such as amplifiers and speakers, which kept me highly motivated in my work.
Following the sound chip project, my next task was developing a CD-ROM reader as a peripheral for ROM cartridge-based gaming consoles. This project aimed to maximize the potential of CD-ROMs, which offered over 100 times the data capacity of traditional ROM cartridges. Eventually, after navigating various challenges, we embarked on developing Sony’s new gaming console. This marked the beginning of what would become PlayStation.
Embedding Computational Power into Semiconductor Chips
──What are some of the interesting technical challenges you encountered during the early days of PlayStation?
When it came to the concept of PlayStation, Ken Kutaragi, the project leader, had a clear vision in his mind. At that time, films entirely created with computer graphics (CG) were just beginning to emerge, and he wanted to bring that fully CG world into gaming.
However, achieving a fully CG-based game presented immense challenges. Initially, I thought that by pre-rendering video backgrounds in CG and placing characters in front, we could create the impression of an interactive, moving background. This approach did indeed produce a fully CG game, but it felt too simplistic. From the player’s perspective, it was almost like you could defeat enemies by just shooting. It made me question whether such a game would truly be engaging.
This led to a change in thinking: we decided to create everything on the screen in real-time CG. For smoother motion, real-time CG requires at least 30 frames per second, meaning each frame must be generated in 1/30 of a second. The smallest unit in a 3D world is a “polygon,” which consists of three vertices, and combining many polygons allows for creating detailed objects. The question then was, how many polygons and computations would be needed? Taking the first demo we created for PlayStation as an example, the dinosaur model was made up of 2,700 polygons. To project a moving dinosaur onto the screen, we needed 243,000 vertex calculations per second (2,700 polygons × 3 vertices × 30 frames).
The First Demo Dinosaur Created for PlayStation
At the time, workstations capable of such processing power cost tens of millions of yen, so recreating this on a home console was a major challenge. Mr. Kutaragi’s idea was to embed the vertex calculation capabilities into a semiconductor chip within the CPU, making mass production affordable. This led to the development and integration of the geometry engine*, which became the key breakthrough and one of the foundational technologies for the PlayStation series.
※Geometry Engine: Software or hardware specialized in coordinate transformation in 3D graphics
While I was directly involved in development up until PlayStation 3 (PS3®), my focus as a software engineer was to provide a user-friendly development environment for game developers. I managed the 3D graphics library, delivering the 3D software library as an API for developers.
The techniques and principles I honed here eventually paved the way for my current specialization in interactive technologies like real-time ray tracing. Currently, I leverage my experience to shape the vision and direction of Sony’s digital interactive technologies. My current interest lies in examining the game industry from the perspective of real-time 3D graphics and understanding how new technologies will transform gaming.
The Evolution of Real-Time Ray Tracing
──Could you tell us the appeal of real-time graphics, real-time ray tracing, and UI technology, as well as your vision for their future?
Games are essentially content that generates scenes and behaviors in real-time. With the evolution of computational power, from the original PlayStation through to PlayStation 5 (PS5®) and beyond, the increasing processing capability allows for more extensive simulations, bringing us closer to true realism.
Let’s look at the evolution of real-time ray tracing as an example. A "ray" refers to a light beam. By processing the light source and object properties, we render 3D graphics on a display. Traditionally, textures on polygons were processed individually to determine colors. In ray tracing, countless rays are simulated by tracing them from the light source, calculating how light bounces, and determining how it reaches the viewer’s eye. This process requires extensive calculations, but by sampling light rays and reducing them to a manageable number for computation, we can simulate reflections and refractions accurately. This makes it possible to achieve realistic and beautiful 3D graphics that align with character movements.
Countless Rays from the Light Source to the Eye
It is anticipated that if computational power increases by tens of thousands of times from current levels, real-time 3D graphics could achieve visual representations indistinguishable from reality to the human eye.
Beyond advancements in computational capabilities, ongoing efforts aim to minimize image degradation. A promising approach involves leveraging AI. For instance, when the number of rays is limited, images can exhibit noise similar to photographs taken in low light. AI-driven denoising techniques have been developed to address this issue. Additionally, PS5 Pro, released in November 2024, features PlayStation Spectral Super Resolution (PSSR), an AI-driven upscaling technology that enhances image resolution by adding an extraordinary amount of detail.
However, certain elements remain challenging to replicate with current technology, notably human beings. Accurately rendering individual strands of hair, subtle movements, and nuanced facial expressions still require improvement. Since humans play a crucial role in games, even slight unnaturalness can trigger the uncanny valley phenomenon, leading to discomfort and detracting from the experience. Achieving realistic human representation and enabling natural control are among the primary challenges we face today.
