Technology that Differentiates Sounds

The ability to distinguish different instrument sounds within a musical performance comes very naturally to most people, as does our ability to focus on the voice of someone we're talking to, even within a crowded and noisy environment. Despite this, replicating this ability with technology proved to be a daunting task, compared to "mixing two drinks together and extracting the flavors of only one."
Sony was among the first to implement deep learning in sound separation technology in 2013 and has continued to dramatically improve the technology's performance ever since. Now, neural networks (a type of machine learning model) are able to analyze the characteristics, frequencies, and changes over time of sounds produced by a guitar, which then allows AI to recognize that sound even when combined with several other audio tracks.

Sony's sound separation technology is already being used across the entertainment space, notably in film. Many older films survive with only a single audio track combining dialogue and sound effects, making it difficult to remaster the film with spatial audio. Sound separation, however, made it possible to extract individual sounds from the original masters when producing the 4K UHD versions of Lawrence of Arabia and Gandhi. Mixing engineers then reconstructed those sounds in a spatial audio format to generate an immersive sound field.
The technology has also been used to remaster animated works, such as Mamoru Oshii's 1985 animated film Angel's Egg. Sound separation was used to produce a 4K remastered version of the film, which premiered in the Cannes Classics section of the Cannes Film Festival in May 2025 and was released in theaters six months later in November.

Sony's consumer products also use sound separation technology to great effect. AI beamforming technology incorporates sound separation as a core element within Sony's headphones for enhancing voice sensitivity, as the technology allows headphones to clearly isolate and transmit the user's voice during calls, even in noisy environments. The technology also allows users to enjoy stereo audio as three-dimensional spatial sound when used in select soundbars.

In recognition of Sony's advanced initiatives in sound separation technology, Yuki Mitsufuji and Shusuke Takahashi of the Ethical Creative & Protective AI Lab. at Sony Group Corporation were awarded the 26th Yamazaki-Teiichi Prize, which honors creative achievements that lead to practical applications. Upon receiving the prize, Takahashi said "It is a great honor and deeply moving moment for us to receive the prestigious Yamazaki-Teiichi Prize. We've been steadily working to create something that can bridge the gap between R&D and, for example, movie theaters, living rooms, and the devices in everyone's hands. Nothing pleases me more than having our efforts recognized in this way. And we could only have done it with the help and expertise from engineers and specialists across many different fields, including film, music, and product development."

A Continually Evolving Technology

Sony's sound separation technology still continues to evolve. Though sound separation is often used to divide music into a set of predefined elements, Sony has succeeded in isolating single elements from a complex audio source produced even by an analog synthesizer.

Since music is composed of melodies, chords, sound effects, and many other audio elements, cues are necessary for extracting individual elements from a piece of music. Sony's newly developed model can use information from a music score, such as a melody or tone, as cues to extract elements from the music. By providing an approximation of the desired sound to be extracted, the system is able to pinpoint the correct sound and isolate it.

Since tones produced by an analog synthesizer are similar in quality to each other, isolating individual elements from synthesizer music is an extremely challenging task. Sony made this technology possible by first building a large dataset simulating the tone of analog synthesizers, which was then used to train AI.

Supporting New Forms of Creation

By making it possible to isolate individual elements from synthesizer music, Sony's sound separation technology enabled new forms of creation by internationally acclaimed sound artist evala at Ars Electronica Festival 2026, a world-renowned media art festival held in Austria this September.

After receiving the Prix Ars Electronica Isao Tomita Special Prize in 2025, evala chose to express his deep respect for the prize's namesake, famed Japanese composer and pioneer of electronic music Isao Tomita, in the new work he would present as an invited artist for the event. Through Sony's sound separation technology, evala was able to extract ever finer samples from Tomita's famed 1974 analog synthesizer arrangement of Claude Debussy's Clair de Lune. He then incorporated portions of these samples, like inherited pieces of DNA, into his performance installation piece Holy Echo.

Exhibited at the St Mary's Cathedral (Mariendom) in Linz, Austria, the installation was placed within the vast space of the cathedral, which stands approximately 134 meters high and extends about 130 meters in length. The installation drew attention in particular for incorporating the cathedral's spatial and contextual qualities as acoustic elements of the work.

Chihiro Nagashima of the Ethical Creative & Protective AI Lab. at Sony Group Corporation, who provided technical support for the exhibit, reflected on the project's challenges by saying, "Separating the countless tones produced by a single analog synthesizer tone by tone, element by element, was incredibly difficult. We had to start by even defining our audio resolution and its smallest unit of separation."

Regarding the installation's success, however, Nagashima says, "by bringing together our research team's combined expertise and thoroughly discussing ideas with evala, we were able to develop a creative, never-before-seen sound separation technology. Being there to witness the moment a new form of expression was created with this technology reminded me of how important it is for creators and R&D members to work together on new technologies."

Speaking about the future of sound separation technology, Mitsufuji explains, "This technology creates new possibilities for both reimagining and restoring past works of art." He adds, "We want to construct an environment where creators can confidently take on new forms of expressing their creativity by making new audio technologies available as opportunities to be taken rather than threats. In doing so, we're able to reimagine past works of music as new next-generation experiences."

Sony will continue to leverage technology to create moving entertainment experiences in collaboration with artists and creators.