Search button in the site




Beyond the limits of musical expression through technology
- Musical Dynaformics -
Sony Computer Science Laboratories, Inc.

Sony's logo is shown on a black background.

The piano is played.

On the background of the image of a man's hands playing the piano, Opening title, "Sony's Innovations & Challenges" and "Stories," and the subtitle, "Beyond the limits of musical expression through technology" and "Sony Computer Science Laboratories, Inc., "Musical Dynaformics"" are displayed.

In a room surrounded by bookshelves, a man in a black shirt is sitting in front of a piano.

On-screen text
Shinichi Furuya, Ph.D.
Sony Computer Science Laboratories, Inc.
"My goal as a researcher is to support the emergence of new creative expression.
If performers discover their own forms of expression,
it would signify that I've succeeded at my goal."

Outside of a building with wooden facades. The signboard hung on top of the glass entrance says "STEINWAY & SONS."

On-screen text
Courtesy of Steinway & Sons TOKYO

In a hall enclosed by wooden walls, a man wearing glasses is playing the piano.
He strikes his fingers strongly on the piano keys.

On-screen text
Kyohei Sorita
Studied in Russia under the recommendation of Mikhail Voskresensky
Entered with the highest score at the Moscow State Tchaikovsky Conservatory
Currently studying under Piotr Paleczny at the Fryderyk Chopin University of Music
Acted as main pianist for the character Sosuke Ajino in the anime TV series Forest of Piano
On-screen text
Liszt: Grandes Études de Paganini No. 3 ("La Campanella")

In a room surrounded by bookshelves, Furuya speaks.

"Piano performance requires fast and intricate playing skills.
However, it's hard to understand how to move your fingers."

The piano music fades out.

Sorita's hands playing the piano. He moves his fingers swiftly and changes his hand position constantly.
In a room surrounded by bookshelves, Furuya speaks with his eyes shining.
In a laboratory, Furuya uses his laptop with a serious look

"Over the centuries, people have said that you have to learn by watching.
But where exactly do you look? It's not obvious.
One big goal for our technology is to visualize and embody these skills,
so that the performer can actually "feel" them."


On-screen text
The goal of this research called "Musical Dynaformics"
is to expand musical expression by unraveling
the mystery of a musician's brain and body
and offering the optimal training.
Sorita and an amateur pianist take part in
a demonstration of Furuya's technology

In a laboratory, Sorita wears a black glove on his right hand.

"I wouldn't think there are sensors in here."

He also wears an armband-type electronic device around his arm and clenches and unclenches his fist.
The glove is made thin; on its back side, there are U-shaped protruding parts attached between the sheaths for fingers.
Each sheath has an opening and his fingertips (above the first joint) are exposed.
The palm part is made of mesh-like material and there is a hole in the middle.

On-screen text
Data Glove
Collects data on the movements
of the pianist's fingers

An exoskeleton-type black robot hand moves automatically. Its movement is the same as that of Sorita's hand wearing the data glove.

On-screen text
Reproduces the movements
on another person's fingers
"This is fantastic!"

An amateur female pianist who volunteered as a research subject wears the exoskeleton. She clenches and unclenches her fist and moves her fingers.

"The teacher puts on the glove.
And the student wears the exoskeleton.
The student's fingers then move in the same way as the teacher's.
You get to experience how the teacher plays intuitively."

Sorita plays the piano.
The woman's fingers wearing the exoskeleton move automatically in the air in sync with Sorita's finger movement.
Furuya says "This is how Sorita plays Double Thirds Etude."
Sorita plays the piano lightly and looks around.

"Let's say you want to create a softer sound like this.
It's hard to express how to do that in words."

Furuya is explaining.
The woman makes a big smile as she looks at her fingers being moved by the exoskeleton.
Sorita is interviewed.

"Furuya is thinking first and foremost about how we use our bodies to create good music.
He's on the performer's side.
We need to listen to what he's saying."

Some staff members are attaching some sensors under the piano keyboard taken out from the main body case.

On-screen text
Keyboard Sensor
Measures the piano key movements optically
"We can measure everything, even half-strokes, on the keys."

Furuya points at the piano.

"You're saying there are sensors inside?"
"Yes, we just put them in."

Sorita is surprised.

Sorita plays the piano quickly.
A value appears on the monitor placed next to the piano.

On-screen text
Number of keystrokes
"Wow, 22 strokes!"

Sorita plays the piano again. And looks at the monitor.

On-screen text
Number of keystrokes

The piano is played.

On a monitor, waveforms that undulate in linkage with keystrokes are displayed.
These waveforms are color-coded by keys and flow from right to left over time.

On-screen text
The keyboard sensor visualizes
the touches of the fingers,
showing how to play the music
"It's like you're pivoting on one foot in basketball,
with your finger in position, you go from "So" to "Re.""

Furuya explains with gestures to Sorita standing in front of the piano. Furuya leans to the right and bends down, and then stretches his body to the left.

Sorita plays the piano in a regular rhythm but fast-paced. Furuya points at the monitor. The monitor displays uniform and high frequency waveforms.

"Your articulation is so consistent, it's scary."

Furuya speaks in a room surrounded by bookshelves.
In a laboratory, Sorita receives an explanation of the waveforms shown on the monitor and plays the piano. He checks the waveforms.

"They say it takes 10,000 hours of practice to become an expert in any given field.
But in reality, the quality of that practice
and the way in which your teacher guides you matter more."

A staff member takes out a white exoskeleton-type robot hand.

On-screen text
Next-generation prototype
More lightweight and easier
to move fingers to more accurately
replicate keystroke movements
"Is this the first of its kind in the world?"

Sorita puts his right middle finger and ring finger through the exoskeleton's (next-generation prototype) sheaths and wears an armband-type electronic device around his right arm.
Furuya is explaining next to Sorita.

If you use it for a long time,
it's much easier to move the middle and ring fingers."

Sorita sits down in front of the piano again.

"I think I can bust out some Brahms then."

Furuya bursts out laughing. Sorita plays a melody.
He stares at the exoskeleton (next-generation prototype).

"This is wild."
"So what we want to do is measure and record your playing.
Then we can reproduce it."
"This can reproduce too?"

Sorita plays the piano with his two right fingers put through the exoskeleton's (next-generation prototype) sheaths using two syllables, "C (Do)" and "D (Re)."
The tempo gradually increases.

On-screen text

Sorita puts his right hand on the back of his left hand.
His two right fingers start to move automatically.

On-screen text
"Ah, this feels like me!
"-Sure does.
This is exactly how I play."

Sorita opens his eyes wide in amazement. He checks the feeling of his fingertips again.
A staff member operates a laptop that is cable-connected to the exoskeleton (next-generation prototype). Sorita's fingers move automatically again.

"When you're actually playing, you're not so aware of your movements.
So by replaying your own keystrokes back through the exoskeleton,
you can understand better than when you are actually performing."

The female research subject wears the exoskeleton (next-generation prototype).
It reproduces Sorita's finger movement that was recorded earlier.
Her two fingers move automatically.
She smiles as her other fingers and arm shakes by reaction.

The female research subject
"You all right?"
The female research subject
"I'm okay!"
"Sorita, your muscles make your keystrokes possible. But it's different for her."
The female research subject
"Wow…so powerful."

Sorita moves his fingertips only.

"I only moved my fingers, but your whole hand was moved."
"Yeah, it did."

Furuya speaks in a room surrounded by bookshelves.

"Practice time is always limited.
I'm hoping that this technology will allow people to practice efficiently
so that they can spend more time on cultivating artistry and expression."

In a laboratory, the female research subject plays the piano with some small instruments attached to her left hand.
Furuya is folding his arms and watching her from behind.
She nods as Furuya talks to her.
Sorita is interviewed.

"There are still many students who wear weights on their fingers
or stay in a room and practice for 20 hours at a time."

Furuya speaks.

"Another goal of our research is to completely eliminate focal dystonia among Japanese musicians."

An image of a pianist playing the piano in sync with a metronome.
Some of the pianist's left hand fingertips wearing a watch are extended outward.
The right ring finger of another person wearing a black shirt is curled into the palm unnaturally.

On-screen text
Musician's dystonia
A condition in which the body does not
respond to the brain's commands
when playing a musical instrument.
Estimated to affect 1 in 50
musicians worldwide

A photo of a woman playing the piano taken from the side. She has a blue electronic device strapped to her head with a headband.
A photo of a man playing the piano. He is wearing some electronics on his right arm and right hand as well as on his head.
A photo taken from above a man with two electronic devices strapped to his head with a headband. These electronic devices are sheet-like style. The one on the left is blue and the one on the right is orange.

On-screen text
Furuya is conducting research for
the prevention of dystonia,
focusing on logical approaches
to practice techniques
and prevention of injury in pianists

In a hall enclosed by wooden walls, Sorita is playing the piano.

On-screen text
Chopin: Largo in E-flat major. B. 109

Furuya speaks in a laboratory.

"I am extremely grateful that artists have been so welcoming toward the idea of using technology.
I believe our research is to preserve musical expression and artistic skill.
We need to pass on the performance skills and creative expression of
today's great musicians to the next generation.
If we don't, then the culture won't evolve.
Sony can use its various technologies to create an archive of artistic skill and creativity,
and use it to recommend the best practice styles.
That's how we can make a difference."

Sorita's hands are playing the piano. His right hand moves softly, and his left hand moves gracefully.
The female research subject is wearing the exoskeleton on her hand.
She is watching her fingertips moving smoothly with a smile.
Colorful waveforms are displayed on a monitor.
Sorita stares at his hand wearing exoskeleton (next-generation prototype).
In a hall enclosed by wooden walls, Sorita smoothly flies his fingertips over the piano keys.

In a hall enclosed by wooden walls, Sorita is playing the piano with a calm expression. The touch of his fingertips becomes progressively soft, as if he is making a more resonant sound.

In a laboratory, Sorita plays the piano quickly. As the music scale increases higher, the frequency of the waveforms shown on a monitor becomes higher also.

"Look at it climb."
"Man, this is fun!"

Movie ends with Sony's logo.