Sports & AI Project
Discovering the unseen charm of sports
Sony showcased a new technology for live sports production featuring sensors and AI
at the Consumer Electronics Show 2020 (CES 2020), an international technology festival.
The project leveraged Sony's unique proprietary pose-estimation technology and high-speed vision sensors with object-recognition
and tracking-process functions to perform real-time analyses of how the ball and players moved during a table tennis match.
That technological fusion opened up unprecedented visualizations, making it possible to
see the ball's flight path and the players’ positions and postures like never before.
The idea for the project originated from a proposal by in-house designers:
"What if we used Sony technologies to make sports broadcasts even more fun to watch?"
That question developed into a joint project uniting Sony designers and engineers, who retrace the trajectory of the effort in this eye-opening story.
Kawano,
Designer,
Sony Creative Center
Sugawara,
Senior Art Director,
Sony Creative Center
Abe,
Engineer,
Sony Semiconductor Solutions
Tahara,
Engineer,
Sony Imaging Products & Solutions
Miyazawa,
Senior Manager,
Sony Creative Center
Asai,
Senior Manager,
Sony Creative Center
Designers uncover the potential of technology
The Sports & AI project sprang from an in-house design-development event known as CARAVAN, which is organized by the Creative Center. How did the project start?
MiyazawaAs Sony designers, we do more than just make product designs. We also translate and edit different technologies available in-house into new experience value in a variety of fields. In our day-to-day work, engineers and researchers frequently consult with us about applications for new technologies in development. We have an in-house event called CARAVAN that started a few years ago to give designers a platform for innovation, an environment where they could actively propose ways to apply certain technologies to particular fields. CARAVAN is an outlet for collaborative creativity: designers explore in-house and external technologies, locate new potential, and then work with engineers, develop their ideas into actual prototypes that they can showcase to a wider audience.
AsaiOne of our exhibits at the 2019 CARAVAN was a prototype that visualized the movement of table tennis balls and players by fusing high-speed vision sensors with pose-estimation technology. We actually didn't start out developing the prototype all together as a team, though. Miyazawa was focusing on high-speed visual sensors, and I was focusing on pose-estimation technology. We were working separately on developing our designs. But along the way, we realized that we could put our technologies together into something even better.
MiyazawaThat's right. Industrial robotics uses high-speed vision sensors capable of shooting 1,000 frames per second, and I was trying to come up with a way to use those sensors to foster new experience value in areas like live sports coverage. I decided to go with table tennis because the players volley at high speeds. Then I asked Abe to work with me because he was the engineer in charge of developing the sensor. With his help, we started working independently on developing a design that’d track the path of the ball as it zipped back and forth.
AsaiOn my end, I was thinking about the future of sports broadcasts and got to wondering: would it be possible to transmit live video together with data on the movements of the players? Could we superimpose computer-graphics (CG) characters on top of them? That's the mindset I was in when I started working with Tahara. He'd developed proprietary pose-estimation technology that could recognize how players' bodies were moving and how their joints twisted—all at high levels of precision—without putting markers on their bodies. We started bouncing ideas off each other, exploring the possibilities of using player-motion data in sports broadcasting.
Along the way, we learned about the prototype Miyazawa was working on and had a flash of inspiration: watching sports could be even more fun if we could merge the two ideas. After combining the ball data from the sensors with the player data from the estimation technology, we used AI resources to visualize everything from the path of the ball to the stances of the players; the approach added a whole new layer to sports videos. That concept led to the creation of a single prototype, which we unveiled at CARAVAN. We received positive feedback from people there. I think the basic idea behind the prototype ties in well with Sony's company-wide purpose, too: "Fill the world with emotion, through the power of creativity and technology." Everything came together great, and we ended up exhibiting what we’d come up with at CES 2020 as the Sports & AI Project.
Expressing the power of technology
in a simple way
Sugawara and Kawano headed up the effort to polish the design for CES 2020. They worked with engineers to create and perfect an experience based on the prototype that they presented at CARAVAN, combining the high-speed vision sensors with the pose-estimation technology.
SugawaraCES 2020 drew broadcast professionals and video creators from all over the world. Since we knew we were going to exhibit our project there, our goal was to give the attendees a clear taste of what Sony's technologies could do and the potential they held for video production. To make that happen, we formulated an approach to highlighting a side of table tennis that people hadn't seen yet by creating a design around the extraordinary capabilities of high-speed vision sensors and pose-estimation technology. While we revamped the technology, we also interviewed people from the table-tennis world and worked with Abe and Tahara, two engineers, to enhance the prototype from CARAVAN. The visual presentation was another part of the effort, and Kawano focused on those aesthetic intricacies.
KawanoFirst, I had to figure out how to design the path of the ball as detected by the sensors. The sensors Abe developed can capture targets at 1,000 frames per second—not just your average speed, by any means. I decided to foreground that high-precision performance in a way that'd be as simple and straightforward as possible: fine, simple lines tracing the path of the ball. We assigned two different colors to the players to make it easy for viewers to follow the rally during the match, and we also showed the position of the ball bouncing on the opponent's side of the table. It was a visualization of the interplay between the two players, a brand-new vantage point on the action: following the course of the ball as it careened and bounced around. After speaking to people with table tennis experience, we also learned that spin rate and variations are key to victory. With that insight, we decided to bring the spin element into the visual expression, too.
AbeThe ball-path design Kawano proposed captured two focal points in clear focus: of the unique appeal of table tennis and the power of the sensors, capable of following the ball at high speeds, with high precision, and in three dimensions. As an engineer, I helped enhance the design, offering suggestions on showing the length of a path and showing the position of a bounce at the optimal time. As we bounced ideas off each other, we gave the presentation an even better finish. Even more impressive was how the visualization of the ball's spin came together. The designers suggested a visualization that'd merge the direction of the rotation with the coordinates of the ball, which really wowed me. I would've never thought of that by myself. That team effort helped us express table tennis.
KawanoThat's not all. One of the strengths of Tahara's pose-estimation technology is that it can even recognize the twisting of joints in three dimensions. In table tennis, the twists and turns of the players' arms and bodies define how the ball moves, so we decided to make the design highlight those kinetic factors. Normal motion capture depicts joints as points and bones as lines, but we decided to go with a different approach. Instead, our visualization would show joints in three dimensions and bones as flat plates—and the results would let viewers see the direction each part of the body faces when the player hits the ball. We had the ideas on paper, but it was hard to figure out whether they’d work in actual practice. I took several joint and bone design suggestions to Tahara and asked him to see if he could apply them to pose-estimation videos showing actual motion. It ended up working, and that’s how we were able to finalize the design.
TaharaAll the AI-driven R&D on pose-estimation technology has helped create plenty of pose expressions, but Kawano's idea of showing the joints in three dimensions and the bones as flat plates was a new one to me. When I applied his renderings to the videos, though, they really accentuated the strengths of our pose-estimation technology—the ability to show the twists in the poses and joints, along with the changes in angles. They captured the twists in the players' bodies just right, too. I think that kind of accuracy makes the visualizations viable as resources to help players improve their forms and other aspects of their game. As I got to see Kawano go about his designs whenever he came to the development area and obsessed over final design adjustments, I couldn't help but feel even more motivated as an engineer to put the finishing touches on the pose-estimation technology. I think we inspired each other—and that synergy propelled the project from the ground up.
Creating new, experience-enriching connections
between people and technology
How did the CES 2020 technological exhibit take place, and how did the visitors react? What kind of developments do you see on the horizon?
AsaiWe wanted visitors at our CES 2020 exhibit to experience the power of Sony’s technology intuitively, so we placed two monitors behind a table tennis match that served as a demonstration setting. The technology analyzed the players' rallies in real time, and one monitor showed their pose-estimated positions. The other monitor showed a synchronized display of the players' posture data and ball-path information, converted into CG characters in real time, to give people a glimpse of the project’s entertainment potential.
We also turned the ball-path analysis from the sensors into a movie to make it easier for visitors to understand. We played it during the match and also projected it onto a large screen above our display booth to attract attention. All in all, the event went over great with attendees. The Sports & AI Project booth was packed with broadcast professionals, and we got a lot of feedback from video creators saying they could see the potential of the technology for their own content, not just table tennis.
MiyazawaLooking forward, we're working on proposals to provide content in other fields such as basketball and golf by visualizing athlete positions via pose-estimation technology. There's also plenty of potential for applying high-speed vision sensors to different types of content. Our project obviously featured high-speed vision sensors and pose estimation technology, but the Sony Group has troves of other cutting-edge technology to work with. We'll continue to translate and edit those technologies through the CARAVAN in-house event so that we can create new value.
SugawaraIn order to create new value, designers like us not only have to know about technology but also need to keep their focus on people. We have to observe the world all the time to learn what grabs people's interests, what moves them. That's how we get personal foresight, which we combine with the right technology whenever the opportunity comes our way—and with that, we draw closer to creating fresh, compelling value. We’re user-interface designers, officially, but our job isn't limited to just designing the content that appears on product screens. Our real job is creating the interfaces connecting Sony technologies with people's lives. For me, the goal of creating new experiences through UI designs keeps me going.
The Sports & AI Project unlocked a new way to experience an athletic contest, augmenting the appeal of sports as entertainment.
Sony designers, together with engineers, will continue to reexamine Sony technologies in
new, exciting applications and offer solutions in a variety of fields.