Jul 24, 2023
Sony’s character conversation AI technology
Conversational AI technology for interactions with characters
Character conversation AI technology
People wish to talk freely with their favorite characters, create AI characters on their own and provide various services using AI characters.
At the Sony Group R&D Center, we are working on the character conversation AI technology to make these dreams come true. Interacting with characters in a more realistic way requires not just natural language processing technology for language understanding and generation but many other underlying technologies as well, including voice signal processing technology for voice recognition and speech synthesis and image processing technology for image recognition and graphic expression.
Deliver exciting experiences by combining the technologies to turn creators’ imaginations into reality with the Sony Group’s entertainment prowess - that’s what we are developing the character conversation AI technology for. In the context of meeting this goal, we are proposing new values using this technology through open demonstration experiments and application development activities.
Sony’s conversation control technology
In building AI characters, we use scenario-based conversations in combination with machine learning-based conversations, which helps realize a wide range of conversations that script writers want to present. First, professional writers create scenario-based, emotionally charged conversations that emphasize the personalities of their characters. Then, machine learning adapts those conversations for a broader range of topics, without ruining the characters’ traits.
Machine learning-based conversations
Scenario-based conversation control is working enough to create AI characters in a small amount of conversation scenarios. Our technology, however, needs to deal with diverse colloquial expressions and many different topics and therefore mixes scenario-based conversations with responses generated by deep learning (DNN). Based on the models learned from massive amounts of conversation data we have gathered on our own, the technology adds and learns the lines that the characters will likely speak. This enables the efficient generation of conversations that reflect the personalities of the characters.
AI remembers conversations with users
We consider “mutual understanding” crucial for vitalizing communication between AI characters and users. As an approach to mutual understanding, we offer a feature for AI to remember the data about users’ preferences and lifestyles (user models) and apply those data for conversations. Our technology uses this user model-based conversation mechanism for conversations with characters.
Business case study of the character conversation AI technology
AI entertainment service “SOKUBAKU KARESHI (Possessive Boyfriend)”
Character of SOKUBAKU KARESHI - Aki Shindo
* The screens presented here are those currently under development and may differ from the screens of the final product. ©Sony Music Solutions Inc. All rights reserved.
Interactive character agent “CHELULU”
©Sony Music Solutions Inc. All rights reserved.
CHELULU remembers the faces of people through face recognition and is able to communicate in a way
that suits the individual it speaks to.
Open experiment of the character conversation AI technology
SWORD ART ONLINE EX-CHRONICLE
AI conversation experiment with Takeru Higa on AI creation and personality development
The video shows conversations between AI reflecting the personality of each individual participant and Alice, a character from the animation.
©2020 REKI KAWAHARA/KADOKAWA CORPORATION/SAO-P Project
©BANDAI NAMCO Entertainment Inc.
A CG model from SWORD ART ONLINE Alicization Lycoris, a home game from BANDAI NAMCO Entertainment Inc., was used.
Participants posted their desired conversations via the Web in advance, and these conversations were used to create AI that reflected the personality of each individual participant. At the event, the conversations between the AI of participants and Alice were demonstrated according to the world view of SWORD ART ONLINE. The participants saw AI speak just like they do, even on topics their posts did not mention, making them feel as if they were in the world of SWORD ART ONLINE.
Live concert by 22/7 (a.k.a. Nananiji)
Surprise event at a regular performance: Sakura Fujima welcomes visitors
We exhibited a demo in which Sakura Fujima welcomed visitors to her group’s regular performance and souvenir shops, saying “Thank you for coming!” In this surprise event, the visitors responded to Sakura’s greeting and she reacted to their response, and their conversation went on and on.
Here is a video showing a demo conversation of our system between visitors and Sakura Fujima.
©22/7 PROJECT Sakura Fujima (22/7) CV: Sally Amaki
Related Link:
- The SOKUBAKU KARESHI (Possessive Boyfriend) project using Sony’s latest conversational AI technology started on Twitter on March 25 (Thursday).
- The conversational AI service “SOKUBAKU KARESHI: Aki Shindo” received the first Kyoto Animation Monozukuri Award in the Technology & Innovation category.
Related Publications:
Silin Gao, Beatriz Borges, Soyoung Oh, Deniz Bayazit, Saya Kanno, Hiromi Wakaki, Yuki Mitsufuji, Antoine Bosselut.
PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives.
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023. - Outstanding Paper Award
---
Silin Gao, Jena D. Hwang, Saya Kanno, Hiromi Wakaki, Yuki Mitsufuji, Antoine Bosselut.
ComFact: A Benchmark for Linking Contextual Commonsense Knowledge.
Findings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
---
Chiaki Miyazaki, Saya Kanno, Makoto Yoda, Junya Ono, Hiromi Wakaki.
Fundamental Exploration of Evaluation Metrics for Persona Characteristics of Text Utterances.
Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 2021.
