Prototyping narrative representation system using a Kabuki dance and legendary story for the narration function of robots

1.Introduction

Some thinkers and philosophers state that communal and personal fiction and narratives play important roles in human societies and mentalities. For example, Takaaki Yoshimoto (1968) presented the idea of “communal illusion” to claim that various social institutions are supported by the people’s common illusion or narrative. Additionally, Yuval Noah Harari (2014) discussed the human ability to construct fiction and narratives as the background or basis of their historical progression. We also believe that humans are essentially animals who are capable of imagination and fantasy. Additionally, according to Schank (1986), Minsky (1988), and Mueller (1990), emotions and stories play an important role in human problem-solving activities. In our research framework, “narratives” are the most powerful tools for humans to organize human imagination and fantasy, as well as emotion, to imaginatively simulate diverse aspects of human life. Therefore, robots, including house companion robots (Kheng, Dag, Kerstin & Michael, 2020), which possess artificial intelligence to be able to interact with humans, must maintain a narrative ability-related imagination. In the future, storytelling robots may be required.

Computer-based narrative generation (Ogata, 2019c; 2020a) has been previously researched. Love, sex, and sexuality are often the most important topics in a story. The themes of love and sex appear as representative narratives and literary works in every age. For example, Genji Monogatari (The Tale of Genji) in ancient Japan and Decameron in medieval Europe treated themes deeply related to love and sex in various hierarchies.11 In the nineteenth century, Madame Bovary by Gustave Flaubert dealt with immorality, and many novels by Soseki Natsume of Japan frequently describe the love triangles between two men and one woman. In the twentieth century, A Farewell to Arms by Ernest Hemingway and Haru no Yuki (Spring Snow) by Yukio Mishima drew a picture of pure love between a woman and a man. Additionally, extraordinary and extreme love and sex have been endorsed, including in the works of Marquis de Sade, Chikamatsu Monzaemon, Yukio Mishima, and Georges Bataille. It is necessary to include such subjects in the stories told by robots in the future.

Recognizing the importance of narrative for humans, as mentioned above, the goal of our studies, including this study, is to automate narrative generation. Specifically, we aim to present the concept of “narrative content works,” including an automated narrative generation function that did not previously exist, and develop a system based on this framework. Furthermore, in one of our future studies, we hope to incorporate the developed functions into robots. Wicke and Veale (2018a) and Meza et al. (2016) introduced technologies focused on the relationship between robots and narrative generation or storytelling. In addition to the study of storytelling Veale (2017), research on human–robot interactions Wicke and Veale (2018b), robotic gestures Ham et al. (2011), educational support for children Kory and Breazeal (2014), movie descriptions Rohrbach et al. (2015), and narrative performances Wicke and Veale (2020) has been conducted. As mentioned below, a characteristic of our narrative generation study is the emphasis on the roles of narrative knowledge, such as the cultural and regional values of narratives and human emotions and behaviors regarding love and sex. We aim to connect narrative knowledge to robot technologies in the future. In particular, we plan to combine narrative generation systems using narrative knowledge with narration and actor robots. Cultural elements are also included in this knowledge because the environmental adaptation of robots is closely related to the cultural environment in which they act. Moreover, future robot technologies can accelerate exchange and communication among diverse cultures. In particular, we deal with narrative and literary works in Japanese culture. Another focal point is love and sex. Future robots that enter diverse human life scenes will necessarily be associated to love and sex. We believe that love and sex robots can function as media for considering various topics in gender studies.

In this study, we deal with narrative generation regarding love and sexuality, aiming at the implementation and application of future narration robots based on the aforementioned concepts. In particular, we deal with kabuki work and the related legendary story as materials and a cultural approach to introduce human elements of love and sexuality into narrative generation. We survey and analyze the characteristics of narrative materials from the viewpoint of the relationships between the positive or bright sides and the negative or dark sides of the narratives. This survey and analysis aim at systematization using a computer program, and this study presents a prototyping system based on a narrative survey and analysis. The following part of this introductory section explains kabuki as an important material in this study and describes the approach to the goal of this study.

In the context of the aforementioned cultural approaches, we focused on the narratives of kabuki (Ogata, 2016; 2018; 2019b; 2020a; 2020b; 2020d) in the process of narrative generation. Kabuki is a representative cultural heritage in Japan that incorporates various types of performing arts from dance and classical performance, such as nō and kyōgen. Kabuki is not only a literary or narrative genre that is different from modern genres, such as those found in novels, but also a collection of traditional myths, legends, folktales, and other narratives, as well as a synthesized genre included in drama, narrative, and literature. Furthermore, because kabuki includes human vivid feelings and eroticism, as well as the primitive and symbolic feeling of love and sex, it can provide many hints and inspiration to approaching love and sex through narratives.

Additionally, although kabuki is a traditional cultural heritage, it is not a traditional geinō (the generic name of performing arts and entertainments) that has been preserved through national protection nor a cultural genre that has been continued as a private business. In contrast, the nation has frequently oppressed kabuki. To survive such disadvantageous conditions, kabuki has positively collected old narratives and entertainment and, simultaneously, contains the innovative energy required to introduce new technologies and ideas for reforming itself. For instance, contemporary kabuki uses new narrative genres, including manga and animation, as its materials and applies various video and computer technologies to representation. In particular, the essential feature of kabuki is the creation of new works through the fusion of tradition and modernity. While many studies have been conducted regarding its preservation and heritage, more recent studies have also investigated kabuki from the viewpoint of technology.

For example, many studies of kabuki have examined how video technology has been introduced to performance. Some researchers have analyzed the preservation of “Shosagoto” (the movements of actors) using motion capture. For example, Oda and Genda (2004) studied the characteristic movements of kabuki using a motion capture system. Omoto et al. (2004) created a model of Minamiza, a traditional kabuki theater, in Kyōto. Kobayashi (2018) focused on actors’ movements, comparing Western dance in the ballet La sylphide with the Japanese dance Kyōganoko Musume Dōjōji to examine the creation of dances. The stage of ningyō jōruri Hidakagawa Iriaizakura was also reproduced remotely at the Tokyo University of Technology using a commercially available robot (desktop robot “Premade AI”). In contrast to these studies, we approached the narrative of kabuki from multiple viewpoints.

The significance of this study is that it treats kabuki as a research object and focuses on the narrative of kabuki, which has not previously been surveyed and analyzed from the viewpoint of systematization using artificial intelligence and cognitive science with respect to automated narrative generation. This is a challenge in intelligence systems research, including in artificial intelligence. Kabuki is interesting in that it is a narrative collection of many traditional Japanese narratives. Moreover, from the perspective of developing a system, the narrative of kabuki is a complex system that includes complicated diverse narratives that can be expressed in multiple forms. Similarly, the precise analysis of structures is an immensely important theme in research involving computer systems.

Kawai, Ono, and Ogata (2020a) prepared a method for approaching the theme, and the current study uses Japanese classical stories, including an old legend and the related kabuki work, to tackle the problem of story generation related to love and sexuality. We treated Kyōganoko Musume Dōjōji (The Maiden at the Dōjōji Temple), which is a famous kabuki dance work created in the eighteenth century. It is one of many works based on the “legend of Dōjōji,” which has also been related to narratives and picture scrolls and is a latter-day description of the legend of Anchin and Kiyohime. Dōjōji is a temple of the Tendai sect in the Kumano area (in current Japan, Hidakachō, Hidaka, Wakayama prefecture). This temple is the subject of the legend of Dōjōji, which has been recorded for over a millennium. Although this legend has undergone many changes over its long history, it is still well known in Japan. Among the versions of this legend, we focus on the kabuki dance Kyōganoko Musume Dōjōji, which is considered to represent the modern form of the legend of Dōjōji. We also examined the stage performance structure of Kyōganoko Musume Dōjōji and the storyline of the legend of Dōjōji.

As an approach to kabuki story generation research, we analyzed the stage structure of the kabuki dance work Kyōganoko Musume Dōjōji Kawai and Ogata (2019) and developed a system to simulate it using a 2D animation (Kawai and Ogata, 2020; Kawai et al., 2020a). Furthermore, as the first step in generating a love story, we analyzed two tales, the legend of Dōjōji and Kyōganoko Musume Dōjōji, and found that there are differences between the two (Kawai et al., 2020b). Moreover, we integrated narrative generation systems, including explanation generation, into the abovementioned simulation system (Kawai et al., 2020c). Kyōganoko Musume Dōjōji is based on the legend of Dōjōji, but as a symbolic kabuki work, the story of the legend is not concretely expressed. Therefore, in this study, we developed a prototype system to complement the story using the legend of the Dōjōji temple. The purpose is to present a system that supplements Kyōganoko Musume Dōjōji using the background of the legend of Dōjōji and to extend the simulation system of the stage performance structure. In this study, we focused on the fact that, although the background of Kyōganoko Musume Dōjōji is the legend of Dōjōji, the rich story of the legend of Dōjōji is not reflected in Kyōganoko Musume Dōjōji, and forms a symbolic dance work. Although Kyōganoko Musume Dōjōji was based on the ancient legend of Dōjōji, all elements of the rich storyline of the legend of Dōjōji were excluded.22 As introduced in Section 2.1, the legend of Dōjōji is dark and cruel. In contrast, Kyōganoko Musume Dōjōji is, if only briefly, colorized by a bright tone. However, the dark aspects from the legend of Dōjōji appear frequently in Kyōganoko Musume Dōjōji. Therefore, although the kabuki dance Kyōganoko Musume Dōjōji is a story of the legend of Dōjōji, it does not directly represent the story of the legend of Dōjōji. However, we found many hidden relationships between Kyōganoko Musume Dōjōji and the legend of Dōjōji. From the viewpoint of love and sex that this study focuses on, although the narrative of the legend of Dōjōji directly represents the contrasting sides of a woman’s love and sexual desire toward a man, Kyōganoko Musume Dōjōji symbolically abstracts them behind the bright and flowery dance scenes.

Fundamentally, this study treats Kyōganoko Musume Dōjōji and the legend of Dōjōji, which is the background for conducting an analysis combining narratives, from the perspective of the contrast between the positive and negative characteristics of the heroine. A study on kabuki by Tamotsu Watanabe (1986) inspired this study. Watanabe’s analysis method focuses on three elements of kokoro (mind) (representing human nature), furi (action (as small movements)) (meaning the genres of dances), and kashi (lyrics) (meaning the genres of songs) that continue to change through the dance of Shirabyōshi Hanako, the heroine of Kyōganoko Musume Dōjōji. We have inherited and developed this concept from the perspective of the system. We began by considering the relationships between Kyōganoko Musume Dōjōji and the legend of Dōjōji to precisely investigate various elements of the story of the legend of Dōjōji, which lurks in the symbolic narrative of Kyōganoko Musume Dōjōji (Kawai and Ogata, 2021; Kawai, Ono & Ogata, 2021a).

In this study, we assume that the mind and lyrics in the scene of Kyōganoko Musume Dōjōji are key to superimposing the legend of Dōjōji and Kyōganoko Musume Dōjōji. Moreover, we aim to develop a system that can express detailed stories based on legends from various parts of Kyōganoko Musume Dōjōji built on the correspondence of the legend of Dōjōji. For example, a bright scene in Kyōganoko Musume Dōjōji is associated with scenes in the legend of Dōjōjis that represent or symbolize Kiyohime’s extreme love and sexual desire. Based on the analysis of the correspondence relationships between two of the narratives according to the three keys of mind, action, and lyrics, we propose a new system that combines Kyōganoko Musume Dōjōji and the legend of Dōjōji as a framework using an originally developed 2D animation system of Kyōganoko Musume Dōjōji. This is an idea in which the scenes or events in the legend of Dōjōji are associated with Kyōganoko Musume Dōjōji based on the dark atmosphere that occasionally appears in mind, lyrics, and action. This study describes the development of two prototyping systems: (1) a 2D animation system that reproduces the stage performance structure of Kyōganoko Musume Dōjōji and (2) a 3D animation system that complements the story of Kyōganoko Musume Dōjōji. Although the former system uses music, background, people, and dance of Kyōganoko Musume Dōjōji, the dance is limited to a simple presentation because the purpose of the system is to represent the entire stage performance structure. These two systems are related to each other. We analyze the scenes of the legend of Dōjōji and associate them with the scenes of Kyōganoko Musume Dōjōji. The system based on the legend of Dōjōji complements the system based on the stage performance structure of Kyōganoko Musume Dōjōji to represent a complementary explanation using visual imagery.

Therefore, this study extends and integrates the previous analysis of Kyōganoko Musume Dōjōji and its systematization and aims to present a novel concept regarding the system according to the storyline of the legend of Dōjōji to complement the former system. Although Kyōganoko Musume Dōjōji is a narrative work derived from the legend of Dōjōji, it does not directly describe the narrative of the legend of Dōjōji. In the proposed complementary system, the simulation system based on Kyōganoko Musume Dōjōji and the narrative system according to the legend of Dōjōji is ultimately combined to grant users an understanding of the relationships between the two narratives and ultimately enjoy the story of Dōjōji as a Japanese traditional narrative. We stated two types of narrative analyses, namely, research that integrates previous analysis results of the stage structures of Kyōganoko Musume Dōjōji and research that analyzes the storyline of the legend of Dōjōji, which ultimately allows the development of two types of systems. Artificial intelligence, which is pivotal to this study, is an engineering branch that has cognitive science as a basis. Because the motivation of this study is to explore the framework of human intelligence, analyses of research objects using various methods also serve as important steps in the development of artificial intelligence.

The remainder of this paper is organized as follows. Section 2 introduces surveys and analyses of Kyōganoko Musume Dōjōji and the legend of Dōjōji. We present several tables representing the storyline of the legend of Dōjōji and the stage performance structure of Kyōganoko Musume Dōjōji. Moreover, we discuss the relationship between the legend of Dōjōji and Kyōganoko Musume Dōjōji considering the duality of Kiyohime’s characteristic in the legend of Dōjōji. Sections 3 and 4 describe a prototype system according to the flow of the system configuration, the algorithm, and the detailed descriptions based on the above analyses and considerations that use a 2D animation system that we have previously sought. Section 3.4 is the central part of this study, which states a system in which the 3D animation system is based on combining the analysis of the storyline of the legend of Dōjōji with the above-mentioned 2D animation system to associate Kyōganoko Musume Dōjōji and the legend of Dōjōji. In Section 5, we present the system implementation, problems with the developed system, and suggestions for future directions. Finally, Section 6 concludes the study.

2.The legend of Dōjōji and Kyōganoko musume Dōjōji

Although Kyōganoko Musume Dōjōji is a narrative work that was derived from the legend of Dōjōji, it does not directly describe the narrative of the legend of Dōjōji. In this study, we focus on the differences between scenes in the legend of Dōjōji and Kyōganoko Musume Dōjōji. In this section, we first explain the legend.

2.3.Associating the legend of Dōjōji with Kyōganoko Musume Dōjōji based on Kiyohime’s duality

In the legend of Dōjōji, the snake is sometimes treated as a negative symbol because Kiyohime is transformed into one by negative emotions (anger and jealousy).44 Kiyohime has two aspects or characteristics, and we can interpret that the legend of Dōjōji is a narrative in which the positive aspect of Kiyohime changes to a negative aspect. Kiyohime’s mind is currently polarized, but there is also a method of classifying the mind and observing changes in them. We believe that a method for expressing the gradation of the mind, which gradually changes from positive to negative, can be generated as a love story. Although Kyōganoko Musume Dōjōji is a kabuki dance work drawn in bright color in general, as shown in Tables 2, 3, and 4, it contains Kiyohime’s negative side. We consider this duality of this work based on mind, action, and lyrics to represent the association between Kyōganoko Musume Dōjōji and the legend of Dōjōji.

Kawai, Ono, and Ogata (in press) focused on Kiyohime’s two-sided features in the story of the legend of Dōjōji, namely, the aspect of a pure girl who loves a young monk and a girl who goes berserk and betrays him. They divided the scenes of the legend of Dōjōji into P scenes corresponding to the former and the N scenes corresponding to the latter. We divided Kyōganoko Musume Dōjōji into P and N scenes, as shown in Tables 2, 3, and 4.

Additionally, Watanabe (1986) mentioned that the mind, action, and lyrics in Kyōganoko Musume Dōjōji individually have complex structures. Even if the “mind” of a scene is “a pure girl,” her action and lyrics are not necessarily consistent with the mind. For instance, the dance showing the “mind” of “a pure girl” can include contrasting “action” and “lyrics.” According to Watanabe (1986), such complexity is an essential characteristic of Kyōganoko Musume Dōjōji. Ogata (2019d) expounded such a narrative feature as a rich form of narrative that is different from the hierarchically organized narrative form claimed by Aristotle (1997). Such a narrative principle indicates that narrative work does not necessarily need to be constructed based on psychological reality. In Japanese literary tradition, for example, the mentalities of many characters of Genji Monogatari have the psychological traits that modern people can sympathize with. Simultaneously, the narrative works of kabuki contain many scenes, episodes, and characters’ mentalities and actions that modern people cannot sympathize with. However, when we view them from the perspective of the representation effects of narrative, kabuki, irrational, at a glance, frequently have impressive dramatic and narrative effects on the audience. Japanese narratives have many elements familiar with “realistic” norms. Kyōganoko Musume Dōjōji is a typical work.

Table 2

Detailed structure of the scene of “Chūkei no Mai”

Table 2

(Continued)

In this study, we consider that the mind, action, and lyrics individually represent the emotions and characteristics of Hanako to analyze them more precisely. According to Tables 2, 3, and 4, regarding the actions and lyrics in 11 scenes, we focused on the parts that have characteristics different from the mind of each scene.

Table 3

Detailed structure of the scene of “Teodori (2)”

Table 3

(Continued)

Kyōganoko Musume Dōjōji has fewer scenes in common with the legend of Dōjōji than Dōjōji Engi, Hokke Genki, and Konjaku Monogatari shū. It has a thin narrative, is abstract, and does not specifically express the story of the legend of Dōjōji. However, some scenes of Shirabyōshi Hanako correspond to those in the legend of Dōjōji. Figure 1 shows the correspondence between the story of the legend of Dōjōji and Kyōganoko Musume Dōjōji. As shown in the figure, the latter uses only two scenes related to the legend of Dōjōji: “Michiyuki” and “Kaneiri.” Both show the true nature of Shirabyōshi Hanako.

Table 4

Detailed structure of the scene of “Suzu-daiko”

Table 4

(Continued)

Figure 1 shows the correspondence between Kyōganoko Musume Dōjōji and the legend of Dōjōji at the macrolevel. Although this figure indicates that many of the parts of the story of the legend of Dōjōji are not included in Kyōganoko Musume Dōjōji several scenes or images in the legend of Dōjōji are reflected in the stage of Kyōganoko Musume Dōjōji. This study focuses on the legend of Dōjōji, which is an essentially dark narrative, being incorporated into Kyōganoko Musume Dōjōji, which is a bright story.

Table 5

Meaning of the components in tables of the stage performance structure of the kabuki dance

Component	Meaning
The flow of the stage performance	The flow of events played on the actual stage.
Time	The time in which the showing of a part of the lyrics is changed to the next.
Characters	People appearing on the stage. Although the central character of Kyōganoko Musume Dōjōji is Shirabyōshi Hanako, many shokes also appear in this stage.
心 [Kokoro: Mind]	The word represents the most essential or symbolic mental state of Shirabyōshi Hanako who is a central character in each dramatic scene.
振り [Furi: Action]	The genres of dances played by Hanako.
歌詞 [Kashi: Lyrics]	The lyrics sung or narrated, written by Fujimoto Tobun.
The scene of the legend of Dōjōji	It indicates whether the associated scene of the legend of Dōjōji is a negative scene or positive scene.
Music genres and characteristics	The genre or features of the music played in each scene.
Katarite [Narrators]/Utaite [Singers]	Narrators or singers of the lyrics and their numbers.
Ensōsha [Musicians]	The musical players on the stage according to the types of instruments and their numbers.
Gakki [Musical instruments]	The types of instruments played on the stage.
下座 [Geza: Background music]/拍子板 [Hyōshiita: Wooden clappers]	Geza is a room that is hidden from the audiences on the left side (as observed from the audience seats) of the stage and the players in the geza mainly play sound effects. On the other hand, Hyōshiita is instruction for a player on the right side of the stage, accompanied by dramatic development.
Scene	The background of the stage.
Place on stage	Each narrative place where one or more dramatic characters exist.
Ishō [Costume]	The costume that Shirabyōshi Hanako wears.
Kodōgu [Small props]	Small props that Shirabyōshi Hanako has.
Stage equipment	Backgrounds and objects on the stage.
Nikutai no point [The point of the dancer’s body]	In each scene, the part of Hanako’s body that is mostly focused on.

For example, the scene of “Chūkei no Mai” shown in Table 2 is a P scene base on “Mind: Girl” (P). This “girl” means a loving pure girl who has a bright nuance. However, the lyrics from one to eight in this scene represent her N emotion to the bell, and we defined these scenes as N. Here, these scenes are associated with N events (scenes) of the legend of Dōjōji, as shown in Table 1. The parts of the mind, lyrics, and actions in Tables 2, 3, and 4 are described.

Next, because the “mind” which is consistently in the scene of “Teodori (2)” in Table 3, is “Manago no musume no shitto” (Jealousy of the girl of Manago), we defined this entire scene as N. However, her action is continuously colored by the bright and P atmosphere of a virgin girl, and the lyrics are hopeful in the middle of the scene (these are continuously P). In contrast, as the lyrics of the last parts (Sections 5 and 6) show her jealousy, these parts are defined as N. Hence, “Taodori (2),” which is bright, is associated with N episodes of the legend of Dōjōji in the final part.

Finally, “Suzu-daiko” of Table 4, in which the “mind” is “girl,” is entirely positioned as a P scene and the action and lyrics also have a P atmosphere in the middle of the scene. However, the lyrics from scenes 12 to 15 changed to an N atmosphere. This last part is associated with the N scene in the legend of Dōjōji.

Fig. 1.

Relationship between the legend of Dōjōji and Kyōganoko Musume Dōjōji.

The above description shows that the episodes of the legend of Dōjōji are fragmentally reflected in Kyōganoko Musume Dōjōji. Figure 2 shows a conceptual image of the relationship between the legend of Dōjōji and Kyōganoko Musume Dōjōji.

Fig. 2.

Image of the legend of Dōjōji that is fragmentally reflected in Kyōganoko Musume Dōjōji.

3.Prototyping system

In this section, we propose a prototype system based on the survey and analysis of both the legend of Dōjōji and Kyōganoko Musume Dōjōji and the analysis of their mutual relationships. It associates our simple 2D representation system for the stage performance structure of Kyōganoko Musume Dōjōji with the same simple 3D representation system based on the legend of Dōjōji. In the proposed association system, the simulation system based on Kyōganoko Musume Dōjōji and the narrative system according to the legend of Dōjōji are combined to elucidate the relationships between the two narratives and grant users the enjoyment of the story of Dōjōji as a Japanese traditional narrative. In the following subsections, we show the system configuration, macroalgorithm including two processes, and the 2D representation system.

3.1.System configuration

The internal structure of the system is illustrated in Fig. 3. As the flow of the system, based on the stage structure code given to Kyōganoko Musume Dōjōji by the user, the stage is reproduced by referring to the image knowledge base for 2D animation. As stated in Section 3.3, the code is a list of commands, such as moving the character and changing the background inside the system. During the 2D animation system performs, different types of systems select a 3D model from the knowledge base according to the corresponding relationship between Kyōganoko Musume Dōjōji and the legend of Dōjōji analyzed in the previous sections, generate a scene of the legend of Dōjōji, and insert it into the stage reproduction of Kyōganoko Musume Dōjōji. Each scene of Kyōganoko Musume Dōjōji is associated with a scene in the legend of Dōjōji using the correspondence method of both narratives based on the method described in the following section. As we can observe from this description, in the system, the output video is divided into two types: one frame represents the scene of Kyōganoko Musume Dōjōji using 2D animation and the other of the legend of Dōjōji using 3D animation. When representing, the system uses the knowledge associated with the specific scenes of the legend of Dōjōji with the “lyrics (kashi)” of Kyōganoko Musume Dōjōji and the “mind (kokoro)” and “action or movement (furi)” expressed in each scene, as mentioned in the previous sections.

From the users’ perspective, this system is used by, for instance, people who are interested in kabuki but do not know the stories of kabuki in detail. Users can see the computer simulation of Kyōganoko Musume Dōjōji, which is bright in general and appreciates the story’s development. Simultaneously, users can also see that the original episodes of the legend of Dōjōji fragmentally appear similar to in a flashback and get explanations according to “mind,” “lyrics,” and “action” of the heroine as keys. In particular, the role of this proposed system is to change the user’s narrative experience to a more complex one and allow multiple experiences through the story of the legend of Dōjōji, which is hidden in Kyōganoko Musume Dōjōji (Fig. 4).

Fig. 3.

System architecture.

Fig. 4.

System image from a user’s perspective.

3.3.2D representation system of the stage performance structure of Kyōganoko Musume Dōjōji using KOSERUBE

The 2D representation system uses part of the functions of the system called KOSERUBE. Figure 5 presents an overview of the KOSERUBE architecture. KOSERUBE comprises two parts: a narrative generation module and a user interface. Each part is implemented using Common Lisp and HSP and connected via input and output files.

KOSERUBE uses the narrative generation mechanism developed in this study. Narrative generation in our system architecture of narrative generation is executed as a structural operation in the story, discourse, and expression. Each process is performed by four main functional components: conceptual dictionaries, story techniques, discourse techniques, and control modules. Conceptual dictionaries provide semantic definitions of the components of an event, which is a fundamental element of the narrative. Story techniques are used to build a story structure. Discourse techniques are generative rules that transform parts of a story structure. The control modules manage the entire generation process. The conceptual dictionaries and mechanisms above comprise integrated narrative generation. A detailed description was provided by Ogata (2018; 2019a; 2020c). Additionally, KOSERUBE mainly uses the above dictionaries and mechanisms. The difference is the story technique and music mechanism. The central technique for the story-generation phase is Propp-based story grammar Ogata (2007). Although it adopts an original method, the music mechanism is fundamentally based on the music mechanism in an integrated narrative generation system.

Fig. 5.

KOSERUBE architecture (used Fig. 1 in Kawai, Ono, and Ogata (2020a)).

The system proposed in this study uses images based on knowledge for each scene and expresses the stage by combining the images of Kyōganoko Musume Dōjōji. KOSERUBE is a folktale-style story automatic generation system. The user selects a folktale story or character, automatically generates it based on the selected element in the system, and returns the generated animation to the user based on the generated story. KOSERUBE was used in the simulation system for the stage reproduction of Kyōganoko Musume Dōjōji. Reproduction refers to the background, people, and music. The music was handwritten in some scenes using a free composition tool based on the score Kineie (2018). There are scenes in which there is no score; hence, existing music is used. We also reproduced a simple dance. Because the purpose of this system is to see the entire stage structure, it is a simple expression of dance. The marked part (Fig. 5) is modified to reproduce the stage performance structure with people, background (stage equipment), music (instrument and performer), verse, and dance.

As stated in Section 2.2, this study was based on a detailed analysis table of “stage performance structure.” As shown in Tables 2, 3, and 4, as its main components, they contain the characters, backgrounds (stage devices), music (instruments, performers, and genres), verses and lines, and the principal conceptual theme of each scene. In KOSERUBE, the words of the dance were entered into the text data and displayed, whereas the background and person were displayed as images. At the back, Nagauta singers and instrumentalists lined up in the same manner as they would on the actual stage. The person singing and performing in each scene is indicated by an arrow, and a spotlight is applied to yield a visually recognizable expression. In addition, the movement of Shirabyōshi Hanako’s dance was expressed.

In the command file, the chapters on the dance to be displayed, reproduction timing of the music, switching of the background image enclosed with the picture database, etc., are described. Animation control is described later. In the proposed 2D representation system, the command uses limited elements of Kyōganoko Musume Dōjōji described in Tables 2, 3, and 4 as examples to generate the animation in each scene. This is a list of 11 scenes in the story of Kyōganoko Musume Dōjōji and instructions that show the contents that express each scene. Figure 6 shows a part of Chūkei no Mai in Table 2, which generates a simple animation with a song “Kane ni urami ha kazukazu gozaru” [I hold a lot of grudges against the bell] based on the code. Table 6 lists the commands in the code shown in Fig. 6. The parts in [] represent the components used.

In KOSERUBE, the actual movement is simply expressed according to the code. In Fig. 7, the command “mov” is used. “mov, 11, v, −50, −25” represents move, character ID, vertical, movement distance, and speed, respectively, from the left. In the first line, the Shirabyōshi Hanako was moved vertically by a distance of 50 and a speed of 25. (The Shirabyōshi Hanako moved up from the current coordinates. Positive values move Shirabyōshi Hanako downward.) Overall, this implies bouncing Shirabyōshi Hanako three times. The right side depicts a screenshot of Chūkei no Mai’s dance (corresponding to Table 2). This illustrates the actual movement of the stage. On the actual stage, the actor moves his hand up and down on the spot. The objective of this research is not to pursue realistic human movements and express them but to visually reproduce the stage performance structure. Therefore, the dance was simplified in KOSERBE.

Additionally, KOSERUBE was expanded to include auditory elements, which were added from a musical score titled Music and Sound in Kabuki. Haikawa (2016) explains debayashi (Nagauta) and sound effects in Kabuki. This system used a music sequencer called “text music Sakura” (https://sakuramml.com/, last accessed on July 9, 2020). This tool can be used to compose and edit music by simply typing notes (C, D, E, etc.) in text form. In KOSERUBE, the auditory element is accompanied by an animation. Additionally, we created music using a free composer called “Wagakuhitosuji” by referring to the score in Shamisen Bunkafu Nagauta Kyōganoko Musume Dōjōji by Kineie (2018). Unlike several composer tools, “Wagakuhitosuji” corresponds to the scale of Japanese music, rather than Western music, and allows composition on shamisen notation.

Fig. 6.

Part of the code entered.

Fig. 6.

(Continued.)

Table 6

Commands used in the code for 2D animation

Command Name	Description Form	Meaning
ch_BG	ch_BG, FileName	Changes the background image on the main stage to the image defined by “FileName.”
Dcs	dsc, Text	Displays the character string designated by “Text” on the display.
Pos	pos, ID, X, Y	Displays the image by “ID” in the coordinate of “X” (width) and “Y” (height).
Mus	mus, 1 or 0, FileName	Plays (“1”) or stops (“0”) the musical file designated by “FileName.”
Mov	mov, ID, v or h, move, speed	The image of “ID,” executes the lateral movement (“h”) and vertical movement (“v”) according to the values of “speed.”
go_action	go_action, /////////	Executes the movement of the image designated by the “move” command.
Reset	Reset	Returns the position of an image to the first state.
Spot	spot, lightID, X, Y	Displays the light by “lightID” in the position of “X” and “Y.”

Fig. 7.

Example of a screenshot of the dance of “Chūkei no Mai”.

3.4.Associating between Kyōganoko Musume Dōjōji and the legend of Dōjōji using 3D representation system

We have developed an experimental system that complements the 2D representation system of Kyōganoko Musume Dōjōji using the 3D representation system of the legend of Dōjōji (Kawai, Ono & Ogata, 2021a), representing the fragmented images of the legend of Dōjōji, which does not appear directly in Kyōganoko Musume Dōjōji. We also considered various application systems for generating explanations as well as for education. Although the system can insert the animation of the legend of Dōjōji into the animation of Kyōganoko Musume Dōjōji, our consideration of their association or correspondence was still insufficient. This study corresponds to the development of research on these associations. Additionally, the system proposed in this study is not only a complementation system but also a prototype toward more advanced goals, including more academic investigation and education systems of cultural heritage. Through the development of the survey and analysis of the association, we will view the possibilities of various systems beyond the complement system of Kyōganoko Musume Dōjōji.

3.4.1.3D representation system of the legend of Dōjōji

The system inserts the scene using the 3D representation system of the legend of Dōjōji shown in Table 1 into the given code and the stage reproduction of Kyōganoko Musume Dōjōji using a 2D animation system based on the knowledge base corresponding to its legend. For all 11 scenes that are reproduced on the stage, the results are returned to the user after scene insertion is completed.

An overview of the 3D representation system was prepared by changing the combination of images from KOSERUBE to a combination of 3D models. In particular, the input information into the 3D representation system is the list of modes to be used and the commands for the positioning and movement of the models. The system reads the list of models, interprets each command according to a sequential order, and outputs the results as a 3D animation. The commands listed in Table 7 are prepared.

The 3D model used was the one used in the “Miku Miku Dance” developed by Yu Higuchi (release date: February 24, 2008). Because this video is also intended to express the structure of the legendary scene of Dōjōji, the detailed movements are not reproduced, and they are symbolically expressed by combining 3D models. The final output video is explained in the next section.

Table 7

Commands and the definitions

Command Name	Definition
Move	Described execution time, object, and the number of changes in x, y, and z for moving the object according to the values.
Setang	Describes execution time, object, and the number of changes in x, y, and z for turning the object according to the values.
Pose	Stops the object according to the inputted time.
Fire	Describes execution time, object, and switch for firing red light from the object’s mouth.
Sitdown	Describes the execution time and object for changing the object to a different model.
Setpos	Describes execution time, object, and the coordinates of x, y, and z for moving the object to the designated position.
End	Finishes the animation.

3.4.2.Associating method between Kyōganoko Musume Dōjōji and the legend of Dōjōji

The way the two stories come and go is expressed using two images, a 2D and a 3D one, with the mind change being key. The correspondence relationships that trigger the return are based on the method described in the following section. Specifically, we state the method of the association between Kyōganoko Musume Dōjōji and the legend of Dōjōji.

As mentioned in Tables 2, 3, and 4 for Kyōganoko Musume Dōjōji, the key terms for associating it with the events in the legend of Dōjōji shown in Table 1 are the “mind” and “action” of Shirabyōshi Hanako and lyrics. This time, the action includes Hanako’s facial expressions and gestures. The “mind” takes the values of P or N as the entire level and the value of “mind” in each scene is the same. Meanwhile, the “lyrics” and “action” action’ take the value P or N in each scene. In this study, we aim to emphasize and represent a scene that has a value different from the value of “mind” for emphasizing their contrast. In the “lyrics” and “action,” we aim to focus on the contrasting parts from the “mind.” However, when the values of “mind,” “lyrics,” and “action” in a scene are the same, one or more events of the legend of Dōjōji having the same value are selected.

From this description, the cases in which “mind” is P and N, and four types of combinations are produced. If “mind” is P [mind/P]:

• mind/P, lyrics/N, action/N (2)

• mind/P, lyrics/N, action/P (1)

• mind/P, lyrics/P, action/N (1)

• mind/P, lyrics/P, action/P (0)

However, if “mind” is N [mind/N]:

• mind/N, lyrics/P, action/P (2)

• mind/N, lyrics/P, action/N (1)

• mind/N, lyrics/N, action/P (1)

• mind/N, lyrics/N, action/N (0)

Additionally, numbers from 0 to 2 are required in the following description.

The association with the scenes of the legend of Dōjōji is based on the above numbers (points). First, in the case of mind/P, the following association is conducted based on the following points:

• 2-point: The N+ scene of the legend of Dōjōji.

• 1-point: The N scene of the legend of Dōjōji.

• 0-point: The P+ or P scene of the legend of Dōjōji.

However, for mind/N, the association is as follows:

• 2-point: The P+ scene of the legend of Dōjōji.

• 1-point: The P scene of the legend of Dōjōji.

• 0-point: The N+ or N scene of the legend of Dōjōji.

In this description, the N+ scene of Dōjōji is the scene that was evaluated as the most negative in the legend of Dōjōji, and N means the second negative scene. The P+ scene is the scene that was evaluated as the most positive scene, followed by the P scene.

In this study, we associate Kyōganoko Musume Dōjōji with the legend of Dōjōji. This method can be implemented in a system that contains the mechanism for moving to the legend of Dōjōji from the simulation system of Kyōganoko Musume Dōjōji. A tentative idea of system implementation based on the above description is shown in Fig. 8: points of the events in the legend of Dōjōji associated when the “mind” is consistently 1(P) or −1(N) in “Chūkei no Mai,” “Teodori (2),” and “Suzu-daiko.” In many cases, contrasting events and scenes are mutually associated.

First, the timing of representing a scene of the legend of Dōjōji in the representation system of Kyōganoko Musume Dōjōji is determined by the user (experimentally, the system can execute the transition at any time without any user inputs). In cases of mind/P and mind/N, common processing was conducted. A corresponding scene is freely selected and displayed without managing, for example, the amount of display based on this point.

Fig. 8.

Associative patterns between the scenes of Kyōganoko Musume Dōjōji and the events of the legend of Dōjōji.

4.Implementation and experiments

Based on the algorithm described in Section 3.4, we implemented a prototype system using Common Lisp, which associates the 2D representation system of Kyōganoko Musume Dōjōji and the 3D representation system of the legend of Dōjōji. When the user pushes the * key on the 2D system, the following algorithm is executed to represent 3D animation (while the 3D representation is executed, the 2D system stops):

Table 9 shows the scenes of Kyōganoko Musume Dōjōji and the scenes of the legend of Dōjōji that are associated with the experimental execution of the proposed system. (The underlined events of the legend of Dōjōji correspond to Figs 9, 10, and 11).

As examples of representation, Figs 9, 10, and 11 show a scene of the legend of Dōjōji that corresponds to a scene on the stage of Kyōganoko Musume Dōjōji.

In Fig. 9, the left shows a scene “Jinjō no hibiki ha shōmetsu metsui” (N) (the “lyrics”) in “Chūkei no Mai” Chukei no Mai’ of Kyōganoko Musume Dōjōji. The “mind” is “girl” (P) and the “action” is “Shirabyōshi” (N). The right associated part of the legend of Dōjōji is the event of “Kiyohime spits fire” (N+). Next, the left scene of Fig. 10 is a scene of

Fig. 9.

Part of output video (1).

“Teodori (2)” of Kyōganoko Musume Dōjōji. The “lyrics” is “Ujigami-sama ga kawaigarashansu” (P), the “mind” is “virgin” (P), and the “action” is “Jealousy of the girl of Manago” (N). On the other hand, the right image shows the event of “Kiyohime, who lives in the private house, falls in love with Anchin at first sight” (P+) of the legend of Dōjōji. Finally, in Fig. 11, the left scene is the image of the “lyrics” “Omoeba urameshiya tote” (N), the “mind” is “girl” (P), and the “action” is “Suzu-daiko” (P). The right part shows the event of “The monks of Dōjōji Temple hide Anchin in the bell of Dōjōji” (N) of the legend of Dōjōji.

Fig. 10.

Part of output video (2).

Fig. 11.

Part of output video (3).

5.Problems and future works

There are five main problems to be addressed. We would like to examine these five points and connect them to promote further development of the system.

Therefore, in the future, we would like to express a wider story by incorporating CGs and robots. This research explores a method that alternates between the scene of Kyōganoko Musume Dōjōji and the legend of Dōjōji. The goal is to combine the generation of stories with CG and seamless expressions using robots. Based on the branching of the scenes, we consider a method in which many story scenes are generated in parallel using the story unit of the legend of Dōjōji.

Fig. 12.

Scene from Hanakurabe Senbonzakura (photographed by Takashi Ogata: August 1, 2019).

As mentioned in Section 1, storytelling robots can be a new form of narrative expression. For example, we propose a kamishibai (paper-play) robot. A kamishibai is a method of storytelling in which the narrator tells a story using pieces of paper with pictures representing the content of each scene. The narrator uses various gestures to draw the listener into the story. The storytelling by a kamishibai robot opens new possibilities. We also believe that the kamishibai robot can be applied to care and nursing care (e.g., Médecins Sans Frontières used the Kamishibai for AIDS prevention Campaign Enjelvin (2018)).

Additionally, Carillon Theatrical Society (https://www.ctsbathurst.com.au/) conducted musical theater productions and other performances, including animation.

6.Conclusion

We have investigated kabuki in the context of narrative generation based on information technologies, including artificial intelligence and cognitive science, as well as literary studies, such as narratology and literary theories. Moreover, kabuki dance work, Kyōganoko Musume Dōjōji, is pivotal to our narrative generation study of kabuki. Although we developed a novel 2D representation system that simulates the stage performance structure of Kyōganoko Musume Dōjōji based on its survey and analysis, this system was only a starting point for expanding the narrative generation study of kabuki and application systems using kabuki materials. As stated in INTRODUCTION, this study, aiming at diverse forms of narrative generation robots, develops a system that reflects the problems of love and sex through a cultural approach toward kabuki and related narrative works. In particular, we focused on the legend of Dōjōji as the background of Kyōganoko Musume Dōjōji and surveyed and analyzed both narratives and their relationships in detail. We concentrated on the positive and negative characteristics of the heroines common to these narratives, called Kiyohime or Shirabyōshi Hanako, to mutually associate these two narratives. These correspond to the contents of the first half of this study. The second half of this study describes the structure, design, implementation, and simulation results of a prototype system based on the above survey and analysis. This system aims to superimpose multiple stories utilizing Kyōganoko Musume Dōjōji, the heroine’s mind and action, and the lyrics sung or narrated on the stage.

In the future, by adding and extending the current deficient knowledge and CG animation, we will expand this prototype system to a completed system and explore various usages of the proposed mechanism and related surveys and analyses as application systems. Furthermore, we believe that it will become necessary for robots to tell stories for human entertainment, nursing, and long-term care. Therefore, we propose not only CG but also robots and to use the works of kabuki and Jōruri rather than Kyōganoko Musume Dōjōji.