Can Computers Understand Picture Books and Comics?

Introduction

How do humans produce creative work? What do individuals imagine, and what do they think of during the creative process? The ultimate purpose of this study is to find an algorithm for the generation of creative, intellectual works, such as novels, comics, and animations, with regard to artificial intelligence. The creative works listed above involve stories and are constructed using sequential components. Further, the popular representation of a narrative utilizes natural language and pictures. For example, novels are constructed using natural language, and this type of language is a very useful means of logically depicting a given situation. However, the number of sentences comprising individual scenes varies considerably, and it is therefore difficult for computers to determine scene boundaries. In addition, comics are constructed using both pictures and natural language, and animations are comprised of pictures and sound based on natural language. Note that pictures constitute an effective means of creating stories, because they can convey postures and positions more directly than verbal or written language. This approach allows stories to become comprehensible to a wider audience, regardless of age or nationality.

In Japan, comics and animations are common in popular culture. A key characteristic of these representations are the deformation or exaggeration of characters and other objects in the story. These visual representations contain lyrical and descriptive aspects to depict each situation clearly. Further, these works are often very intellectual, and some popular authors have produced exceptional creative pieces.

Story creation is something of which the vast majority of humans are capable; however, computers can neither create nor understand stories. The authors are very interested in the process of story construction, particularly as regards the development of appropriate models to facilitate computer story creation. Therefore, our goal is to define complete models of story creation for use by computers. However, it is difficult to define models for the entire process and every objective, therefore the authors suggest models for a specific process of creation. Hence, the authors have conducted several studies regarding computational creative stories, focusing on the topics listed below.

• •

  Story Generation and Picture Book Generation: Semi-automatic story generation using log data.

  • o

    Continuous transitions are automatically created and are given to writers in order to write unexpected stories that cannot be created by humans only. The aim of this section is to propose a computational method of coherent and unexpected story creation. and various picture book generation.

• •

  Comic Generation: Automatic comic generation focusing on the relationship between transitions of stories and expression of pictures and comic analysis based on two of their features.

  • o

    Continuous transitions expressed by pictures are modeled and the relations between characteristic expressions and the patterns of stories are analyzed. The aim of this section is to suggest computational models for comics focusing on the expressions in pictures.

In this chapter, these studies are introduced using examples, in order to demonstrate the fundamental techniques of computational narratology and the challenges involved in the quantification of story creation.

This chapter is enhanced one of “Can Computers Create Comics and Animations?” (Ueno, Fukuda, & Mori, 2016). Compared to our previous chapter, the authors introduce three major enhanced topics in this chapter.

• •

  Picture Book Generation

• •

  Analyzing Four-Scene Comics by Deep Learning Methods

• •

  Constructing Original Four-Scene Comics for Machine Learning