Associating semantically structured cooking videos with their preparation steps

The dramatic growth in the amount of multimedia data in recent years has caused techniques for analyzing those data to become increasingly important. To make use of information contained in multimedia data to the fullest extent, attention has been focused on techniques for processing multiple types of media in an integrated manner. Among the various kinds of videos that exist, the authors selected cooking videos, which are closely related to everyday life, to propose a technique for associating the videos with preparation steps described in related text‐based materials referred to simply as textbooks. With educational videos having related textbooks such as cooking program videos, the textbooks are easier to consult than the videos but the videos contain useful visual information that cannot be represented in the textbooks. Therefore, integration of the video and textbook is expected to result in high‐order video structuring and indexing. In this paper, the authors first analyze the video structure and define the video blocks that are to be the video units used for association. They also analyze the preparation steps described in the closed captions and textbooks and propose and define an association technique based on keyword extraction. In addition, they show through evaluation experiments that association in terms of video block units or, in other words, video indexing, can be performed with high precision by using the proposed technique. They also show that limiting the subject matter and skillfully incorporating relatively simple elemental techniques enables a practical level of precision to be obtained. © 2005 Wiley Periodicals, Inc. Syst Comp Jpn, 36(2): 51–62, 2005; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/scj.20131