MANPU2016 Invited Talk: Manga Generator, a future of interactive manga media

1. Manga Generator, a future of interactive manga media MANPU2016: Invited Talk The First International Workshop on coMics ANalysis, Processing and Understanding 1 Akihiko SHIRAI, Ph.D Kanagawa Institute of Technology, JAPAN @o_ob

2. Akihiko SHIRAi in 3 min YumeNaviLive , pitch talk for high school students.

3. Akihiko SHIRAI, Ph.D (Engineering) Department of Information Media, Kanagawa Institute of Technology (KAIT), Japan Chair of Laval Virtual ReVolution (2006~) Executive Commttiee of International collegiate Virtual Reality Contest (2002~) ACM SIGGRAPH ASIA 2015 Youth Program Liaison AKIHIKO SHIRAI, Ph.D in Engineering, has obtained a bachelor of Photo Engineering and master of Image Processing from Tokyo Institute of Polytechnics. Afterwards he worked for Canon and Criterion as a game development consultant to distribute RenderWare, a multi-platform graphics middleware for the game industry until 2001. He went back to academia to study intelligent systems and obtained a Ph.D. in the Tokyo Institute of Technology in Japan in 2004 with research concerning the “Tangible Playroom”, an entertainment system for young children using haptics, a floor screen and a physics engine. He was a R&D researcher at NHK-ES in Japan, focusing on the next generation’s TV production environment before moving to ENSAM Presence & Innovation Laboratory in France from 2004 to 2007 for R&D of a Virtual Reality theme park project. He worked for National Museum of Emerging Science and Innovation (Miraikan), Tokyo Japan, as a science communicator and exhibition planner from 2008 to 2010. Starting from 2010, he works in the Information Media Department at Kanagawa Institute of Technology (KAIT) as an associate professor.

4. Manga Generator, a future of interactive manga media * Motivation from Student competition project to International new media * Technology of Manga Generator * IVRC and Laval Virtual * Entertainment systems and Sensory motor play * Research, Results and Deployments * Collaboration with professional manga * Player analysis * Contents toolkit * Future * MGV: Multiplex Digital Signage * Next generation as “Interactive Manga Media”

5. Manga Generator Project

6. Slide - 1 Motivation of Manga Generator • Entertainment system using VR technology. • Enjoyable to people for all. • Exploring future of “media mixed” creation. • Tell Japanese manga culture to the world.

7. Slide - 2 Japanese manga culture

8. Slide - 3 Japanese manga culture • “Shonen” category for boy’s Ø “Dive into the manga world” is a boy’s dream. Ø Children play “Make-Believe play” as a hero. • Animated manga (Anime) Ø It’s a quite big media. However, it is not an interactive content. Imagination is not needed for audience

9. Slide -4 Daiki UMEDA “Real-time manga-like depiction based on interpretation of bodily movements by using kinect”, 2012 Related Research “Comic Chat”, 1996“Comi Po!”, 2010 “Manga camera”, 2012Masaki Hayashi “Research on automatic manga generation based on TVML script”, 2002

10. Prototype alpha Version beta1 Version beta2 Version RC1 Laval Version Slide - 7 2012/8 2012/9 2012/10 2012/11 ・ Manga-like Visual ・ Automatic Layout Adjuster ・ Set a story ・ Dynamic Mood Selector ・Set a multi-story *IVRC2012 semifinal *IVRC2012 final *TV Appearance：TBS *TV Appearance：TV TOKYO Road map Better expression using user posture 2013/3

11. Slide - 1 Scenario 「Break the WALL」「Crisis of the Earth」「American Comics taste」

12. Slide - 9 Very simple and controllable mathematics, it can perform in real time (30fps or more). Kinemotion

13. KinEmotion

14. * KinEmotion: context controllable emotional motion analysis method for interactive cartoon generator, SIGGRAPH '13 ACM SIGGRAPH 2013 PostersArticle No. 75 Slide - 1 KinEmotion Better expression using user posture

15. Slide - 12 LAVAL VIRTUAL Version •Located is Laval city •Linking to Facebook

16. Slide - 13 Contents design •Laval Virtual 2013 •Touristic promotion •TV program

17. Slide - 11 Awarded • IVRC2012(International virtual reality contest) • Asia Digital Art Award • Asia student creative competition • Laval Virtual 2013 Award

18. Slide - 14 Conclusion in 2013 • Entertainment system using VR technology • Enjoyable to people for all ages • The future of the media mixed creation • New promotion system • Tell the world of Japanese manga culture

20. What is IVRC? * International collegiate Virtual Reality Contest establish in 1993 by Prof. Susumu Tachi * Contest of interaction and virtual reality, which fuses art and technology * Honors the greatest Virtual Reality and Interactive works of students * New education system through contest

21. What is IVRC? * Organized by Virtual Reality Society of Japan * Supported by Ministry of Economy, Trade and Industry and Ministry of Education, Culture, Science and Technology * Ambassade de France au japon, Service pour la Science et la Technologie * Held at Miraikan, National science museum of Emerging Science and Innovation Virtual Reality Society of Japan http://www.vrsj.org/

22. IVRC: contest of the students, by the students, for the students l IVRC is the acronym of “International collegiate Virtual Reality Contest”, which has been held since 1993 and is the contest of Interactive products using advanced technologies such as virtual reality and robots. Our English name changed for internationalization in 2004, because we began to invite awarded teams from the French VR contest “Laval Virtual”. l When you hear the words such as “Virtual Reality” and “robot”, you should feel that special expensive machines are necessary to realize them. However, there are many ideas which can be actualized only with handmade machines. The main purpose of this contest is for participants to gain deeper understanding of and familiarity with interactive technologies through activities such as planning interactive systems, making devices themselves, and improving quality of their contents. l IVRC has not only the function of contest but also the function of education. It aims to construct the new education system of the 21st century to develop excellent human resources, who can think, learn, and work by themselves. planning technology design Team management

23. Call for Submission Deadline of submission 2nd stage, Demonstration Final stage, Demonstration SIGGRAPH/ Laval Virtual Online submission Improvement phase 1st stage, Paper review Development phase IVRC Flow Chart 4 +1 projects Annual Conference of VRSJ DCEXPO: Digital Content Expo April Late June or Early July September October

24. * Semi final at the annual conference of the Virtual Reality Society of Japan (9-11th September, Tokyo) Demonstration in academic conference * Grand final in Digital Contents EXPO 2015 22-25th October, Miraikan, Tokyo. Demonstration in Public * U-18: Youth section * International Video Selection for the final stage

25. The First Presence at ACM SIGGRAPH 2002 * IVRC Committee helps the champion team submit to SIGGRAPH as a reward to the winner. Ø“Virtual Chambara”, SIGGRAPH2002 “Virtual Chambara”, SIGGRAPH 2002

26. History of IVRC Presence at SIGGRAPH * 2003 THE DIMENSION BOOK * 2004 Fragra / Dis-Tansu * 2005 Straw-like user interface / Kobito: Virtual Brownies / Ton2 * 2006 bubble cosmos / INVISIBLE ~The Shadow Chaser~ / Powder Screen: A Virtual Materializer …..

27. History of IVRC Presence at SIGGRAPH * 2007 BYU-BYU-View / CoGAME * 2008 Ants in the Pants / Landscape Bartender * 2009 Baby type robot “YOTARO” / Back to the mouth / Funbrella: making rain fun * 2010 Haptic Canvas * 2011 Mommy Tummy * 2015 Childhood/ vibroSkate

28. International Cooperation with Laval Virtual “The winner of the IVRC Award in France will be invited to the Tokyo stage of the IVRC in Japan as a challenger. Similarly, the winner of the Laval Virtual Award in Japan will be invited to the final stage of Virtual Fantasy, category “Demos”, student competition of Laval Virtual, in France as a challenger. Conditions of invitation will be defined every year by each part of this agreement.”

29. Agreement on International Cooperation at Laval Virtual 2004 IVRC and Laval Virtual will maintain an award partnership for three years (2004-2007): the Laval Virtual Award at the IVRC in Japan, the IVRC Award at “Laval Virtual’s Le Village de la Creations” in France. J.F. Fontaine (Laval Virtual) Prof. S. Tachi (IVRC)

30. International Cooperation with ETC, CMU “The best work certified by CMU among other works created as student project will be invited for the final contest of IVRC. The invited work will be assessed in the final contest in the same way as other selected works in the preliminary contest of IVRC and will be entitled to receive each award of IVRC.”

31. Partnership in SIGGRAPH 2010 Don Marinelli Executive producer of ETC

32. http://ivrc.net/ Carnegie Mellon University (CMU) Entertainment Technology Center (ETC) and International collegiate Virtual Reality Contest (IVRC), in recognition of the growing importance of the roles Virtual Reality and Entertainment Technology play in the 21st century, have come to the agreement: Agreement on International Cooperation in Virtual Reality and Entertainment Technology Michelle Macau (ETC) Ambassador Agent for Donald Marinelli Executive Producer of ETC Prof. S. Tachi (IVRC)

33. 3D moves in 30-year cycles VR moves 10 years after３D crazes 1st 3D Craze （the1950s) Dawn of VR (the1960s) 1940 1960 1980 2000 20201920 Dawn of 3D （the1920s) 2nd 3D Craze （the1980s) 3rd 3D Craze （the2010s) 2nd VR Wave (the2020s) ? 1st VR Wave (the1990s) movie movie Theme-park computer game TV movie movie

34. Virtual Reality in the 1990s * 1989 VPL Research Data Glove and Eyephone. “Virtual reality" was first used. * 1990 Santa Barbara Conference (The Engineering Foundation) Virtually all the researchers were summoned. * 1991 ICAT (International Conference on Artificial Reality and Telexistence) was held in Japan. * 1992 Committee on Virtual Reality was established in National Research Council. Its report was published in 1995. * 1993 IVR（Industrial Virtual Realty) was held in Japan. * 1993 VRAIS (Virtual Reality Annual International Symposium) and Research Frontiers in Virtual Reality were held. They gathered together to form IEEE Virtual Reality Conference. * 1993 IVRC（International Virtual Realty Contest) was held. * 1996 VRSJ (Virtual Reality Society of Japan) was established. * 2001 IEEE Virtual Reality Conference was held in Japan. * 2008 National Academy of Engineering (NSF) announced 14 grand engineering challenges for the 21st century on February 15, 2008. Enhance Virtual Reality is chosen as one of the 14 challenges.

35. 23 years… * More than 200 projects have been realized through 23 IVRCs including 21 projects selected as SIGGRAPH E-Tech. * Average 5 persons for each project mean over 1,000 persons have deeply experienced and understood VR! * IVRC students become staffs, and staffs now become world famous professors, designers, creators, and engineers.

36. IVRC Alumni Members

37. 21 CENTURY'S GRAND ENGINEERING CHALLENGES * National Academy of Engineering announced 14 grand engineering challenges for the 21st century on February 15, 2008. http://www.engineeringchallenges.org/ * The final choices fall into four themes that are essential for humanity to flourish -- sustainability, health, reducing vulnerability, and joy of living. * Tremendous advances in quality of life have come from improved technology in areas such as farming and manufacturing. If we focus our effort on the important grand challenges of our age, we can hugely improve the future. * Enhance Virtual Reality is chosen one of the 14 challenges.

38. Second VR Wave（the 2020s ？) VR Challenges * Autostereoscopic Display * Haptic VR * Augmented Reality * Telexistence * Human Augmentation

39. 3D moves in 30-year cycles VR moves 10 years after３D crazes 1st 3D Craze （the1950s) Dawn of VR (the1960s) 1940 1960 1980 2000 20201920 Dawn of 3D （the1920s) 2nd 3D Craze （the1980s) 3rd 3D Craze （the2010s) 2nd VR Wave (the2020s) ? 1st VR Wave (the1990s) movie movie Theme-park computer game TV movie movie

40. Next-generation “INTERACTIVE CONTENTS”

41. Laval Virtual

42. Laval Virtual Academic Conference “ACM VRIC” Award, Competition, Public Day

43. * Keynotes * Workshop + Aural Presentation ACM VRIC: Virtual Reality International Conference Dr. Pattie Maes, MIT Media LaboratoryPr Alain Berthoz, College de France Virtual avatars and real brains: how do they interact? AR and VR for learning and collaboration Dr. Oliver Bimber, Johannes Kepler Univ. Linz Light-Field Technology: A Revolution to Imaging and Display Dr. Carolina Cruz-Neira, Emerging Analytics Center, University of Arkansas at Little Rock Beyond Fun and Games: VR as a Tool of the Trade Christine Perey, AR for Enterprise Alliance The Future of Work with Augmented Reality Dr. Skip Rizzo Director for Medical Virtual Reality Institute for Creative Technologies - Research Professor USC Davis School of Gerontology and USC Keck School of Medicine Department of Psychiatry & Behavioral Sciences Beyond Video Games: A Virtual Reality Revolution in Behavioral Health

44. * Workshop + Aural Presentation ACM VRIC: Virtual Reality International Conference

45. Special Session on Friday 14:00-14:20 Opening Address “Real-Virtuality”, Prof. Akihiko SHIRAI (KAIT) 14:20-14:30 “IVRC to Laval Virtual / Jack and the Beanstalk”, Mr. Shota SUGIMOTO, (Univ Keio) 14:30-14:50 “Embodied Media”, Prof. Kouta MINAMIZAWA (Univ keio) 14:50-15:00 “Metamorphosis Hand: Interactive Experience of Embodying Virtually Transformed Hands”, Nami Ogawa (Univ of Tokyo) 15:00-15:30 “VR to Superhuman Sports” Prof. Masahiko INAMI (Univ of Tokyo) 15:30-16:00 Round table and debate, exchanges with the audience ACM VRIC: Virtual Reality International Conference http://superhuman-sports.org/

47. [definition] Entertainment system is System which affects to human amusement. Video game, museum exhibition, entertainment VR, media arts, e-sports, broadcasting, social network… 1. Device and/or System 2. Service and/or Contents 3. Evaluation methods Research in Entertainment system

48. * Jean Piaget (1896-1980) Swiss clinical psychologist known for his pioneering work in child development. * Sensorimotor play * Symbolic play * Rule Play “Sensory-motor Play”

49. Revolution

51. Research, Results and Deployments *Collaboration with professional manga *Player analysis *Contents toolkit

52. Manga Generator Tottori version

53. Tottori Manga Kingdom

54. Player Analysis in TEPIA permanent exhibition

55. Contents toolkit

56. In Taiwan * Hoping Jones version in Taiwan Manga Expo

57. Concept: Common issue in living room

58. * ExPixel is a Pixel-Shader Technology that can show multiple images on one screen at same time. * ExPixel * Extended Pixel * X-Pol (FPR for 3D) * Extra Image… What is ExPixel

59. Related Works: Multiplex display * Direction based separation * “Bois-Clair” (Gaspar Antoine, 1692) * “Vari-Vue” (Victor Anderson, 1962) * “Lumisight Table”(2005), ”UlteriorScape” (2008) by Kakehi * Time/Shutter based technologies * “Snared illumination” (fakespace lab, 2004) * “ThirdEye” (Pranav Mistry, 2009) * “SimulView” (SONY, 2010-) * Polarization based technologies * “FairyFinder” (Hachiya, 2005) * “Mysterious Pond”, “A visibility control system for collaborative digital table” (Kitamura, Sakurai. 2009) * Hidden but not shown for naked eyes * Patent: JPA2005-092227 (Semiconductor Energy Lab, 2005)

60. *Adding a new value of 3D devices. *Realizing multiplex-hidden imagery on a consumer 3D flat panel without any hardware modification. *Providing content creation environment for wider creators. Challenge

61. Scritter to ExPixel: Scritter NAGANO, K., UTSUGI, T., HIRANO, M., HAMADA, T., SHIRAI,A., AND NAKAJIMA, M. 2010. A new ”multiplex content” displaying system compatible with current 3d projection technology. In ACM SIGGRAPH 2010 Posters, ACM, New York,NY, USA, SIGGRAPH ’10, 79:1–79:1. Multiplex by two projectors.

62. Scritter to ExPixel: ScritterH Wataru FUJIMURA, Yukua KOIDE, Robert SONGER, Takahiro HAYAKAWA, Akihiko SHIRAI, Kazuhisa YANAKA, "2x3D: Real-Time Shader for Simultaneous 2D/3D Hybrid Theater", ACM SIGGRAPH ASIA 2012 Emerging Technologies, Singapore, 2012 Hidden imagery with two projectors.

63. ScritterH, Projector based hidden imagery * Merge an arbitrary image A for naked eyes and an image B that you would like to hide to generate a different image C and multiple- project B and C. * Image A visible to naked eyes while image B with glasses. * Figure from ScritterH project

64. * Naked eye: Left + Filter: Right. * It is Implemented as a GPU based shader. * Good for theater but not suitable for consumer usage. 2x3D: Hybrid 2D + 3D display Wataru FUJIMURA, Yukua KOIDE, Robert SONGER, Takahiro HAYAKAWA, Akihiko SHIRAI, Kazuhisa YANAKA, "2x3D: Real-Time Shader for Simultaneous 2D/3D Hybrid Theater", ACM SIGGRAPH ASIA 2012 Emerging Technologies, Singapore, 2012

65. * Scritter/ScritterH doesn’t need synchronized devices. Only needs polarization filter to separate two channels. * However, it still needs specific products such as DMD projector, polarization supported screen. * It is difficult to prepare two projectors as a stuck. ScritterH, Projector based hidden imagery

66. Algorithm of ExPixel (1/3) With a circular polarization filter Contrast compression

67. Contrast compression 0 2550 255 input output 0 2550 255 input output 0 2550 255 input output 𝑎"#$ a’.rgb = a.rgb × (1.0 - 𝑎"#$) +𝑎"#$ b’.rgb = b.rgb × 𝑎"#$

68. Algorithm of ExPixel (2/3) With a circular polarization filter Inversed gamma correction

69. * Obtain image C with considering display gamma. Inversed gamma correction c.rgb = (𝑎' − 𝑏' ) + , gamma correction

70. Algorithm of ExPixel (3/3) With a circular polarization filter Even odd selection

71. Line-by-line: Even odd selection R L Image A Image B R L R L R L Display P polarization S polarization P polarization S polarization P polarization S polarization Normal Use: Parallax images ExPixe Use: Arbitrary images

72. Line-by-line: Even odd selection Only even lines Only odd lines

73. * Everybody can make multiplex hidden content with Unity 3D Content Creation tool 1) Import Package 2) Set Images 3) Run!!

74. Multi-language presentation Naked Channel (Janpanese) Hidden Channel (English)

75. UbiCode Naked Channel (Advertisement) Hidden Channel (QR code)

76. POLAR: Physical Optical Layered AR AR marker on the main content

77. POLAR: Physical Optical Layered AR AR marker on the hidden channel

78. Using Scene Naked Eye Hidden Eye Experience Presentation Listener Presenter Only presenter can check scenario. Classroom No hint Hint available Show hints for student who needs hint. Museum Children Adult Explications for different age. Shop Non member Member Show special price for registered member. Sightseeing Children Adult Explications for different interests. Medical Patient Doctor Supplementary information which should not be shown to patients. Municipal services Local residents People from abroad Communication tool between domestic people and foreigner. Various applications

79. * Converter * XNA * Static image * Video * Unity * Static image * Video * Game * AR * Presentation * Player Shader model 3.0 Converter * Processing * PHP * Static image * FPGA Hardware Content creation tools

80. *Realizing multiplex-hidden imagery on a consumer 3D flat panel without any hardware modification. *Providing content creation environment for wider creators. *Adding a new value of 3D devices. Conclusion

81. *Why it cancel the hidden image while canceling image and hidden image are not on the same line? *Resolution of display is very high that people can’t recognize pixels. *It is not recognized as a pixel, (because it’s same size as pixels!!) ExPixel = Contrast compression + sub pixel

82. Contrast compression Before After

83. * Contrast * It needs to reduce each image’s contrast according to current algorithm. * Resolution * Vertical resolution became half, but horizontal resolution can be kept actual amount. * Image with black background for Naked Image * After pass contrast compression step, black color level 0 is pushed up to gray color level 127. Weak Points

87. Georg Wilhelm Friedrich Hegel « Vorlesungen über die Ästhetik » (Lectures on Aesthetics) 1835–8 1. Architecture 2. Sculpture 3. Painting 4. Music & Dance (=performing art) 5. Poetry Definition of “Arts” Modern philosophy in https://fr.wiktionary.org/ 6. Photography 7. Cinema 8. Television & Drama (=Broadcasting) 9. Bande dessinée, Manga, Comic 10. Video Game

88. Prehistric Manga Media

89. Manga Generator Pro by Progmind Inc

90. * We would mention thanks to all contributors of Manga Generator project, Yukua KOIDE, Wataru FUJIMURA, Yuto NARA, Genki KUNITOMI, Kosaku OGAWA, Ayumi ANDO, Ryo OKAMOTO, Hisataka SUZUKI and Rex HSIEH. * The project also continue parallel with “Manga Generator Pro” by Progmind Inc. * www.progmind.jp/service/manga- generator/ ACKNOWLEDGMENTS

92. Q: Original Authors says yes for Manga Generator? A: We have afraid with our respect for the authors but authors are already trusted their publishers and Photoshop toolkit helped the understanding. Q: Dynamic scenario is possible? A: Yes, “Manga Generator KAI” project successes to implement dynamic scenario. https://twitter.com/mangageneratorK Discussion from audiences

MANPU2016 Invited Talk: Manga Generator, a future of interactive manga media

Akihiko Shirai

MANPU2016 Invited Talk: Manga Generator, a future of interactive manga media