SIGGRAPHの論文を書き終えました！

以下、PDF流し込み

“Real Baby – Real Family” – VR Entertainment Baby Interaction System

ABSTRACT
This research, “Real Baby – Real Family” – VR Entertainment Baby
Interaction System, is a project aimed at generating virtual baby
avatars from any 2D facial images. By pulling speci€c facial features
from photographs, analyzing them, and then merging the obtained
data together, the virtual baby avatars whose face closely resembles
that of subjects has been successfully created. Furthermore, this
experience contains various interactions with physical and virtual
baby avatar aimed at provoking thoughts regarding to solving social
issue of rapid aging society.

MOTIVATION
VR is a cross point of visual arts and experience. Character and/or
avatar must have interactivity in VR. Haptics and force feedbacks
are also important channel. If the emotion can be integrated and
implemented into a computer generated experience with a rich
channels, we can extend a meaning of current virtual reality contents.
this project, “Real Baby – Real Family” – VR Entertainment
Baby Interaction System, can bring user to a life with a baby which
has an audio, visual, and holdable tangible feedback. It may solve
declining birthrate by changing relation and understanding of birth
by a condenced moment of VR expernence with users’ baby.

RELATEDWORKS
Below are some of the previous researches that deals with child care
and/or real time image manipulation. Neuro-Baby[7], Infanoid[1],
Kismet[2], Cog[5], Babybot[3], YOTARO[4]. Amongst these researches
YOTARO can be considered a work that emulates a realistic
baby. YOTARO is a baby robot that can simulate a runny nose
and many different emotions. However, because YOTARO’s face
is that of an illustration and not a real-life portrait, it looks as if
the baby is not related to the subjects. this research generates a
baby from portraits of heterosexual or homosexual couples playing
it thus subjects get a baby looking just like the subjects.
Face2Face[6] is another research we look into. It uses monocular
facial reenactment in real-time to manipulate the target video into
acting out the facial motions of the source. this research deals
with real time video manipulation and not still image manipulation
however, and it can only manipulate the facial motion and not the
age progression of the portraits.

DESIGN
This VR research realized a virtual family enabling the subjects
to communicate with a baby through haptic, visual, and audio
feedback while wearing a head-mounted display (HMD). According
to our hypothesis, creating a believable baby resembling the subjects
will make them more emotionally invested with the experience.
Following this theory we created a virtual baby generated from
subjects’ photos. The most important part of this research; however,
is that we constructed a physical baby that can interact with the
audience. ”Real Baby – Real Family” is unique for allowing subjects
to (1) hug and interact with the baby physically, (2) creating a baby
looking just like the subjects, (3) contains audio components.

3.1 Holdable robot device
In order to hug a baby doll without experiencing discomfort while
wearing the HMD, there is a need for high precision position tracking
of the doll. However, this is hindered by image recognition
ability of an one eyed camera which slows down the position tracking
speed. Our proposed method uses multi-point image sensor
implemented in the controller of HTC Vive to enable high precision
position tracking. This synchronizes the physical and virtual
movement of the baby and allows subjects to hug the baby without
getting visually disoriented.

3.2 Baby Face Generator
Below are three characteristics of our baby face generator: (1)
Generating a face fitting the subjects’ skin tone, (2) Creating the
baby’s face from multiple photos, (3) Inverse the age of the subjects
to fit that of the baby.
3.2.1 Average Baby Face Generation. It determines the skin tone
of the baby by averaging the skin tone of the two photographs. This
system also morphs the colors and shapes of the faces using OpenCV.
Lastly it collects 16 baby face images from each of the following
categories using Google Image Search: Japanese, European, and
African baby.
3.2.2 Get Face Landmark Index. In order to morph the many
photographs, it is necessary to obtain the index. In the proposed
method, we use the Dlib of open source library, from the results that
have been learned by the data set iBUG-300-W, to get the points
of each part of the face. As a result of fig.1(Left), the eyes, noses,
mouths, eyebrows, feature points, and the 68 points with index
composed of contour, are all morphed automatically. Here it can be
seen that morphing are concentrated in the mouth and even parts
of the face.
3.2.3 Inverse Age Progression. Inverse age progression is the
process of generating a final image from the averaged age of portraits.
This is all done without collapsing baby portraits obtained
in Average Baby Face Generation. The implementation uses a Dlib
and OpenCV. It obtains feature points performed at Dlib, performs
image generation by passing the coordinate data of the feature
points OpenCV side. Dlib automatically sorts the feature points
obtained and treated them as a landmark. Its index is unchanged
in all of the facial images. By setting the contribution ratio with
respect to the index number, it is possible to process each selective
parts. The color components are produced by Average Baby Face
Generation, the present process is a concept that only features the
specified parts inherited. If carried out well as deformation of 3D
modeling, contour around the chin and other aspects should also
be considered. The generated facial image is used by Live2D.
3.3 Display of Haptics Synchronized with Voice
When holding the baby robot doll, the baby robot is strengthening
the interaction by vibro of Vibro transducer Vp2 (Vp210) which was
mounted in the baby doll. This vibro is controlled by voice. Only
by passing the band-pass filter using a Fourier transform effective
frequency band (5Hz, 200Hz) and extracted as the vibration data
onto an output audio can it perform real-time tactile presentation
through HMD attendant controller (Vive Controller). However,
since it’s the only vibrator that is built in the standard Vive controller,
we felt it’s electric current output which vibrates the baby
doll was insufficient after many exhibitions. To fix this problem we
mounted the Vp210 onto Vive controller to improve the oscillating
capability.
4 FUTURE OF REAL BABY
“Real Baby – Real Family” enable many meaningful interactions
with baby avatar such as hugging, giving milk, calming down, and
causing him/her to burp. Currently the project is being refined
according to feedback received during public exhibitions such as
the baby behaves differently from an actual baby and inclusion of
changing diaper being strongly desired due to the importance it
played in raising infants. It is our wish to enable aspiring parents
and parents-to-be to experience the joy and essence of caring for
a new-born as well as raising the question of nurturing the future
generation of humankind, and merging virtual baby with emotional
driven interactive scenarios aimed at provoking thoughts of raising
a baby may be the solution to rapid aging society nowadays.
Acknowledgement: Special thanks to Fujitsu Social Science Laboratory
for helping us realizing this project. VR Entertainment
Application Knowledge (VREAK) Center, and Kanagawa Institute
of Technology.

おわりに、PDFを流し込んだあとの文字化けを直す作業が心に響きました….