Concept
Smart Mirror is a magical interactive item that helps people choose
and manage clothes in their daily life. The mirror contains some
magic elements which attract people who are fond of pop culture. The
mirror is installed in people’s homes and is connected with the
user’s wardrobe.
The main interactions include picking occasions, changing clothes,
and communicating with the mirror. When the user wakes the mirror up
by holding the wand(mirror controller), the current weather is shown
in the mirror(snow, rain, and wind). Then the user tells the mirror
which occasion they would like to go to so that the corresponding
clothes will be filtered. Users point their upper and lower bodies
with the wand to change tops and bottoms. There is a parrot at the
upper right corner of the mirror, used to give feedback or
communicate with the user when picking outfits. The parrot can shake
its head and wave wings to represent agreement or other opinions. At
the same time, the parrot will give feedback by talking with users
to guide them find the best outfit effectively. The number of times
the clothes have been worn is displayed on the mirror, enabling
users to know how many clothes are worn regularly and how many are
unused or forgotten when users switch between clothes. The parrot
also reminds users to organize their unworn clothes from time to
time, allowing them to use and manage their clothes more
effectively.
Technical development
As the main part of the mirror, we use a large display screen and
Kinect camera. The effect of the mirror is achieved by presenting
the user on the screen through a camera. The Kinect is connected to
the computer, and Unity is used to implement most functions. In
order to present the effect of wearing clothes, the team uses the
skeleton system to track the human body and add 3D clothes models on
the joints of the human body. At present, we only have 3 models of
set clothes for use to choose. The team uses the "main camera" in
Unity to take pictures and upload these pictures on the screen to
implement the screenshot function. Also, to simulate different
weather, we add the weather effect between the "main camera" and the
"Kinect screen" (rain, snow). To implement the change of the scene,
we apply the background removal script to remove the real background
from the Kinect camera and add pictures instead, like the function
of changing background in Zoom (Online meeting software).
In the remote control part, the team originally planned to use ESP32
as the controller and let Unity detect the controller to implement
the interaction of changing clothes. However, because of the time
limitation, this part couldn't be finished. Instead, the team uses
Unity to detect the joint of the hand of the user and uses the
gesture track script. Therefore, the change of clothes can be
controlled by detecting the user's movement by waving the wand left
and right.
There is another device, which is a bird on the side of the mirror.
An Arduino Uno board and the three servo motors are placed inside
the bird doll. Arduino Ide is applied to control three servos to
simulate the bird's actions(shake its head & fan its wings). There
is also a speaker in the bird to give feedback to users. In order to
simulate the bird giving feedback, we recorded some feedback and
processed the sound into a sound close to that of a parrot through
voice changer software. Then, according to the keys, the parrot doll
is controlled to play positive feedback and swing wings, as well as
give negative feedback and shake hands.
Final Statement
Exhibit Experience
The smart mirror attracted a large number of visitors and provided engaging experiences for them. Participants included students, professionals, and children. As the exhibition was held outdoors, the light and wind had an impact on the imaging, and the "mirror" presented blurred portraits. In the afternoon, the portrait became clear enough to display for the exhibition. When visitors arrived, the team first introduced the main functions of the mirror and then asked them to stand in front of it and interact with it based on our instruction.
Public Response
The outcome of the test of the prototype can be considered similar
to a fully functional prototype, as almost all the visitors cannot
identify that if the team is simulating the parrot feedback as an
AI. Most of the visitors are thrilled with waving the wand to change
the clothes, and since the outfits are 3D-modeled, people could not
help but turn their bodies and pose in front of the mirror. However,
due to technical issues, users currently cannot see the back of
chosen clothes.
In addition, visitors enjoyed taking selfies and making comparisons
with hand movement. But, not all users can immediately understand
how to take a snapshot. The current interaction is that users need
to use their left hand to control a cursor and grab and release a
camera icon. Some users choose to grab the camera with their hand
but not the cursor at the first attempt.
Surprisingly, most male testers show great interest while trying the
device, although currently, it only provides female outfit choices.
It was more common for male users to shake their bodies and do sexy
poses to see how the dress was moving. There is a chance that this
device can explore male user's outfit choice boundary, which is male
users cannot wear a dress.
Some users can't hear the feedback from the parrot clearly because
the tone of the parrot has been changed. It is good to make the
voice sound like a parrot, but it also needs to be clear enough to
be understandable.
Future Work
In the future stage, the priority is to implement all the simulated
parts. Therefore, ESP32 can be developed more and applied to the
device as a controller. AI technology and Amazon Lex(Voice
recognition) can be applied inside the “parrot” to provide users a
real communication experience. The device will also need the network
to display the real-time weather.
According to the response from the exhibition, there is evidence
that male users are still needed to be considered in the group
users. The mirror will present more styles of clothes, especially
for males, so that users can switch different kinds of clothes more
flexibly.
