Smart Mirror is a magical interactive item that helps people choose and manage clothes in their daily life. The mirror contains some magic elements
which attract people who are fond of pop culture. The mirror is installed in people’s homes and is connected with the user’s wardrobe.
The main interactions include picking occasions, changing clothes, and communicating with the mirror. When the user wakes the mirror up by holding the wand(mirror controller), the current weather is shown in the mirror(snow, rain, and wind). Then the user tells the mirror which occasion they would like to go to so that the corresponding clothes will be filtered. Users point their upper and lower bodies with the wand to change tops and bottoms. There is a parrot at the upper right corner of the mirror, used to give feedback or communicate with the user when picking outfits. The parrot can shake its head and wave wings to represent agreement or other opinions. At the same time, the parrot will give feedback by talking with users to guide them find the best outfit effectively. The number of times the clothes have been worn is displayed on the mirror, enabling users to know how many clothes are worn regularly and how many are unused or forgotten when users switch between clothes. The parrot also reminds users to organize their unworn clothes from time to time, allowing them to use and manage their clothes more effectively.
As the main part of the mirror, we use a large display screen and Kinect camera. The effect of the mirror is achieved by presenting the user on the screen through a camera.
The Kinect is connected to the computer, and Unity is used to implement most functions. In order to present the effect of wearing clothes, the team uses the skeleton system
to track the human body and add 3D clothes models on the joints of the human body. At present, we only have 3 models of set clothes for use to choose. The team uses the
"main camera" in Unity to take pictures and upload these pictures on the screen to implement the screenshot function. Also, to simulate different weather, we add the weather
effect between the "main camera" and the "Kinect screen" (rain, snow). To implement the change of the scene, we apply the background removal script to remove the real
background from the Kinect camera and add pictures instead, like the function of changing background in Zoom (Online meeting software).
In the remote control part, the team originally planned to use ESP32 as the controller and let Unity detect the controller to implement the interaction of changing clothes. However, because of the time limitation, this part couldn't be finished. Instead, the team uses Unity to detect the joint of the hand of the user and uses the gesture track script. Therefore, the change of clothes can be controlled by detecting the user's movement by waving the wand left and right.
There is another device, which is a bird on the side of the mirror. An Arduino Uno board and the three servo motors are placed inside the bird doll. Arduino Ide is applied to control three servos to simulate the bird's actions(shake its head & fan its wings). There is also a speaker in the bird to give feedback to users. In order to simulate the bird giving feedback, we recorded some feedback and processed the sound into a sound close to that of a parrot through voice changer software. Then, according to the keys, the parrot doll is controlled to play positive feedback and swing wings, as well as give negative feedback and shake hands.
The smart mirror attracted a large number of visitors and provided engaging experiences for them. Participants included students, professionals, and children. As the exhibition was held outdoors, the light and wind had an impact on the imaging, and the "mirror" presented blurred portraits. In the afternoon, the portrait became clear enough to display for the exhibition. When visitors arrived, the team first introduced the main functions of the mirror and then asked them to stand in front of it and interact with it based on our instruction.
The outcome of the test of the prototype can be considered similar to a fully functional prototype, as almost all the visitors cannot identify that if the team is
simulating the parrot feedback as an AI. Most of the visitors are thrilled with waving the wand to change the clothes, and since the outfits are 3D-modeled, people
could not help but turn their bodies and pose in front of the mirror. However, due to technical issues, users currently cannot see the back of chosen clothes.
In addition, visitors enjoyed taking selfies and making comparisons with hand movement. But, not all users can immediately understand how to take a snapshot. The current interaction is that users need to use their left hand to control a cursor and grab and release a camera icon. Some users choose to grab the camera with their hand but not the cursor at the first attempt.
Surprisingly, most male testers show great interest while trying the device, although currently, it only provides female outfit choices. It was more common for male users to shake their bodies and do sexy poses to see how the dress was moving. There is a chance that this device can explore male user's outfit choice boundary, which is male users cannot wear a dress.
Some users can't hear the feedback from the parrot clearly because the tone of the parrot has been changed. It is good to make the voice sound like a parrot, but it also needs to be clear enough to be understandable.
In the future stage, the priority is to implement all the simulated parts. Therefore, ESP32 can be developed more and applied to the device as a controller.
AI technology and Amazon Lex(Voice recognition) can be applied inside the “parrot” to provide users a real communication experience. The device will also need
the network to display the real-time weather.
According to the response from the exhibition, there is evidence that male users are still needed to be considered in the group users. The mirror will present more styles of clothes, especially for males, so that users can switch different kinds of clothes more flexibly.