We are using a combination of Unity and Arduino, simulate the
natural environment of the ocean and show the creatures, landscapes
and phenomena. In order to bring a good experience to the user, this
project allows the user to operate gloves and a stress ball to
control the tuna in the display.
The main equipment includes gloves, stress balls, and a display screen. The glove is equipped with a gyroscope to detect the inclination of the hand to control the direction of the tuna. The fingertip part of the glove is equipped with a pressure monitor to detect the pressure of the fingertip, so as to simulate the feeding behavior of tuna. A pressure detection device is installed inside the pressure ball to detect whether the pressure ball is under pressure, so as to make the tuna move forward. In addition, there is a vibration device inside the glove to alert the user if the tuna has touched other entities in the scene.
When the project starts, users need to put on gloves and hold the stress ball in their hands, and then they can see the scene simulation of the underwater world appear on the display screen.
When users want the tuna to advance, they need to squeeze the pressure ball with their hands, and each squeeze can move the tuna forward a certain distance. When users want to turn the tuna, they just tilt their palms. Leaning to the left means to turn left, and leaning to the right makes the tuna turn to the right.
After manipulating the tuna close to the prey, the user needs to touch the fingertips of both hands to generate enough pressure to make the tuna eat the prey. When the tuna touches other objects in the scene, such as stones, the vibration device in the glove will vibrate to remind the user.
The original prototype consists of a marine environment in Unity and
a physical model with sensors on Arduino circuit playground or
external. The connection between Unity and Arduino board is realized
by Ardity. The specific sensors and technologies used are as
1.Arduino circuit playground's own accelerometer controls the turning of fish (as before). Change to one-handed control.
2.Flex sensor controls the way the fish moves forward. Flex sensor is similar to bone, which connects users' hands and wrists and can sense their activities. The user's hands bend left and right once in a group, and each group will control the fish to advance a certain distance. The speed of left and right bending determines the speed of advance. This way can not only vividly simulate the movement of the fish, but also control the swimming speed of the fish.
3.The press sensor located at the mouth of the toy fish controls the way of eating. When the user squeezes the fish's tongue with his finger, the fish will perform the action of "eating".
4.Unity scene adds pop-up prompt box to remind children. When fish eat other low-level creatures, the pop-up window will tell them what they just ate. When the fish touches the shark, the motor inside the toy fish will vibrate to remind children that it is a dangerous creature. When children try to eat a shark or are about to be eaten by a shark, a pop-up window will appear and warn them, but the system will not show the process of fish being eaten. This picture and experience is unpleasant for children. We hope that children will learn about marine life in a relaxed environment and have more positive thinking.
In addition, the fish controlled by children will vibrate when they touch the shark, but the pop-up window will not appear. They can decide whether to leave or eat the shark by themselves. Only when they try to eat the shark or are about to be eaten by the shark will the pop-up window appear and warn them. If a child sees the fish he controls being eaten by a shark, it may make the child feel sad. Therefore, we try to prevent this from happening by directly informing them. Although we give children tips and warnings, we still hope to give them the opportunity to explore and try on their own initiative.
5.When the fish eats the prey, the microphone that comes with Arduino circuit playground will play pleasant sound effects to cue the child a proper predation has been completed.
On the day of exhibition, we were so excited that we could show our product to the public. We set up our place by 12:00pm, there was something wrong with our projector, but luckily our tutor brought us a good one.
We also prepared some snacks for the audience.
After 1:00pm, we got our first audience, they were impressed by our product and played for a while. And then, our teaching team came to experience and mark our product. After marking, we basically welcomed many students audience to experience and ask them for suggestions.
But, our flex sensor was broken, since someone treated our Dory fish so harshly, we then decided to fix this during the break before 5 pm.
From 5 pm to 7 pm, we welcomed our photographer first.
Then, there were many professionals or people from industry visited us and provided many valuable suggestions.
After 7 pm, our tutor announced the prizes for some groups, congratulations to them! Before 8 pm, we packed up all the settings.
Our team enjoyed the whole process of the exhibit since it gave us a chance to get some valuable feedback from professionals and learn how to deal with emergency situation since we needed to fix the circuit.
Feedback & Response
In general, the evaluations we received from the teaching team and audiences are positive and constructive. The visual effects displayed in the prototype, the simulation, and the natural role-playing and control methods received a lot of affirmation and appreciation. In the interaction with virtual reality, the synchronized vibration and sound feedback of the device in hand also surprised some audiences. During the exhibition, we also received valuable feedback, which mainly focuses on the following aspects:
1.The requirement for more interactive content in the visual scene. The current prototype can only demonstrate the most elemental functions, including an edible jellyfish and a shark that cannot be preyed upon. Other fish, plants, hills, lights, and bubbles in the scene serve more as aesthetic visual effects or pictures than as entities with interactive functions. This demand is reflected in the fact that the majority of the participants at the exhibition, after completing their interactions with specific jellyfish and sharks in practice, usually tend to look for other objects in the scene, such as schools of jellyfish or seaweed, to perform the action of predation. Secondly, the collection and reward mechanism of the game itself was also mentioned by individual viewers. For example, if the glittering pearls in the scene can be added to the prototype gameplay design as collectible props or achievements, it is likely to motivate our target group to engage in interactive activities and explore the virtual world.
2.Functional upgrades to suit the workplace. One of the professionals who coined for us highly praised our prototype and offered ideas and some possibilities about specific application scenarios. He believes that our concepts and products are suitable for use in museums as a means of popularizing marine knowledge. In this case, The physical interaction scenario of the product will expand from a small place to a hall or exhibition room. A wide venue means more individuals can be involved in prototype interactions as well as group activities and collaborations. After a group discussion, we believe that this idea is more relevant to the proposed concept than the original implementation, which can be taken as a direction for the future development of this product.