Project Concept and Intended Experience
Shadow “Play” is an interactive art installation that brings the traditional art - Chinese Shadow Play, closer to the public. People can choose their preferred scenes and shadow puppetries, and then interact with the scenes by manipulating actual shadow puppetries. The visual effects of the different objects within the scene are triggered depending on how people manipulate the shadow to move around on the curtain.
Shadow "Play" uses a projector to simulate the light source of traditional shadow play and restores the story scenes of traditional shadow play by directly projecting scene frames. At the beginning of the interaction, people are presented with a short video introduction, including the roles people play in it as well as tips for manipulating shadow puppetries and instructions for the next step at the end of the video. We provide four scenes divided by seasons -- spring, summer, autumn, and winter, as well as four different shadow puppetries for people to choose. People can use the same shadow puppetry to explore different scenes and change them at any time. When people manipulate shadow puppetries on the curtain and interact with objects from the scene, the objects produce visual and sound feedback. The visual and sound feedback of the triggerable objects is designed according to the characteristics of different seasons. The purpose of this is to restore the sense of storytelling in traditional shadow play through this specific perspective. Because in traditional shadow play, shadow puppetries may be used repeatedly, but the same characters advance different storylines in different plays. When people choose scenes and shadow puppets according to their preferences, they are also choosing different "stories" they will “see”. While people manipulating the shadow puppetries on the curtain, they can also explore the scene as it should be for that season through visual and sound feedback.
People are intended to feel closer to traditional arts as they interact with Shadow "Play". Intangible Cultural Heritage (ICH) is playing a significant role in human society, as the distillation of the precious history and culture of mankind, an explanation of human behaviour in the present, and a historical guide to the future. The traditional art performance as one of the forms of ICH, it has certain requirements for performers, which to a certain extent raises the barriers for the public to know them and widens the distance between art and the public. The technology applied here, is to provide people another role while appreciating traditional arts – not only as audiences, but also as performers. It aims to make traditional arts more accessible to a wider audience through immersive experiences and novel interactions and offers a potential method for preserving the disappearing ICH.
Technical Development
Within the development of Shadow “Play”, there were three stages where major changes were made and influenced our final delivery. These there stages can be formed into the following:
1. Conceptual Stage
2. Prototype Stage
3. Final Exhibit
In the conceptual stage, tangential learning was suggested as the theoretical support to communicate the interaction medium to the traditional arts. We also did research on previous work to identify the gap between the practice and theory, to better form the interaction flow for Shadow “Play”. In terms of that, we proposed to use hand shadows to trigger the corresponding traditional shadow puppet character on the curtain, and people could interact with the scene by moving their hand shadows. To interact with the scene, we adopted a machine-learning method to capture the location of the users’ hand shadows. If the location of the users’ hand shadow is closed enough to the location of the interactive elements in the scene, the animation would be triggered. The users can access the visual and sound responses from our design.
In the prototype stage, the concept has changed due to user feedback and technical limitations. Instead of applying hand shadows as the interaction medium, which users felt was hard to connect with the traditional arts, Shadow “Play” changed to restore the actual shadow play performance. We applied actual shadow puppetries for people to manipulate and interact with the curtain. Shadow “Play” applied Hall Effect sensors to detect the magnetic shadow puppetries. We stuck the Hall Effect sensors at the location of our interactive elements and put the magnet behind the shadow puppetries. The Hall Effect sensors would be triggered if the puppetries are close to the elements. Then, the users will receive the visual and sound effects from the scene. In the final exhibit stage, several detailed changes were made from the last stage related to the intended experience. The Hall Effect sensors as the detection are still in use. However, from the feedback of the teaching team, the interaction journey might be hard to deliver the intended experience due to the lack of integrity of the actual interactive experience. In terms of this, we proposed a background introduction video at the beginning of the interaction. And provides more selections of scenes for people to choose from instead of giving them a specific setting with limited interaction steps. For this change, we used Lilypad as the colour recognition sensor to detect which season card people choose. And this also allows people to freely change scenes during the interaction. Finally, Shadow “Play” integrated the features of traditional Chinese Shadow Play and the technology to deliver our intended experience.
Technical Description
Shadow “Play” comprises four parts: scroll, scene, shadow puppetries and screen, which all come together to provide the intended experience of Shadow “Play”. The users can choose the season of the scene from the scroll. To achieve the interaction journey, we built two modules for the technical implementation part, including the Input from Arduino sensors and the Output for responding to the user's behaviour. The scene part is made by Unity and is used for providing sound and visual feedback to the users. Furthermore, the puppetry part is a tool that helps the users to interact with the scene. Lastly, the screen part is the physical device, which allows the users to interact with the elements in the scene.
The scroll part includes four hall effect sensors, one colour sensor, Flexible Strip with LEDs, and one button. Firstly, we adopt these four hall effect sensors to detect whether puppetries are on the scroll. We use this sensor because it’s easier to implement and is more stable compared with other sensors, such as light sensors. This action will be captured if the users take one of the puppetries. Then, the Flexible Strip with LEDs is lighted to offer visual feedback to the users. Secondly, to offer the function to let the users select the season of the scene, four cards corresponding to four seasons are placed on the scroll. The users can pick up the corresponding card and place it on the colour sensor to select a certain season of the screen. Since the colours of the card are fixed, we believe the colour sensor is the best choice to implement this function. Lastly, the button is built into the scroll. The users can watch the introduction video by pressing it.
We built the shadow play scene part with Unity since we aimed to provide motion and sound feedback when users interact with the scene object. As we aim to create an immersive learning experience, we adopted a learning-based and task-oriented interaction as an intended experience. Below is the development flow that we applied in the scene part:
1. Construction of the basic scene.
2. Create animations and event triggers.
3. Add input functionality to object and event trigger to obtain the key code input.
4. Building of corresponding animation feedback.
5. Implement SceneManager to control the switch between scenes.
6. Apply sound resources to each scene and object.
In the basic scene, game objects are sorted into different layers in Unity so that we can edit game objects separately. First, we added various layers and made EventSystem control layers without affecting unrelated objects. We then added animation to items we intended to give users dynamic feedback. Followed by Implementing the animation on trees, clouds, and birds. By adding animation to items we intended to give users interaction feedback. Next, we applied a C# script event trigger to control the animation, which uses InputSystem as a functionality library. InputSystem is a functionality which can get input such as click and key press actions. So that we can use getting input to trigger the objects as our intended experience(feedback). For instance, the leaves would fall after the hand shadow collides with the tree object. InputSystem is also used as a controller to let the user switch the scenes.
The physical shadow puppetries is made up of two parts: design and construction. We used Adobe Illustrator and Adobe Photoshop to create them. First, we used Adobe Illustrator to create the outline of the puppetries. Then we imported it into Photoshop to turn each part of the puppetries into a different layer for adding color and material. After designing the puppetries to mimic the traditional Chinese shadow making method. We used Photoshop to divide and print the limbs of the puppetries. Afterwards, the plastic sheet was glued to the limbs with double-sided tape to increase the thickness. With the cooperation of two people, we used a hole punch to punch holes in each limb joint. Finally, we threaded wooden sticks into the sides of the shadow to manipulate it.
The shadow puppetries are the devices that allow the users to offer input to our scene. We stick two magnets behind each puppet. Two for each puppetry is because, after user testing, three magnets make the puppetry too heavy to use, and the location of the puppetry is hard to detect if there is only one magnet.
The screen part consists of a projection curtain and eight hall effect sensors. Hall effect sensors provide an intuitive and interactive experience, removing barriers of complexity that could deter a modern audience. We place four locations of the interactive elements on the screen. Each location has two magnet sensors to increase the chance of detecting the puppetries with the magnet behind them. Once the sensors are triggered, they would send a certain input to the Unity scene. Therefore, the application of Unity is to provide motion and sound feedback when users interact with the scene object. Also, to create animations and event triggers.
In conclusion, we mainly use the hall effect sensors and Unity to implement this design. Since hall effect sensors can precisely offer the correct inputs for the scene, they act as a bridge between the users and the shadow world. Furthermore, we adopt Unity to offer the users various sound and visual feedback. Hence, we believe we can provide the intended experience for the users through the above technologies.