Multimodality In Clouds Over Sidra Essay.

Multimodality on Clouds Over Sidra

Discuss about the Multimodality in Clouds over Sidra.

Multimodality is defined as the relationship shared between communication and representation which goes beyond the realm of language. With changes in society, new technologies and media, addressing communication have evolved over the years. It has indeed become crucial for practice, teaching and researching in practical and academic areas. For such analysis a variety of methodologies, frameworks and techniques have been evolved (Machin 2016). In recent years communication using messages and texts, problems arising from using semiotic resources, interacting with each other and with language. For instance, sound design and cinematographic and production resources, page layout and image text, aural and visual art, dress, proxemics, gaze, and gesture have led to some necessary challenges. This essay focuses on the Virtual Reality film Clouds Over Sidra, told in a narrative fashion to depict the journey of a young girl Sidra, to highlight the type of multimodality used.

From Within (2016), the short video Clouds Over Sidra, produced by Aaron Koblin and Chris Milk in their production company VRSE, Sidra a 12-year-old girl from Syria, narrates her experience of her life living away from her home country. Throughout the video, she narrates to the audience describing each scene to the audience starting from her birth to her present life in a refugee camp in Jordan (Rethinked.org 2016). The short video which is a documentary has been presented in a 360-degree Virtual Reality (VR) video format and is around 8 minutes long. If the video had been a simple narrative format, it wouldn't have caused an immersive experience. 360-degree video format enables the viewer to directly interact with the video to see the full surrounding in a given scene, and VR format allows the viewer to directly view the three-dimensional spaces of the living characters. From Within (2016), a typical example of the scene is when she is watching television in a room; the audience can watch the television by sitting beside her, looking through a window. When the user sees down, the user finds himself or herself sitting on the same ground as that of her. This evokes emotion in a much deeper way than passively viewing the video. Journalists, storytellers, and producers will soon find themselves experiencing reality through immersive 360 videos. Biocca and Levy (2013) stated that Storytelling has evolved from text narration to a sensory experience. What it started from simple visual, sound was added later, and then came augmented and virtual reality which ultimately mixed with a sense of touch from wearables and other inputs. This creates a deeper narrative and theme (Dunleavy and Dede 2014).

Technological Advances in Multimodality

VR is a relatively new technological tool and is much more than just a video game peripheral. It connects a user to other humans who were unheard before in any form of media. It also changes the perception of each other. VR mode was also used for the first time to treat a cancer patient in Britain where people could see the live-stream of the surgery while finding themselves right in the operation theater (Meyercancer.weill.cornell.edu 2016). Kim et al. (2013) pointed out that this is a new mode of communication where a machine makes some more connected, empathetic and compassionate, and ultimately more human. For instance, one can experience the Colosseum as it was in 70 AD, or feel the plight of the day-to-day life of Syrian refugees in Jordan through the new technological mediums. They are still in their infancy but by 2020 it is predicted that one can witness a live event or a breaking news right from the spot where it is all happening. Karutz and Bailenson (2015) argued that live studio audience, documentary, and current affairs journalism will be presented this way. More and more stories can be experienced through the visually rich medium of 3d holographic AR projections, interactive VR spaces, where the environment can be mimicked to that of a virtual classroom, bedroom and living room. It will let a user move and interact with the objects for entertainment and education purposes once the gesture is introduced in the 3D mixed and virtual reality worlds. Emma-Ogbangwo et al. (2014), expressed concern that the current limitation preventing it from happening is the restriction of internet bandwidth and advanced video processing, that will create believable virtual worlds. Coleman.edu (2016) states Oculus Rift and HTC Live are some of the popular VR headset makers in the market right now. Digital experiences will undergo a breakthrough than just simple touch interactions and will be present in every medium (refer to Appendix 1). People will emulate their interaction with the digital content like they do physically in the real world (Liu, Yin and Zhang 2012).

Hardt (2007) stated that the Multimodal analysis comprises the analysis of all forms of communication. It particularly focuses on the text which includes the integration and interaction of various semiotic resources or modes of communication, to achieve the meaning of the text. The different resources interactive computing resources are software and digital media hardware). Technological products used like sound and image recording, architecture, writing, painting and carving (OHalloran and Smith 2012). Semiotic action and proxemics of other bodily resources are bodies, hand and face; and aspects of speech like vocal and intonation characteristics. Semiotic resources come with their constraints and affordances, both in combination and individually, analytical variations lie nature and terms of the media, scope, and detail of the analysis, and the difficulties arising from its implementation across media (Poria et al. 2015). For dealing with these challenges two major strategies can be identified – explore theory using analysis of text as both illustration and test of general principle discussions. The other strategy is to examine original texts and working on them before on to generalizations. The first method makes the theory problematic by not assuming, and most of the work is about integrating and comparing knowledge and practice from other different theoretical traditions (Benjamin 1970). Multimodal studies apply to the method when a person takes into consideration the vast range of academic and disciplinary traditions for the relevant multimodality. Theoretical generalization is usually applied in this way to explain various text types and texts, to illustrate and test, and derive from general principles (Burset, Bosch and Pujola 2016). Djonov and Zhao (2013) pointed out that analytical detail is of utmost importance in this approach, and the challenges of reproduction, analysis, transcription and accessing publication are quite apparent..          

Challenges and Approaches in Multimodal Studies

The film Clouds Over Sidra is in simple terms a short documentary film, and if released on a social video platform called Youtube or Facebook, would or would not have made much impact. The question then raised is how to create a short film that stands unique from the rest and can still tell a simple story to which the audience can relate emotionally (Kaiser 2016). To differentiate this movie from other works, the creators looked for a multimodal approach, which means to distribute the film (media) through many platforms like the with.in website, VR headset (modal) (Kiros, Salakhutdinov and Zemel 2014). To create the full multimodal experience, the film instead of having a video and someone else narrating, took a different approach. The narration is done by the same person the film is focused on, which makes use of the girl’s (Sidra) voice (media) to capture the audience. Hearing from a first-person perspective is more engrossing than hearing from a professional narrator (Taylor 2013). The directors Gabo Arora and Chris Malik took it a step further and implemented VR so that the audience can explore the scenery in each scene which creates an immersive effect. The voice, in particular, narrates every scene and being a 12-year-old young girl; the audience finds the voice more pleasing to hear and at the same time understand the emotions and feelings of the character by the tone used (Wall and Crevecoeur 2016). There are few scenes in the film where the voice complements the overall scene and helps make the scene much more impactful (Bateman 2013). For instance, at 0:52 (refer to Appendix 2) Sidra describes her shanty living place and her family while the viewer can explore the scene by rotating the video on the website or via VR and understand her living condition. From 3:16 to 4:13 (refer to Appendix 3 and 4), she describes the leisure habits of the Syrian men (Within 2016). This scene is commendable for the fact that unlike traditional films there is almost minimal to nil editing and cut-shots, and the viewer has fully fledged control of the scene to see what is going around (Ohanian and Phillips 2013). In the book Engagements with Narrative, the author Janine Utell describes the use of first-person narration in media create for a more autobiographical approach, and a form of direct communication with the viewers (refer Appendix 5) (Utell 2015).

Conclusion 

The value and applications of multimodal text analysis are enormous. Understanding of multimodality is essential to the study of classroom education and discourse. The addition of new techniques and knowledge from multimodal semiotics science shows new methods to the study and development of computational science, and multimodal text analysis is crucial in a wide variety of fields. Virtual Reality and Augmented Reality devices combined with haptic feedback of sensors from wearables and other input have created a new level of immersion to the storytelling and journalism, evoking a deeper feeling in the process and more connectedness among others. 

References

Bateman, J.A., 2013. Multimodality and Film. The Encyclopedia of Applied Linguistics.

Benjamin, W. 1970, ‘The Work of Art in the Age of Mechanical reproduction’, In H. Arendt & W. Benjamin (eds), Illuminalions, Cape, pp.219-226.

Biocca, F. and Levy, M.R. eds., 2013. Communication in the age of virtual reality. Routledge.

Burset, S., Bosch, E. and Pujolà, J.T., 2016. A study of multimodal discourse in the design of interactive digital material for language learning. New perspectives on teaching and working with languages in the digital era, p.163.

Coleman.edu. (2016). Virtual Reality Comes to Coleman, Thanks to Mr. Jeep and Harvard University | Coleman University. [online] Available at: https://www.coleman.edu/blog/?p=1065 [Accessed 5 Sep. 2016].

Djonov, E. and Zhao, S., 2013. Critical multimodal studies of popular discourse. Routledge.

Dunleavy, M. and Dede, C., 2014. Augmented reality teaching and learning. In Handbook of research on educational communications and technology (pp. 735-745). Springer New York.

Emma-Ogbangwo, C., Cope, N., Behringer, R. and Fabri, M., 2014, June. Enhancing user immersion and virtual presence in interactive multiuser virtual environments through the development and integration of a gesture-centric natural user interface developed from existing virtual reality technologies. In International Conference on Human-Computer Interaction (pp. 410-414). Springer International Publishing.

Hardt, M. 2007, ‘Foreword: What affects are good for’ in Halley, J. & Clough, C.P., The

Affective Turn: Theorizing the Social, Duke University Press, pp.9-13.

Kaiser, A., 2016. Do Documentary Films Constitute A Social Science.Political Analysis, 18(1), p.6.

Karutz, C.O. and Bailenson, J.N., 2015. Immersive virtual environments and the classrooms of tomorrow. The handbook of the psychology of communication technology, 32, p.290.

Kim, M.J., Wang, X., Love, P.E.D., Li, H. and Kang, S.C., 2013. Virtual reality for the built environment: a critical review of recent advances. Journal of Information Technology in Construction, 18(2), pp.279-305.

Kiros, R., Salakhutdinov, R. and Zemel, R.S., 2014. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539.

Liu, Y., Yin, Y. and Zhang, S., 2012, August. Hand gesture recognition based on HU moments in interaction of virtual reality. In Intelligent Human-Machine Systems and Cybernetics (IHMSC), 2012 4th International Conference on (Vol. 1, pp. 145-148). IEEE.

Machin, D., 2016. Introduction to multimodal analysis. Bloomsbury Publishing.

Meyercancer.weill.cornell.edu. (2016). Home | Sandra and Edward Meyer Cancer Center. [online] Available at: https://meyercancer.weill.cornell.edu/ [Accessed 18 Sep. 2016].

ÓHalloran, K.L. and Smith, B.A., 2012. Multimodal text analysis. The encyclopedia of applied linguistics.

Ohanian, T. and Phillips, N., 2013. Digital filmmaking: the changing art and craft of making motion pictures. CRC Press.

Poria, S., Cambria, E., Hussain, A. and Huang, G.B., 2015. Towards an intelligent framework for multimodal affective data analysis. Neural Networks, 63, pp.104-116.

Rethinked.org. (2016). Technology | rethinked…*. [online] Available at: https://rethinked.org/?cat=1083 [Accessed 5 Sep. 2016].

Taylor, L., 2013. Voice, body and the transmission of the real in documentary theatre. Contemporary Theatre Review, 23(3), pp.368-379.

Utell, J., 2015. Engagements with Narrative. Routledge.

Wall, B. and Crevecoeur, Y., 2016. Narrator's Voice: The Dilemma Of Children's Fiction. Springer.

Within. (2016). Clouds Over Sidra: A Virtual Reality (VR) film | With.in (360 video). [online] Available at: https://with.in/watch/clouds-over-sidra/ [Accessed 5 Sep. 2016].