As promised in the last article, we will continue to analyze the cinema of the future in its immersive form. Immersiveness is certainly linked to the experience it can create in the viewer. In this article we will therefore talk about experiential cinema through old and new storytelling , first person , greater involvement of the spectator and the five senses , and more data and artificial intelligence .
What is a media, or “mediated history”
The characteristics of traditional cinema
Each medium has advantages and disadvantages. Or, we could say, possibilities and impossibilities. Going into detail, we quickly see the five peculiar characteristics of the media at the center of our attention: cinema .
- Static and linear narrative structure
- Single or double mode
- Episodic
- Third person perspective
- Passive audience
Static and linear narrative structure
It is the basis of cinema: a unidirectional narrative structure with beginning , development , climax and end . Fixed and static, it obviously cannot be modified by the viewer. Sure, there are Flashbacks or Flashforwards , but they don't fundamentally change this approach to storytelling.
There is usually a causal chain: each step narrated in the script leads to a subsequent step that is a consequence of the previous one.
And movies , of course, don't change over time . Except for very rare cases of errors or problems that occurred after the film's release (e.g. Kubrick cut the last minutes of The Shining after its theatrical release), these remain the same from the moment of release, and forever.
A good analysis was done in 1992 by Prof. David Pinault in Story-Telling Techniques in the Arabian Nights .
Nonlinear structure in traditional cinema
In postmodern cinema there is actually a concrete attempt to modify this linear structure. Often an attempt is made to "intertwine" the story, leaving the viewer to use intuition and irrational instinct to understand the film, while logic and drama are dissolved and diluted.
The linear structure often develops into a multi-part symphony with a non-linear structure. In the following sections we will better understand how this will develop even further into experiential/interactive cinema, forming various personalized narratives (which is reminiscent of video games).
Single or double mode
By this we mean that cinema involves at most one or two senses , sight and hearing . Clearly current cinema is always dual, as we can only include silent films in the "single mode" category.
Over the years, various attempts have been made to improve the involvement of the other senses. Let's think of great directors who manage to convey a sense of taste to the viewer, for example in films such as Eat, Drink, Man, Woman , Ratatouille or Mid-August Lunch . Clearly, it's a trick on our brains. But ultimately, cinema itself is.
Episodic
According to Prof. Jason Mittell in his book Complex TV: The Poetics of Contemporary Television Storytelling , current cinema and media are generally episodic; that is, they tend to develop around an event or series of interconnected events. This is even more true, of course, in the world of news media.
A story is told based on facts and events, real or fictitious. It is told from the narrator's point of view, which makes it easier for the viewer to enjoy but at the same time distances him empathetically from the story.
Third person perspective
This perspective is an important feature in our analysis. In fact, although possible, first and second person perspectives have always been little used in traditional cinema . In the history of communication in general, it was abundantly used only in the radio of the first half of the 20th century.
Heart of Darkness, first-person radio adaptation
For example, considerJoseph Conrad 's Heart of Darkness . It was adapted for radio in 1938 by director Orson Welles (famous for making Americans believe they were under Martian attack through the radio show The War of the Worlds , in 1938). The goal was to have the protagonist tell the story directly in the first person. Interesting how Welles himself, new to cinema at the time, tried to persuade RKO Pictures to make the film version.
It had all the potential to become one of the greatest films of all time and, perhaps, raise public awareness of issues that, if misinterpreted, led to the Second World War the following year. But it was the use of the first person, in addition to the political themes little loved by the majors , that probably pushed Hollywood not to consider its feasibility. It was a drastic break from the rules, and the world wasn't ready for it yet.
In fact, Coppola tried to catch up in 1979 with Apocalypse Now , only loosely based on the novel “Heart of Darkness” as it was set in Vietnam and not Africa. It is certainly too late to see peace in Europe and the world.
The first person in the history of cinema
There are some sporadic cases of filmic use of first-person narration, especially in its early years. Cases that are sometimes little known, but which in some way have tried to change the way we see things. First of all I think of Dr. Jekyll and Mr. Hyde from 1931, better known in Italy as Doctor Jekyll and Mr. Hyde , by Rouben Mamoulian .
Other cases were William Dieterle 's 1934 film The Firebird . Or The Lady in the Lake filmed in 1947 byRobert Montgomery , and Dark Passage by David Goodis from the same year.
Two other examples, decidedly more modern, we will see shortly in the section on experiential cinema, as they are more useful for a comparison with the cinema of the future.
I have only cited examples of first-person works, because talking about third-person works would be impossible or useless... These are practically almost all existing films. And then because, from a perspective of creating the more experiential cinema of the future, I believe that these ideas must be well taken into consideration.
Then you'll tell me what you think, I'm interested in being closer to the technical world than to film criticism.
Passive audience
As we have said, the story in current or past media is told in the name and on behalf of the narrator . Whether it's the writer or the director, the journalist or the speaker, everyone shows you what they want. Have you ever wanted to look at something from the side, or behind the camera, but the director didn't show you?
This happens especially often with sports on TV, which is perhaps why television itself is among the first media to have “progressed” in a direction decided by the viewer . In Italy, as Wenner Gatta mentions , when the “ Spidercam ” was inserted on the football fields, the spectator could choose with the remote control whether to see the match with the classic direction or only via the spidercam. And from there, above all, Sky continued to develop the technology a lot by exploiting multiple broadcast channels for the same event.
After a study of the cinema that has been, and still is, we will finally analyze the characteristics of the new cinema, projected towards the future, to also understand how to overcome the problems of the last 127 years .
The characteristics of experiential cinema: the future
The five characteristics that we will be able to find, all or in part, in the cinema of the future, according to what technology currently makes available, can be:
- Immersiveness
- Interactivity, non-linearity and sociality
- Multisensory presentation
- Algorithmic, personalized in real time thanks to data
- First person perspective
Immersive cinema
We already know some partially experiential media. Let's think about immersive virtual reality and augmented reality platforms . As we saw in the last article , these fully fit into the line of continuity between the real and virtual world hypothesized by Paul Milgram.
Immersion means “ enveloping the user in a real physical space using augmented or mixed reality on a portable or wearable device, also including haptic interfaces ”.
I found one of the first practical examples in the paper “ Situated Documentaries: Embedding Multimedia Presentations in the Real World ” , by Tobias Höllerer , Steven Feiner and John Pavlik and taken from the 1999 International Symposium on Wearable Computers .
Image from “ Situated Documentaries: Embedding Multimedia Presentations in the Real World .”
These “environmental documentaries” were entirely based on wearable devices, to integrate novels and documentaries into real-world locations . The system is very reminiscent of current AR glasses, obviously with the technology available in 1999. It was a backpack with a GPS tracker and a 360° video camera (developed by Shree Nayar , of Columbia University), a type of portable computer with graphics, audio and video, and augmented reality glasses capable of reporting points of interest in the real environment. It was called MJW ( Mobile Journalist Workstation ).
The glasses could also reproduce rudimentary 360° videos superimposed on reality, and the gaze was the main pointing system. By staring at an object in the real world for at least half a second, it was selected, producing related information and multimedia files. You could also travel through time by tapping the desired year on the portable display.
A great example of what augmented reality will be 20 years later. AR is in fact developing rapidly, becoming common use since 2018 following the implementation of the native ARKit API in iPhones (which, among other things, I believe to date is the latest innovation worthy of note in the smartphone field).
The first point of our cinema will therefore be immersion. But certainly a different immersion from what we've seen so far. The basis will in fact be the story told, the cinema. We will not immerse ourselves in the world, we will not immerse ourselves alone or in virtual communal environments . The immersion will be produced by the 360° screen, the stereoscopy, and the "room", or dome, with internal elements directly linked to the story narrated .
The screenwriters will have to be really good, to develop plots that make the viewer feel "involved", without being obvious. Obviously, the first thing that comes to mind: keeping the protagonist of the story constantly seated, who must impersonate us spectators. It is at best a starting point that should not be underestimated, but above all the brainstorming will be really interesting in the early stages.
Interactive, non-linear and social
Word to the viewer ; choice . These may be the key words for the non-linear structure of experiential cinema; which is therefore more complex.
Although without particular rules, it maintains its own logic based on temporal sequence (or order ), duration and frequency , which are the concept of division proposed by the French essayist Gérard Gennette in the field of literary fiction , later introduced in the field of film criticism by Andre Gaudreault and David Bordwell .
The order of events in stories
In this section we open a general parenthesis on the stories; in fact, there are characteristics in common with future immersive cinema. Gennette clarified that the event could have occurred:
- before narration ( analysis or flashback)
- in the future (therefore only announced or expected, the prolepsis ).
- Again, events can be told in a different order from how they happened ( anachrony ), used to make the story more compelling.
- More rarely there may be a movement between one narrative level and another ( metalepsis ).
An example is the "author's metalepsis", a sort of passage of the author from the outside to the inside of the story, or vice versa if a character becomes a narrator.
Another literary example is that of the poet Virgil , who "kills" Dido in canto IV of the Aeneid , orDiderot who writes in Jacques the fatalist : "Who could stop me from marrying the Master and cuckolding him ?". Both examples taken from Armando Mollica Bonivento 's doctoral thesis at Ca' Foscari University (in Italian), which I invite you to read for greater understanding.
The duration of events in stories
So, after the order (event that happened before, future event, narration in a different order from the real one or passage between different levels of narration), we have duration . Also definable as the "rhythm", the "speed" with which events are told.
We can mainly divide it into four types:
- ellipses (very accelerated rhythm), with frequent chronological jumps;
- summary (relatively fast paced), in which a story is summarized into its main points. They can be of variable length.
- scene : relatively slow, it is the classic almost real-time narration. An example is the dialogues;
- descriptive : no progress in the story, we stop to describe a certain moment.
Clearly, these types can be combined. We can have, for example, a summary inserted within a dialogue.
The frequency of events in stories
Frequency is nothing other than the relationship between how many times a certain event occurs in reality (even invented), and how many times it is told. If, in practice, the same event is narrated several times (or the same statement by a character is repeated).
I leave you the link to an interesting article on the subject , from which the following image is taken.
The cinema of the future has an intersubjective non-linear structure
After this long interlude, let's try to understand why the story, or the screenplay, is decidedly more complex for the cinema of the future. In fact, all these elements must be inserted within an intersubjective non-linear structure . That is, the story can move forward and backward in time, in an environment common to other viewers who may want to make different choices than ours.
Intersubjectivity is the biggest problem to solve in writing new screenplays . In fact, if interactivity is a concept already well known thanks to video games, shared interactivity between multiple people, with necessarily a screen that shows the same images to everyone, presupposes that a small democracy is created within the room.
The intersubjective nonlinear structure has a great advantage: it respects the viewer . It grants the right to choose one's own story, to judge the morality of certain scenes. The spectator becomes the center of the film as well as a part of it. Current cinema is decidedly too one-sided, and if it has lived unchanged for so many years it is only due to the simplicity (inherent in its characteristics) of using it as a means of political and commercial propaganda .
Thinking about the average use of cinema, which is also and above all a moment of relaxation and not very active entertainment, we must not, however, fall into the temptation of inserting excessive "gamification", transforming it into a video game. I mean, nowadays we go to the cinema to relax with friends or family, to pass the time without thinking too much. And choosing involves thinking… This is why interactivity must be limited and not even mandatory, and probably will not be the top priority in creating the cinema of the future.
We will see in some future articles why interactive cinema has not been successful, albeit in a few past experiences. But it's related to this.
Multisensory presentation
Experiential media may seem like something new in recent years, but that's not entirely true. For centuries, humanity has developed its own unique characteristics, improving available technology in small but steady steps.
Wearable devices to engage the senses
According to what Prof. John V. Pavlik reported in the book " Journalism in the Age of Virtual Reality ", the first experience of "wearable" devices can be summarized in the Chinese invention of the ring/abacus , a wearable measuring instrument dating back to the Qing dynasty of the seventeenth century.
Subsequently in Europe, in 1780 the "pedometer", a step counter, was developed, arriving in 1965 with the (unsuccessful) American attempt to create the first exoskeleton ( Hardiman ) to allow humans to lift up to 650 kg.
In recent years, developments have certainly been much faster, also thanks to the logarithmic scale of the technology which rarely stops once started. To understand, did you know that in 2004, just 18 years ago, the GoPro wearable camera came out… And that it even used 35mm film?
Cinema stimulates us physiologically and sensually
My body is not just an object among all objects, but an object sensitive to all others, reverberating to all sounds, vibrating in all colors, and giving words their primordial meaning through the way it receives them.
Maurice Merleau-Ponty in Phenomenology of Perception
At the beginning of this paragraph I included the trailer for the film Piano Lessons , Jane Campion 's 1993 masterpiece. I chose it as an excellent example of how current cinema tries, in more or less orthodox ways, to trick the brain to involve senses that are not directly involved (touch in this case). I also invite you to review the last article that talked about Matthew Shifrin and his Legos for the blind.
This is the magic of cinema. Art in this sector has reached unimaginable heights, even if we try to think of how to go further. How to physically stimulate the other senses. Although already in the 1940s, the philosopher Siegfried Kracauer wrote:
The material elements that appear in films directly stimulate the material layers of the human being: his nerves, his senses, his entire physiological substance.
Siegfried Kracauer
How to involve the five senses in cinema
How then to involve the five senses, or at least more than two, in cinema? It will be necessary to proceed gradually, as technology allows. Let us therefore consider the experiments done in the past, to then understand how WE can involve the five senses for our spectators.
Touch
Meanwhile, touch : William Castle , in 1959 filmed the horror film The Tingler . He inserted a vibrating device called “Percepto!” in the seats of some cinemas, which was synchronized with the action. Castle himself was an evil genius… Before the screening of Macabre in 1958, he had everyone given a $1,000 insurance policy in case of death or fright during the film. Then during the 1959 film House on Haunted Hill , he brought a phosphorescent skeleton above the audience. Based on a pulley system called “ Emergo ”.
And finally there is “ Illusion-O ”, launched with the film13 Ghosts : all elements in the frame, except the ghosts, were subjected to a blue filter. The ghosts on the other hand had a red filter, and were superimposed on the frame. The public received cards with red and blue filters: looking through the blue filter, the ghosts could not be seen. Through the red filter, however, they could be seen.
Still on touch, of course the modern 4D cinemas that we all know have entire vibrating seats, as well as ropes that touch the legs, fans for the wind effect (hot or cold) and water sprays for humidity.
Smell
Smell : One strategy was adopted by John Waters for the 1981 film Polyester . This is the “Olorama” system, basically cards with numbers to scratch off. Each number has a smell (rose, pizza etc…), which can be smelled at the required time (a small number appeared on the screen).
Another 1960 attempt was Smell-O-Vision , used only in Scent of Mystery produced byMike Todd Jr (son of the famous Mike Todd Senior, author of Around the World in 80 Days ). Smell-O-Vision involved introducing up to 30 evocative odors into the audience via tubes leading to individual seats in the theater, with perfume bottles held on a rotating drum.
Today a company has revived the concept in a modern industrial production, called Olorama . It would be nice to collaborate with; objectively, the system seems much more functional than olfactory cards (which were actually only reused in a couple of children's films).
The criticism he received is interesting. According to the Times , some viewers complained of delays between the smell and the scene, others found the odors mixed unpleasantly, Henny Youngman said he didn't understand the film because he had a cold.
Other inconveniences to pay close attention to are nausea and headaches caused by too strong and persistent fragrances, possible discomfort, distractions and heavy air.
For completeness of information, I quote Walter Reade Jr of AromaRama . Key difference with Smell-O-Vision? Simply, the AromaRama used the air conditioning system to diffuse aromas. Smart.
Eating with someone else is a way of saying: “I'm with you, I like you, let's form a community together”.
Thomas C. Foster
Taste
We have seen touch and smell, obviously taste is missing. Even if progress has been made towards taste "transmission" systems (through objects to lick), I don't think they are yet developed enough and, above all, we are not yet ready to welcome them. And I don't know if we ever will be...
However, during some screenings of Willy Wonka and the Chocolate Factory , Wonka chocolates were provided to viewers . And from here I came up with the idea of providing the projected food directly to the spectators. A double advantage: for identification in the film, and for the economy of the cinema and the customers who would respectively sell and buy better food than popcorn and Coca-Cola.
I soon discovered that this idea is not exactly original: in 2012 in London, more precisely in Notting Hill, Edible Cinema took place. It was a collaboration between the Soho House team, organizer Polly Betton and experimental food designer Andrew Stellitano . In practice, everyone present had numbered bags and glasses containing food and drinks, on the seats there was also a menu explaining the meals, and a woman appeared at the side of the screen during the film to indicate when to eat or drink each number.
Algorithmic, personalized in real time thanks to data
In a data-centric society, cinema cannot stand idly by. Of course, respecting privacy and possibly without using this data for not very noble purposes.
A great use could come from geolocalization inside the dome, to possibly send different signals to different spectators. But, above all, it will be possible to take into account the direction of the gaze of the latter to understand what is more interesting, and to have a more passive, and therefore less tiring, "input device", due to the interactivity we have just talked about.
Other anonymized data, such as moment-to-moment physiological responses, can be useful for developing subsequent films and gauging audience reactions. Which, in a subsequent development of experiential cinema, can also be exploited within the story itself (for example, interactively managing volumes to increase or decrease human reactions).
Women's Aid, a great example of algorithmic interactive advertising
On the occasion of World Women's Day in 2015, an interactive blow-up depicting the face of a woman victim of violence was installed in the Canary Wharf business center in London. A facial recognition camera was used to update a counter and change the image every time a passerby paid attention to the advertisement. It was a typical example, albeit in the advanced advertising sector, of using data to modify the result obtained.
Artificial intelligence for storytelling
Already today many stories are created with artificial intelligence, as the American Associated Press does for example. The Times also created an algorithmic robot, the QuakeBot , which automatically acquires data from the US Geological Survey , the US earthquake analysis organisation, to automatically write the complete article with magnitude, epicenter and time. The human editor just needs to check it for accuracy and publish.
These robots, these artificial intelligences, are led to the development of increasingly engaging, interactive and multi-sensory stories. This will be fundamental for the cinema of the future, to assist human screenwriters in writing increasingly complex and autonomously unmanageable stories.
Furthermore, they will be able to exploit the data present in the world, and in the cinemas themselves. To create "tailor-made" stories for the target audience.
Artificial intelligence itself is, and will increasingly be, used in technical video production. Google, for example, has the Jump compiler (well described in their paper ), which takes care of the stitching of 16 high-quality video streams to obtain a complete 360° VR video. The main result achieved was the elimination of a large part of the latency , thus increasing the sense of reality of the image shown.
Ultimately, AI will certainly be the center of attention. And, to avoid controversy, it will also have to be used with kid gloves, focusing on privacy and the importance of the human being.
First person perspective
We already know some partially experiential media. Let's think about immersive virtual reality and augmented reality platforms. As we saw in the last article , these fully fit into the line of continuity between the real and virtual world hypothesized by Paul Milgram .
These are clearly in the first person, since the real protagonist of the story is ourselves. The experience is given by contact (albeit still virtual) and direct observation of objects and events in the ways we like most.
Experiential cinema will often have to be first-person. This is unlike most films present and past, which instead aim to tell us stories with eyes external to the story. But ours will be a different first person: we will be the spectators, the main character. Recently, in 2016, the film Hardcore , directed by Russian musician Ilya Najšuller , had good success. Evidently producing cinema from a cultural background external to it helps to take risks and innovate it.
All Hardcore is shot in first person. I believe that the success is deserved, and it is a good point of reference for the cinema to come. A way of narrating video games, which invites us into someone else's life, looking at the world through their eyes. Well, I think the only difference is that it will have to be our life, inserted into the new cinema. I know, it's scary, but the writers shouldn't even tell us...
I also want to mention another film, not entirely in the first person (the protagonist is seen in various scenes, removing the effect of total identification) but which has a decidedly more constructed and engaging plot than Hardcore: Enter the Void , from 2009, directed by the Argentine director Gaspar Noé .
Two films that I recommend you see, first because it will be a special and different experience. Second, to get a taste (albeit small) of what cinema will be like in a few years.
To identify everyone in the history of experiential cinema
How can you create a film for many, which can realistically represent the life of each of them? Once again, the screenwriters will have to do a great job of multiple introspection. Creating engaging but generalist stories, inserting characters that will also be new for the protagonist of the story. No known relatives, maybe a distant cousin we didn't know we had. Nothing else.
In 2015, on the occasion of the fourth international conference on Design , User Experience and Usability in Los Angeles, Aaron Marcus collected in a useful book a series of information on the current relationship between information technology, the virtual world and human beings. Starting from the assumption that each of us has our own culture and knowledge, design (and therefore those who create it) must leave us the freedom to experiment, to have doubts and thoughts, must build a world accepted by all the personal cultures of the spectators and, at the same time, propose their own idea of the universe.
We want and need to reflect, to understand. And this absurdly leads to hyperrealism , that is, to a construction of the virtual world that is very faithful to the real world, in which to experiment and be present firsthand.
The importance of sociality in the cinema of the future
We have seen, talking about the first-person perspective, that technology now allows us to decide on a place, in a given historical moment, and live there virtually. But one thing will be important: we will have to experience it with others .
Sociality is a theme that should not be underestimated: there is no life alone . To reconstruct the real in the virtual, other human beings (and living beings in general) must therefore necessarily be present. And a concrete interaction with them.
Ultimately, reality is a concept inherent in our mind. Many things are real to you and me, while others are only real to one of us. “Mediated” reality has an intrinsic complexity in this, mainly because it must be able to reproduce a synthesis of our personal realities.
Let me explain better with an example: Aurora has the habit of calling her mother in case of problems. In his reality, the mother is always present. He gives her advice, hugs her, offers her constant support. Marco , on the other hand, has a bad relationship with his mother. She has always had problems, he tried to help her for years but without success.
Aurora's reality requires a loving and very present mother, whereas Marco's rejects her. The screenwriters of the new cinema will therefore have to be able to fall into hyperrealism, without causing moral accidents with any of the spectators. How do you solve this? Important characters can, or perhaps should, be ambiguous. An ambiguous character that allows everyone to "accept" him by mixing his own internal reality with the reproduced and therefore "mediated" one. In short, see them as you wish.
Ultimately, the “static” part of the real world, like trees and houses, is always there. And it's easily reproducible. On the other hand, the human part, the social part, represents a decidedly more complex choice, although not insurmountable.