Multimodal Texts

Importance of multimodal texts

Multimodal texts are important because, these days, communication is shaped by a range of different, and overlapping, technologies. It is essential that we are able to understand multimodal texts so that we can take advantage of this.

Furthermore, combining different modes of communication to create meaning is an essential skill. Understanding them helps us develop a critical perspective of the media we consume.

Modes of multimodal text

There are five modes that can be combined to create multimodal texts:

• Linguistic
• Visual
• Gestural
• Spatial
• Aural

Linguistic

The linguistic mode refers to any sort of written or spoken text.

The linguistic mode is the most common mode because it can be used in a variety of ways. Think of all the places where you find written or spoken texts!

Visual

This refers to images and characters, such as marks, signs, and symbols. The visual mode involves the use of color, layout, style, size, and perspective.

Think of road signs, for example. They seem purely visual because they don't use any language or sound. Signs rely on the visual mode to get their message across. However, they also use the spatial mode. That's why we can say that signs are multimodal texts.

Gestural

The gestural mode is about movement and how we interpret it. Examples of gestural modes include facial expressions, hand gestures, body language, and interactions between people.

The gestural mode is important in face-to-face conversations, in the theater, concerts, and other live performances, as well as in YouTube videos and other platforms. This mode is usually combined with the other modes.

Spatial

The spatial mode refers to the organisation of elements in a space. It also deals with the physical closeness between people and objects.

Here are two examples of the spatial mode:

Fig 1. - Theatre stages are examples of a spatial mode

The stage is on a platform and the audience sits in chairs below, facing it. Here, the power is in the hands of the performers. This spatial order allows the audience to forget about their surroundings and to focus on what is happening on stage.

The audience sits in a circle that surrounds the stage. The performers and the audience are on the same level. This spatial order allows each audience member to get a unique view of what is happening on stage. Additionally, sitting in-the-round makes the viewers aware of each other, and enables their participation.

Both of these uses of the spatial mode create different experiences for the audience.

Aural

The aural or audio mode focuses on sound. This includes music, sound effects, ambient noises, silence, the tone of a person's voice, the volume of a sound, emphasis, and accent.

Some examples of the aural mode are radio programs, podcasts, and audio plays.

Examples of multimodal text

Multimodal texts are compositions that incorporate multiple modes of communication, including written text, visual images, audio, and sometimes even interactive elements. Here are a few examples:

1. Graphic Novels: These blend written text with visual images to tell a story. For example, Maus (1991) by Art Spiegelman.

2. Websites: Most websites are multimodal, combining text, images, video, and interactive elements.

3. Film: Films use moving images, spoken words, sound effects, and music to create a multimodal experience.

4. Video Games: Video games combine visuals, sound, text, and interactive gameplay.

5. Infographics: These visual representations use images and text to explain complex information quickly and clearly.

6. Advertisements: Most advertisements, whether print or digital, combine text, images, and sometimes sound to sell a product or service.

Let's look at the different types of multimodal text along with these examples.

Types of multimodal text

There are four types of multimodal text:

Print-based multimodal text

This is a printed text that combines different modes.

Comic books, graphic novels, picture storybooks, posters, leaflets, magazines, and newspapers are all examples of print-based multimodal text. They combine the linguistic mode and the visual mode.

Digital multimodal text

This is a virtual text that combines different modes.

Digital texts can be internet-based, such as social media content, blog posts, vlogs, and online games. They can also be found offline. Some examples are films, animations, e-books, video games, and presentation slides.

Digital multimodal texts can combine different modes. For instance, films and video games normally combine the linguistic, visual, gestural, and aural modes, whilst blog posts and e-books usually only use linguistic and visual modes.

Live multimodal text

This is a text that is performed live.

Live multimodal texts include theatre, opera performances, and concerts. Book premieres, poetry readings, and lectures are also examples of live multimodal texts. These usually combine four of the five modes: linguistic, gestural, spatial, and aural. Sometimes they can combine all five modes.

Transmedia multimodal text

This is a text that tells a story or sends a message through a combination of media platforms (each of which uses various modes.) A transmedia text is a multimodal text within a multimodal text.

Each added platform in a transmedia multimodal text further develops the story or idea and enhances the user experience.

Some games can be effective transmedia multimodal texts:

Both of these are examples of transmedia multimodal text.

Multimodal text analysis

To analyse a multimodal text, you should first recognise its different modes and be aware of the context they appear in. Then, you can proceed by pointing out those modes and using them as evidence to make an argument. You should also describe the modes in detail.

Multimodal text analysis example

Here is an example analysis of a multimodal text. In this case a street sign:

Fig 2. - A stop sign is an example of a multimodal text

The use of the linguistic mode –the written word 'stop'– reveals the message of the street sign. The drivers who see the sign need to stop driving. The visual mode supports the linguistic mode: the layout of the sign is white letters on a red background which is framed in white. This simple but effective layout draws attention and the use of red enhances the importance of what is written (that it is important to stop).

The spatial mode positions the sign on a road in what seems to be the countryside, where it is visible and not obstructed by other signs or buildings. This tells us that the drivers can immediately notice the sign and do as it says: to stop. This is important to do in the space where the sign is situated and will prevent accidents.