The use of multimodal representations in scientific journals, school science textbooks, newspapers and popular magazines is now very common. The science textbooks in current use in secondary schools invariably contain words and visual images, for example, schematic drawings, photographs, and mathematical and statistical graphs. Indeed the practice of combining image and language text have been an essential element in conveying scientific information to audiences from expert scientists, to school students and the general public long before as Lemke (2002) reminds us when 'Galileo inserted a tiny drawing of Saturn, with the rings that he first saw in place of the word Saturn in a line of written text.'
There have been attempts to theorize regarding the use of visual displays in science texts. Lynch (1990: 153-154), for instance believes that "visual displays are more than a simple matter of supplying pictorial illustrations for scientific texts, but they are essential to how scientific objects and orderly relationships are revealed and analysable".
Furthermore, Lemke (1998) found that scientific meaning throughout history is most commonly made up of two or more modes of representation. Lemke (1998) further acknowledged that today these representations of science have 'fused them inseparably',
'Scientific literacy is not just the knowledge of scientific concepts and facts; it is the ability to make meaning conjointly with verbal concepts, mathematical relationships, visual representations, and manual-technical operations.' (p. 38)
As there is a growing recognition that learning science in school entails understanding and linking verbal, visual and mathematical modes to develop knowledge of scientific concepts and processes, it follows that the authors of science textbooks would combine different modes to express the concepts and facts of the meaning intended.
If the assumption that it is 'absolutely normal and necessary' as Lemke (2002) suggests to 'interpret the verbal text in relation to other semiotic formation, and vice versa.' It would clearly be identified by the analysis of the intersemiotic semantic mechanisms by which the illustrations, diagrams, graphs and written text are working in conjunction on the page.
In this assignment, I briefly illustrate the approach by analyzing a multimodal text extracted from a science textbook that follows the National Curriculum Science at Key Stage 3. The excerpt is from Chapter 9 Environment Chemistry: Is global warming happening? by Levesley et al.(2008).
In order to demonstrate that these two semiotic systems, the visual and verbal modes complement each other, to realise an intersemiotically coherent multimodal text, I will use a similar approach that Royce (2002) used drawing on the social semiotic, metafunctional view of communication advanced by Halliday's (1994) systemic functional linguistics model, with a specific emphasis on the intersemiotic ideational meanings to discuss how each type of resource contributes to meaning-making. In order to analyse a multimodal text, Royce (2002) suggests beginning with 'an image and examining its encoded ideational features by asking a series of questions.' These questions are based on Halliday's (1994) functional categories of represented participant, process circumstances and attributes. The answers for these categories produce descriptive glosses for the image which Royce (2002) refers to as 'visual message elements (VMEs).' The final step in the process that he suggests is 'to look at the lexical choices to see how the visual ideational choices relate semantically to the written ideational choices.'
Analysing the Greenhouse effect image
The image represented in the textbook offers a scenario on which to think and in which to locate, add, and identify the main entities involved in the greenhouse effect. The image is initially used to represent what happens in nature, and thus it facilitates the sharing a common representation of global warming in our environment.
The image also facilitates the actions of representing changes of the sun's energy, locating them and making them dynamic. To communicate this kind of information, the textbook, and the scientific community in general, uses arrows. When arrows are added to the diagram, this begins to show what we know on the greenhouse effect in nature.
The diagram uses different arrows that give different meanings to the changes they represent. According to Kress and Van Leeuwen (1996), arrows are a graphic tool to represent a process in a narrative diagram: An arrow can represent force, energy, velocity etc. giving meaning to phenomena so varied as sunbeams, wind blow, superficial circulation of water, or a change in state such as absorption. In the case that we are analyzing the textbook uses arrows to change the location or direction of the sun's energy.
The textbook initially makes three fairly straight arrows which represent the changes in the solar energy (labeled solar radiation) when they reach the surface of the earth. There are a further three arrows that show the different paths the solar energy could take once deflected off earth's surface. All arrows coloured red to indicate the temperature (hot) and the high velocity at which the sun's radiation is travelling.
The first deflected arrow reaches high into space representing the path heat might take when it is reflected and lost. The second takes a slightly shorter path but still extends out of the earth's atmosphere representing the warm heat that the Earth radiates. The third arrow takes on an even shorter path in the same direction as the second arrow but curves back to the Earth representing being 'absorbed by carbon dioxide in the atmosphere and re-radiated back to the surface.
At the moment in which arrows become reflect from the Earth's surface they mark the space direction of the change that they represent, they are incorporating an iconic component (Lemke, 1999), they are correlating patterns of behavior and their visual representation, and they are facilitating greater complexity showing the relations of transitivity between the different entities represented. This constitutes both a description and an interpretation of the phenomenon to communicate the idea of changes in the sun's heat energy and the factor that carbon dioxide contributes to the process of global warming.
Analysing the Line Graphs
Figure 1.2 is a statistical graph representing the changes in mean temperature from 1860 to 2100. The graph uses a solid red line to indicate the temperature record on a certain date. The data record is extended for future years with a dashed line. The predictions indicate a sharp increase in temperature levels.
Figure 1.2 Figure 1.3
Figure 1.3 is a statistical graph representing the changes in the levels of carbon dioxide in the last thousand years. The graph uses a solid blue line to indicate the recorded levels of carbon dioxide.
Both graphs indicate a sudden rise either in carbon dioxide or temperature which may indicate to the reader the dynamic impact global warming is having on our environment. Figure 1.1 would have presented less of an impact if the prediction hadn't been added.
Analysing the 'Global Warming' Text
The multimodal text analysed from a science textbook (Levesley et al., 2008) deals with subject matter common to textbooks in general science and physical geography texts: global warming. The text (see the Appendix) contains three sets of visuals; a drawing of the greenhouse effect, a coloured line graph on the right-hand side in the top corner indicating changes in mean world temperature from 1860 until 2100, a second line graph on the left-hand side that is half way down the page indicting how carbon dioxide levels have changed over a thousand years and near the bottom of the page a drawing of the greenhouse effects. There are two purple-coloured boxes with questions and one yellow-coloured box with 'you should know â€¦' statements for reflection on knowledge retained from studying information from the multimodal text.
The three visuals reveal a range of 'visual message elements (VMEs)' both in terms of both whole diagram and in terms of the individual participants represented within the visual field. At the level of the whole figure are the VME the greenhouse effect image and the two graphs (realized by the verbal heading and the graphic images), and at the level of individual represented participants are the VMEs; climate, earth, and gas. These are realized not only through labeling of text but also visually by the red arrows on the Figure 1.1 and the lines on Figure 1.2 and Figure 1.3 which give a sense of direction or that some action or process is taking place e.g. carbon dioxide traps the sun's heat, rising temperatures are linked to increasing carbon dioxide levels.
The results of the analysis can be expressed by referring to the lexical chain analysis (Table 1 & Table 2) and moving back and forth from the visual to the lexical item. In a similarity to Royce (2002) findings almost every sentence in the verbal aspect of the text contains lexical items that relate semantically in some way to the two graphs and the image of the greenhouse effect. It also reveals that the visual have not be placed on the page randomly but have been placed to work to produce a coherent multimodal text.
TABLE 1- Intersemiotic Meaning for the Whole Diagram
Lexical items and intersemiotic relation
global warming; environment
weather; sea; coastal
temperatures; carbon dioxide
carbon dioxide; gases; forests; gas
carbon dioxide; gases; sun's heat
greenhouse effect; global warming
carbon dioxide; carbon dioxide
TABLE 2- Intersemiotic Meaning for the Represented Participants
In this assignment, a tentative framework for the analysis of two types of visual displays common in general science texts and a science text were used to show evidence that the multimodal aspects of the textbook are meaning-making. As may have been made clear in the discussion, the visual images in the science text are not redundant with language with language in meaning-making; they extend and complement it. Visuals are as Lemke (2002) put it "freeze into place what from time immemorial has been a function of bodily gesture and posture" which provide our students with the ability to make sense from the meaning of scientific text. However, Lemke (2002) provides us with some words of wisdom regarding text becoming more multimodal:
"Purely textual literacy will survive, but it will not continue to hold the dominant place that it has in the past. Students in all subjects need to know how to critically interpret and analyze â€¦ schematic drawings and diagrams, and other visual resources in relation to verbal text, and vice versa."
Indeed as Lemke (2002) suggests that the literacy education can learn a great deal from literacy as it has been "the pioneer of this multimodal fusion."
Word count: 1810
Figures 1.1, 1.2 and 1.3: Copyright Â© 2002 from Exploring Science 9 by Mark Levesley, Sandra Baggley, Julian Clarke, Steve Gray and Penny Johnson. Pearson Education Limited. Reproduced by permission pending from Pearson Education Limited 2002
Goodman, S. (1996) Redesigning English new texts, new identities: Visual English, London: Routledge.
Halliday, M. A. K. (1994) An introduction to functional grammar (2nd edition.), London; Edward Arnold.
Kress, G. and van Leeuwen, T. (1996) Reading Images: The Grammar of Visual Design, London: Routledge.
Kress, G., Ogborn, J. And Martind, I. (1998) A satellite view of language: some lessons from Science Classrooms, Language Awareness, volume 7. No. 2/3 pp.65-91.
Lemke, J. (2004) The literacies of science: Crossing borders in
literacy and science instruction: Perspectives on theory and practice. Newark DE: International Reading Association and National Science Teachers Association.
OToole. M. (1994) The language of Displayed Art. London: Leicester University Press.
Royce, T. (2002) Multimodality in the TESOL Classroom: Exploring Visual-verbal Synergy, TESOL Quarterly Volume 38, No. 2
Stenglin, M. and Isdema, R. (2001) Analysising English in a Global Context: How to analyse visual images, London: Routledge.