The proposed project is situated at the interface of experimental literature, multimodality, technology and translation. In itself, multimodality is not new. After all, at least as far as texts are concerned, mono-modality does not exist as such (Baldry 2000; Baldry and Thibault 2006:58). The study of sign systems beyond the printed word, however, did not receive serious academic attention until the second half of the twentieth century. In early theorisations, the hierarchical order between verbal and non-verbal modes of expression is clear: the former (i.e. the printed word) is seen as a higher-order entity than the latter (e.g. pictures). This view is advanced, for example, by Barthes (1977a:39), who perceives images as a "floating chain of signifieds" that has to be 'fixed' by verbal language through interpretation. Images, accordingly, are to be interpreted by recourse to the verbal text (a process that Barthes terms 'anchorage'). It is the verbal text that "holds the connoted meanings [of images] from proliferating, whether towards excessively individual regions â€¦ or towards dysphoric values" (ibid.). The "projective power of pictures", in other words, is to be restrained and neutralised by the "repressive value" afforded the written word (ibid.:40).
It is only since the publication of Kress and van Leeuwen's (2006) ground-breaking Reading Images that visual signs have come to be seen as semiotic entities in their own right. They now constitute "an ensemble of modes" (p. 41) that includes such aspects of textuality as the physical medium of writing, font type and layout participating in meaning-production as an active ingredient of a holistic text. In this semiotic view, texts are seen as "objects which result from a variety of representational and production practices that make use of a variety of signifier resources organized as signifying systems" (ibid.:216). These systems are called "modes", which, together with different types of media - "signifier materials", "the surfaces of production (paper, rock, plastic, textile, wood, etc.)", "the substances of production (ink, gold, paint, light, etc.)" and "the tools of production (chisel, pen, brush, pencils, stylus, etc.)" - constitute the materiality of a text (ibid.).
Kress and van Leeuwen's (2006) seminal contribution lies in their foregrounding of the extra-linguistic aspects, and in particular the visual dimension, of discursive constructions. They thus opened up multiple and exciting avenues of application in disciplines as disparate as literacy studies (Street, Pahl and Rowsell 2011), anthropology (Howes 2011), design (Björkvall 2011) and fine arts (Mavers 2011; West 2011). In textual studies, multimodality has often been researched within the general field of discourse analysis (e.g. Kress and van Leeuwen 2001, 2006; Scollon and Scollon 2003; Baldry and Thibault 2006; Royce and Bowcher 2007). Media texts have been the dominant focus, and more recently attention has turned to new media and audiovisual texts that tap into sophisticated digital technologies, such as Internet websites (Jones 2011), mobile devices (Leander and Vasudevan 2011) and video games (Lemke 2011). In this regard, literary texts have generally been neglected.
A recent attempt to fill this gap was made by Gibbons (2012), who analyses case examples of experimental English fiction within the framework of multimodal cognitive poetics. She taps into several concepts from visual communication and multimodal studies, two of which are relevant to my proposed study, namely, embodiment and multiple sensory perceptions. Embodiment, a central concept in cognitive linguistics, is the nexus that ties linguistic structures to the daily interaction between our bodies and the physical world. In multimodal experimental literature, the body - both of the text and the reader - is intimately involved in the construction and enactment of the literary experience. Through the process of embodiment, the body of the reader interacts with the materiality of the text in performing the act of meaning production. In the event that the physical body is not directly implicated, multiple sensorial faculties invariably come into play. A distinctive feature of experimental literature is its simultaneous engagement of our sensorial capacities, particularly the verbal, visual, auditory and tactile. This cognitive phenomenon has found support in neuroscientific studies, which inform us that the norm is for our senses to operate in tandem, rather than independently, to produce "a unified representation" of the sensory world (Ghazanfar and Schroeder 2006:278). This engagement of multiple senses produces a unique type of literature the critique of which cannot be based on its verbal component alone. Such literature, which deploys multimodality in its structuration, needs to be deconstructed on the basis of the interaction between the text and the plethora of senses invoked and played out.
Gibbon's (2012) work is part of a more systematic programme in the Western literary establishment to locate experimental literature in a theoretical context. The recent publication of The Routledge Companion to Experimental Literature (Bray, Gibbons and McHale 2012) is the culmination of this programme. Based on literary works and movements in Europe, this important reference maps the development of experimental literature since the early twentieth century, as well as its contemporary manifestations in the digital age in the form of web fiction and code poetry. To date, no work of this nature and scale has been seen in the Sinophone context, although there have been sporadic efforts along this line, particularly with regard to the materiality of the Chinese signifier. For example, Yau (1992, 1997) discusses the gestural images that underpin the construction of Chinese characters and idiomatic expressions, although not strictly in the literary context. A more relevant work is that of Chang (1992), who looks at 'scriptorial performance' in literature by focusing on how the visuality of the Chinese graph is exploited in the writing of three Taiwanese novelists. Arguing for a graphemic perspective on literary writing, Chang foregrounds such nonverbal dimensions of texts as typography, orthography, punctuation and spacing and critically considers their subversive effects on the conventional written code of Chinese. Exploring the poetry of two Chinese writers from different generations and locales, Yeh (2008) espouses a 'poetics of noise' that focuses on aural discordances in modern and contemporary writing and celebrates such poetics as a radical effort to challenge established literary boundaries. These studies, and a few others, show that the nonverbal and multimodal aspects of Chinese literature have already caught the attention of some Chinese literary scholars, a fact underscored by the 2005 special issue of Chung Wai Literary Monthly, which centres on the theme of biosemiotics (although it must be stressed that the articles in this issue do not deal primarily with Chinese material). Research in this area is, however, still inadequate and far from systematic, thus constituting the first gap that the proposed project seeks to fill.
The second gap that the project intends to address is the connection between translation and multimodality in literature. At the confluence of the two lies 'intersemiotic translation', as famously coined by Roman Jakobson (1959/2004) and defined as "an interpretation of verbal signs by means of signs of nonverbal sign systems", otherwise termed 'transmutation' (p.139). Although intersemiotic translation emerged as part of Jakobson's tripartite conceptualisation of translation (the other two concepts being 'intralingual translation' and 'interlingual translation'), it has remained unexplored until relatively recently and even then in rather restricted domains. A cursory survey of five major references in translation studies - three handbooks (Gambier and van Doorslaer 2010; Malmkjær and Windle 2011; Millán and Bartrina 2012), an encyclopaedia (Baker and Saldanha 2009) and a translation companion (Munday 2009) - shows that none contains a separate entry on intersemiotic translation, and only one an entry specifically on 'multimodality and translation'.
As a concept, multimodality does appear in some of the aforementioned references, but usually in association with audio-visual translation. This is in line with ongoing research in the field, where digitally mediated texts (from film subtitles to video games) have become fashionable in multimodal translation studies (Gambier 2007; Remael, Orero and Carroll 2012). Indeed, the link between multimodality and audiovisual texts is so strong as to render the terms 'audiovisual translation' and 'multimodal translation' (as well as such terms as 'multimedia translation', 'screen translation', 'media translation', and the like) almost synonymous in the literature (Chiaro 2008). In addition to audio-visual translation, advertising translation also invokes the concept of multimodality (Torresi 2009), particularly with respect to cultural problems in adapting the visual aspects of advertisements (Smith 2008; Torresi 2008). In the context of cultural adaptations for commercial purposes (as, for example, in product localisation [Pym 2010]), intersemiotic translation is construed as "a particularly effective instrument when the very image, not only of a given product but of a whole brand and the values it aspires to embody, must be adapted to different target cultures" (Torresi 2008:68).
The literary genre has not received the same attention in this regard, with the exception of children's fiction, in which illustrations are seen as 'translations' of the verbal text (Pereira 2008). The translation of children's literature into another language raises the question of how images interact with texts in complex ways across cultures (Alvstad 2008; Kansu-Yetkiner and Oktar 2012). It is now a commonplace that such non-verbal elements as pictures, sound and movement (Oittinen 2008), visual paratexts (Gerber 2012), paraverbal behaviour (Nord 1997) and punctuation (Malmkjær 1997) need to be taken into account in translating this sub-genre. Within the broader field of literature encompassing serious literature, theatre and cinema, Poyatos (2008) explores the reader's (and translator's) sensory-intellectual investment in his/her reading of literature and its implications for the translator's task. Focusing on the interaction between the verbal and non-verbal elements in literary communication, including visuals, sound and kinesics, Poyatos (2008) provides copious examples from (primarily English-language) literary texts, but unfortunately not from their actual translations. Interesting though it is, this work is nevertheless based on expert contemplation, and, although translation is a recurring theme, Poyatos remains very much focused on the non-verbal nuances of original texts rather than on their translations.
Having said that, his work advances new frameworks and hypotheses that are valuable to the study of the materiality of literature. One aim of the empirical work to be undertaken in the proposed project is to test some of these hypotheses and frameworks, particularly Poyatos's (2008) sensory-intellectual model of reader-book interaction, which constitutes a physical conditioning element of the reading experience (Poyatos 1997). The project will also examine other nonverbal elements operating within the text that mediate the reading act in both original and translated texts, including orthography (Haas 1970), typography (van Leeuwen 2006; Nørgaard 2009) and punctuation (Poyatos 1981). The important outcome of this endeavour will be to supply Chinese-language material to the relevant research literature, in which there is a dearth of data beyond those in Western languages.
The proposed project thus represents an attempt to sketch out the contours of experimental Chinese literature, with an eye on multimodal works that employ translation as a mode of composition. It will depart from previous studies by focusing not only on how the multimodal features in a given text are reproduced in translation, but also - and perhaps more predominantly - on the interlingual and/or intersemiotic operations that underlie experimental literary production. In other words, I am interested in translation in two senses of the word: in its more conventional sense, it is the transcreation of a text written in one language (the source language) in another language (the target language); in its more conceptual sense, it becomes translational, that is, the manifestation of trans-boundary discursive practice within the creative fabric of a literary work. Translation in the latter sense is conceived not as a technical act that is external to a text and applied to it, but rather as an inherent mechanism that constitutes its very form.
In this connection, the project is an extension and expansion of my earlier work in translation and multimodality in experimental Chinese literature, in which I explore how text, translation and technology work together in engendering post-modernist reading experiences. In Lee (2011a, 2011b), I examine the bilingual poetry collection Pink Noise, one of the most ambitious attempts in deconstructive poetics to date by a Chinese author. In this collection, machine translation is employed in a most uncanny way: whilst illogicality and non-fluency are often seen as negative attributes that evidence the incompetence of computer programs in translating 'softer' texts such as poetry, Hsia Yü, the author of Pink Noise, exploits those very attributes to create translated Chinese poems that push the limits of intelligibility. In this literary experiment, Hsia revels in the arbitrariness and materiality of meaning production in two ways: first, the 'original' poems are composed as a kind of pastiche by juxtaposing random lines plucked from the Internet; second, these poems are translated using a computer program and then modified on the basis of the new contexts arising from the automated translations. In so doing, the poet subverts the authority of the 'original' texts by allowing the machine translator maximal intervention in their formation and by constantly compromising the perceived boundaries between (creative) original and (translated) copy. The resulting translations are full of collocational and syntactic disjunctures, from which emerge a noise aesthetics (cf. Yeh 2008) that challenges both the sensibility/sensuality of the reader and traditional assumptions about literary meaning.
In the aforementioned studies, I analyse the notion of materiality in Hsia's poetics at two levels. The first entails the physical constitution of the book, including its medium (vinyl in the case of Pink Noise) and such typographical aspects as layout and colour. I consider how the reader's impeded bodily engagement with the book has the effect of slowing down - at times even breaking down - the reading process, and how this dovetails with Hsia's concept of incommunicability in literary meaning. The second level has to do with the poet's fetish with the written sign. I argue that through the literal, unwieldy machine translations, Hsia directs the reader's attention away from 'meaning', focusing it instead on the corporeality of the word itself. Machine translation is thus not just a technical process in generating bilingual poetry but is also central to the articulation of a material poetics.
More recently (Lee 2012), I turned to experimental literary works based in Taiwan and Hong Kong that exhibit strong intersemiotic potential; that is, they employ at least two modes of representation in a translational relationship. These works include a collaborative project in which the essence of local poems is 'translated' into visual installations; a case of cyber-literature in which Chinese poetry is translated into English and presented cross-medially (the original and translated texts are 'read' in tandem on-screen and on soundtrack); and another case of cyber-literature in which a written poem that plays with the materiality of the Chinese script is intersemiotically transposed into an animated video on YouTube. Interlingual translation also figures in each of these cases, but to varying degrees of prominence. Based on these examples, I hypothesise that with the proliferation of digital technologies have emerged various forms of experimental literature that sometimes tap into the operations of translation, as broadly defined. These new forms necessitate "a multimodal perspective on translation and a translational perspective on multimodal expression" (Lee 2012:14), which together can reveal the interplay between verbal and nonverbal modes of composition in textual practices.
Considerable data remain uncovered as far as translation and experimental Chinese literature are concerned, and the theoretical consequences for literary and translation studies deserve greater critical attention. The Chinese scholar Wang Ning (2009a) has made some recent forays in outlining an 'iconographical turn' in literary/cultural studies. A critical part of this turn is what Wang calls 'cross-cultural interlingual-intersemiotic translation', which effectively is a broader type of intercultural textual practice that encompasses translation and transcends semiotic borders (2009a, 2009b:195-237). This new mode of translation practice crosses three types of boundaries: those between languages, those between cultural traditions and those between different arts and disciplines (Wang 2009a:42). This proposal answers Wang's call for the study of intersemiotic translation in the contexts of picture theory and iconographical criticism (Mitchell 1994). In this project, I will look into the interlingual and intersemiotic operations that transpire in Chinese literary experiments, with a view to articulating these operations in terms of post-structuralist conceptions of signification. In so doing, the study will contribute to the field of translation/literary studies by furthering research into an enlarged perspective on translation practice (Tymozko 2007) using texts by Chinese authors and by positioning multimodal Chinese writing/translation as part of the iconographical turn in literary and cultural studies.
(b) Research plan and methodology
This research will involve a series of case studies on multimodal Chinese literature. The data that inform it are hence literary in nature and multimodal in form. To demonstrate the intersemiotic potential of Chinese literature, my sources will be varied and wide-ranging, including, inter alia, conventional print publications, image writing, cyberspace literature, cross-media performances and literary artefacts. The criteria for text selection will be as follows. (1) Do the literary works foreground their materiality by using two or more media platforms and/or by invoking two or more sensory faculties? (2) Do the works in question enact a dialogue or 'translation' between different languages, cultural traditions, and arts and disciplines? (3) Is technology involved in the literary production in question with the aim of achieving certain corporeal effects?
Methodology: Stage I
In terms of methodology, there are two themes running through my previous research that I will further develop on the basis of new data: the embodiment of literature and the discursive function of translation in experimental writing. On the theme of embodiment, I will expound on the materiality of contemporary Chinese literature by focusing on the two following aspects.
Sensory-intellectual reader-book interaction. Here I will draw on Poyatos's (2008) classification of different types of sensory perceptions that mediate our first interaction with texts. These include direct sensory perception and synaesthesial perception, covering the visual, olfactory, tactile and kinaesthetic senses. This set of features will be used to elicit the various sensory-intellectual features that manifest themselves in my data, on the basis of which I will describe the ways in which Chinese literature becomes embodied through our senses.
The Chinese script as material signifier. Here I will draw on research in selected aspects of information processing in literary reading, such as orthography (Haas 1970), typography (van Leeuwen 2006; Nørgaard 2009) and punctuation (Poyatos 1981) to examine how experimental writers exploit the graphemic features of the Chinese script to create sensuous effects. I will also look at what I call 'anti-communicative writing', a stylistic invention characterised by non-referentiality/narrativity, syntactic rupture and intense reader involvement, all aimed at foregrounding the materiality of the written sign.
The role of technology in the material constitution of literary products will also be explored in (i) and (ii) above. The project will apply as its working definition of technology any process by which the task at hand is executed by applying certain techniques or tapping into resources that can enhance it, either in terms of efficiency and/or effectiveness or in terms of its aesthetic properties. Thus, the scope of technology considered in this project will encompass digital platforms such as machine translation, as well as non-digital manifestations such as certain publishing techniques, and of course writing itself (Liu 2010).
Methodology: Stage II
The second theme of the project is translation, here conceived as "a kind of biliterate performance that forms part of a more holistic literary communication" (Lee 2012:5), rather than merely as a textual act that transmits messages from one language into another. This definition will allow me to take into account instances of the translational that do not manifest conventional modes of language transfer. As previously noted, this study will be about the translational (the concept of translation) as much as about translation (the act of translating). By locating translation as a performative concept within the context of experimental Chinese literature, it will be possible to theorise on textual practices that foreground, and hence perform, a discourse of translation within them, which may or may not be accompanied by acts of translation proper.
More specifically, in delineating the kinds of translation that occur in multimodal Chinese literature, I will test and possibly refine Wang's (2009a:42) taxonomy of 'cross-cultural interlingual-intersemiotic translation' on the basis of my data:
"it must be the translation crossing the boundary between different languages";
"it must be the translation or interpretation crossing the boundary between different cultural traditions; and
"it must be the translation or interpretation crossing the boundary between different arts and disciplines".
Based on this work, an open-ended typology of experimental Chinese literature will be derived, and is likely to take the following form. It should be noted that the specific categories will depend on the actual literary data found and that an individual work can fall into more than one category (a case of synaesthesia).
Verbal-aural interaction, where a piece of literary work is communicated intersemiotically through words and sounds, and where interlingual translation and digital technology may be involved.
Verbal-visual interaction, where a piece of literary work is communicated intersemiotically through words and images, and where design and digital technologies may be involved.
Verbal-tactile interaction, where a piece of literary work is communicated intersemiotically through words and materiality, including the physical constitution and presentation of the work, where kinaesthetic interaction between text and reader may be involved.
Methodology: Stage III
I will contemplate the theoretical importance of multimodal Chinese literature, with reference to post-structuralist discourses on meaning and signification. In her intriguing book, The Freudian Robot, Lydia Liu (2010) ruminates on the theoretical link between literary imagination and technology. In particular, she looks at literary experiments that play with 'stochastic processes' (such as James Joyce's novels) and calls for a philosophical understanding of a machine-mediated world of communication. Taking this as a point of departure, I will investigate how the complex interactions among literary communication (or non-communication, as the case may be), translation and technology may be conceptualised in post-structuralist terms. In this connection, the study will draw upon the following concepts related to literary signification.
The three 'D's of 'meaning': deferral, différance, dissemination (Derrida 1974, 1981, 1982) and how the notion of meaning is treated, or 'mistreated', as part of the multimodal aesthetics of experimental literature.
The 'death of the author' (Barthes 1977b) and how the authority of original texts is constantly subject to compromise in multimodal translation.
The 'pleasure of the text' (Barthes 1976) and its relationship to the materiality of multimodal literature.
Experimental Chinese literature and the digital age: the paradox of thinking the 'machine' and the 'event' in tandem (Derrida 2002) and how digital writing challenges conventional literary theory (Liu 2010).
Methodology: Stage IV
Finally, the study will assess the current position of multimodal experimental Chinese literature by surveying key discourses, including textbooks, anthologies and histories of modern/contemporary Chinese literature. On the basis of the theoretical significance of experimental literature in a post-structuralist, digital age (see Stage III), the study will argue for its formal recognition in literature and translation research and teaching agendas. Email interviews with Chinese experimental writers may be solicited at this point to provide an insider's perspective on the issue.
September-November 2013: Review of the literature
December 2013-February 2014: Material collection and collation
March-May 2014: Complete Stage I
June-August 2014: Complete Stage II
September-November 2014: Complete Stage III
December 2014-February 2015: Complete Stage IV
February-August 2015: Collation and analysis of all results
August 2015-August 2016: Writing of results in monograph. The reporting and dissemination of the partial results will take place throughout the research period.