The paper is mainly theoretical by providing an literature overview over the history of human-computer interaction and the state of art in research on conscious and unconscious brain activities. We relate these unconscious mental processes to attitudes, emotions, and overt behaviour. Based on a communication theory very popular in Germany, we analyse and describe the possibilities to design persuasive technology (from small wearable to large mixed reality installations). As a result we introduce and discuss Kansei mediated communication. We zoom in on emotions and user experience, the difference between consciousness and unconsciousness, and argue for the primacy of the unconsciousness as the probably most powerful control mechanism for human behaviour.
User experience, communication, unconsciousness, persuasion, design
Human-Computer Interaction (HCI) has evolved over half a century by now (Myers, 1998), starting with Licklider (1960; Licklider & Clark, 1962; Sutherland, 1963). Although the history of HCI is rich and complex, we will shortly summarise some of the major trends that are: (1) man machine interaction, (2) personal computing, (3) cooperative computing, (4) social computing, and (5) most recently cultural computing. The history of HCI goes back to the 60s. Originally it was about Man-Machine Interaction (Meadow, 1970) and the emergence of the Personal Computing (PC) technology in the 70s (Card, Moran, & Newell, 1983). In the 80s, HCI was investigating media rich computing with the focus on networked computer mediated interaction (e.g. computer supported cooperative work - CSCW) (Schmidt & Bannon, 1992; Grudin, 1994). Interactive multimedia was and still is mainly about design for attention and memory, mainly in a learning context (Meera & Roger, 1992), but less for rich interaction in an emotional and social setting (Wensveen, Overbeeke, & Djajadiningrat, 2000). More recently, at the turn of the century, HCI was about the social computing trend with community mediated interaction (Schuler, 1994), i.e. such as internet based applications (e.g., on line communities) (Preece, 2000). With mobile, portable and ubiquitous technology, HCI is looking at more personalised and intimate interaction with positive experiences (Lugano, 2010). Different concepts have emerged in recent years for future directions of HCI (most prominent ones in alphabetical order): ambient intelligence (Aarts & Marzano, 2003), entertainment computing (Nakatsu & Rauterberg, 2009), mixed-reality (Milgram & Kishino, 1994), nomadic (Rheingold, 2002), tangible interaction (Hornecker & Buur, 2006), ubiquitous (Weiser, 1991), and several others. In general all these new directions have some common properties: (1) the disappearing computer; (2) the ease of use and positive experience and; (3) the building of social communities. The computer is no more the centre of interest, nor is it the focus of user's attention alone. It is the running application and the benefits and effects these have on the user that matter (Bekker, Sturm, & Barakova, 2010). Finally, we propose as a new paradigm for HCI, coined Kansei Mediated Interaction (Nakatsu, Rauterberg, & Salem, 2006). Kansei Mediation is a form of multimedia communication that carries non-verbal, emotional and Kansei information (e.g., unconscious communication) (Rauterberg, 2010). It is a combination of Kansei Communication (i.e., 'content') and Kansei Media (i.e., 'form'). The main research objectives in Kansei Mediated Interaction are the underlying almost unconscious social and cultural determinants (Tosa, 2000a; Rauterberg, 2007). Already quite at the beginning of HCI research a communication theoretical view was introduced (Licklider & Clark, 1962; Oberquelle, Kupka, & Maass, 1983; Rohr & Tauber, 1984), and the possible design challenges for a proper interface design identified and a 'meta communication' channel recommended.
Communication Theoretical View
The design of conventional interactive systems is based on the assumption that mainly logical information is sent, received and processed in human communications. This is a rather restricted interpretation of the situation. Non-verbal communication and emotions play a major role in the interpersonal communication that we are engaged in and enjoy everyday. Moreover, the exchange of higher-level Kansei information during communication is only attainable in highly sophisticated and mutually enjoyed communication (Nakatsu, 2002; Tosa & Nakatsu, 2002). Communication can be about understanding, about exchanging emotions, moods or sharing experiences. The communication features semantics, syntax construction or a variety of levels of perceptibility. The user perception and action are at the sensory and cognitive level, the motor and synesthetic (perception across senses) level or the autonomic level.
Conventional media technologies have been designed to primarily handle multimedia, informative, logical communications based on logic and aiming for the user's understanding of a message. This is a narrow view on the communication capabilities of humans and Kansei Mediation affords new opportunities. It is a style of lifelong learning that possesses a rich combination of communication channels to let conscious and unconscious information flow freely (Tosa, 2000b; Rauterberg, 2007).
We will now discuss how human-human communication can be delivered unconsciously using modalities that are almost not perceived consciously but nevertheless processed. The model of Schulz von Thun (1981) distinguishes four important dimensions of any message communicated among humans (see Figure 1): (1) content ("what are the facts"), (2) appeal ("what does s/he want me to do, think, feel, etc"), (3) relationship ("what does s/he think about me") and (4) self-disclosure ("what kind of person I am"). To communicate the sender sends messages: about the subject, which tell something about him/herself about his perception of the relationship with the receiver and which appeal to the receiver to change in some way. One could say the sender talks with four different 'beaks'. To understand the sender well, the receiver should listen with four 'ears' and each 'ear' should be tuned to what the corresponding 'beak' tries to say. If the 'beak' and the 'ear' are not in tune with each other, this is one of the main causes for misunderstandings among humans; but not only among humans these kind of epidemic misunderstandings appear (Veer & Vliet, 2003).
Figure 1 The four basic dimensions of human to human communication [solid line = conscious; dashed lines = unconscious] (Schulz von Thun, 1981)
One of the basic assumptions of the communication model of Schulz von Thun is that human behaviour is always influenced by all four levels at any time in any communication. These four levels can be split into two parts: (1) knowledge and facts ('conscious part' = content dimension) and (2) emotional aspects ('sub- or even unconscious part' = appeal, relationship and self disclosure dimensions). To successfully influence the behaviour of a receiver, one must try to understand each other at all two levels. To be congruent throughout the communication, one should formulate messages in a way consistent with one's personality, attitudes and values. Merely teachers' tricks on how to influence others will not work. The model of Schulz von Thun is clearly based on humanistic psychology (Buhler, 1971). The kind of communication should be in agreement with the needs of the social situation. Meta communication can help to learn how to communicate more effectively in a given situation and to recover from misunderstandings. This meta communication exists in two different ways: (1) the emotional aspects as a contextualisation for the what (i.e. factual message) accompanying each communication act (Gottman, Katz, & Hooven, 1996; Scherer, 2003), and (2) the explicit communication about the way how to communicate with each other; the self referential aspect of communication in social systems (Schulz von Thun, 1981; Luhmann, 2008). Of course it is always a challenge to transfer communication theoretical concepts developed for human-human interaction to human-system interaction, but I think it is still a fruitful approach.
The integration of multiple, multimode and Kansei Media can enable a type of communications that is neither biased towards cognition, nor biased towards awareness. The experience users will have of such combination is labelled Kansei Communication. Kansei Communication is mainly based on the inclusion of non-verbal (e.g. emotional) information throughout interaction (Hassenzahl & Tractinsky, 2006) (Nakatsu, 1998, 1999) (Tosa & Nakatsu, 2000).
According to Oatley and Johnson-Laird (1996) human mental and social life is fundamentally influenced by emotions. Emotions integrate bodily changes, planned action, social relating, and subjective experience. In their original theory (Oatley & Johnson-Laird, 1987) they proposed integrating the biological approach to emotions (Darwin, 1872) and the cognitive science approach to emotions (Simon, 1967). Later Oatley and Johnson-Laird (1996) proposed that emotions are communicative: They are brain signals setting the brain into distinct modes reflecting priorities of goals and predisposing toward appropriate classes of behaviour. Such behavioural actions include perceivable expressions communicating emotions to other people. In Rauterberg (2010) I discuss an extended idea to argue for a separate communication channel between human's unconscious subsystems, and I argue for emotions as the 'voice of the unconsciousness'. Before I discuss the relationship to persuasive system design, I will provide an overview over user experience and later introduce the 'power of the unconscious'.
According to Alves Lino, Salem and Rauterberg (2010) the concept oft user experience has been explored since the middle 90's, but, so far, there is little agreement on a unified notion or view on what exactly defines user experience (Law, Roto, Vermeeren, Kort, & Hassenzahl, 2008). While early perspectives framed user experience mainly as a result of user interface design, many researchers believe that positive user experience comes from the value and meaning of the product concept itself (Roto, Rantavuo, & Väänänen-Vainio-Mattila, 2009). User experience focuses on "non-utilitarian aspects of human-technology interactions, shifting the focus to user affect and sensation" (Law et al., 2008, p. 2396). This is one of the reasons why it seems to be not so easy to develop appropriate criteria for evaluation. In the context of interactive systems, Forlizzi and Battarbee (2004) have explored three types of experiences:
Experience, referring to the constant stream of 'self-talk' that happens while we are conscious, and that is based on the holistic knowledge one has acquired during the course of his/her life. E.g. while walking in the park or performing light housekeeping.
An experience, referred to as something more unified, that could be articulated or named, having a beginning and an end, and could eventually constitute a learning point from the holistic point of view. E.g. a particular event or a particular product that one interacted with at a certain moment.
Co-experience, which involves creating meaning and emotion together, or shared with others. Shared experiences allow a range of interpretation by others involved, and therefore personal experience is mutually influenced by sharing and expressing meaning through social interaction. E.g. interacting with others during a particular event and context.
In addition, Hassenzahl and Tractinsky (2006) defined user experience as the affective consequence of mainly the following three aspects:
The user's internal state (e.g. pre-dispositions, expectations, needs, functionality, mood, etc.),
the characteristics of the designed system (e.g. complexity, purpose, usability, functionality, etc.), and
the context in which the interaction occurs (e.g. social setting, meaningfulness of the activity, voluntariness of use, etc.).
Such a definition aims on how to improve the quality of the experiences rather than merely preventing usability problems. Designing a user experience is about addressing the relationship between an application running on an interactive system that can be represented as a series of computations and experiences that are essentially internal mental events (Davis, 2003). In this perspective one has to understand that the experience cannot be the media, the services or the environment, it can only be part of the interaction, as a 'communicative' process linking the user internal state with the various states of the system and environment, the alter other. The challenge is then about the design, selection, rendering and manipulation of the environment and the services offered to trigger a desired user experience.
The user experience delivered out of an interactive environment is therefore about affect and motivation (Camurri, Volpe, Poli, & Leman, 2005). Affect to have certain emotions and, motivation to perform a certain action or adopt certain behaviour (Ortony, Norman, & Revelle, 2005). Within an interactive environment the user experience combines interaction and perception experiences (Salem, Nakatsu, & Rauterberg, 2009). The interaction experience relates to the relation between the user and the environment as an interactive system and a physical space. The perception experience has as input a combination of multimodal visual, audio and sensory stimuli.
After reading the book of Nørretranders (1998) I realized how important subliminal perception and sub- or even unconsciousness is in the social realm. With mixed reality technology we can relate the perspective of persuasive technology as introduced by Fogg (2003) to sub- and unconscious user experiences because we want to elicit and guide predetermined user behavior through adequately designed affordances (Tosa, 1993; Smets, Stappers, Overbeeke, & Mast, 1995). Fogg coined the term 'captology' as an overlapping area between persuasion and computing technology, affecting user intention and behavior. He differentiates between two levels of persuasion. The first is "the overall persuasive intent of a product" on a macro level. The second involves parts of these products such as dialogue boxes that "incorporate smaller persuasive elements to achieve a different overall goal" on a micro level (Fogg, 2003, pp. 17f). The persuasion all happens on a conscious level, at least Fogg does not mention the aspect of sub- or un-consciousness. Kansei mediated communication broadens the perspective of persuasive technology by not only addressing user intention and behavior, but also emotions and cultural values (Tosa, Matsuoka, Ellis, Ueda, & Nakatsu, 2005; Bartneck, Hu, Salem, Cristescu, & Rauterberg, 2008; Hu, Bartneck, Salem, & Rauterberg, 2008). Kansei mediated communication does not only address the established conscious communication, but works also specifically through the sub- and unconscious (Nakatsu et al., 2006; Rauterberg, 2008; Aart, Bartneck, Hu, Rauterberg, & Salem, 2010).
In sociology, Johnson (2008) identifies the micro (individual actions or interactional systems), meso (organizational systems) and macro level (societal systems). When drawing the analogy between persuasive technology and sociology, a similar structure can be identified. The micro level would then remain on the level of direct interaction such as dialogue boxes, tangible interaction objects or other human-system interaction elements. This level influences user's experience through single actions. The meso level would incorporate Fogg's macro level of products, as a set of interactional aspects forming organizational systems. As such, the meso level affects user behavior and intentions as outlined by Fogg. The macro level is then the societal systems level in the form of social and cultural computing, addressing cultural values and affecting among others the self-concept (Kooijmans & Rauterberg, 2007). This macro-level analogy is also supported by a description of modern societies "characterized by increasing levels of reflexivity or self-reflection and the development of procedures for deliberate implementation of change" (Johnson, 2008, p. 489). In the context of his article cultural computing (Tosa, Matsuoka, & Thomas, 2004; Tosa et al., 2005; Tosa, 2006) can thus be seen as a macro level implementation of persuasive technology for mixed reality environments.
According to Epley (1998) subliminal means 'beneath the threshold of conscious awareness' (and supraliminal means 'above the threshold of conscious awareness'). Although easy to define, there was and still is considerable controversy and debate regarding how unconscious influences should be measured. The paradox of consciously not being aware about unconscious effects has traditionally been resolved using indirect methods. With these indirect methods we can show that stimuli can influence a person's thoughts or judgments even when they are unable to identify or recognize the stimulus (see e.g. Kooijmans & Rauterberg, 2007). In an experiment where you were flashed the word 'love' the subliminal influence would be evidenced if you were unable to indicate that you were flashed the word 'love' but you were nevertheless faster to identify words related to 'love' (e.g. 'kiss', 'sex', 'partner') than those that were unrelated (e.g. 'leaf', 'rotund' , 'car'). This is an example of a research approach that can find a correlation between measures of conscious and unconscious responses.
According to Cheesman and Merikle (1985) two general classes of measures are identified: subjective and objective. Subjective measures of awareness rely on participants' self-reports of their perceptual experiences or conscious processing via introspection. Questions like 'Did you recognize the word that was presented to you?,' or 'Do you think the word that was presented influenced your judgments?,' are examples of subjective measures of awareness. Objective measures rely on a participant's ability to discriminate between particular stimuli. Such measures included forced choices between two or more stimuli (e.g. 'Which of these three stimuli have you seen before?'), or judgments regarding the presence or absence of a particular stimulus (e.g. 'Was this one of the stimuli you were presented or not?'). Measures of objective awareness provide a lower threshold for conscious awareness than subjective measures. According to Epley (1998) it seems eminently reasonable that if you want to find out if someone is unaware of a particular stimulus, you just ask them. It should be evidence enough if they say, 'I have no idea what you presented to me'. It seems to be clear from the many experiments reported and summarized by Dijksterhuis, Aarts, & Smith (2005, p. 86) "that subliminal perception does much more than bring about small semantic effects. Subliminal perception can elicit affective responses. and it can influence both social judgments and overt behaviour." The remaining question is how to design subliminal stimuli to guide the user's behaviour for achieving an intended experience? If the subliminal stimulus does not go further than influencing attitudes, I will label this 'unconscious perception'; but if it goes through to overt behavioural effects, I call this 'unconscious persuasion'.
From Consciousness to Unconsciousness
Consciousness is a topic for which either there exist no acceptable description, definition and explanation or, and this depends on one's point of view, there are far too many and far too divergent ones. Most definitions from the western world are results of two major elements: (1) The emergence of western religions: Judaism, Christianity and Islam, and (2) Descartes Dualism. The first have established the existence of a self and thus of a soul as one of the foundation dogmas upon which their doctrines have been built. The second has created a schism between mind of body that does not necessarily exist and that has been a key, not necessarily a correct one, in the western world understanding of consciousness. Even today's literature is full of reference to the mind and the brain as if it has been established beyond doubt that there was indeed a separation. The illusory Cartesian self is more and more challenged by biological and neuro-functional evidences that point to consciousness as an emergent property of competing and successive brain processes (Baars & Gage, 2010). Not as a separate property or entity that is associated with humans and lies beyond the reach of science and investigation.
Table 1. Some Conscious and Unconscious phenomena and their relevance in Kansei Mediation. Adapted from (Ortony, Clore, & Collins, 1988).
Novel, Significant, Informative
Routine, Non-Significant, Non-Informative
Immediate, Declarative, Episodic
Long Term, Procedural, Semantic
Consciousness nevertheless can be defined according to accepted attributes (see Table 1). It is a state of mind, a mental property associated with the perception of self-awareness, subjective experience, the sentience faculty and the sapience capacity (Khatri & Ng, 2000). Self-awareness is the ability to recognise that one exist, one has a body, a personality and inner thoughts. Subjective experience is about acknowledging how an experience is like. The sentience faculty is related to the capacity one has to perceive and feel. Sapience is the capacity to act with intelligence. Within the context of Kansei Mediation we shall focus on consciousness as an experience and the actual perception of such experience (Varela, Thompson, & Rosch, 1991). Consciousness could be divided into two subsets: access consciousness and phenomenal consciousness. Access consciousness relates to the availability of information for a particular purpose, such as an action. Phenomenal consciousness is the experience of a given state (Hofmann & Wilson, 2010). Thus in essence 'what is happening' and 'how I feel it' are two faces of consciousness. A conscious experience could be described as the emergence of temporary, partially overlapping and successive global states of the brain. Thus consciousness could be described as the temporal integration of this continuum of experiences into one single stream.
There are different levels of consciousness experience even within awake individuals. A good example to illustrate this is having two people on a bike. The one holding the handle bar and cycling will have a much richer conscious experience of the ride then the one sitting at the back and not even knowing where the journey is leading. We could describe the first experience as being active and the second as being passive. It seems that consciousness is more than the sum of some components. Consciousness is probably needed for at least two high level purposes, the pursuit of pleasure and the establishment of relationships with others (Noort, 2003). Consciousness is probably also the seat of a very important function: prediction (Lane, 2000).
Figure 2. The cognitive architecture how verbal and non verbal behavior is initiated. Adapted from (Hofmann & Wilson, 2010).
Unconsciousness and consciousness play a particular role in the initiation and performance of voluntary and involuntary actions (see Figure 2). Both action and expression originate in the unconscious triggered by the sensory inputs. They are then either vetoed or approved by the consciousness or directly unconsciously executed. Some expressions are not voluntary and are very difficult to be consciously vetoed and prevented from occurring, for example blushing and crying. Noort, Hugdahl and Bosch (2005) discuss several experiments that suggest a special role for unconscious emotional information processing in HCI.
In the dualist approach advocated by Descartes, the mind is conscious and the body unconscious. There was and still is a primacy of consciousness over unconsciousness (Baars, 2003); however this topic is of growing interest to cognitive neuro science and neuro psychology (Hofmann & Wilson, 2010). Traditionally the unconscious activities of the human mind are seen hidden under and are controlled by consciousness (the word oppressed is often used) (Nakatsu et al., 2006). In the emergent view however , there is no such separation between mind and body, and consciousness seems to be an emerging property of unconsciousness (Rauterberg, Hu, & Langereis, 2010). It is moving away from the concept of conscious oppression of perception and expression into the concept of emerging perception and expression (Brosch, Pourtois, & Sander, 2010; Dijksterhuis & Aarts, 2010; Hofmann & Wilson, 2010).
The power of the Unconscious
When we think of being conscious, we think of being awake and aware of our surroundings (Rauterberg, 2008). Being conscious also means being aware of ourselves as individuals. Mostly, people tend to think of being conscious as being alive. We tend to think that the person should be responsive to the surrounding environment to be conscious. Being in a coma is considered to be the opposite of conscious, so called non-conscious or unconsciousness. There are at least three forms of consciousness for humans: (1) the conscious state; (2) the subconscious state (also called 'phenomenal consciousness'); and (3) the unconscious state. In the scope of this paper the unconscious state is fully operational and functional for a normal human living as a collection of parallel background processes, we are just not aware of, e.g. activities of the cerebellum, particular hormone levels, or reflexes. The subconscious can be turned into conscious (i.e. by paying attention to subconscious activities in the 'global working memory'); the unconscious normally is not available to the conscious, at least not through self introspection. The remaining question is how - if at all - does the unconscious communicate with the conscious? Is it just passing filtered information through to the conscious? Or is it a mainly independent and parallel activity?
The conscious part of the brain is investigated already for a long time. One of the important results is the limitations of the information processing capacity of the short-term memory (Baddeley, 1994). In his classic paper Miller (1956) found that the conscious information processing capacity is limited to seven (plus or minus two) chunks or dimensions (see the discussion by Simon, 1974 ). This conscious part is mainly described as the short-term or working memory to emphasize its role in decision making and controlling behaviour. Consciousness is a topic for which either there exist no acceptable description, definition and explanation or, and this depends on one's point of view, there are far too many and far too divergent ones. Most definitions from the Western world are resulting in a dualism (Boyle, 1983). This dualism has created a schism between mind and body that does not necessarily exist and that has been a key, not necessarily a correct one, in the Western world understanding of consciousness. Even today's literature is full of reference to the mind and the body as if it has been established beyond doubt that there was indeed a separation (Dreyfus & Dreyfus, 1984). The illusory Cartesian self is more and more challenged by biological and neuro functional evidences that point to consciousness as an emergent property of competing and successive brain processes. Of course, at least the unconsciousness and may be the consciousness too, play a role in the initiation and performance of actions (Kim, 2000; Dijksterhuis & Aarts, 2010). Both action and expression originate in the unconscious (Libet, 2003). They are eventually then vetoed by certain neural structures to achieve self control (Brass & Haggard, 2007). Actions and expressions not vetoed are then performed (Libet, 1999). In the dualist approach, the mind is conscious and the body unconscious. Interestingly enough, recently one of the pioneers of artificial intelligence started to recognize the importance of the sub-, respectively the unconscious (Schank, 2009). The unconscious activities of the human mind are hidden under and are controlled by consciousness (the word oppressed is often used). In the emergent view however, there is no such separation between mind and body, and consciousness is said to be an emerging property of unconsciousness. It is moving away from the concept of conscious oppression of perception and expression into the concept of emerging perception and expression.
An iceberg can serve as a useful metaphor to understand the unconscious mind (Scherer, 2005; Dijksterhuis & Nordgren, 2006). As an iceberg floats in the water, the huge mass of it remains below the surface. Only a small percentage of the whole iceberg is visible above the surface. In this way, the iceberg is like the mind. The conscious mind is what we notice above the surface while the sub- and unconscious mind, the largest and most powerful part, remains unseen below the surface. The unconscious mind holds all awareness that is not presently in the conscious mind. All activities, memories and thoughts that are out of conscious awareness are by definition sub- or even unconscious. Scherer (2005) assumes that a large part of emotions functions in an unconscious mode and only some parts will emerge into conscious. But what does this relationship between unconscious and conscious look like?
Recent investigation in cognitive psychology of the conscious and unconscious are promising. Most of the brain's energy consumption is not used for processing responses to external stimuli as usually assumed; but what is this enormous amount of brain energy then for? (Raichle, 2006). One promising aspect of unconscious information processing is finding optimal solutions in the multidimensional sensor and knowledge space of the unconscious for controlling behaviour by situated forecasting throughout the near and far future (Rauterberg, 2010). Any input (subliminal, supraliminal, or none of both) that can contribute to optimise action planning and future forecasting, will probably be highly appreciated by the unconscious.
In persuasive design several methods mainly combine psychology with careful preparation (Wikipedia, 2009). Experts in marketing and sales and other professional persuaders, are commonly trained to work within a carefully prepared conceptual framework. They operate on a series of contingency plans structuring and clarifing the user, resp. customer interaction for them.
Pentland (2008) argues that our impression, attitude, and even behaviour is strongly influenced by intonation, say, or the fluctuating pace and amplitude of the voice of our communication partner. This normally happens within the first few seconds, guided by signals we are not consciously aware of. "Human behaviour is much more predictable than is generally thought," says Pentland (reported in Buchanan, 2009). Person's responses can often be explained by "non-linguistic behaviours of other people and simple instincts for social display and response, without any recourse to conscious cognition". Pentland's second channel of human communication (in our concept the 'non-verbal' behaviour) acts in parallel with reasoning that based on rational thinking and verbal communication. This is extremely important in human affairs.
With modern mobile technology we can monitor these social signals in spontaneous, natural settings in a way and on a scale that was not possible before. Normally it is quite difficult to collecting this kind of spontaneous data. Pentland and his group have developed wearable devices that include electronic badges, modified personal digital assistants and specially configured mobile phones. These field measure instruments can track not only a person's location, but also her or his biological and behavioural details, i.e. intonation, heart rate, upper-body movements, etc. These kinds of data can then be analysed for patterns of meaningful social signals; the 'honest signals' as Pentland coined them to put the argument forward that is seems quite difficult to consciously control all these non-verbal output pattern.
A popular framework for subliminal persuasion is based on neuro-linguistic programming (NLP) (Grinder & Bandler, 1976; Bandler & Grinder, 1979). A core aspect in NLP is the analysis of people's maps to represent their ontology, summarized by the sentence 'The Map is not the Territory'. One of the central concepts of NLP is the primary representational system (PRS). The maps of people are represented by the five senses (in alphabetic order): auditory (A), gustatory (G), kinesthetic (K), olfactory (O) and visual (V). V, A, and K are considered to be the major senses; individuals differ in the way they employ and maintain their PRS. The PRS of a person can be observed in his choice of predicates, and in the direction of the person's eye movements. According to Bandler and Grinder is the matching of the customer's PRS by mirroring or pacing their verbal and non-verbal behavior. Matching aspects can be speech, gestures, body posture, breathing, blinking, etc; we should tune in on our counter partner's PRS and thereby enhancing communication, rapport, trust, understanding, etc. If we apply NLP concepts to our communication model from above, we have to replace the sensory modalities in the PRS with our four communicative dimensions: content, appeal, relationship, and self disclosure (see Figure 1). The remaining challenge is how to design interactive systems with this kind of appearance and behavior.
Much of our everyday life is determined not only by our conscious intentions and deliberate choices, but also by unconscious perceptual and mental processes afforded and triggered by our environment. The scientific literature is full of experiments demonstrating that conscious explanations for our behaviour are often just post rationalizations after the action has taken place. It seems to be naive to take conscious verbal communication as the primary way that people respond to each other. In the future, this kind of persuasive design research can be expanded to look in much more detail at human experience and social interactions: individually, in small and even in larger groups.
Acknowledgement: I have to thank my colleagues here in Eindhoven, and in particular Naoko Tosa and Ryohei Nakatsu for all the inspiring discussions.