Research: Book project Blogs
White Papers
Reading Notes
Observations
Research

About     Social Interaction Design     Writings     SxD Blog


Through the Wire: Excerpts

This chapter takes a look at the way in which technologies used for communication distort and transform our expressive faculties. The question then being, what happens to communication? I am on the hunt for the ambiguities created by the introduction of technology into human interactions. In later chapters I look for stratagems used to cope with those ambiguities and distortions.


Features from the face

By their selection of one human sense among others, technologies enable and amplify a given set of possible facial features. Features of course are expressive, and are deployed in normal interaction for a wide range of purposes. A technology might amplify a sense; but it doesn't necessarily follow that it has amplified its expressivity. And it may amplify a sense without enhancing its receptivity. Take the ear, in the case of the phone, as an example. While the ear assumes a privileged position vis a vis other senses (namely, the eyes), it's ability to discern acoustic information is compressed and focused on what comes through the mouthpiece at the other end. Surrounding noises and their spatial location are bracketed out (for practical purposes of concentrating attention on the speaker's voice). The voice, which would be mute without the mouth, is heard disembodied from the mouth and the face to which it is attached. We can hear a laugh be we can't make out a smirk.

Selection of facial features brackets our ability to discern facial expression (even when it's produced at the other end). And facial features are critical in co-present interactions to the manufacture of what are called "paralinguistic markers." Cues, in other words, to what is going on. Clues and hints supplied in face to face interaction that provide interactants with a faster and more effective path to discerning what's happening. There is another dimension to the artificial isolation of perceptual senses, and that has to do with our tendency in face to face interactions to mirror our partners' expressions with the unconscious intention of establishing a shared rhythm and intensity. We naturally scale or tune our attention levels up and down with any interaction, and do that through subtle shifts in facial gesture and expression. Mirroring makes of the face an impressive as well as expressive device, a dynamic and changing prop in the pursuit of an engaging encounter. Bracketing out visual access to the face of course eliminates this possibility. Turns taken during an exchange are played out in a game whose players are forced to play at catching up to where the other(s) have moved to. The sense of being in synch is lost, and what remains to do is strive for a more literal understanding of what's going on, and to avoid any potential stumbles that may result from a step taken in the wrong direction.


Affective capacity

I will make little attempt here to characterize the role of human affect or emotion in communication. That we are moved, and that we move others, goes without saying. What importance that has in different domains of activity is a matter of discussion. Media can be easily accused of blocking the flow or expression of emotion in communication, and of flattening it out if you will. Some might say that media force communication to become more rational. Asynchronous media represent communication through use of a secondary or recording medium. How does this translate the passage of emotional content? What affect makes it through text? Through voice mail? Through SMS, instant messaging, and so on? Naturally, a great deal of affect can pass through those media and in ways that are sometimes quite moving indeed. What secondary media do block, however, is the co-production of affective states among participants in real time. Involvement of a recording medium of any kind by definition precludes the achievement of shared emotional co-presence.


Secondary medium

By nature of their commitment to record and encode communication with accuracy, secondary media seek as much transparency as possible. If it were at all possible today, we'd be looking at holographic projections of one another as did our famous friends in Star Wars. We'd leave nothing out that we couldn't record. Text, the preferred medium of most asynchronous media, involves a language of transcription whose characteristic ambiguities have been tried and tested for centuries (though granted, literacy is a fairly recent invention). The question thus facing a secondary medium involves that which it does make possible and which direct or synchronous media don't, and that would be propagation. Can the message be forwarded or passed along easily? Will it be understood out of context? Will it lose anything as it travels?

Secondary media are somewhat unique in their ability to serve as vehicles of communication in and of themselves. A joke, video clip, and quote can be passed from one person to the next over email with or without accompanying commentary and context-setting. Language creates a statement and the system by which to understand it. As long as the statement follows grammatical rules, and is thus meaningful in a literal sense, it can communicate even to unintended recipients. Secondary media not only allow us to create documents or artifacts that communicate in our absence. Combined with the power of networks, secondary media take part in the circulation of communication. All kinds of messaging, from statements to questions, inquiries, announcements, comments, remarks, rumors, jokes, and more get passed around over networks.

Asynchronous media, though compromised in their ability to convey tone and facial texture, benefit from the combined powers of digital recording, computing capabilities, cost-effective distribution networks like the internet, and wireless devices that speak in internet protocols. The lossless preservation of messages in the digital domain, for example, permits us to put messages into endless circulation over the net. Messaging, as a different form of speech than its more honest synchronous elder brother, can reach an almost unlimited audience without any decay or degradation. Search capabilities applied to text enable us to mine past interactions and to pull them up as if drawing a book from the shelves of a library. Messaging preserves a degree of historical accuracy, one admissible as evidence in the courts, even though contexts of interaction have long since disappeared. As a form of information, asynchronous media benefit from the precision of written language. And in some cases, from events triggered by messaging transactions that can be mined in the future for the purposes of additional recollection. Written language is accurate in spite of its limitations. The filtering effect it has on the production of meaningful statements gives us a level of clarity. It is this aspect of written language that first made it useful for accounting purposes, and against which poets and thinkers battle in an effort to extract ever more eloquence from the juxtapositions of dead words.

And yet of course the ambiguities of written speech are undeniable. Intent is among the first layers of meaning to go, for the use of writing systems in communication denies the presence (and the communication that comes from the pragmatics of situated interactions) of interaction its power to bind speaker and hearer. Truth, as a measure of the speaker's sincerity also suffers a screening effect when the visible mouth is translated to paper. Any reference to objective states of the world will require independent verification if it is at all suspect-an effort most of us forego in the interest of efficiency. And the speaker's normative authority to make whatever normative claims he may put in writing must be taken at its word, or confirmed independently also. A preacher man in writing wears no visible cloth. Even the signature line adapted to email has lost its signature value. These ambiguities may short circuit the possibilities of asynchronous messaging. And yet we continue to play-sometimes with unexpected results-with those very ambiguities. We insert jokes and irony into messages as flat as an ironing board. We tease and flirt, smirk, grin, cajole and otherwise engage many of the physical gestures available to the gregarious through hints implicit or explicit in text. Asynchronous media, because they use secondary media for message encoding, are different and singular forms of speech. Utterances are divorced from their speaker and from the speech situation itself. Does this eliminate their illocutionary force, their ability to bind speaker and hearer? What is the difference between a message received by its intended recipient and an unintended recipient? Certainly there is a difference, though it is not in the message itself. Perhaps this is where the proliferating power of the internet is most confusing. If message propagation increases the possibilities of talk, but not always of talk directed by speaker to hearer, then what are the characteristics of this new talk?


Selection of attention

Contingent on a medium's perceptual apparatus, or selection of one sense over others, is its ability to convey the expression of attention. We make ourselves present to others by directing our attention to them in such a way as to support the mutually established focus of activity. We're either in or we're out, and the difference can be seen without great difficulty. (The possibilities of appearing to be in while in fact being out, or of appearing to be out, while being in, as in the cases of faking, lurking, and eavesdropping, are not sanctioned. Bystanders are generally "in".) Once in, the activity in which we are engaged provides tacitly understood parameters that inform the distribution of attention. Such things as who has the floor, for how long, and when it is given up. These are not rules per se, but extremely fluid guidelines that shift with the very interaction, as the interaction may itself pass through any number of transformations. In general, participants focus their attention on the speaker, frequently providing him or her with a cue to give up the floor when his time is up. Thus the speaker is obliged to tend his occupancy with a regard for his audience's interest, tenancy coming at the price of his audience's attention. And the audience is obliged to pay attention while the speaker has the floor, knowing that each may in turn expect the same. The speaker scales up his expression, the audience scales up its attention levels, and the two are off on a ride that will see many ups and downs before coming to an end.

Introduce media into an encounter and it becomes impossible for participants to engage in such a dynamic exchange of attention. Participants have to make adjustments in order to accommodate the distance introduced into the encounter by the medium. In the case of asynchronous media, which operate by messaging (through an additional recording medium), attention is provided in strips that belong only to one user at a time, and is not integrated in the form of a shared real-time experience. It becomes critical then for a medium to display some amount of attention, either by encoding it in symbolic or signaling devices, or by giving the speaker and listener a connection through which they can give and take attention themselves. Because so much of an interaction hinges on one's ability to monitor its progress, the bracketing of attention by media can produce stress (not knowing how to act), misunderstanding, and other kinds of distancing effects. It might be argued that the higher the level of structure in a given encounter, the more it is ritual or ceremonial, for example, the less likely that media can successfully translate it. Media are better suited to loosely structured interactions. Ceremony and ritual involve bodily as well as facial control to such an extent that most participants come to it specially dressed.

Media also intervene in the user's ability to manifest his or her availability to others and to situations. While this is different from the display of attention critical to unfolding a real-time encounter, it is no less important. We make our presence availability known by directing our attention such that others know "where we stand." (And we shift our position while standing, of course!) Media assume that we are available, and in fact offer us only the choice between a categorical yes or no. On or off. The bias inherent in media of course favors our being as on as the medium itself. This dirty little secret of technology can cause users a great deal of stress and interruption, though we can hope that future generations of technology will respond better to our need to express presence availability with more than ones and zeroes.


Distortion

Media produce so many distortions across the full range of human communication that any attempt to catalog them would be foolish. And any proper attempt to account for distortions would be remiss if it did not also account for the little compensations and bigger stratagems we employ in response. Media distort at the level of perceptual selection through the process of amplification and the internal rearrangement of our perceptual apparatus. They distort the relationship between interactants in terms of presence and its expression, and communication and its direct or indirect transmission. By creating means of access, they distort our ability to make ourselves available or unavailable. Media distort relations among people by changing flows of information, access to information, and access to individuals at points whose equidistance may show no respect for hierarchy or relational structure. Media produce distortion at all levels: intra-personal, interpersonal, and organizational/systemic.


Recourse during failure

Failure is a possibility in any kind of communication, though in face to face interaction, our proximity to the other and to the encounter itself permits us a good view of what's going on. Through command over our own face, familiarity with the context of a speech situation, and competence in the language spoken, we can sidestep mistakes and misunderstandings by turning to ritual or system level meanings for help. But communication is always a joint pursuit of arriving at a sense of what is going on. And the extra layer of separation that mediation inserts into interaction complicates the process of achieving successful communication. Misunderstandings related to the medium itself might occur, as might misinterpretations of things said or spoken about. For media not only bracket out the help system of ritual and physical co-presence, they create new ambiguities of their own technical nature.

Communication can fail on many fronts. One or all interactants involved may fail to achieve a sense of what is going on. This would be a very basic failure, as it prevents communication from proceeding any further. Interactants may know what is going on but fail to agree with one another. This is a second order failure, for it is not so much a failure of communication (as a system) but a failure of the participants to agree, and thus may obtain from any number of differences in positions, interests, expectations, and so forth. And then there are third order failures, in which participants, having reached agreement, fail to achieve the results or purpose of their communication. Add technology to the mix, and not only does the risk of any of these kinds of failures increase. The possibility of technical failure emerges also.

We take the possibilities of failure into account when we communicate, as we take the possibility of technical failure into account when engaging technologies. In matters of communication, we can take recourse to other systems of interaction. In the case of a first order failure, we can use language itself, ritual systems of meaning, and paralinguistic expressions to rephrase a statement in order to better package our intended meaning. Access to ritual, to a situation and its contextual meanings, to vocal and auditory equipment give us a wide range of options by which to clarify misunderstanding and by which to better convey what is going on. Second order failures, not being a matter of failed communication systems, must be addressed by finding agreeable positions. Participants must find the expression acceptable to each of them. Third order failures may require participants to seek their objectives again, or in a different way altogether. Social realities provide us with an almost infinite number of ways to arrive at successful communication, and in fact a great deal of the pleasure of interaction obtains from the creative negotiation of options and alternatives.

Given that access to these alternatives is an important factor in successful communication, the introduction of technology creates several problems. The bracketing of face work, of situatedness, in short the bracketing of presence, eliminates some of our more immediate options. And the introduction of new and technical ambiguities creates the possibility of technical failures for which there may be no interactional help system. Insofar as we anticipate the likelihood of failure when we communicate, we may attenuate what we say through technology on the basis of what we can do when we encounter failure. In many cases we switch from mediation to face work when the failure might be of a technical nature. For to use the technology again as a means of undoing the cause of failure might not get us anywhere new. If an email is incapable of conveying the delicacy of tone or expression required, for example, we might be better off by communicating in person. If the success of an interaction hinges on the delivery of a message, we might use the phone to check that it has arrived. We use redundancy to ensure success.

It is important that technical systems of communication give us access to these redundancies. Generally, email, phones, and physical points of presence provide an adequate set of alternatives. But our increasing tendency to favor systems like email for the sake of cost reduction-witness trends in customer service and support-often comes at a heavy price. Given a reduced set of options in the case of communication failure, and the additional complications produced by technology itself, people my rightly opt out of communication in the first place. The risk of failure can create stress and anxiety. And in situations limited by technology we might choose not to communicate at all rather than expose ourselves to the trouble of having to negotiate mediated ambiguities and technical problems. And insofar as access to any party might be constrained by the limitations of the medium of access itself, the frustration experienced by those who choose not to engage in communication at all will damage relationships ultimately far more important. Systems designed for technical efficiency do not interface with the experience of human interaction in the same way, for ambiguity cannot be reduced out of human communication.


Silence

In any kind of social interaction, most silences are pregnant with meaning of some kind. They can be awkward, poignant, even ceremonial. Silence provides a pause in communication, a rest or pit stop away from the traffic of social commerce. But silence in mediated interaction can be confusing. Given that a communication medium's natural state is silence, how are we to know if a particular silence is a meaningless gap in activity or a meaningful absence, rejection, or avoidance strategy? At what point does a hiatus become awkward? When does the suspension of communication fade into real distance and separation? And how do we know when a participant's disappearance from mediated interactions is real, and when it's not? We don't, of course, and this is simply one of the conditions of mediated communication that we are forced to contend with. Silence is far more ambiguous in situations of mediated talk. It can mean everything or nothing at all, the difference here being that there is often no way, through the technology, of knowing which it is.

Silence has duration and it has reference. What is the length of a silence? The longer a silence endures, ordinarily, the more decay occurs in communication. Silence becomes the state of communication, and the absence to which it corresponds displaces the presence shared between correspondents. Long silences, however, may not indicate problems in a relationship any more than they might indicate problems of communication. Relationships, and communication too, go into suspension at times for reasons that may have little to do with the relationship itself, but which have practical origins. So the duration of a silence is not a key to its meaning. Reference might be a more accurate way to unpack the message behind the absence. What started the silence? How did it begin and with whom? Knowing what the silence is a response to, or to what it refers, might be a more accurate way of reading its message, if it has one. Silences appear during interactions as a way of spacing out statements, as products of the positioning between actors (particularly when one or more feels uncomfortable), and as effects of the disposition towards one another that individuals take up in a group situation. The ritual or system level of group interactions helps participants to ascertain the meaning of these silences. A crowd falls silent in a moment of directed recognition, of an event belonging to their ritual gathering, or of a transformation within the crowd itself (an event within the crowd). An individual falls into a very different silence during a private tete a tete, tripping into emptiness or self awareness, falling out of conversation for what was said-the next statement being unspoken for reasons of what its utterer cannot bring himself to say. Silences belong to interactions and in some cases are statements themselves, responses to the situation at hand or to things actually expressed.

Where technical or mediated silences differ is in the difficulty correspondents have in the process of interpretation required to fashion a silence into meaning. Neither the duration nor the reference of a silence provide adequate signposts. The interaction is often missing a clear beginning and ending, its frame robbed of markers. The open state of interaction enabled by the medium itself produces intentional and accidental silences alike. And the ease with which we can reach out for interaction through media makes it possible for us to hurry towards clarification when the silence becomes disproportionally loaded. The dislocation of presence and presence availability negotiation that comes with mediated interaction means that we have to work through the confusion of a communication that shows bias towards expression and against reception. We're better able to reach out, to ask, to call upon. And less able to grasp the meaning of absence, silence, disappearance. Communication stutters and falters as interactants struggle to determine the value of a silence, pause, or suspension. And the intermittent nature of the connections realized by connectivity technologies tends to produce ever more ambiguous interruptions and gaps.

We become trapped in this uncertainty the moment we make ourselves technically accessible. While it may seem that communication is compressed by the fact of technical mediation, silences and absences exceed mediation. The shortage of meaning in one is the excess of meaning in the other. We do our best to manage our way through the intermittence of mediated talk, ultimately engendering further talk as a means by which to resolve what's ambiguous and unclear. Insofar as we all need to ascribe meaning to silence, to position it either within a chain of statements, a conversation, or interaction, or to obtain from its duration and span a sense of continuity, of presence or of absence, technical silences are a serious breach of understanding and may have unintended consequences for individual relationships. We get better with time and practice at knowing what to make of silence. But as with so many other aspects of technical-human and technical-social interfacing, no human effort can ever compensate for what cannot pass through the wire.


Attention Sustain

Who doesn't enjoy people watching every now and then? Just simply watching others interact, guessing at what's going on between them, perhaps taking a stab at who they are, where they might be from, what they do, and so on can be endlessly entertaining. Our cities have places for people watching. Cafes and street-corners, intersections and other places in which human traffic seems to reach a higher level of animation, part public performance, part intimate and private. People watching is entertaining because people produce such a rich set of body and facial movements and gestures when they interact (and possibly differently so when they interact in public). We read the exchange of attention between them as each takes his or her turn at the floor. We almost become engaged in our own way as the position of speaker and hearer, speaker and audience shifts back and forth, conveyed sometimes by as little as a wipe of recognition or acknowledgement across the face of one or the other.

This exchange of attention in face to face encounters serves to unfold the positioning of each interactant to the other, as well as to the encounter itself. What happens when media get involved? For the functional effectiveness of a medium would seem to rest to some degree at least on the medium's ability to conduct this dynamic of attention getting and giving between participants. Mediation of course brackets the face, and insodoing eliminates our ability to show and to monitor attention in real time. We can pull it out of the voice heard in a phone call, but not see it in the ear of the listener. Media intervene in the flow of attention further if they subdivide and displace it from the real time or co-temporal unfolding of interaction. Attention is sent, not shared. Or rather, and more accurately, it is shifted. When we engage in mediated interactions we shift our attention to the medium itself, even if only for a short period of time. I have argued elsewhere that this shift involves the attending to the mode of production over the production itself. Machinic forms require and occupy attention. There is no driving without paying some amount of attention to the car itself.

A medium's effectiveness at engaging interactants in their exchange, and in successfully transmitting and enabling their communication, can be observed in part by the degree of attention shifted to the mode of production. Not only because the more aware an individual is of the mode of production, the less transparent or present she is to the interaction itself. But also because the more attention the individual shows to the medium, the more her communication is a response to its constraints, its enabling, in short its functionalities and technicalities. Attention committed to the materiality and other features and functions of technologies of communication structures how people say what they say and how they hear what they hear. It becomes as much a part of communication as awareness of posture and decorum might be a part of ceremony and ritual.

Faces were designed, if they were designed, for the exchange of attention. Which is why we get so much out of people watching. Machines have no such capacity of perception, and only a fraction of the capacity for expression. And whatever attention our technologies appear to give us, is always and only a simulation. The shift of attention to the mode of production itself that comes with use of technologies for communication ironically creates a second order attention deficit problem. Not only now has the exchange of attention from face to face been shut out or redistributed to technical means, but the technology's ability to sustain the capture of user attention fast becomes an issue of communicative efficacy. For when we find our attention drifting from a technology, when it falls from our awareness, or when it slips our mind entirely, our access and availability for interaction goes too. Phones have ringers and email beeps in order to get our attention. If we wish to hear the call or notice an incoming message, we are bound to providing technology with our attention. And technologies fight for attention. Email and instant messaging applications compete with a computer's other functions and applications. Phones compete with physical and social contexts, each of which constrain our capacity for diversion and availability to interruption.

It is likely that you spend less time people watching and more time technology watching. It's happening to all of us, and it's the price paid to be available and accessible through technical networks of connectivity. Whether this price is higher when paid in real terms or in opportunity cost is a question we should ask, knowing of course, that by the time we get our answer we may have forgotten why we asked it in the first place.