Research article on human factors design issues in communication technology, human computer interaction and computer mediated interaction

About Social Interaction Design Writings SxD Blog

Through the Wire: Excerpts

"Our private sphere has ceased to be the stage where the drama of the subject at odds with his objects and with his image is played out: we no longer exist as playwrights or actors but as terminals of multiple networks. Television is the most direct prefiguration of this, and yet today one's private living space is conceived of as a receiving and operating area, as a monitoring screen endowed with telematic power, that is to say, with the capacity to regulate everything by remote control." Jean Baudrillard, Ecstacy of Communication, pp. 16-7

We have argued that our adoption of technologies unfolds through a process of disembedding and reembedding-a disembedding of practice from traditional ways of doing things, and a reembedding through the development of new and mediated practices. The machine and social worlds constrain one another, and new practices emerge as a marriage of unfolding social convention and technical possibility and functionality. The more the social world is governed by or responds to established practices of interaction, the more we will find those practices inform our use of technology. In the case of unstructured interactions, such as personal uses of communications technologies, our analysis becomes much more difficult. We have to find the ways in which individual uses manifest personal habits and routines. In the absence of a social structure, what frameworks can we lean upon by which to observe new uses and practices?

What strange machines are these, which couple us to one another, creating connections, in the same time or in different times? What are these assemblages of relation, how do they work and what are they capable of? Our relationships to one another already transcend space and time. They already persist through the spans of absence that we take from one another. We find ways, in our personal relationships to friends and family, to colleagues and acquaintances alike, to envision our meaning for one another when we are absent physically. Being apart does not sever a relationship, any more than not speaking means we have nothing to say. Periods of absence and distance are an intrinsic part of interpersonal relationships. But the absence of ritual or ceremony, the absence of structure, from relationships as well as from interactions makes the analysis of mediation suddenly more challenging. We cannot point to the use of mediation and find activity coordination, the satisfaction of organizational goals, the completion of work-related tasks and responsibilities, or the transmission of critical business information. Personal relationships involve temporalities marked by the individuals who compose and execute them. Styles, signatures, personality and character come to mark interactions and the periods between them. Only we know when it has been too long between interactions. Only we know how to interpret a silence or a pause, only we can unpack the messages written between the lines of an email, or tucked into the punctuation of a text message.

These machines of connection create proximities unique to their users. And yet technology has its materiality and its functionality. By what it can and cannot do it creates the conditions of communication. In many ways the persistent nature of interpersonal relationships appears at least to transcend the interventions of technologies. The understanding that we have with a friend, for example, allows us to survive the little hiccups that might occur now and then, and that are a part of daily life. Our commitment to one another and to the friendship itself get us through misunderstandings by providing a foundation to fall back on. We demonstrate a commitment to resolving issues with an orientation towards understanding. Disagreements rarely threaten the friendship itself, but are a part of its development, and a process of getting to know one another better. Our emotional connections with one another transcend the words we speak, and there's always something more to the interaction than meets the eye. Its through this grasp of where another person is at, in spite of what he or she might actually say, that friendship exceeds relations experienced with strangers or acquaintances. To be close to somebody is not a matter of physical co-presence, though of course it's difficult to imagine that we might get close to others through non-physical means only. Being close requires not only face time but face commitments. The communication we have with our friends involves the kinds of speech and action that create commitments and obligations out of mutual trust. We make promises to each other. We choose not to betray each other's positions or feelings. We protect and serve our friendships. The loyalties we experience in these kinds of relationships have to do with the bond that ties us to a shared future. Insofar as we have history with our friends, these bonds may take only superficial interations to maintain. They are in a sense analogous to Goffman's ceremonies, the highly structured social rituals that serve to reproduce social convention and order.

In friendships and personal relations then, the face generally wears a smile. It looks back at us with recognition, and provides a welcoming embrace. The smile in the eyes begs for a return. A glance is exchanged, one for the other. An economy of trust prevails over the interaction. Faces watch one another, as they watch out for one another. We gather with friends in order to share in the experience of good and supportive facetime, to feel and enjoy the kinds of interaction that allow and hopefully encourage us to be ourselves in the company of others. An unstructured order underlies the provision of face in these kinds of situations. For our face is as important to others as theirs is to us. Facework maintains and produces. It carries and provides emotions, taking and giving, offering and requesting. The bonds that tie good friends can be seen written on their faces and in the wide range of expressions they find that they generate spontaneously in performances among the comfort of companions. An order no less an order for its light spiritedness. Bonds strong enough to survive challenges of disappointment and frustration. Connections and proximities that endure the misunderstandings of poor communication, of gossip or rumors, secrets and sometimes betrayals. And moments of real anger. "Listen to me when I'm talking to you." That's a filial expression. Perhaps more complicated than friendship. Mother and son, the communication of order seen quite visibly on the face of an angry parent and a hurt child. "Ye-smother." Stern faces and disciplinary faces. Faces of authority and punishing faces. Filial relationships, being hereditary and not a matter of individual choice, incorporate a wider range of affect than the trade of friendships and exercise of friendly relations.

Emotions belong to the inner self. How then could they be the currency of relationships? And yet they are. Faces distribute emotion, happy and sad, and all those in between, among friends and family. Mouths speak, eyes gleam, through faces upon which are written the feelings that motivate us. Faces serve us as vehicles of proximity. Through them and what we write on them we get close, we move nearer and farther. Movement of affect finds expression and thus a manifestation to others on the face that gives our words a home front.

"You talking to me?" DeNiro's character directs the question at himself by way of the reflective surface of a mirror. He sees in his own face the challenge to respond. "You talking to me?" The first thing a face learns to do is how to turn another face. "Hey, you talking to me?" it's not for the back of a head that faces turn other faces. If we could see through a face to what lies behind it, we'd surely be just as happy to do it through the back of a head as from in front of it. A face will only give away what its bearer chooses to show, and what its viewer is able to detect, and then some. That "some" being the layer of communication in which we don't participate consciously, but which finds its way into expression and which we betray... My face betrayed my feelings. A good reason perhaps for why I would rather not show my face. The face that serves me as a vehicle for my relation to others is at once a mask and a window. That I cannot remove it is a given. How well I can make it, put on a face, may depend as much on my friends and their knowledge of me as on my own ability to hide my true face. Like a screen that separates the outer and the inner, the face is a membrane through which affects pass. From within us to those around us. And from them to us. Like it or not, our faces provide a measure of recognition, and serve as testimony to the circulation of affect from one face to another. It is the zone upon which affects are made recognizable, that is, where they accumulate social or common meaning. The face makes visible, and thus sensible, the reception of affect. "Look at me while I'm talking to you," says the disciplinarian to the student, intending to gather from the student's face a hint of recognition, first, and then an acknowledgement and submission to the meaning of what's said. The face provides some measure of the effectiveness of communication. In the case of friends and family, of communication among people who know one another well, faces reflect the successful transmission of the deep and meaningful. They give away the passage of the affective element of communication and interaction.

For all that our faces give away in the presence of others, we keep a good many feelings to ourselves. Relationships are all a matter of imbalance, entanglements and webs of asymmetry. Structures that lean this way and that. We are on the move at all times. Our relationships too are in motion. We come and go, rarely the same. Differentiating ourselves, creating ourselves, inventing or discovering ourselves (choose your path) through our negotiation of commitments to those close to us. Being close is an inner feeling. A sense that where there is absence there is nonetheless close proximity. An understanding and a familiarity. With what we each are thinking, doing: with where we are each at. We hold this proximity dear, protecting it at times as if it were a sacred territory, a secret place, or a private privilege. Jealousies can quickly build around the violation of privileged relationships. For these kinds of close proximities are economies, and as such are not open to participation by all. Close relationships trade in affect and emotion. Their communication creates opportunities for the expression, reception, and distribution of affect.

The illocutionary power of speech is described as the power of speech to move the hearer. Affect may ground the force and intention of speech and interaction. It moves us, as speakers, to move others, as hearers. This is communication as binding, the kind of interpersonal exchange that has less to do with the manifest content of an interaction and more to do with the capacity for communication to produce results within an intersubjective world shared by those involved. We communicate in many cases to express and to a certain extent to resolve the movements that flow or run within us. Ambiguities that may surround our feelings about other people, for example, and that may perhaps characterize or give substance to how we think they think of us. Ambiguities produced by our existential separation from one another, caused by the fundamental impossibility of communication to ever create more than a bridge between people, binding them but not uniting them in proximity to one another. Affect, held on the inside, is a turbulent language, a field of emotional forces and fluxes that ebb and flow with the tides of our interactions, moved by the sun and moon of our closeness and separation from the intersubjective world of relationships.

There is very little in media theory that gives us a language with which to describe the experience of affect. We have few containers that seem adequate to both technical and emotional contents. Affect is one of those domains left often to psychology, where interactions and experiences are treated in the context of face to face interactions. The very practice of (psychoanalytic) analysis insists on facework, as it insists also that interpersonal issues be addressed with the honesty and sincerity that only physically co-present interaction seems able to provide. And yet we all recognize the inability of facework to guarantee resolution of our personal troubles. Our issues with others are more often than not an amalgamation of inner feelings attributable to self image and wellbeing as to material problems encountered with other people. It has long been known that conflict management is not simply a matter of getting people together to talk through their issues. Increased interactions sometimes lead to more conflict, as can increased proximity. Communication is not a guarantee of cooperation and understanding, as separation is not a guarantee of peace and non-interference. There is no satisfying description of proximity as a measure closeness. We shall have to create our own descriptions, our own terms and approaches. New proximities involving the mediation of affective relationships, the transmission or communication of affective or emotional contents, the expression and reception of messages and statements linguistically embedded but emotionally laden require us to fashion new terms for our inner world as well as the interactive world. This is especially true for the case of asynchronous communication, which operate by delay and relay and which also force affective contents into a secondary medium that is bound by its own internal grammar and signification.

Affects are physical sensations. They are the inner movements of sensation that belong neither to perception (touch, taste, sight, hearing) nor to kinesthetic experiences (of physical movement), and as we mean them here, they are prior to emotions. Affects can rise up spontaneously, utterly unattached or unrelated to physical stimulus or engagement. While our experiences of these kinds of sensations may often be triggered by a thought, memory, or by something we have just seen or heard, affects are not contingent on stimulus. They are forces of sensation of a pure kind. They move through us and fall into relation with images of memory and recollection, with sounds, words, phrases, with tastes, and smells, and with physical touch and movement, creating and giving recognizable shape or form to our experiences in a way that we feel them, and that we recognize the feelings we have in them. As affects, fall into relation with experience we recognize a sensation's meaning. We are able to place it, to connect it with other sensations, memories, and experiences. Affect is the force of movement prior to these meanings, the pure and unconnected motion interior to human experience. It gives us all our capacity to feel and what we sense, to relate to events in a way that has a power, a power of movement and intuition, that gives us a grasp and insight. Affect is not a common sense, though it gives us each the power to sense. Affects fall into relations with experience in a personal and individual manner, coming to expression through our personality and character, as tendencies to humor, wit, irony, reason, compassion, empathy, profundity, and so on. With affect we have the ability to perceive the world and to experience real events with a sense that gives them a force, as those events fall into relation with our interior movements of affect. And we have the ability to create, to express, to manifest and exercise passionately and forcefully. Affects fall into relation with the world as it is experienced as well as with the world as we create or produce it. Affect runs equally through the dream world, and the imaginary (as we all know that there is nothing less "real" about these experiences from an affective perspective). In fact, the mentally ill often experience their reality as a mix of inner and outer worlds in which the affective dimension of experience makes the "wrong" distinction between worlds. Mentally ill are those people who find difficulty conforming to a social reality that is not as real for them as it is for others. And which doesn't exercise the normative force on their experiences that it does for the rest of us.

Affects attach, relate, link and couple. Affects create chains and series, lines, relations. As the movement of affect picks up it might attach to an object in the real world, or an image in the interior world. It might become invested in a memory, a projection, or a gesture. It might fall into relation with other affective movements and in combination become an emotion. Such as jealousy, greed, or paranoia. And as an emotion, it might settle with a person, harden and fasten itself to particular experiences and memories, soon becoming a personality or character trait. A jealous person, a greedy person, a paranoid person. It might find objects, connect with people, establish techniques or methods by which to realize its dominance: the husband jealous of his wife's tennis coach; gambler unable to play a last hand until a particular card has shown its face; paranoid employee afraid to make eye contact at the watercooler.... It is only through the relations established with affective investments that affects obtain attributes and attributions. Their abilities are undetermined and amorphous until they fall into relation with actions and impressions. Emotions are affects in relation with one another and with an experience, be it recurring or not, be it interior or exterior, attached to an object, person, or image, sound, song, taste, or smell. Emotions are the recognizable manifestation of affect, formed affects, or formations of affective movement having taken qualities and representations. Affects and emotions alike are grounded in an embodied experience of the world, a sensibility that is prior to the physical but that involves the physical in its unfolding and connecting. And while they are individual, they are what animate our interactions with others and with the world. The face, of course, is the site upon which we express the greatest range of affect. Through acculturation and socialization, affects in the form of emotions obtain ways to make themselves clear through expressions shared as a language, a visual and facial range of expression recognizable to and shared by others. And it's for this reason that communication always seeks to turn a face. It is in sharing of face, and recognition of expression, that we align and bind with one another in spite of the enormous intersubjective separation and distance that gives human experience its poignancy.

We have already exceeded the linguistic approach to communication by our passage through pragmatics, arguing that interaction involves meta-linguistic aspects of communication that cannot be found within language systems alone. We must now exceed pragmatics by attempting to recover the force of affect somehow. If we are to map the ambiguities produced by mediated proximities, if we are to develop a language for the experiences of asynchronous communication, characterized as they are by a stammering and stuttering system of delays and interruptions, persistence of presence and suspension of interaction, then we must come up with a framework of proximity that can better account for the displacement of affect and its dissociation from linguistic interaction. Linguistic analysis assumes a more or less direct line between agency and expression. Bracketing the second-order analysis of psychology, which has demonstrated that we do not always know what we are saying insofar as we are not privy at a conscious level of our own intentions, linguistics presumes that we do, for the most part, mean what we say. Words and statements do not betray us. Rather, they serve us, and we them. Speech is a medium in which we express our intentions. Ambiguities arise in human interaction because of a residual uncertainty that can be pegged to intentionality and language's inability to capture the full spectrum of intentionality. We continue to speak to one another, says the linguist, in order to resolve what is always left as the residue of ambiguity unassimilable by language. Ambiguity is the intentional left over, the remainder that cannot find form and expression in speech. The affective domain is the intensional, as opposed to the intentional. It is that inner movement of sense prior to agency and so prior to expression. We need this idea of an intensional domain because we need to supplement the expressive form of communication with an impressional form. We have developed this need, in turn, because mediated interactions involve a higher degree of impressional involvement than do face to face interactions. Our experiences of asynchronous communication can, and often do when they involve proximities of emotional closeness, produce interior relations, relations of self to self, formed or attached to images of the other who is not physically or acoustically present. Asynchronous communication puts us in relation to others in the form of acceptable hallucinations.

There's an intriguing schizophrenization involved in the use of asynchronous mediation, a dissociation of speech or talk from presence that forces us to address new ambiguities-technical affects obtained from the distorting effect of technology. If speech in face to face interaction is a medium of expression more or less commensurate with our intentions, the experience of asynchronous proximities unfolds upon a field of impressionality and sets forth intensional movements within us that compensate for what's missing by hallucinating, inventing, and imagining interpretations and impressions. The analyst will say that ordinary human communication satisfies the following conditions: I am talking to you here and now. And the analyst will say that the schizophrenic breaks one or more of these rules. I, Joseph Stalin, am talking to you here and now. I am talking to you, my captor, in this Soviet gulag, in 1943... Is email a schizophrenic mode of interaction? Not insofar as it is accepted, and not insofar as it makes no claims to an alternative reality. In its disembedding of speech from the face, and of interaction from co-temporality, it certainly shows a few schizophrenic tendencies. I am writing/talking to you/my image of you here/there and now/later. Of course I'm not trying to suggest that email is a sign of cultural insanity, or that text messaging shows a culture at odds with reality. But if we take the simple route and treat such asynchronous interactions as extensions of a writing system (mail, correspondence, letter writing) we're missing out on an opportunity to understand the profoundly new technical-human relationships involved the mediation of human communication.

Asynchronous media cannot turn a face any better than a schizophrenic can. It is the dislocation of message and response, of call and answer that makes asynchronous communication unique. Perhaps some day in the not so distant future there wont be any asynchronous media formats out there. It's certainly within the realm of possibility if not likelihood. Wireless developments could soon guarantee that all text messages (and soon audio and video) will arrive at the speed of light. Then perhaps we will call it phonetyping, typecalling, or phonewriting and everyone will be doing it. It is more likely that when instant messaging is possible across all media we will engage in asynchronous communication even more so. For the frustration that comes with being in constant touch will generate an even stronger need for low intensity asynchronous interaction. The delay between sending a message and receiving an answer puts us into relation with the impressional and intensional domains of experience because we are now either composing speech to ourselves, or reading speech to ourselves. Asynchronous correspondence is an activity engaged in privately, internally, through and with impressions and interior affective flows. These flows now fall into relation with writing as well as with the medium, for they must do so in order to either produce or receive expression (messages) with any interpretive accuracy. What was meant by that last message? Why is she taking so long to get back to me? Am I being sensitive, or was that last message a bit cold? Or is he a poor emailer? These are technical affects loosed by the medium, affective flotsam created by the separation of speech and writing, face and presence/expression. They are byproductions of asynchronous interaction that belong neither to the medium nor to the sender. These are not existing meanings, or meanings that exist per se. Nor are they strictly our interpretations of messages, for some of these affective byproducts begin with silences, delays, and interruptions. They are intensional and impressional because they relate to the interpretive movements of communication and interaction that call forth interior images, recollections, memories, as well as hopes, desires, and fears. It is for this, and nothing else, that email and chat have proven so effective at seduction. For this that rumors, particularly those that prey on get rich quick schemes, have been so effective over email and the web. Asynchronous media put us in new relations to ourselves just as they put us in new relations to others. They force us, by unpacking messages and separating the contents from the intents, through the impressive and the impressional, where the spontaneity of expression finds its mirror image: a spontaneity of impression. We argued above that affects were formless until they fall into relation with their attributes, and now we understand why we made such a detour past the conventional language of emotion and intention: asynchronous media create new interior relations of the self, our sensitivities (as movements of affect triggered by the dissociated presence and expression of an other), words and utterances, timings and absences in the form of impressions equal to the form of expression. There is an experience of mediated interaction that is as profound in what we do not see, in what is not expressed, as there is in what we do see, read, and write. Messages make an impression, make impressions. Those impressions are not explained by the intentional discourse, speaking, or writing of the other. They are described by the intensional movement of affect within an impressional space that works as much with what is not there (silences, pauses) as with what is there (words, utterances). Linguistics may attend to communicated utterances, and pragmatics to the effects of speech in the context of co-presence performances, but neither covers the new proximities of asynchronous communication.

What strange new talking machines, these asynchronous couplings of writing and speech, expression and delay, reading and impression. What curious mutterings we get from messages that can be as much a production of self talk as a form of communication or interaction. What odd mumblings we get in the form of text messages, with their characteristically off angle declarations and statements, abridged and punctuated to fit within an 80 character limit. The stutterings of messages sent and then curiously resent or referred to, sometimes even through another medium as when a mother calls her son to ask if he has received her email. And of course the quotes and samplings. What do we make of those, those messages in which the words belong to another but are delivered verbatim and sometimes without commentary. Are these acts of possession? The differences unique to asynchronous communication are found in the relations and connections it creates between self and expression, expression and impression, and the binding of speaker and hearer through linguistically mediated interaction. How do asynchronous formats stretch relations, or rather, what happens when they do? Are the connections established during moments of asynchronous interactions integrated into our real world experiences? Are they separate because their mode of production is separate? Or do we integrate these different worlds internally, and experience them as the same, or at least, as related?

We began this section asking what was new of these new proximities. And particularly, how we might describe the experience of being close to another person in a language adequate to the needs of emotional connection as well as mediated connection and interaction. Asynchronous interactions do establish new kinds of proximities, proximities in which one's relation to the other person is experienced sometimes as much by and through participation in absence as through participation in what's given for presence. We are all always and already separate and disconnected. These proximities in which we experience the formation of new kinds of connections effectively produce new formations of power and influence, trust and exposure, intimacy and loneliness. The micro couplings of affect in the intensional and impressional produce recognizable forms of organization as they scale. Authority, which may originate in the face of the god, then the ruler, the religious figurehead, the head of the clan, the father, reassembles itself along interior and intensional connections and is perhaps for some no less effective, legitimate, or coercive in email than in real life. The seductive and intimate date is for some no less convincing, alluring, or mobilizing in email than when sitting across the dinner table. We do not necessarily lose our relations to others by conducting relations through asynchronous interactions. We shift and reconstitute them along new lines, lines that pick up internal images and emotional blocs, that fill the spaces and gaps of interaction with meanings and projections, that anticipate consequences based on memory or internal authority figures as much as voices raised. Proximities of the impressional and the expressional form in aggregations as well as in one-off encounters. The relations here are stretched through time and space in a different manner than they would be without mediation, but they are stretched nonetheless and especially so. These mediated presencings have new intensities. Some place a call that we answer immediately. Others may seek but have no call at all. Take spam, a product of email whose only claim on attention is realized as we move it from our inbox to the trash can. These are relations that are thin when measured by the degree of presence realized through them, but no less capable of moving one another. We find loyalties among web-based communities, seductions in chat rooms, and secrets traded over email. These are interpersonal and social phenomena that belong traditionally to face work situations and communication in which speech is governed by a strong co-dependence on context and practice. Asynchronous interaction leaves us more yet to discover. As a production format of interaction, its possibilities obtain along untraditional unconventional axes.