Social Augmented Experiences leveraging geoawareness and human and machine intelligence to create real time information brokerages, combined with an augmented reality view, can create a new opportunities to reimagine our relationships with each other and our environment.

This Summer, I have been on a blogging hiatus, which has meant I haven’t been sharing as frequently and, unfortunately, the second half of two conversations I had earlier this year, both of which have much influence my thinking on social augmented reality, have languished in private mode -  part 2 of my talk with Bruce Sterling (see Interview with Bruce Sterling, Part I: At the 9am of the Augmented Reality Industry, are2010, and part 2 of my conversation with Anselm Hook - Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, Part 1. Time to get caught up on some blogging!  The lightly edited transcript of Part 2 of my conversation with Bruce Sterling is posted in full below.

Bruce Sterling has been blogging all the key developments in augmented reality (amongst other topics of interest!) on his Wired Blog, and he brought my attention to Boskoi the Ushahidi based app for Android phones, augmented foraging pictured in use above – for more pics see fightthegooglejugend.

Augmented Reality and Real Time Information Brokerages

Picture above is the path the “nomads” took through the Westhaven cryptoforest with Pieter Bol,co-auteur of the book Biological Globalisation and Theun Karelse of Urban Edibles Amsterdam “who presented his ‘augmented foraging’ app Boskoi.“  For more see, The Cryptoforests of Utrecht and, Westraven Psychogeography, 6 June 2010. Note: Cryptoforests: 1) Urban forests hidden from view 2) Urban fallows that might or might not be considered as forests 3) Gardens gone wild)

My interest in the Ushahidi family of ideas was already fired up by a conversation with Anselm Hook early this year.  We discussed a number of Ushahidi related projects, Swift, Crisis Filter and Anselm’s project Angel, Augmented Reality, and my own keen interest in an open, real time, distributed platform for augmented reality – ARWave.

The Ushahidi platform and the related project Swift has pioneered the real time brokerage of information with people acting in curatorial roles or matchmaking roles coevolving with machine assisted matching to connect wants to haves.  Ushahidi uses multiple gateways including SMS, and Twitter.  But the Ushahidi family of ideas is extremely interesting when combined with augmented reality and suggests many new possibilities for social augmented experiences, as Anselm pointed out, for human to human communications, human to civilization communication, and human to environment communications (e.g., perhaps, how machine intelligence can help bridge the difference in time scale that Kate Hartman explores in her, Research for Glacier-Human Communication Techniques).

Ushahidi, which means “testimony” in Swahili, is a website that was initially developed to map reports of violence in Kenya after the post-election fallout at the beginning of 2008. It is now an open platform with a wide range of applications and growing developer community.  See What is the Ushahidi Platform? from Ushahidi on Vimeo.

Swift - a project that emerged from the Ushahidi dev community, is a human sensor/real-time brokerage for dealing with emergencies, enabling the filtering and verification of real-time data from channels such as Twitter, SMS, Email and RSS feeds.

Boskoi – augmented foraging is the first app,  I have seen, to begin linking Ushahidi with augmented reality – although I don’t think there is a full augmented view for Boskoi developed yet?

“The whole point of AR is to see things from a different point of view…”

Click to enlarge poster from upcoming ARWave demo at Software Freedom Day – for more see below

I am often asked what augmented reality brings to the table with respect to location based social networking, which is on the verge of going mainstream in smart phone apps like Four Square. While the first part to my answer is usually to explain what is unique to augmented reality.

As Bo Begole notes, the full vision of AR requires machine perception technologies to detect the identity and physical configuration of objects relative to each other to accurately project information alongside/overlaid with a physical object (see this post on the PARC Blog by Bo Begole on the difference between AR and ubiquitous computing – thank you Rouli for bringing my attention to this).

But it is only in recent months that we have begun to see the kind of tools that make this possible become freely available to developers – see my interview with Jay Wright of Qualcomm here.   Also see this post on Bundler: Structure from Motion for Unordered Image Collections an open source system that allows the creation of 3D point clouds from unordered image collections, e.g. internet image collections.  We now have many tools available to move mobile augmented reality beyond the recent crop of apps relying on GPS and compass alone for positioning into a new era of vision assisted AR apps that will increasingly bring the full vision of AR into our daily lives.

Further, the integration of visual search applications like Google Goggles and Kooaba which can detect the identity of particular objects will add another vital tool to machine perception technologies enabling AR “checkins” on potentially anything in the physical world around us, and more fuel to the Gamepocalypse (e.g. it would be easy to turn every trash can in the city into a basketball hoop as we discussed at the ARNY meetup last month).  And soon, the Pandora’s Box of  facial recognition (Google Goggles have the capability though it is not released to the public yet) will open up.

Jesse Schell described the importance of AR in a nutshell in his keynote for are2010:

“The whole point of AR is to see things from a different point of view…How can there be a more powerful art form than one that actually changes what you see?”

But how AR matures as a social experience will be the key to Jesse’s suggestion that:

“Augmented Reality will be one of the things that fundamentally define the 21st century”

There are many interesting forms of AR that are not reliant on a tight registration between media and physical objects – several are put forward by Bruce in the convo below.  And, it is likely we will see AR eyewear as an occasional useful accessory to a smart phone long before we have the sexy, affordable augmented reality eyewear worn that we wear throughout the day.   These speech to text glasses would be a very useful and viable accessory to a smart phone right now for the hearing impaired.

For the moment, as Bruce notes, some of the most interesting and useful augmented experiences to date have not been in the cell phone space:

“There are other aspects of AR besides the cell phone space. There’s Total Immersion’s big display screens. There’s the web-based fiduciary stuff. And there’s projection mapping. And then there’s experience design just for people who need their reality augmented for whatever personal or social reason.”

On of my favorite social AR experiences is this SMS Slingshot.

But I have been excited for a long while about the intersection of mobile social augmented reality, real time communications, and ubiquitous computing see Total Immersion and the “Transfigured City:” Shared Augmented Realities, the “Web Squared Era,” and Google Wave.  And I have described in many places why I think real time, open, distributed communications for AR are so important to developing social augmented experiences – see the slides for my talk at Augmented Reality Event here, here and here for starters.

ARWave at Software Freedom Day 2010, September 18th 2010

Thomas Wrobel and Bertine van Hovell will demo the first ARWave Android client at Software Freedom Day this weekend!

A number of people have asked me, (including Bruce), What will be the future of ARWave now that Google Wave is no longer a stand alone application?  Yes, the recently announced release of Wave in a Box (see here and here) is very exciting for the ARWave team.

The ARWave Android client is the first open AR client built on an open, real time, distributed platform -  based on a server that anyone can download and set up, currently the “FedOne” server but Wave in a Box, hopefully, will be even easier to deploy.  Wave in a Box seems perfect for ARWave’s needs – for more see the WiaB Google Group here.  And for more information on the ARWave client -  click to enlarge the poster below, see the ARWave concept video here, and for more, and how to get involved see arwave.org.   Props to Thomas Wrobel and Bertine van Hovell (posters below from demo for Software Freedom Day), Mark Evin, Davide Carnovale, and Markus Strickler, for all their hard and brilliant work on ARWave.  Also to JCPT the open Android 3D engine that has saved a lot of work!

click to enlarge slide

Social Augmented Experiences that Matter

My ideas on the future of social augmented experience have been deeply informed by the the conversations I had with Bruce Sterling and Anselm Hook this year.

Bruce Sterling notes in the conversation below, location based social apps like, Four Square, are interesting because they are not “urban geography like Google’s satellite stare from above,” but rather “groups of citizens are doing portraits of their own region.” Augmented Reality, with its of lauded power to make the invisible visible is, of course, is the ideal tool for “citizen portraits”  to the next level.  Cory Doctorow described to me three years ago (see here) an “inverse surveillance society,” enabled by an augmented view  – “where all the data from the positional and temporal characteristics of all the objects that we own were in aggregate visible and available so that we can mix and match them remix them understand them and have more agency in the world.”

It is very cool to go back to reread this conversation now that it is becoming possible to build the kinds of apps Cory described, and Bruce Sterling envisioned in Shaping Things (see Amazon.org  page 111).

click to enlarge

My  conversation with Bruce earlier this summer (see below) took place on the heels of are2010 – Augmented Reality Event.   See the video of Bruce’s keynote, “Bake a BigPie,” here,  and the final keynote, “Seeing,” by Jesse Schell (see video here) in which Jesse riffed on AR and the man with the X-ray eyes.  Both these awesome talks are still fresh in my mind.  Bruce noted how we should pay attention to augmentations for people and situations that could really use some augmentation… and not get too fixated on the coming of AR Goggles.  He elaborated on this in our conversation (again full transcript below):

“Well, it’s a matter of deciding whose reality it is that you’re trying to augment. I’m not trying to be a bleeding heart about it, but obviously there are people in our society right now with reality that could really use some augmentation. They are mostly disadvantaged people. They are vision impaired, or maybe they have autism. They might be senile and just can’t remember where they put their shoes. These are people who could really use some help, right?”

So, start with people who really need sensory or cognitive help. Before you turn our geeks into Superman, why don’t you try turning some people who are harmed into more functional individuals? Then you’ll be able to learn how to do that. Then maybe you can ramp it up to these Nietzschian heights of the superb Man With the X-ray Eyes. Whatever.”

What will make AR interesting and useful long before and long after we see the full vision of AR eyewear manifest is its social aspects.  Bruce points out:

“My argument would be that if you want people to be more sensitive toward certain, say, issues and problems, it’s better to find the people who are already sensitive to those issues and problems, and give them a bigger stake in your augmentation system.”

“Say that I am really worried about public health. Well, if I have a lot of nurses that are using my system, people who are aware of my issues, then I could be walking around and I’ll see a lot more tags saying, “This is where he got food poisoning!” “In this shooting gallery, many people have caught AIDS!” Or, you know, “Tuberculosis has been spotted over here in this building.”

At that point, I could simply share their knowledge and get some social intelligence. As opposed to trying to amp the basements of my little hacker-mind and drag stuff up that’s escaped my conscious attention.”

Finding new ways to broker information – bring together needs with haves and different participants, empowered and disempowered  is., as Anselm discussed with me, one way to change our view of human to human, human to environment and human to civilization communication (particularly in light of this  “sobering account of how open data is used against the poor in Bangalore” that as @timoreilly noted recently OpenData Empowering the Empowered).

The key idea in a crisis filter, Anselm noted,  was to break up the participants into different kinds, to connects wants with haves:

“There are people who are in  situation.  We call them citizens.  And then there are reporters, people who report situations back to Twitter.  And then there are curators, people that canvas Twitter looking for important Tweets.  And then there are first responders, people who take the curating collection of responses and then act on them.”

This kind of brokerage between people acting in a curatorial role or matchmaking role with each other can be extended into and coevolve with machine assisted matching as Anselm explains.

It is also a vital part of creating social augmented experiences that matter.

One of Anselm Hook’s projects, which is called Angel is the the most radical expression of connecting wants with haves in that the idea is that “you have a situation, you broadcast that situation, and help magically appears.  You don’t even sign up for a service.  You just get help …

As Anselm explains this is the same idea of a brokerage for dealing with emergencies, but applied to the long tail of crisis response.  As Anselm describes it:

“I am interested in personal crisis.  ‘I lost my cat.  Help.  I can’t find where my kid is.  I am out of gas.  I have a flat tire.  My house is on fire.  My aunt is trapped in the bedroom.’  The kind of personal crisis that is just as important, but is not enough to get a national movement to help you…

I will publish this conversation with Anselm in full in an upcoming post.

Zorop – an ARG for World Peace

If you want to be part of a really exciting experiment to reimagine our relationships with each other and can be in San Jose this weekend, I highly recommend exploring this “rabbit hole”.

Thank you Gene Becker, Lightning Laboratories and Ubistudio for sending me this invite:

“Ken Eklund (@writerguygames) is developing a wonderful game for the 01SJ Biennial called ZOROP, aimed at creating World Peace(!). Some of you might know Ken from his work on the amazing ARGs EVOKE and World Without Oil. Anyway Ken, along with his collaborator Annette Mees, are furiously working to get ZOROP ready to go for the Sept 17th premiere at 01SJ.

Are you intrigued? I thought so, and here are your next steps down the rabbit hole:

> Check out http://zorop.org to learn about the game

> Follow @ZoropPrime to watch it unfold: http://twitter.com/zoropprime

> ‘Like’ ZOROP on FB for a different view: http://www.facebook.com/pages/Zorop/141140772593618

> Become one with the game; consider volunteering as a Zoropathian: curious@zorop.org

> Head down to San Jose on the 17th, play the game, and ride the ZOROP Mexican Party Bus. Seriously.”

Interview with Bruce Sterling

Click on image above to see video clip from from brucesflickr

[Note the first part of this interview is here and I broke in anticipation of Part 2 just as I started experimenting with an idea Joshua Kauffman – an advisor and entrepreneur working on design in the public sphere gave me for an interview technique – the All Souls College one-word question interview.  Although apparently they recently scrapped it and I am not very good to sticking to a single word!]

Tish Shute: We were talking about these proximity-based social work networks like Foursquare and Gowalla and how they may influence the emergence of social augmented experiences.

So Joshua’s suggestion for the first word was “territorialization” e.g. how do these new mobile social experiences like Foursquare, and the observation that actually rather than breaking down territorialization – which would be a good thing, tend to support territorialization…

Bruce Sterling: Yeah, they’re re-intensifying it in a very odd, electronic fashion.

Tish Shute: Yes.

Bruce Sterling: It’s not true of projection mapping or the webcam fiduciary display stuff. But with the handheld stuff, and especially the urban informatic stuff, it really can’t help but take on a local flavor. Layar is like “Augmented Dutch Reality.”

And TonchiDot is “Augmented Japanese Reality.” It’s hard to imagine a Layar interface going gangbusters at Tokyo. Whereas the TonchiDot interface, which is so clearly influenced by Anime and cartoon graphics…. Maybe it could find some niche of hipsters in Amsterdam hash bars…

Stuff that’s socially generated by people on the ground, as with Foursquare and Gowalla, is bound to take on a regional influence. Right? It’s like the New York hipsters who were early adopters of Foursquare. They’re not mapping New York! They’re mapping Hipster New York.

It’s all about Williamsburg and places where 24-year-olds go to drink… They found a demographic niche there. These guys are building the service for them. They’re people who are willing to work for Foursquare for free, because they want to wear the little king hat.

Tish Shute: I got the far far away badge ‘cos I live on the Upper West Side!

Bruce Sterling: But that’s not urban geography, right? I mean, that’s not like Google’s satellite stare from above. That’s a group of citizens doing a portrait of their own region. You’re going to see interesting things happen because, of course, people who use Foursquare elsewhere are going to check into New York, and they’re going to look at the “New York Foursquare.” They’re going to be aliens who interact with Foursquare people in New York and annotate what they’re seeing.

Tish Shute: Oh! Yes. Good point.

Bruce Sterling: That Foursquare community has a certain émigré soul. It’s different from the normal émigré soul of simple tourists on New York. So you’re friend there is right about the territorialization.

Tish Shute: Yes, Joshua Kauffman is a smart guy! Yes I am interested to see what interesting kinds of deterritorializations proximity based social networks and the hyperlocal view of augmented reality might bring, not just the new territorializations.

Bruce Sterling: It’s not the intense kind of territorialization, like gangs putting down graffiti markers and beating people up. It’s an inherent regional character that comes with using peer production to build your database.

Tish Shute: We were discussing whether AR could break down the walls between people – people who share the same physical space but actually inhabit different territories even if they are sitting on the table next to you.

Bruce Sterling: You know, I just wrote an article for my Italian magazine column. I think I mentioned this to you – a report about ARE 2010. I titled it, “Chicks Dig Augmented Reality.”

Tish Shute: [laughs]

Bruce Sterling: There is a very heavy social element to AR, and a phone based element. So the question is: Why would a woman wear a fiducial marker? Like our Metaio speaker at ARE2010 who had a fiducial marker on her lapel pin.

Tish Shute: Right. Lisa!

Bruce Sterling: Why would a woman go out in public with her Facebook profile on her body?

Tish Shute: Well I can think of some reasons…

Bruce Sterling: So that men will approach her, of course.

Tish Shute: Yes the core of all successful social networks is always a form of dating app.

Bruce Sterling: You do it a social icebreaker. It’s like: I’m a woman, I’m sitting here alone, and you can sort of glide by and, you know, take a snap of me. Then you retreat and have a beer with your friends and you work up the courage, and then you come and say, “So! Susan! I understand you like bicycling! And, boy, me too!” Right?

Tish Shute: There are all kinds of social barriers between people in cities that AR might be helpful in breaking down. An extreme example is the dilemma you actually quite often face as a New Yorker as you walk around a city. There are people asleep on the pavement and you don’t know if they’re dead or alive.

Bruce Sterling: Right.

Tish Shute: And you sort of like have this awful ethical dilemma of like, “Am I walking by someone I should be shaking by the shoulder, right, to wake them up so they don’t die, right?”

Bruce Sterling: Yes.

Tish Shute: You said in your keynote that we should pay attention to augmentations for people and situations that could really use some augmentation..

Bruce Sterling: Right. There actually is such an app in Britain right now. I posted about it: two Augmented Reality schemes for rubbish and hobos.

Tish Shute: Right. Yes I saw that!

Bruce Sterling: “Any sufficiently advanced technology is indistinguishable from garbage and hobos.” You don’t need to personally find out whether this hobo is worth your help. What you need is a good way to report the hobo to a hobo check-up service. They come in, and they look on their own database or supply a database to you, or a facial recognition unit, whatever. The service says: “Oh, well. That’s Fred. He’s a paranoid schizophrenic. He always sleeps in that alley. Let him be.”

The same goes for the rubbish — although I don’t want to compare rubbish to hobos. In fact, people do go out with their AR kits and take pictures of abandoned garbage bags and broken glass. They upload them with geolocated tags for the local garbage guys. Guys who are sitting around doing pretty much nothing because they don’t know where the rubbish is.

And they will come out and get the rubbish! I mean, they just deputize guys to go out and follow these alerts. Right?

But nobody predicted — least of all me — that you were going to have a high-tech Augmented Reality system that consisted of removing rubbish and derelicts. Right? But rubbish and derelicts always go profoundly under-reported. It’s just hard to get people’s attention.

But it’s very easy to set up a system so that, if you get ten reports on the same piece of rubbish, that’s going to work its way to the top of the stack. That’s why I was trying to get AR people away from the romance of the hottest app for the shiniest machine. More toward a design stance that’s more user-centric.

Where are the actual problems about stuff that we perceive? Stuff we can’t do anything about? Or people whose mechanisms of perceptions are harmed. They could be doing good work, being more participative, if they didn’t, basically, walk around without their glasses on.

Tish Shute: Well this leads well into the second word, Joshua suggested was interesting spring board – sensitivity.

On the one hand we can do these things for people who maybe need the augmentation because they have difficulty with one or another sense, e.g., their eyes are not functioning, or their ears are not functioning. But on the other hand, we can’t cross the social bridge to communicate with people who are temporarily disempowered in relation to the rest of society e.g. hobos and people who sleep on the streets of New York City.  And even though Augmented Reality could potentially be helpful it can even be more disempowering to the already disempowered.

Bruce Sterling: Right.

Tish Shute: But re “sensitivity” – does augmentation increase or decrease our sensitivity? This is a problem that Will Wright brought up [see video of Will Wright’s keynote at are2010], e.g, the problem of parking HUDs getting in the way of your intuitive parallel parking skills. The Lexus that takes driving control from you when you look back, ‘cos it knows that you’re looking at the road, and it starts to brake. Right?

Bruce Sterling: Right.

Tish Shute: The fact that the problem with technology is that it makes us less sensitive, right, augmentations sometimes get in our way?

Bruce Sterling: I suppose that’s true. But I’ve heard that said about practically every medium. Especially television.

Everybody wants to blame machinery for their lack of morality. It’s hard to top something like the Kitty Genovese killing in New York. This sort of legendary New York horror story from the 1960s. A woman is stabbed to death in public, no one does anything.

Tish Shute: Right.

Bruce Sterling: I don’t think that our media is making us any less humane or more callous.

Tish Shute: All right. Oh no! I see what you’re saying. Perhaps I misrepresented what Will was suggesting by putting it that way. The question is perhaps more how do we get the sensitivity into the technology. Human bodies are fantastically sensitive and sensory.

Bruce Sterling: Right.

Tish Shute: And we have these like sensitivities. For instance, How could augmentations of reality be like a blush ? You definitely want an interaction that’s not just this data being pushed at you. But what is the data that counts, right? Will shows a slide often of an iceberg with the tip of the iceberg which is the conscious mind.

Bruce Sterling: Oh, I see. Yeah.

Tish Shute: And underneath it is all the preconscious stuff that really counts, right? Any thoughts on that?

Bruce Sterling: I did take interest in that. Will has obviously been spending a lot of time studying cognition.

Tish Shute: Yes.

Bruce Sterling: I’m not convinced that AR has got a lot to do with that. There is certainly a trend there. There are a lot of people who want to do body hacks and brain hacks. I can imagine AR being used for that purpose, but it seems like a niche application. What is the point of our accessing even more stuff that’s outside of our consciousness?

Tish Shute: One of the things he is talking about is game dynamics, is it? The role of the imagination in play. For example, he shows the high dynamic range photos that make the world magical. Something you want to engage with playfully. This he points out increases a sense of agency because you are encouraged to engage and to play with the world.

Bruce Sterling: Well, I’m a literary guy. Italo Calvino did a lot of writing about this. He talked about the classics of literature. Why do we read the classics? Calvino said we do not read, but reread the classics. And the reason we do that is that, at first, we read a classic book and we think, “Boy, this book is really good.” Then, five years later, we read it again and we think, “Boy, this is a really good book, and it’s got so much more in it than I thought it had when I was 18.” Then we read it again at 28, and it’s like, “OK, now I really seem to understand this book, and it means something to me now that I didn’t know when I was 18 and 25.”

What you are doing through that access is learning something about yourself. So Will is arguing is what I really need is like a better augmentation. So that I can go in there and sop up the book all at once. I can grab every cultural nuance in it, instead of the stuff that’s sliding past me because I’m 18 and kind of young and hasty. Maybe I could have certain words and phrases helpfully underlined, that are like, “OK, well, this part is problematic for you.” In some sense, that’s not allowing me to be 18.

I’m never going to have the experience of my own maturation against this text, because I’ve devoured it all in one gulp.

My argument would be that if you want people to be more sensitive toward certain, say, issues and problems, it’s better to find the people who are already sensitive to those issues and problems, and give them a bigger stake in your augmentation system.

Tish Shute: Yes the social augmented experiences are going to be the most valuable.

Bruce Sterling: Say that I am really worried about public health. Well, if I have a lot of nurses that are using my system, people who are aware of my issues, then I could be walking around and I’ll see a lot more tags saying, “This is where he got food poisoning!” “In this shooting gallery, many people have caught AIDS!” Or, you know, “Tuberculosis has been spotted over here in this building.”

At that point, I could simply share their knowledge and get some social intelligence. As opposed to trying to amp the basements of my little hacker-mind and drag stuff up that’s escaped my conscious attention.

Tish Shute: Interesting that seems to bring us to another kind of repetitive theme in AR, the people tend to pigeon hole it as “merely” a visual interface. But actually, it’s the intersection, isn’t it, of social intelligence and augmentation.

Bruce Sterling: Well, it depends entirely on how you design the system. If I’ve got a military augmented reality, I would expect that to be mostly about urban fighting. It’s going to be about kicking in a door and shooting terrorists. If I pull that helmet off my head and put that on the head of an emergency worker or a cop, I’m going to get a militarized cop or a militarized emergency worker.

Tish Shute: Well the histories of the two great mass media of the twentieth century – TV and the atomic bomb were intertwined, and I suppose the evolution of ubiquitous media, augmented reality and urban warfare is already intertwined too.  So how can we encourage augmented realities to move beyond military roots that is common to much technology and into more peaceful urban realities?

Bruce Sterling: Well, it’s a matter of deciding whose reality it is that you’re trying to augment. I’m not trying to be a bleeding heart about it, but obviously there are people in our society right now with reality that could really use some augmentation. They are mostly disadvantaged people. They are vision impaired, or maybe they have autism. They might be senile and just can’t remember where they put their shoes. These are people who could really use some help, right?

So, start with people who really need sensory or cognitive help. Before you turn our geeks into Superman, why don’t you try turning some people who are harmed into more functional individuals? Then you’ll be able to learn how to do that. Then maybe you can ramp it up to these Nietzschian heights of the superb Man With the X-ray Eyes. Whatever.

Tish Shute: Did you notice that a couple of apps actually like TagWhat have apps geared towards people with disabilities – I haven’t had a chance to check it out.

Bruce Sterling: I’m sorry, I wasn’t looking at their tags.

Tish Shute: I was discussing this with Joshua who mentioned Zachary Lieberman’s Eye Writer, which is for people with locked-in syndrome. Do you know that?

Bruce Sterling: Sure. And people appreciate that because the poor guy, he’s laid up with Lou Gehrig’s Disease. Now they’ve given him a way out. AR is like a spark of new hope that gives his life meaning. What’s wrong with that?

Tish Shute: Yeah. And Tim Byrne using Sixth Sense for Autism is interesting.

Bruce Sterling: Let’s consider it the other way. Let’s say this graffiti writer there, instead of him being sick and weak, let’s say he’s an athlete. So I want to make him into a super-human graffiti writer. I want him to run around graffiti-tagging the entire town before dawn. Is that a good idea? Do we need that? Super human, super taggers? What if he’s going to spray up stencils of Nietszche? I kinda wonder whether the game is worth the candle.

Tish Shute: Yes I suppose it is not a great social scenario to be always augmenting the lives of the elites! Hmm, the third single word interview question is “homophily,” and earlier were you’re saying that we’ve kinda got to accept this is very much part of AR – as how it works, because hyperlocal experiences gets created by local communities – that up to know have tended to be homophilies.

Bruce Sterling: Well, I think that’s easily handled with some design thinking. You’ve got to do some user observation and show some sympathy with the user, and to be aware that you’re designing for the user and you’re not designing for yourself.

In a field as young as this, it’s mostly geeks building cool stuff for geeks. In a lot of ways, it’s a “can you top this” contest. That’s OK, but it’s not good design to be your own client all the time. It’s like writing novels to amuse yourself, or sitting on the porch singing the blues on your own guitar with only yourself to hear.

Tish Shute: What will it take for AR mature out of this “geeks building cool stuff for geeks” phase do you think?

Bruce Sterling: It’s necessary to master some of the tools first. I think of the way the web has developed over the years. When the World Wide Web first appeared, it was just for physicists, and was all line commands and quite unstable and difficult. Then you got usability studies, and things like Ajax and so forth. It’s a very painstaking thing.

We’re not best at building interfaces for the best computer scientists. Web 2.0 was built from things like watching people cry while they were trying to fill out insurance forms. “Well, why are you so upset?”

“Well, I got to the end of the webpage, and then it said I took too long, and it cut me off and now I have to start all over!” Jesse James Garrett, right? Benefactor of mankind.

If you’re experienced, you think: “Why don’t I build a little module here, and kind of move the form over here, then I’ll periodically update it with some asynchronous Java and XTML.” And people are like, “Gee, how odd.” But that really works for real people. It comes from studying what people want to do. Whereas, the current AR approach to a problem like the insurance form would be like, “I will give you the ability to record the entire insurance form, and it will flash before your eyes!” OK great, that’s a cool hack, but I don’t really need X-Ray Eyes to fill out my insurance form. What I need is a more user friendly interface.

Tish Shute: Well it seems like we are moving into the terrain of Joshua’s fifth word “ventilation,” – if I understand it rightly – it is at least partially the antidote to territorialization because it’s this idea that a place needs air so we come out of our hermetically sealed boxes of the way we relate to a place and what kind of augmentation would bring more oxygen to that space.

There was an interesting moment in the Auggies because when Maarten Lens-FitzGerald presented the guerrilla shopping Layar and basically Mark Billinghurst and Jessie Schell who spoke first didn’t seem too impressed. They didn’t want to walk to shopping – that was what web shopping did, it saved us from walking to shop… but I felt, to me you picked up on something which might have some bearing on “ventilation” in that this AR shopping Layar was kind of squatting Prada – a favela chic AR shopping thing?

Bruce Sterling: I wasn’t sure if I was interpreting what Maarten had in mind by that. But I think Maarten sees his structure accurately as an experience thing rather than a mapping thing. I think he’s proudest of things like the Berlin Wall app on Layar, as opposed to Layars that help you go get a hamburger. It’s like…so when Layar inserts parasitic augmented shopping over other people’s real shopping? That was rather a subversive thing.

I think the key there is that his client is called “Hostage T-shirts,” right? I mean it’s actually kind of a transgressive little hippy T-shirt store that Layar can dump anywhere in the world. Layered right over, say, Versace and Prada. I don’t know what becomes of that effort. And I’m not sure about the term “ventilation,” because that’s a term of art I haven’t heard much.

Tish Shute: Maybe it’s like in a cafe. Ventilation would mean we were able to communicate with all these different categories of people that we normally would be unable to connect to, even though we might be sitting only a few feet apart.

Bruce Sterling: So it means ventilation in the bottles of our homophilies.

That’s not a personal problem for me. I commonly live in foreign cities and, you know, and spend a helluva lot of time talking to strangers at conferences. So I don’t think I’d have that particular tight little social island problem.

Tish Shute: Of the three judges at the Auggies, you seemed most enthusiastic about the Layar entry.

Bruce Sterling: It may be they’re not as familiar with the business models of locative AR as I am, and as Maarten is. It was kind of a subtle in-joke he was making about Layar’s own business model there.

Tish Shute: How do you explain that?

Bruce Sterling: Well, you know, Layar’s in the business of selling software to make mapping and urban structures into ecommerce.

The ideal way to do that obviously would be to move the richest customers into the most expensive shops in the most rapid way possible. Or at least distribute them in the directions they want to go, a la Google. Whereas this app that Maarten was talking about puts big barnacles in the way that are selling punk t-shirts.

Tish Shute: Right! Right!

Bruce Sterling: The Dutch are a bit subtle in their humor. I rather imagine there’s a lot of discussion in Layar’s inner circle about exactly what they want developers to do with their platform. They’re going to have considerable political difficulty deciding who can have a Layar key and how you discipline people when they start doing weird stuff. “The Oakland Medical Marijuana layar.”

Tish Shute: Well, finding nudists is one of the top layars at the moment.

Bruce Sterling: You know, obviously so. And finding narcotics in Amsterdam, or a prostitution layer. I warned them nine months ago this was bound to happen. I’m sure they’re aware of it. I don’t think Layar wants Google’s style of cool, technocratic detachment.

Tish Shute: But that’s pretty difficult to do in current augmented reality because we don’t have all the mathematical voodoo for full on AR search yet, do we?

Bruce Sterling: Well, you can hire it out. Somebody’s going to do it, if they get interested enough. There’s Nokia-Yahoo. Nokia-Yahoo! just did a big corporate deal…involving Nokia’s mapping system and Yahoo’s localization. So the Nokia-Yahoo! mash-up is called Nooo! Or could be called Yahno. Yakia! Unfortunately ridiculous names.

Tish Shute: It’s interesting because you mentioned the spiders’ mating problem at Google. They’ve got all the pieces to make this kind of level of AR obviously right now. But they actually haven’t done it yet.

Bruce Sterling: There must be at least some discussion in Google, but the same goes for Microsoft. I’m frankly baffled by Microsoft, because it’s just full of insanely brilliant people. What the hell are they doing in there? Name one serious innovation that’s come out of their labs in five years. They make Integral Research look dynamic. It’s really kind of sad.

Tish Shute: It’s a very curious situation with AR though, because AR more than any new technology relies on these big hordes of data particularly for the mapping, right? And only the big four have the data – although we are beginning to see upstarts, Earth Mine, Simple Geo… Did you get a chance to meet Di-Ann Eisnor from Waze – real-time maps and traffic information based on the wisdom of the crowd.  Waze is a very interesting project that is a potential giant killer.

Bruce Sterling: No, I didn’t talk to them. I’ve seen people speculate that Earthmine and Apple are going to make an allegiance. I guess if you’re thinking that urban informatic mapping is a super big thing for AR, that must be true. But I’m not convinced that’s necessarily the case. People have pointed out that you can just use Google Maps, and you don’t have to walk around with a little visor. There are other aspects of AR besides the cell phone space. There’s Total Immersion’s big display screens. There’s the web-based fiduciary stuff. And there’s projection mapping. And then there’s experience design just for people who need their reality augmented for whatever personal or social reason. [dog barking]

Tish Shute: Right. Oh, I’m in the middle… My son’s come. What a good hair cut!

Bruce Sterling: Hi, there.

Tish’s Son: Hi.

Bruce Sterling: How’s it going, sir? Good to see you…

Tish’s Son: Good.

Tish: [laughs]

Bruce Sterling: Yeah. Nice looking shirt. I like the back of it.

Tish Shute: That’s from the American Shaolin Temple. [laughs]

Bruce Sterling: All right. Awesome. Kung Fu geek shirt.

Tish Shute: Yup he is a bit of Kung Fu Geek. He and his dad did an iPhone app on it for Yu-Gi-Oh, for Yu-Gi-Oh scoring.

Bruce Sterling: Awesome. Plenty of Pokémon-style combat in Yu-Gi-Oh.

Tish Shute: Yeah. Well, it’s interesting because you’ve talked about this aspect. That all of this, the Pokémon aspect of AR hasn’t kicked in yet. But it’s obviously a match made in heaven to some degree, isn’t it?

Bruce Sterling: One would think so, yeah. The whole little kid gaming thing. What does that have to do with Google or Bing? You don’t need a massive database for stuff like that.

Tish Shute: Yeah, you’re right. But good tracking, mapping and registration requires a lot of mapping…

Bruce Sterling: Well, our current tracking, mapping and registration requires that. Maybe there’s some other way to hack it that we don’t know about yet.

Tish Shute: That’s a very interesting point. We always have to stretch the way we think about mapping… perhaps its a real-time understanding of the location you’re in…perhaps the map is being negotiated through several social processes?

Bruce Sterling: There are maps, and then there are maps. There’s a kind of artillery map where you need to know the precise location of target spaces. And then there’s the kind of social map where I’m really looking for the IN-N-OUT Burger where my sister went last Tuesday. That’s a different system.

Tish Shute: And I think AR, at the moment, we’re getting the most out of the social maps certainly. And the other [machine perception technologies to detect the identity and physical configuration of objects relative to each other to accurately project information alongside/overlaid with a physical object] is still kind of the big dream, isn’t it?

Bruce Sterling: They say that men never ask for directions and women never read maps. Clearly, the genders have different ways of navigating the world. Who’s to say what manner of augmenting our experiences is hottest? I’m not convinced that today’s rather rigid geolocativity is really what our society wants from that particular service. Maybe what we want is something more folksy. Some useful nudge in the right direction as opposed to grids with 200 meters here and instructions to turn such-and-such.

Besides, there’s other hacks we haven’t considered. We’re very dependent on GPS, but just suppose all those satellites are blown out of the sky in a solar storm. Would we really want to give up mapping? Wouldn’t we just come up with some other nifty hack? Radio beacons, let’s just say. Atomic clock timers in towns. Or maybe just little QR codes on lampposts that give you the exact location of that lamppost, and just click the thing and have it calculate where you are.

Tish Shute: Yes the NextHope OpenAMD project had a clever way of triangulating location indoors.

Bruce Sterling: Well, GPS is there and people all want to use it. It’s got good API, so of course you want to. And the guys who are good at doing it are real geolocative freaks. But the mere fact that we can do it this way, and that you can make it pay, doesn’t mean that it’s the ultimate way to provide that service to people. It’s like saying that Egyptian hieroglyphics must be the greatest way to write, because we’ve got a lot of them and they’re hard to learn. What if somebody comes along with an alphabet? It’s going to be a little embarrassing.

Tish Shute: Yeah, that’s a very good point. Now, this is a more simple ordinary question about the event. YDreams went off the map in the Auggie voting, and walked away with The Auggies. No one doubted that that was the most…

Bruce Sterling: I don’t know. I thought those Occipital guys with the panoramic painting…. That was hairy. I would have been tempted to give them the prize myself, actually.

Tish Shute: And what did you like best about that? Because I agree. I love Occipital.

click to enlarge

Bruce Sterling: I thought it was a more technically difficult stunt than the hand registration thing. Using a hand as a 3-D cursor is hot, but not like painting a panorama in 3-D in real time. That was an impressive technical feat.

Tish Shute: And they hinted at the 2.1.1 AR, more AR version of that. What do you see coming out of that as possibilities?

Bruce Sterling: Well, I’d heard of YDreams, so I wasn’t stunned. But I’d never heard of those guys. I wonder what else the heck they’ve got in the attic.

Tish Shute: very cool stuff…

Bruce Sterling: Well, more power to them. But clearly YDreams was the popular favorite. And who couldn’t like it? It was just so AR.

Tish Shute: It was so AR and so gorgeous.

Bruce Sterling: It was pretty, actually.  Except for their ugly menu button and poor font choice.

Tish Shute: Oh, yes. You didn’t like that, did you? [laughs] But with the Occipital panorama, what do you see the next stage of that?

Bruce Sterling: Well, obviously quicker and faster. Quicker and faster and more accurate in a network. Let’s just say I’m in New York and you’re in New York and I’m calling you for help. And you say where are you? I just whirl around like this and I mail it to you on a Google Wave. And you whirl around like that, and then we compare the two panoramas and do an instant triangulation. And you say: I’m over here on this red dot of your screen.

Tish Shute: Yeah, exactly.

Bruce Sterling: We’re navigating with panoramas by having two connected panoramas and considering the difference.

Tish Shute: Yeah, very interesting…

Bruce Sterling: Not shabby, right?

Tish Shute: Not shabby at all.

Bruce Sterling: If you could do it in real time.

Tish Shute: Then the other thing I missed because I was going to meet Will was I missed the Launch Pad competition. Did you catch that?

Bruce Sterling: I didn’t see it either. I thought of another app though.

Tish Shute: Oh!

Bruce Sterling: You’ve got a panorama maker in your home office, and it just scans the office 24 hours 365 and tags anything that moves, right? OK, where’s the clipboard?  At 8:15 it was over here. Now it’s vanished. Now another object is viewed over here. So, logically, ping, you hit it with a sticky light and there it is, right?

Tish Shute: Oh, that’s cool also knowing what has changed in any environment would be a big enabler for a lot of AR visions.

Bruce Sterling: I’m sure there are many other things you could do with panoramas.

Tish Shute: My jet lag is beginning to kick in big time – so many ideas to pursue from are2010 – those panoramas are very exciting though.

Bruce Sterling: Oh, well, it’s all right. We can augment reality! I’ve got three heads and six hands!

I met Stowe Boyd, of Microsyntax.org at Jeff Pulver’s 140 Characters Conference which convened in the middle of a perfect storm for the State of NOW (more mundanely known as the real time web) as thousands of tiny Twitter pipes became a vital conduit for the historic events occurring in Iran (picture on left, Stowe Boyd, from Brian Solis‘ Flickr here, and on the right, Bruce Sterling, presenting at reboot11 from scriptingnews‘ Flickr here).

But, as Clay Shirky pointed out, re Twitter and Iran:

“It’s incredibly messy, and the definitive rules of the game have yet to be written. So yes, we’re seeing the medium invent itself in real time.”

Stowe Boyd is managing director of Microsyntax.org, a non-profit investigating the embedding of structured information within microstreaming applications, particularly Twitter. It is a communitarian project so if you are interested you should get involved – see Stowe’s #140conf. presentation, “The evolution of Microsyntax.” Stowe is an architect of “flow” and a webthropologist of the State of NOW.  I had the opportunity to talk with him at the conference (see the full conversation below). We talked not only about some of the practicalities of implementing microsyntax but about how “the web of flow” produces a fundamental shift in how we communicate, and who we are.  As Stowe Boyd put it:

“You use these tools, and you are changed. And it’s just a question of how long you use them and the longer you use them, the more you use them, the more changed you are. When people shift to a basis of sociality around connection with other people as opposed to mass affiliation, it’s different. It’s completely different. Your whole system of ethics, the way you judge the world and decide what’s important is different. And not only different it’s better. It’s a better way to deal with the world.”

As Wyclef Sean (@wyclef) remarked at #140conf, “Twitter just cuts the middle man in everything.”

At the 140 Characters Conference it was hard not to be captivated by the energy and optimism arising from the successful use of Twitter by Iranians to communicate in the aftermath of the election.  But the subsequent repression in Iran, in which the regime took advantage of central infrastructure controls to silence Iranian twittering (we have similar network technologies in place here in the US), leaves a big question that came to the fore after the conference:

While these real time applications give us the ability to leverage network effects in totally new ways, and they have enormous potential to make our lives better, do we need to give more thought to the infrastructure they rely on?

The videos for the 140Conf are up now. If you haven’t already seen them, after watching Jeff Pulver’s intro to The State of NOW a great place to start is the “Twitter as a News Gathering Tool” (Part 2).  Also see Ann Curry Defends Foreign Correspondents, Twitter; Rick Sanchez Defends CNN and Brian Solis’ post on techcrunch. Christopher R. Weingarten (@1000TimesYes), “Twitter and the End Of Music Criticism,” and Moeed Ahmad’s (@moeed), Gaza in Focus, are two of several must see presentations. The #140Conf was an extraordinary event.  Jeff Pulver orchestrated a brilliant cast of characters and a manifestation of social media “hybrid vigor” that was exhilarating to be part of.

A “Director’s Cut” of #140conf will be re-broadcast (Monday, June 29th and Tuesday, June 30th) at 11AM EST / 8AM PST – http://140conf.com/watchit. Some of the speakers will be tweeting while their session is being re-broadcast (see The Jeff Pulver Blog for more).

(picture above from Brian Solis’ Flickr here)

In a serendipitous convergence of events I found myself in the front row taking photos for Brian Solis (@briansolis) see Brian’s post, “Is Twitter the CNN of the New Media Generation.” I like my photo of Jack Dorsey (@jack) Twitter founder – the lens of my own camera would never have allowed for this one!

I was also sitting close to Stowe Boyd (@stoweboyd), who out of all of attendees at this jam packed event was one of the people I had most hoped to connect with.

Talking with Bruce Sterling about Squelettes, Twitter, Favela Chic, and Gothic High Tech

I have been following the microsyntax.org effort that Stowe has been leading since this post by Bruce Sterling (@bruces) on Pachube Feeds which contained this challenge:

“(((Extra credit for eager ubicomp hackers: combine this [pachube feeds] with Googlewave, then describe it in microsyntax. Hello, 2015!)))”

Stowe pointed out in our conversation at #140conf, that Microsyntax.org is in one sense a very narrow project but on the other hand it’s very broad, because every sort of information that you can imagine is going to be streaming through Twitter and related [real time] applications.

Or as Aaron Straup Cope put it to me: “This is ultimately the “magic word” problem, which is essentially the semweb vs. google-is-smarter-than-you problem.”

There are a bunch of crystal ball posts up at the moment looking into the future of the real time web…. for example, this post on threeminds.organic (via @timoreilly and @buckybit) asking whether we need page rank for people and not just sites…..and this post on readwriteweb that asks is the state of now the harbinger of doom to walled gardens like Facebook. And there seems to be an arms race starting around real time search.

But Bruce Sterling (@bruces) in his cover story for Interactions Magazine examines some of the blinkering on “two inherently forward looking schools of thought and action [design and science fiction].” He writes:

“We have entered an unimagined culture. In this world of search engines and cross-links, of keywords and networks, the solid smokestacks of yesterday’s disciplines have blown out.”

While I was writing up this post, I found myself up at the crack of doom (4 am EST) with insomnia I attribute to a tweet from Mark Vanderbeeken @vanderbeeken which I (@tishshute ) retweeted:

“Internet of Things – An action plan for Europe,” (This EU Doc. cites @agpublic ’s Everyware) http://bit.ly/16uiu3 via @vanderbeeken“

(I wish I had used the new microsyntax in Tweetdeck RE (for more on RE see Stowe Boyd’s post here) then I would have been able to find @vanderbeeken’s original tweet just now.)

So after a quick scan of the EU paper on the internet of things, and in a “here comes everybody” pre-dawn state of mind, craving oracular pronouncement, I impulsively shot an email to Bruce Sterling.

[Note: the following is an asynchronous exchange – not synchronous as a Google Wave would have made possible. Also I have pulled the conversation out of the original email format. Lars and Jens Rasmussen of  Google Wave seem to have hit the nail on the head when they “set out to answer the question: What would email look like if we set out to invent it today?” (see this excellent post by Tim O’Reilly on Google Wave)]

Tish Shute: I shouldn’t be up at 4am EST sending you more questions but I began reading The “Internet of Things – An action plan for Europe,” http://bit.ly/16uiu3 before I went to sleep and woke up thinking: “How can we work on an action plan for everybody?” ((Another highlight of 140Conf. was Kevin Slavin’s talk on “Things that Twitter –  “sensor aesthetics are streamy”)).

Bruce Sterling: *Everybody?  What, all 6,706,993,152 of us?

Tish Shute: How does, “it’s all about the data,” and “google’s smarter than you” thinking versus “bottom up”/”personal informatics”/”sem web” get worked out in the internet of things?

Bruce Sterling: *I’d be guessing via mergers, acquisitions, lawsuits and police crackdowns, but you never know.  You might have a massive financial collapse where innovations like this start coming out of slums and favelas.  I heard such a great term at LIFT last week:  ”Favela Chic.”  That’s when you are totally penniless and without commercial prospects of any kind but still wired to the gills and big on Facebook.

Photo of Bruce Sterling at Lift 2009 by Centralasian

Tish Shute: Could you elaborate on your comment:

“Also, this stuff they’re discussing: this is like all kindsa trouble ten years from now.” (from your post http://www.wired.com/beyond_the_beyond/2009/03/spime-watch-data-shadows/)

Bruce Sterling: *Okay: you know how much trouble SMS messages are in Iran right now, even though ten years ago, cellphones were only for foreigners and rich guys in Iran?  Kinda like that.

Tish Shute: You wrote here: “The idea of living in *abandoned prototypes* or giant failed larval  husks is very contemporary, very New Depression. Very “Favela Chic…”

And:

“Ocasionally squatters move into “squelettes” and bring in some breeze-block, corrugated tin and plastic hoses, transforming squelettes into high-rise favelas. This doesn’t work very well because it’s tough to manage the utilities, especially the water.”

Tish Shute: So what happens when we rely on Google & Twitter repurposed as our main means to access our government?  Not only repressive regimes can cut these utilities off, even though Twitter was asked to delay maintenance so that the Iranian Twitters could keep flowing, Michael Jackson brought Twitter down.

Bruce Sterling: *Google and Twitter aren’t going to last long enough to become main means of an access to government.  It’s not that Google and Twitter go away and we return to a previous status quo, however.  It’s that they are ramshackle digital expedients that get replaced by  even more ramshackle digital expedients.

In the meantime the stuff we used to call “government” gets similarly destabilized.  It’s been privatized, or offshored, or turned into a hollow shell.

Tish: Shute: So is Twitter a squelette (like all our favorite internet platforms, including Google Wave which we haven’t even had a chance to squat yet)? And is microsyntax our breeze-block, plastic hose and corrugated tin-  – very Favela chic but vulnerable to the vagaries of Michael Jackson’s life and death, and deadly shut downs and snooping by repressive regimes that control the underlying utilities? (Squelettes, as Bruce Sterling points out, are:  “one of those coinages like “Prada Goth” that spring out everywhere once they are pointed out.”)

Bruce Sterling: *We can draw a distinction here:  “Gothic High Tech” is the top-end version, while “Favela Chic” is the low-end.  “Gothic High Tech” would be the likes of a “repressive regime” which finds itself forced to conduct cruel, secret, spooky, Guantanamo cyberwars… it’s pretending to transparency, accountability and open elections, while below that surface is a weird, torchlit, Gothic hall of mirrors where invisible hands wreck banks, impoverish the civil population and kidnap people.

It’s “Gothic” because of its magnificent, elaborate appearance — very “Castle of Dracula” — but that no longer maps onto its panicky, extremist, transgressive behavior.

Gothic High Tech doesn’t live in “squelettes.”  Gothic High Tech lives in fancier, more respectable structures called “stuffed animals.”  A stuffed-animal used to be a functional building. From the outside it looks pretty much like it always did, maybe even “conservative.”  Inside it’s half-retrofitted with aging, Frankenstein machineries, already outmoded, rapidly decaying.

A “stuffed animal” might, for instance, be a “savings and loan” where the behavior of the present-day inhabitants involves no actual saving and no actual loaning.  Instead the inhabitants are on television negotiating a position in a crisis narrative and living on bailouts, while, every day, the cobwebs get a little thicker.  “Regulatory capture” is stuffed-animal activity.  “Failed states” and “hollow states” are stuffed animals.

“Favela Chic” is the same basic activity, but with much less money and institutional clout.  In “Favela Chic” nobody bothers to ask for bailouts.  They know the state has failed, or they themselves are engaged in weird activities they prefer to hide from the authorities.   “Favela Chic” lives within openly failed structures, or else in half-structures that are in “permanent beta” and falling down as rapidly as they can be erected.  Favela Chic is bottom-up, open-sourced, heavily networked, subversive and piratical.

There’s a certain amount of class-transition between Gothic High Tech and Favela Chic — like, Twitter was Favela Chic and is heading straight for Gothic High Tech.  But there’s much less transition than there used to be, because of income differentiation — the tiny faction of Gothic moguls “own” what’s left of most of the wealth, which they themselves are rapidly destroying.  The general trend is not toward increasing global prosperity.  The precarity is becoming general.  The Favela beckons for everybody.  That’s where most of the planet’s population lives already, and it’s certainly where most of the young people live.  The idea of a “developing world” needs to be reversed; the end game is in the “developing world” and the rich nations are heading there.

Tish Shute: It seems to me that Twitter and the real time web of flow is a revolution in our means of communication presenting awesome opportunities.  But, are we squatters in an infrastructure that is hard to manage?

Bruce Sterling: *Yes. I’d go farther and say that we are squatters in an infrastructure that methodically destroys previous systems of management.  Especially itself: the closer you are to a revolutionary real-time web flow, the faster you have to reboot.

Tish Shute: And what is the answer to the question at the end of your cover story for Interactions:

“The winds of the Net are full of straws. Who will make the bricks?”

Bruce Sterling: *I frankly have no idea.  The storm-gusts are rising in a hurry and we are in for a whole lot of straws.

*I would point out that, if we could make up out minds about what kind of bricks we wanted, we could make them at tremendous speed.  We’re not helpless: our productive capacity is frankly fantastic.  Clearly we’ve lost the thread and can no longer explain what we’ve done to ourselves or how we get out of our fix.  But we might surprise ourselves.  21st century Favela Chic is no mere favela, and Gothic High Tech isn’t just Gothic, it’s also very high tech.  We’re in a Depression and it’s gonna last, but this is no 1930s Depression.

Talking with Stowe Boyd

Photo from Stowe Boyd’s Flickr stream, “Little” Tower Of Babel, Pieter Bruegel the Younger. It is also a slide from his presentation, “The evolution of Microsyntax.”

[Note: Most of this conversation took part in a busy foyer at #140conf and various people joined in the conversation at different points.  I have cut out these other conversations and tried to maintain the thread of my own questions in the transcription.  But this may have resulted in a sense of choppiness and discontinuity in places.]

Tish Shute: You have been on the front-line of so much web innovation, but, perhaps, you could give me a little back story on how you came to take the lead with microsyntax.org.

Stowe Boyd: Well, I’ve been on twitter 990 days or something. But long before Twitter became a commonplace household word, I’ve been advocating what I’ve been calling flow application, based on the streaming metaphor – the notion that you’d have a stream of updates coming from people that you chose to follow, which is now being called the asymmetric follow model. Years and years ago I postulated that that model was going to come along and completely change all future significant social applications. Back in the late nineties, I introduced a term “Social tools” and said social tools were going to come along and change the way the web worked. So I have a history of being 4 or 5 years ahead of what actually happens.

Microsyntax is sort of an interesting outgrowth of that. In a way it’s a very narrow area, in the sense that it’s focusing on these information patterns, the way that people want to encode information in the twitter stream or in the realtime stream of other apps. So it’s very narrow in the sense that it doesn’t immediately include all sorts of other things like these sports figures talking about how to market their services or whatever. But on the other hand it’s very broad, because every sort of information that you can imagine is going to be streaming through twitter and related applications.

We saw examples today of plants demanding water or DJ’s posting their set lists as they’re playing them, devices or equipment talking about its status, video stream from surveilance cameras. Everything you can possibly imagine will find it’s way in that stream. It’s all going to be encoded in different ways and grappling with that is actually an interesting problem. But more importantly it’s better for us as a community of users if we try to approach it in some systematic fashion. That’s the purpose of Microsyntax.org – this nonprofit. The concept of microsyntax is immediately evident to people who use Twitter, and that is we have a whole bunch of conventions that have emerged, and we have some places where it would be nice if conventions did emerge, but we don’t have them yet. And the idea of creating a nonprofit to do it is a sensible thing to do. So I decided I’ll go along with the request that others have made, because other people asked me to do this. So that’s a little unusual for me.

The Web of Flow

Tish: What first attracted my attention to Microsyntax.org was Bruce Sterling’s post suggesting combining pachube feeds with Googlewave and then describing this in microsyntax. Why do you think Bruce Sterling posed this particular challenge?

Stowe: Well, because he sees that everything is moving into the web of flow. Everything is moving out of the web of pages. In the next ten years we’re going to cease to experience the web as we do now, which is as a bunch of pages and we move around from link to link. And that’s what browsers are about. They help us move from page to page on the web. But Twitter, and before it the minifeed and instant messaging and a handful of other really interesting applications, have suggested a completely different web where information flows from other people to you through streaming mechanisms.

And the really interesting stuff that comes to me now on a daily basis is streaming to me through Twitter, not through my RSS reader, not me wandering around figuring out what to google, news or something. And that’s an indicator of the fact that that’s the hottest, coolest way to do it now, and means that in the future it will be “the way” that it’s done. So there will still be a web of pages out there, but it’ll exist like an archive. And we won’t experience the web that way in general because, “why would I go to the web page and see the guy’s blog post on his page, when it’s been served up to me 16 other ways?” And most importantly I’ve found it initially in some client, because somebody recommended it to me, and I resolved it in a hover window in my Twitter client. I’d never go to the page. I comment on it here…

Tish: I like your framing,  “the web of flow…”

Stowe: Well it’s also that one of the characteristics is the tempo is different. I actually wrote a post about this, that I think it’s fundamentally important. It’s not really gotten much drift yet. I think it’s too hard for people to think this way. They just can’t get it.

The dimension that’s really most interesting is the transition from secret to private to public. The fact that Twitter is inherently public as a default is a breakthrough. I mean there’s nothing else like this. The first time that the idea, except for the blogosphere itself which is the concept it’s built on,  the inherent notion is that you’re publishing stuff and anyone can get access to it. But the tempo thing really matters, the fact that it’s near synchronous so your perception of what you feel like you’re doing is you feel like you’re in a stream of updates from friends. We know that. But the sensation is dramatically different than your close personal relationship with your inbox, which is email. Email is secret, closed, and the sense is the context is that it’s an inbox, like the one on your desk. And you are boxed in by that, and you’re not actually feeling like you’re dealing with people. You feel like you’re dealing with the inbox.

Tish: This was only present in boxes as you say – chat rooms, IM, IRC, MUDs, Virtual Worlds but they all had that realtime experience going on.

Stowe: Yes instant messaging, chat rooms, etc. they were private. You had to invite people. The update paradigm on instant messaging was backwards. It said I want to follow this guys updates, but you had to get his permissin to do it. That seemed like a sensible thing in the mid ’90s when people worried about privacy and so they made it private. And private is not good, actually.

Tish: IRC is exactly like twitter but it’s off in closed worlds…

Stowe: Yes you have to know about them. You can’t just stumble across them, you have to be invited or give the password. It’s another closed model. But instant messaging is the father of all this, or the mother, depending on which way you look at it. But that fundamental last thing, it’s based on a quote by Gabriel García Márquez which is, “All people have three lives. they have a public life, a private life and a secret life.” And we are philosophically moving from a time where things were primarily secret (pre internet) to a time where things were primarily private which is web 1.0 into this new web where things are going to be primarily public and open and immediate. So we are building the scaffolding real fast to allow that to happen. And it’ll take us away from the old web. The old web will go down there. Everything’s built on dirt right? Do you see very much dirt in cities? No. No. The dirt is all concealed. It’s down there. If you want to go find it you can dig underneath the floor, and there’s dirt under there. But most people don’t spend very much time down there we send professionals down there to put plumbing and pipes underneath and we experience the world like this.

Tish: I met Eric Horvitz (Microsoft Researcher) at Where 2.0.  He is interested in community sensing and ideas about how people can share data in a win win way (see here). Do we need to work out ways to make sure people’s relationship to their data is not just to have it harvested by others for profit or repression?

Stowe: I’m interested in this actually. I recently wrote a piece about the governance of Twitter and for the purpose of your question let’s just go along with the premise that Twitter’s going to continue to be benevolent, and everything will be open, and everything will be public and everyone can do whatever they want with it. Well there’s a tremendous amount of things that people will want to do, but most of the things that they will set about doing to begin with will turn out to be irrelevant.

People will want to measure sentiment and all this other stuff, for example. And they’ll do that and they’ll coerce a lot of big brands and so on to pay money for these services. But the thing that’s going on with the now web, my web of flow is that people are disconnecting from self identity based on mass affiliations. So ultimately the more you spend your time doing this, you don’t give a s**t about brands. Nike – I could care less.

So there is defection from the mass media. We heard it today. There’s people here who were like booing these media guys, who think they should be held up as gods because, “Oh I’m one of the first to use Twitter on TV.” Well F*** you, I don’t give a s***. I don’t watch television. Every hour that people spend on the internet is an hour they do not spend watching television. It’s a direct and one to one correlation. Sure people still want to get their fill of whatever, the NBA playoffs, but significantly less than ever before. Which is why they’re increasingly irrelevant.

So the idea that some magicians are going to come along, figure out how to mine this data to find out how I feel about my automobile? I do not have a close personal relationship with an automobile. I don’t. And increasingly people won’t affiliate that way. They won’t bond with their stuff like that. That’s why I say most of this information won’t be helpful. It’ll be interesting sociologically. Webthropologists will be able to make it interesting – and marketing people, who are trying to figure what’s going on, might be able to do the right thing. But if they’re trying to take it and make it do something for them… They’re going to try to take it and use it to change us? To control us? It’s like that line in The Labyrinth,  “you don’t have any power over me anymore.”

Tish: You are actually saying something much more radical than say community sensing or that we need to store our own data. You seem to be saying that in some ways it doesn’t matter whether you store your own data or your data’s in the cloud (although Iran seems to be showing how centralized network control can be a powerful tool of repression).

Stowe: Most of the things that they’re going to try to use it to do won’t work because we’re not the same anymore. It’s inevitable. You use these tools, and you are changed. And it’s just a question of how long you use them and the longer you use them, the more you use them, the more changed you are. When people shift to a basis of sociality around connection with other people as opposed to mass affiliation, it’s different. It’s completely different. Your whole system of ethics, the way you judge the world and decide what’s important, is different. And not only different it’s better. It’s a better way to deal with the world. And these guys are still hoping that the old rules hold, but they don’t. They just won’t.

Tish: This is  rather a broad question. But one of the things that Kevin Slavin brought up in his talk is about things that tweet – your plant is tweeting, your shoes are tweeting, your house is tweeting. Twitter is a natural medium for the internet of things and what Kevin Slavin calls the “streamy aesthetics of sensors.” But with all these things that are tweeting people have had a lot of problems with filtering that kind of flood of tweets.  For example, I may want to listen to a tweet from my plant telling me it needs water when I am actually at home and can do something about it. But I may not want to listen to my plant whining about being thirsty all the time. Can microsyntax help? Or is this a place for those appliances you mentioned earlier?

Stowe:  There’s a whole other category of stuff having to do with priorities – this isn’t really a microsyntax – of different times of day when you’re involved in different activities. You may be more or less interested in different collections of Twitter streams. And the notion of how you go about dealing with that is – it could semi-microsyntactical, but maybe it isn’t at all. Maybe it’s all just having to do with the way that clever client apps work. So maybe if you have a super duper Tweet Deck, and you say it’s evening time and I’m in my evening mode, so a whole bunch get blocked and a different group of people, for example, your Parcheesi evening friends get enabled, and at the weekend when you have time to do house care you listen to your house.

I don’t think this is a microsyntactical issue. I don’t think this is an issue of what’s embedded in the stream except as a notion of priorities. There’s a lot of people who would like to have a mechanism to indicate priority. But I can’t think of any effective way to do it that wouldn’t immediately be abused. Of course anything can be abused. This guy thinks that this is high priority, but maybe once again it’s one of these sort of mutual dimensions where they want to indicate it’s high priority but I say I only believe in priorities from certain people.

But still there might be a case to be made for allowing people to put some kind of indication of priority in a tweet, so that there is a hope that it could rise out of the clutter. I talked about some things that I’m interested in that are just purely operational. One of these things I want to get people to build, in Tweet Deck, but it could be in any kind of a client, I want to be able to say don’t let this tweet go away. So I’m getting them to build the pushpin. So I can put a pushpin in the thing and it’ll stay at the top, or stay at the bottom, wherever I put it. And then I can respond to it later, because if I don’t respond to it right now, in most places it goes bye, and then you’ve got to go search for it – a pain in the ass.

Then I say if I’m going to have pushpins I want to have a record of all the things that I’ve push pinned – a history of pushpins. But it’s all client based. It’s got nothing to do with what’s in the text.

Tish: And knowing how many of your followers had already got a particularly tweet from somewhere else which would be very useful has to be done as an appliance…

Stowe: Yes that’s sort of a downstream metrics kind of thing.

Microsyntax is not the answer to every kind of thing. Like, appropriately dealing with hash tags in a sensible fashion is not purely a function of how we use them. But some of it is the structure itself. That’s why I came up with the subtags model. So everybody at South by South West tagged everything southbysouthwest, so if you searched for it there were 150,000 hits a day. So it was useless. But if people had used the subtags model, or something else like that, you could have searched for the subtag. So you could have searched for south-by-southwest.parties or south-by-southwest.thirtytwo-bit which was a particular party.

And so if you have sensible tools that are doing a better job of aggregating information around more complicated ways of structuring hash information, then we can get past the fact that brute force search just isn’t going to work. It just won’t work. For example somebody going through the stuff from today all the stuff that says #140conf but they want to find just the stuff that had to do with media, they wont be able to do it. They’ll have to do it manually. So some of that is better syntax. But some of it is better tools. I mean somebody should go build a better hashtags.org.

Tish: And in terms of creating a web of flow not all of what we need can been done within the Twitter messages – it has to be done in the client and external applications…

Stowe: Yes, there’s this class of applications that listen very diligently to what you’re doing in Twitter. The primary mechanism of how you influence the app is doing stuff in Twitter. You can always go to the app and look at it and fool with it. But, if in fact, the preponderance of your interaction is, it’s listening or talking to you in Twitter – I call that an appliance, to distinguish it from these other apps. Any external application might provide you with the mechanism to dump information into Twitter, but you have to go to the app to do the primary kinds of interaction. In fact major functionality may not be available at all in Twitter or maybe no functionality, except for like Brightkite allows you to dump stuff into Twitter. But the idea is that primarily you do it there. Or there’s a very limited thing like you get with Brightkite, you can send a message saying, “I’m somewhere.”

Tish: Should location be put into tags?

Stowe: I don’t think that location should be put into tags. In other words, if I talk about Paris, then using hashtags is sensible. Or I’m talking about Sherlock Holmes and his relationship to London. It’s a conceptual thing – like talking about Heaven. It doesn’t actually have to exist on the planet somewhere. But it’s really different if you say I am in New York City right now or the more interesting case I think really is, “I am going to be in Boston colon next week” or June 15 dash 17. And I want that information to be available to everybody or a select group of my friends, or just to myself and have it find it’s way into my calendar. But that’s really different than saying “I’ve always enjoyed it when I visit HASH New York.”

Tish: I liked Kevin Slavin’s  phrase “the streamy aesthetics of sensors.” I guess streamy aesthetics is something you have given a lot of thought to?

Stowe: First of all I read a lot of poetry, so I believe in poetics in reading and writing. But I don’t think punctuation marks really degrade that dramatically. I mean it’s OK to have periods and exclamation marks and commas, and things can still be poetic. I think it’s important to try to dream up microsyntax that doesn’t take your eyes off the content, the stuff that people are really trying to say. So that’s why for example I hate L: as a location queue because anything that has letters in it, if you’re not supposed to say them, – if you’re not mentally supposed to say them, or if you’re not supposed to say them if you read it aloud, causes you to do a stutter step when you’re reading the tweet.

But if you use punctuation marks, special characters at various points or placement conventions, like where do things appear in order in a tweet, those things don’t have the same toe stub, that I think really ugly syntactic conventions would. So it’s possible to make these things pretty. For example I’m testing out trying on various conventions for what do you do with a re-tweet. If you want to re-tweet it, if you actually want to have people see it, and then you want to make your own comment. So the question is how do you separate the two? So, RT – guy’s name and then text. Well then how do you know where his text ends and my text begins. So certain things don’t work for me. I mean like a comma is not enough because there might be a comma in the text. And a period doesn’t work because there might be multiple sentences. So it has to be something else.

Tish: And aren’t there confusions that arise because there are already conventions of usage…

Stowe: Yes, I have problems with angle brackets, for example. Sometimes when the tweets wind up in not particularly smart rendering systems, it gets confused because it thinks they’re html. For example, somebody was using the open angle bracket, and even though it’s just text, and it’s not html, when I took that tweet and put it in a blog post, it thought it was the start of an html tag, and so it disappeared. You could use an html escape character but that’s the kind of thing that causes problems. The other problem is there are other ways that it’s been used a lot. People have used this as the thing to introduce the comment that they’re making after a re-tweet.

Tish: There must be very few characters not being used for other things?

Stowe:  Yes but for example, when we use geoslashes there’s a blank in front of it, or it’s the first character in the tweet – so in that particular example  it is similar because slash is used for other things. But, in all the places where it is used, generally there’s a character that precedes it – like “w/o” for without or a fraction or a long list list of  these options. [Geoslash is microsyntax for user location using slash (‘/’) — as in ‘just arrived /SFO’ or ‘heading to /New York: tomorrow/’ for more see Stowe’s post here.]

When I was rooting around for a character I looked for a long time.  And also I wanted to make sure that the slash was easily reachable on cell phones, which, for example, angle bracket isn’t. So if you’re on a phone and you want to say I’m here – I don’t know how far you have to go on your phone, but it isn’t in the first eight characters of Symbian. I looked carefully to make sure it wasn’t a common character that people use widely in everyday speech like commas and semicolons and exclamation marks, but was still easily used. There are still other alternatives. It’s not the only one. There are cases to be made for all of these things – pros and cons for all of them.

Anyway I was making the case of experimenting with different things for this re-tweet, “Here’s my comment.” And I was trying all sorts of stuff like double colon, I tried all kinds of things I wanted to see what it looked like. So starting this week I used the solid bar, the upright bar. It sets it off. It really feels like there’s a divide. There’s a cleavage point, and that’s that guy and this is this guy. So I’m going to write it up as one of the candidates. Some people use square brackets and many other things. There are many personal conventions but nothing has become a real convention, accepted as the norm.

[ Note: Our conversation ended here as the presentations had resumed at 140 Characters Conference ]