Social Augmented Experiences leveraging geoawareness and human and machine intelligence to create real time information brokerages, combined with an augmented reality view, can create a new opportunities to reimagine our relationships with each other and our environment.

This Summer, I have been on a blogging hiatus, which has meant I haven’t been sharing as frequently and, unfortunately, the second half of two conversations I had earlier this year, both of which have much influence my thinking on social augmented reality, have languished in private mode -  part 2 of my talk with Bruce Sterling (see Interview with Bruce Sterling, Part I: At the 9am of the Augmented Reality Industry, are2010, and part 2 of my conversation with Anselm Hook - Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, Part 1. Time to get caught up on some blogging!  The lightly edited transcript of Part 2 of my conversation with Bruce Sterling is posted in full below.

Bruce Sterling has been blogging all the key developments in augmented reality (amongst other topics of interest!) on his Wired Blog, and he brought my attention to Boskoi the Ushahidi based app for Android phones, augmented foraging pictured in use above – for more pics see fightthegooglejugend.

Augmented Reality and Real Time Information Brokerages

Picture above is the path the “nomads” took through the Westhaven cryptoforest with Pieter Bol,co-auteur of the book Biological Globalisation and Theun Karelse of Urban Edibles Amsterdam “who presented his ‘augmented foraging’ app Boskoi.“  For more see, The Cryptoforests of Utrecht and, Westraven Psychogeography, 6 June 2010. Note: Cryptoforests: 1) Urban forests hidden from view 2) Urban fallows that might or might not be considered as forests 3) Gardens gone wild)

My interest in the Ushahidi family of ideas was already fired up by a conversation with Anselm Hook early this year.  We discussed a number of Ushahidi related projects, Swift, Crisis Filter and Anselm’s project Angel, Augmented Reality, and my own keen interest in an open, real time, distributed platform for augmented reality – ARWave.

The Ushahidi platform and the related project Swift has pioneered the real time brokerage of information with people acting in curatorial roles or matchmaking roles coevolving with machine assisted matching to connect wants to haves.  Ushahidi uses multiple gateways including SMS, and Twitter.  But the Ushahidi family of ideas is extremely interesting when combined with augmented reality and suggests many new possibilities for social augmented experiences, as Anselm pointed out, for human to human communications, human to civilization communication, and human to environment communications (e.g., perhaps, how machine intelligence can help bridge the difference in time scale that Kate Hartman explores in her, Research for Glacier-Human Communication Techniques).

Ushahidi, which means “testimony” in Swahili, is a website that was initially developed to map reports of violence in Kenya after the post-election fallout at the beginning of 2008. It is now an open platform with a wide range of applications and growing developer community.  See What is the Ushahidi Platform? from Ushahidi on Vimeo.

Swift - a project that emerged from the Ushahidi dev community, is a human sensor/real-time brokerage for dealing with emergencies, enabling the filtering and verification of real-time data from channels such as Twitter, SMS, Email and RSS feeds.

Boskoi – augmented foraging is the first app,  I have seen, to begin linking Ushahidi with augmented reality – although I don’t think there is a full augmented view for Boskoi developed yet?

“The whole point of AR is to see things from a different point of view…”

Click to enlarge poster from upcoming ARWave demo at Software Freedom Day – for more see below

I am often asked what augmented reality brings to the table with respect to location based social networking, which is on the verge of going mainstream in smart phone apps like Four Square. While the first part to my answer is usually to explain what is unique to augmented reality.

As Bo Begole notes, the full vision of AR requires machine perception technologies to detect the identity and physical configuration of objects relative to each other to accurately project information alongside/overlaid with a physical object (see this post on the PARC Blog by Bo Begole on the difference between AR and ubiquitous computing – thank you Rouli for bringing my attention to this).

But it is only in recent months that we have begun to see the kind of tools that make this possible become freely available to developers – see my interview with Jay Wright of Qualcomm here.   Also see this post on Bundler: Structure from Motion for Unordered Image Collections an open source system that allows the creation of 3D point clouds from unordered image collections, e.g. internet image collections.  We now have many tools available to move mobile augmented reality beyond the recent crop of apps relying on GPS and compass alone for positioning into a new era of vision assisted AR apps that will increasingly bring the full vision of AR into our daily lives.

Further, the integration of visual search applications like Google Goggles and Kooaba which can detect the identity of particular objects will add another vital tool to machine perception technologies enabling AR “checkins” on potentially anything in the physical world around us, and more fuel to the Gamepocalypse (e.g. it would be easy to turn every trash can in the city into a basketball hoop as we discussed at the ARNY meetup last month).  And soon, the Pandora’s Box of  facial recognition (Google Goggles have the capability though it is not released to the public yet) will open up.

Jesse Schell described the importance of AR in a nutshell in his keynote for are2010:

“The whole point of AR is to see things from a different point of view…How can there be a more powerful art form than one that actually changes what you see?”

But how AR matures as a social experience will be the key to Jesse’s suggestion that:

“Augmented Reality will be one of the things that fundamentally define the 21st century”

There are many interesting forms of AR that are not reliant on a tight registration between media and physical objects – several are put forward by Bruce in the convo below.  And, it is likely we will see AR eyewear as an occasional useful accessory to a smart phone long before we have the sexy, affordable augmented reality eyewear worn that we wear throughout the day.   These speech to text glasses would be a very useful and viable accessory to a smart phone right now for the hearing impaired.

For the moment, as Bruce notes, some of the most interesting and useful augmented experiences to date have not been in the cell phone space:

“There are other aspects of AR besides the cell phone space. There’s Total Immersion’s big display screens. There’s the web-based fiduciary stuff. And there’s projection mapping. And then there’s experience design just for people who need their reality augmented for whatever personal or social reason.”

On of my favorite social AR experiences is this SMS Slingshot.

But I have been excited for a long while about the intersection of mobile social augmented reality, real time communications, and ubiquitous computing see Total Immersion and the “Transfigured City:” Shared Augmented Realities, the “Web Squared Era,” and Google Wave.  And I have described in many places why I think real time, open, distributed communications for AR are so important to developing social augmented experiences – see the slides for my talk at Augmented Reality Event here, here and here for starters.

ARWave at Software Freedom Day 2010, September 18th 2010

Thomas Wrobel and Bertine van Hovell will demo the first ARWave Android client at Software Freedom Day this weekend!

A number of people have asked me, (including Bruce), What will be the future of ARWave now that Google Wave is no longer a stand alone application?  Yes, the recently announced release of Wave in a Box (see here and here) is very exciting for the ARWave team.

The ARWave Android client is the first open AR client built on an open, real time, distributed platform -  based on a server that anyone can download and set up, currently the “FedOne” server but Wave in a Box, hopefully, will be even easier to deploy.  Wave in a Box seems perfect for ARWave’s needs – for more see the WiaB Google Group here.  And for more information on the ARWave client -  click to enlarge the poster below, see the ARWave concept video here, and for more, and how to get involved see arwave.org.   Props to Thomas Wrobel and Bertine van Hovell (posters below from demo for Software Freedom Day), Mark Evin, Davide Carnovale, and Markus Strickler, for all their hard and brilliant work on ARWave.  Also to JCPT the open Android 3D engine that has saved a lot of work!

click to enlarge slide

Social Augmented Experiences that Matter

My ideas on the future of social augmented experience have been deeply informed by the the conversations I had with Bruce Sterling and Anselm Hook this year.

Bruce Sterling notes in the conversation below, location based social apps like, Four Square, are interesting because they are not “urban geography like Google’s satellite stare from above,” but rather “groups of citizens are doing portraits of their own region.” Augmented Reality, with its of lauded power to make the invisible visible is, of course, is the ideal tool for “citizen portraits”  to the next level.  Cory Doctorow described to me three years ago (see here) an “inverse surveillance society,” enabled by an augmented view  – “where all the data from the positional and temporal characteristics of all the objects that we own were in aggregate visible and available so that we can mix and match them remix them understand them and have more agency in the world.”

It is very cool to go back to reread this conversation now that it is becoming possible to build the kinds of apps Cory described, and Bruce Sterling envisioned in Shaping Things (see Amazon.org  page 111).

click to enlarge

My  conversation with Bruce earlier this summer (see below) took place on the heels of are2010 – Augmented Reality Event.   See the video of Bruce’s keynote, “Bake a BigPie,” here,  and the final keynote, “Seeing,” by Jesse Schell (see video here) in which Jesse riffed on AR and the man with the X-ray eyes.  Both these awesome talks are still fresh in my mind.  Bruce noted how we should pay attention to augmentations for people and situations that could really use some augmentation… and not get too fixated on the coming of AR Goggles.  He elaborated on this in our conversation (again full transcript below):

“Well, it’s a matter of deciding whose reality it is that you’re trying to augment. I’m not trying to be a bleeding heart about it, but obviously there are people in our society right now with reality that could really use some augmentation. They are mostly disadvantaged people. They are vision impaired, or maybe they have autism. They might be senile and just can’t remember where they put their shoes. These are people who could really use some help, right?”

So, start with people who really need sensory or cognitive help. Before you turn our geeks into Superman, why don’t you try turning some people who are harmed into more functional individuals? Then you’ll be able to learn how to do that. Then maybe you can ramp it up to these Nietzschian heights of the superb Man With the X-ray Eyes. Whatever.”

What will make AR interesting and useful long before and long after we see the full vision of AR eyewear manifest is its social aspects.  Bruce points out:

“My argument would be that if you want people to be more sensitive toward certain, say, issues and problems, it’s better to find the people who are already sensitive to those issues and problems, and give them a bigger stake in your augmentation system.”

“Say that I am really worried about public health. Well, if I have a lot of nurses that are using my system, people who are aware of my issues, then I could be walking around and I’ll see a lot more tags saying, “This is where he got food poisoning!” “In this shooting gallery, many people have caught AIDS!” Or, you know, “Tuberculosis has been spotted over here in this building.”

At that point, I could simply share their knowledge and get some social intelligence. As opposed to trying to amp the basements of my little hacker-mind and drag stuff up that’s escaped my conscious attention.”

Finding new ways to broker information – bring together needs with haves and different participants, empowered and disempowered  is., as Anselm discussed with me, one way to change our view of human to human, human to environment and human to civilization communication (particularly in light of this  “sobering account of how open data is used against the poor in Bangalore” that as @timoreilly noted recently OpenData Empowering the Empowered).

The key idea in a crisis filter, Anselm noted,  was to break up the participants into different kinds, to connects wants with haves:

“There are people who are in  situation.  We call them citizens.  And then there are reporters, people who report situations back to Twitter.  And then there are curators, people that canvas Twitter looking for important Tweets.  And then there are first responders, people who take the curating collection of responses and then act on them.”

This kind of brokerage between people acting in a curatorial role or matchmaking role with each other can be extended into and coevolve with machine assisted matching as Anselm explains.

It is also a vital part of creating social augmented experiences that matter.

One of Anselm Hook’s projects, which is called Angel is the the most radical expression of connecting wants with haves in that the idea is that “you have a situation, you broadcast that situation, and help magically appears.  You don’t even sign up for a service.  You just get help …

As Anselm explains this is the same idea of a brokerage for dealing with emergencies, but applied to the long tail of crisis response.  As Anselm describes it:

“I am interested in personal crisis.  ‘I lost my cat.  Help.  I can’t find where my kid is.  I am out of gas.  I have a flat tire.  My house is on fire.  My aunt is trapped in the bedroom.’  The kind of personal crisis that is just as important, but is not enough to get a national movement to help you…

I will publish this conversation with Anselm in full in an upcoming post.

Zorop – an ARG for World Peace

If you want to be part of a really exciting experiment to reimagine our relationships with each other and can be in San Jose this weekend, I highly recommend exploring this “rabbit hole”.

Thank you Gene Becker, Lightning Laboratories and Ubistudio for sending me this invite:

“Ken Eklund (@writerguygames) is developing a wonderful game for the 01SJ Biennial called ZOROP, aimed at creating World Peace(!). Some of you might know Ken from his work on the amazing ARGs EVOKE and World Without Oil. Anyway Ken, along with his collaborator Annette Mees, are furiously working to get ZOROP ready to go for the Sept 17th premiere at 01SJ.

Are you intrigued? I thought so, and here are your next steps down the rabbit hole:

> Check out http://zorop.org to learn about the game

> Follow @ZoropPrime to watch it unfold: http://twitter.com/zoropprime

> ‘Like’ ZOROP on FB for a different view: http://www.facebook.com/pages/Zorop/141140772593618

> Become one with the game; consider volunteering as a Zoropathian: curious@zorop.org

> Head down to San Jose on the 17th, play the game, and ride the ZOROP Mexican Party Bus. Seriously.”

Interview with Bruce Sterling

Click on image above to see video clip from from brucesflickr

[Note the first part of this interview is here and I broke in anticipation of Part 2 just as I started experimenting with an idea Joshua Kauffman – an advisor and entrepreneur working on design in the public sphere gave me for an interview technique – the All Souls College one-word question interview.  Although apparently they recently scrapped it and I am not very good to sticking to a single word!]

Tish Shute: We were talking about these proximity-based social work networks like Foursquare and Gowalla and how they may influence the emergence of social augmented experiences.

So Joshua’s suggestion for the first word was “territorialization” e.g. how do these new mobile social experiences like Foursquare, and the observation that actually rather than breaking down territorialization – which would be a good thing, tend to support territorialization…

Bruce Sterling: Yeah, they’re re-intensifying it in a very odd, electronic fashion.

Tish Shute: Yes.

Bruce Sterling: It’s not true of projection mapping or the webcam fiduciary display stuff. But with the handheld stuff, and especially the urban informatic stuff, it really can’t help but take on a local flavor. Layar is like “Augmented Dutch Reality.”

And TonchiDot is “Augmented Japanese Reality.” It’s hard to imagine a Layar interface going gangbusters at Tokyo. Whereas the TonchiDot interface, which is so clearly influenced by Anime and cartoon graphics…. Maybe it could find some niche of hipsters in Amsterdam hash bars…

Stuff that’s socially generated by people on the ground, as with Foursquare and Gowalla, is bound to take on a regional influence. Right? It’s like the New York hipsters who were early adopters of Foursquare. They’re not mapping New York! They’re mapping Hipster New York.

It’s all about Williamsburg and places where 24-year-olds go to drink… They found a demographic niche there. These guys are building the service for them. They’re people who are willing to work for Foursquare for free, because they want to wear the little king hat.

Tish Shute: I got the far far away badge ‘cos I live on the Upper West Side!

Bruce Sterling: But that’s not urban geography, right? I mean, that’s not like Google’s satellite stare from above. That’s a group of citizens doing a portrait of their own region. You’re going to see interesting things happen because, of course, people who use Foursquare elsewhere are going to check into New York, and they’re going to look at the “New York Foursquare.” They’re going to be aliens who interact with Foursquare people in New York and annotate what they’re seeing.

Tish Shute: Oh! Yes. Good point.

Bruce Sterling: That Foursquare community has a certain émigré soul. It’s different from the normal émigré soul of simple tourists on New York. So you’re friend there is right about the territorialization.

Tish Shute: Yes, Joshua Kauffman is a smart guy! Yes I am interested to see what interesting kinds of deterritorializations proximity based social networks and the hyperlocal view of augmented reality might bring, not just the new territorializations.

Bruce Sterling: It’s not the intense kind of territorialization, like gangs putting down graffiti markers and beating people up. It’s an inherent regional character that comes with using peer production to build your database.

Tish Shute: We were discussing whether AR could break down the walls between people – people who share the same physical space but actually inhabit different territories even if they are sitting on the table next to you.

Bruce Sterling: You know, I just wrote an article for my Italian magazine column. I think I mentioned this to you – a report about ARE 2010. I titled it, “Chicks Dig Augmented Reality.”

Tish Shute: [laughs]

Bruce Sterling: There is a very heavy social element to AR, and a phone based element. So the question is: Why would a woman wear a fiducial marker? Like our Metaio speaker at ARE2010 who had a fiducial marker on her lapel pin.

Tish Shute: Right. Lisa!

Bruce Sterling: Why would a woman go out in public with her Facebook profile on her body?

Tish Shute: Well I can think of some reasons…

Bruce Sterling: So that men will approach her, of course.

Tish Shute: Yes the core of all successful social networks is always a form of dating app.

Bruce Sterling: You do it a social icebreaker. It’s like: I’m a woman, I’m sitting here alone, and you can sort of glide by and, you know, take a snap of me. Then you retreat and have a beer with your friends and you work up the courage, and then you come and say, “So! Susan! I understand you like bicycling! And, boy, me too!” Right?

Tish Shute: There are all kinds of social barriers between people in cities that AR might be helpful in breaking down. An extreme example is the dilemma you actually quite often face as a New Yorker as you walk around a city. There are people asleep on the pavement and you don’t know if they’re dead or alive.

Bruce Sterling: Right.

Tish Shute: And you sort of like have this awful ethical dilemma of like, “Am I walking by someone I should be shaking by the shoulder, right, to wake them up so they don’t die, right?”

Bruce Sterling: Yes.

Tish Shute: You said in your keynote that we should pay attention to augmentations for people and situations that could really use some augmentation..

Bruce Sterling: Right. There actually is such an app in Britain right now. I posted about it: two Augmented Reality schemes for rubbish and hobos.

Tish Shute: Right. Yes I saw that!

Bruce Sterling: “Any sufficiently advanced technology is indistinguishable from garbage and hobos.” You don’t need to personally find out whether this hobo is worth your help. What you need is a good way to report the hobo to a hobo check-up service. They come in, and they look on their own database or supply a database to you, or a facial recognition unit, whatever. The service says: “Oh, well. That’s Fred. He’s a paranoid schizophrenic. He always sleeps in that alley. Let him be.”

The same goes for the rubbish — although I don’t want to compare rubbish to hobos. In fact, people do go out with their AR kits and take pictures of abandoned garbage bags and broken glass. They upload them with geolocated tags for the local garbage guys. Guys who are sitting around doing pretty much nothing because they don’t know where the rubbish is.

And they will come out and get the rubbish! I mean, they just deputize guys to go out and follow these alerts. Right?

But nobody predicted — least of all me — that you were going to have a high-tech Augmented Reality system that consisted of removing rubbish and derelicts. Right? But rubbish and derelicts always go profoundly under-reported. It’s just hard to get people’s attention.

But it’s very easy to set up a system so that, if you get ten reports on the same piece of rubbish, that’s going to work its way to the top of the stack. That’s why I was trying to get AR people away from the romance of the hottest app for the shiniest machine. More toward a design stance that’s more user-centric.

Where are the actual problems about stuff that we perceive? Stuff we can’t do anything about? Or people whose mechanisms of perceptions are harmed. They could be doing good work, being more participative, if they didn’t, basically, walk around without their glasses on.

Tish Shute: Well this leads well into the second word, Joshua suggested was interesting spring board – sensitivity.

On the one hand we can do these things for people who maybe need the augmentation because they have difficulty with one or another sense, e.g., their eyes are not functioning, or their ears are not functioning. But on the other hand, we can’t cross the social bridge to communicate with people who are temporarily disempowered in relation to the rest of society e.g. hobos and people who sleep on the streets of New York City.  And even though Augmented Reality could potentially be helpful it can even be more disempowering to the already disempowered.

Bruce Sterling: Right.

Tish Shute: But re “sensitivity” – does augmentation increase or decrease our sensitivity? This is a problem that Will Wright brought up [see video of Will Wright’s keynote at are2010], e.g, the problem of parking HUDs getting in the way of your intuitive parallel parking skills. The Lexus that takes driving control from you when you look back, ‘cos it knows that you’re looking at the road, and it starts to brake. Right?

Bruce Sterling: Right.

Tish Shute: The fact that the problem with technology is that it makes us less sensitive, right, augmentations sometimes get in our way?

Bruce Sterling: I suppose that’s true. But I’ve heard that said about practically every medium. Especially television.

Everybody wants to blame machinery for their lack of morality. It’s hard to top something like the Kitty Genovese killing in New York. This sort of legendary New York horror story from the 1960s. A woman is stabbed to death in public, no one does anything.

Tish Shute: Right.

Bruce Sterling: I don’t think that our media is making us any less humane or more callous.

Tish Shute: All right. Oh no! I see what you’re saying. Perhaps I misrepresented what Will was suggesting by putting it that way. The question is perhaps more how do we get the sensitivity into the technology. Human bodies are fantastically sensitive and sensory.

Bruce Sterling: Right.

Tish Shute: And we have these like sensitivities. For instance, How could augmentations of reality be like a blush ? You definitely want an interaction that’s not just this data being pushed at you. But what is the data that counts, right? Will shows a slide often of an iceberg with the tip of the iceberg which is the conscious mind.

Bruce Sterling: Oh, I see. Yeah.

Tish Shute: And underneath it is all the preconscious stuff that really counts, right? Any thoughts on that?

Bruce Sterling: I did take interest in that. Will has obviously been spending a lot of time studying cognition.

Tish Shute: Yes.

Bruce Sterling: I’m not convinced that AR has got a lot to do with that. There is certainly a trend there. There are a lot of people who want to do body hacks and brain hacks. I can imagine AR being used for that purpose, but it seems like a niche application. What is the point of our accessing even more stuff that’s outside of our consciousness?

Tish Shute: One of the things he is talking about is game dynamics, is it? The role of the imagination in play. For example, he shows the high dynamic range photos that make the world magical. Something you want to engage with playfully. This he points out increases a sense of agency because you are encouraged to engage and to play with the world.

Bruce Sterling: Well, I’m a literary guy. Italo Calvino did a lot of writing about this. He talked about the classics of literature. Why do we read the classics? Calvino said we do not read, but reread the classics. And the reason we do that is that, at first, we read a classic book and we think, “Boy, this book is really good.” Then, five years later, we read it again and we think, “Boy, this is a really good book, and it’s got so much more in it than I thought it had when I was 18.” Then we read it again at 28, and it’s like, “OK, now I really seem to understand this book, and it means something to me now that I didn’t know when I was 18 and 25.”

What you are doing through that access is learning something about yourself. So Will is arguing is what I really need is like a better augmentation. So that I can go in there and sop up the book all at once. I can grab every cultural nuance in it, instead of the stuff that’s sliding past me because I’m 18 and kind of young and hasty. Maybe I could have certain words and phrases helpfully underlined, that are like, “OK, well, this part is problematic for you.” In some sense, that’s not allowing me to be 18.

I’m never going to have the experience of my own maturation against this text, because I’ve devoured it all in one gulp.

My argument would be that if you want people to be more sensitive toward certain, say, issues and problems, it’s better to find the people who are already sensitive to those issues and problems, and give them a bigger stake in your augmentation system.

Tish Shute: Yes the social augmented experiences are going to be the most valuable.

Bruce Sterling: Say that I am really worried about public health. Well, if I have a lot of nurses that are using my system, people who are aware of my issues, then I could be walking around and I’ll see a lot more tags saying, “This is where he got food poisoning!” “In this shooting gallery, many people have caught AIDS!” Or, you know, “Tuberculosis has been spotted over here in this building.”

At that point, I could simply share their knowledge and get some social intelligence. As opposed to trying to amp the basements of my little hacker-mind and drag stuff up that’s escaped my conscious attention.

Tish Shute: Interesting that seems to bring us to another kind of repetitive theme in AR, the people tend to pigeon hole it as “merely” a visual interface. But actually, it’s the intersection, isn’t it, of social intelligence and augmentation.

Bruce Sterling: Well, it depends entirely on how you design the system. If I’ve got a military augmented reality, I would expect that to be mostly about urban fighting. It’s going to be about kicking in a door and shooting terrorists. If I pull that helmet off my head and put that on the head of an emergency worker or a cop, I’m going to get a militarized cop or a militarized emergency worker.

Tish Shute: Well the histories of the two great mass media of the twentieth century – TV and the atomic bomb were intertwined, and I suppose the evolution of ubiquitous media, augmented reality and urban warfare is already intertwined too.  So how can we encourage augmented realities to move beyond military roots that is common to much technology and into more peaceful urban realities?

Bruce Sterling: Well, it’s a matter of deciding whose reality it is that you’re trying to augment. I’m not trying to be a bleeding heart about it, but obviously there are people in our society right now with reality that could really use some augmentation. They are mostly disadvantaged people. They are vision impaired, or maybe they have autism. They might be senile and just can’t remember where they put their shoes. These are people who could really use some help, right?

So, start with people who really need sensory or cognitive help. Before you turn our geeks into Superman, why don’t you try turning some people who are harmed into more functional individuals? Then you’ll be able to learn how to do that. Then maybe you can ramp it up to these Nietzschian heights of the superb Man With the X-ray Eyes. Whatever.

Tish Shute: Did you notice that a couple of apps actually like TagWhat have apps geared towards people with disabilities – I haven’t had a chance to check it out.

Bruce Sterling: I’m sorry, I wasn’t looking at their tags.

Tish Shute: I was discussing this with Joshua who mentioned Zachary Lieberman’s Eye Writer, which is for people with locked-in syndrome. Do you know that?

Bruce Sterling: Sure. And people appreciate that because the poor guy, he’s laid up with Lou Gehrig’s Disease. Now they’ve given him a way out. AR is like a spark of new hope that gives his life meaning. What’s wrong with that?

Tish Shute: Yeah. And Tim Byrne using Sixth Sense for Autism is interesting.

Bruce Sterling: Let’s consider it the other way. Let’s say this graffiti writer there, instead of him being sick and weak, let’s say he’s an athlete. So I want to make him into a super-human graffiti writer. I want him to run around graffiti-tagging the entire town before dawn. Is that a good idea? Do we need that? Super human, super taggers? What if he’s going to spray up stencils of Nietszche? I kinda wonder whether the game is worth the candle.

Tish Shute: Yes I suppose it is not a great social scenario to be always augmenting the lives of the elites! Hmm, the third single word interview question is “homophily,” and earlier were you’re saying that we’ve kinda got to accept this is very much part of AR – as how it works, because hyperlocal experiences gets created by local communities – that up to know have tended to be homophilies.

Bruce Sterling: Well, I think that’s easily handled with some design thinking. You’ve got to do some user observation and show some sympathy with the user, and to be aware that you’re designing for the user and you’re not designing for yourself.

In a field as young as this, it’s mostly geeks building cool stuff for geeks. In a lot of ways, it’s a “can you top this” contest. That’s OK, but it’s not good design to be your own client all the time. It’s like writing novels to amuse yourself, or sitting on the porch singing the blues on your own guitar with only yourself to hear.

Tish Shute: What will it take for AR mature out of this “geeks building cool stuff for geeks” phase do you think?

Bruce Sterling: It’s necessary to master some of the tools first. I think of the way the web has developed over the years. When the World Wide Web first appeared, it was just for physicists, and was all line commands and quite unstable and difficult. Then you got usability studies, and things like Ajax and so forth. It’s a very painstaking thing.

We’re not best at building interfaces for the best computer scientists. Web 2.0 was built from things like watching people cry while they were trying to fill out insurance forms. “Well, why are you so upset?”

“Well, I got to the end of the webpage, and then it said I took too long, and it cut me off and now I have to start all over!” Jesse James Garrett, right? Benefactor of mankind.

If you’re experienced, you think: “Why don’t I build a little module here, and kind of move the form over here, then I’ll periodically update it with some asynchronous Java and XTML.” And people are like, “Gee, how odd.” But that really works for real people. It comes from studying what people want to do. Whereas, the current AR approach to a problem like the insurance form would be like, “I will give you the ability to record the entire insurance form, and it will flash before your eyes!” OK great, that’s a cool hack, but I don’t really need X-Ray Eyes to fill out my insurance form. What I need is a more user friendly interface.

Tish Shute: Well it seems like we are moving into the terrain of Joshua’s fifth word “ventilation,” – if I understand it rightly – it is at least partially the antidote to territorialization because it’s this idea that a place needs air so we come out of our hermetically sealed boxes of the way we relate to a place and what kind of augmentation would bring more oxygen to that space.

There was an interesting moment in the Auggies because when Maarten Lens-FitzGerald presented the guerrilla shopping Layar and basically Mark Billinghurst and Jessie Schell who spoke first didn’t seem too impressed. They didn’t want to walk to shopping – that was what web shopping did, it saved us from walking to shop… but I felt, to me you picked up on something which might have some bearing on “ventilation” in that this AR shopping Layar was kind of squatting Prada – a favela chic AR shopping thing?

Bruce Sterling: I wasn’t sure if I was interpreting what Maarten had in mind by that. But I think Maarten sees his structure accurately as an experience thing rather than a mapping thing. I think he’s proudest of things like the Berlin Wall app on Layar, as opposed to Layars that help you go get a hamburger. It’s like…so when Layar inserts parasitic augmented shopping over other people’s real shopping? That was rather a subversive thing.

I think the key there is that his client is called “Hostage T-shirts,” right? I mean it’s actually kind of a transgressive little hippy T-shirt store that Layar can dump anywhere in the world. Layered right over, say, Versace and Prada. I don’t know what becomes of that effort. And I’m not sure about the term “ventilation,” because that’s a term of art I haven’t heard much.

Tish Shute: Maybe it’s like in a cafe. Ventilation would mean we were able to communicate with all these different categories of people that we normally would be unable to connect to, even though we might be sitting only a few feet apart.

Bruce Sterling: So it means ventilation in the bottles of our homophilies.

That’s not a personal problem for me. I commonly live in foreign cities and, you know, and spend a helluva lot of time talking to strangers at conferences. So I don’t think I’d have that particular tight little social island problem.

Tish Shute: Of the three judges at the Auggies, you seemed most enthusiastic about the Layar entry.

Bruce Sterling: It may be they’re not as familiar with the business models of locative AR as I am, and as Maarten is. It was kind of a subtle in-joke he was making about Layar’s own business model there.

Tish Shute: How do you explain that?

Bruce Sterling: Well, you know, Layar’s in the business of selling software to make mapping and urban structures into ecommerce.

The ideal way to do that obviously would be to move the richest customers into the most expensive shops in the most rapid way possible. Or at least distribute them in the directions they want to go, a la Google. Whereas this app that Maarten was talking about puts big barnacles in the way that are selling punk t-shirts.

Tish Shute: Right! Right!

Bruce Sterling: The Dutch are a bit subtle in their humor. I rather imagine there’s a lot of discussion in Layar’s inner circle about exactly what they want developers to do with their platform. They’re going to have considerable political difficulty deciding who can have a Layar key and how you discipline people when they start doing weird stuff. “The Oakland Medical Marijuana layar.”

Tish Shute: Well, finding nudists is one of the top layars at the moment.

Bruce Sterling: You know, obviously so. And finding narcotics in Amsterdam, or a prostitution layer. I warned them nine months ago this was bound to happen. I’m sure they’re aware of it. I don’t think Layar wants Google’s style of cool, technocratic detachment.

Tish Shute: But that’s pretty difficult to do in current augmented reality because we don’t have all the mathematical voodoo for full on AR search yet, do we?

Bruce Sterling: Well, you can hire it out. Somebody’s going to do it, if they get interested enough. There’s Nokia-Yahoo. Nokia-Yahoo! just did a big corporate deal…involving Nokia’s mapping system and Yahoo’s localization. So the Nokia-Yahoo! mash-up is called Nooo! Or could be called Yahno. Yakia! Unfortunately ridiculous names.

Tish Shute: It’s interesting because you mentioned the spiders’ mating problem at Google. They’ve got all the pieces to make this kind of level of AR obviously right now. But they actually haven’t done it yet.

Bruce Sterling: There must be at least some discussion in Google, but the same goes for Microsoft. I’m frankly baffled by Microsoft, because it’s just full of insanely brilliant people. What the hell are they doing in there? Name one serious innovation that’s come out of their labs in five years. They make Integral Research look dynamic. It’s really kind of sad.

Tish Shute: It’s a very curious situation with AR though, because AR more than any new technology relies on these big hordes of data particularly for the mapping, right? And only the big four have the data – although we are beginning to see upstarts, Earth Mine, Simple Geo… Did you get a chance to meet Di-Ann Eisnor from Waze – real-time maps and traffic information based on the wisdom of the crowd.  Waze is a very interesting project that is a potential giant killer.

Bruce Sterling: No, I didn’t talk to them. I’ve seen people speculate that Earthmine and Apple are going to make an allegiance. I guess if you’re thinking that urban informatic mapping is a super big thing for AR, that must be true. But I’m not convinced that’s necessarily the case. People have pointed out that you can just use Google Maps, and you don’t have to walk around with a little visor. There are other aspects of AR besides the cell phone space. There’s Total Immersion’s big display screens. There’s the web-based fiduciary stuff. And there’s projection mapping. And then there’s experience design just for people who need their reality augmented for whatever personal or social reason. [dog barking]

Tish Shute: Right. Oh, I’m in the middle… My son’s come. What a good hair cut!

Bruce Sterling: Hi, there.

Tish’s Son: Hi.

Bruce Sterling: How’s it going, sir? Good to see you…

Tish’s Son: Good.

Tish: [laughs]

Bruce Sterling: Yeah. Nice looking shirt. I like the back of it.

Tish Shute: That’s from the American Shaolin Temple. [laughs]

Bruce Sterling: All right. Awesome. Kung Fu geek shirt.

Tish Shute: Yup he is a bit of Kung Fu Geek. He and his dad did an iPhone app on it for Yu-Gi-Oh, for Yu-Gi-Oh scoring.

Bruce Sterling: Awesome. Plenty of Pokémon-style combat in Yu-Gi-Oh.

Tish Shute: Yeah. Well, it’s interesting because you’ve talked about this aspect. That all of this, the Pokémon aspect of AR hasn’t kicked in yet. But it’s obviously a match made in heaven to some degree, isn’t it?

Bruce Sterling: One would think so, yeah. The whole little kid gaming thing. What does that have to do with Google or Bing? You don’t need a massive database for stuff like that.

Tish Shute: Yeah, you’re right. But good tracking, mapping and registration requires a lot of mapping…

Bruce Sterling: Well, our current tracking, mapping and registration requires that. Maybe there’s some other way to hack it that we don’t know about yet.

Tish Shute: That’s a very interesting point. We always have to stretch the way we think about mapping… perhaps its a real-time understanding of the location you’re in…perhaps the map is being negotiated through several social processes?

Bruce Sterling: There are maps, and then there are maps. There’s a kind of artillery map where you need to know the precise location of target spaces. And then there’s the kind of social map where I’m really looking for the IN-N-OUT Burger where my sister went last Tuesday. That’s a different system.

Tish Shute: And I think AR, at the moment, we’re getting the most out of the social maps certainly. And the other [machine perception technologies to detect the identity and physical configuration of objects relative to each other to accurately project information alongside/overlaid with a physical object] is still kind of the big dream, isn’t it?

Bruce Sterling: They say that men never ask for directions and women never read maps. Clearly, the genders have different ways of navigating the world. Who’s to say what manner of augmenting our experiences is hottest? I’m not convinced that today’s rather rigid geolocativity is really what our society wants from that particular service. Maybe what we want is something more folksy. Some useful nudge in the right direction as opposed to grids with 200 meters here and instructions to turn such-and-such.

Besides, there’s other hacks we haven’t considered. We’re very dependent on GPS, but just suppose all those satellites are blown out of the sky in a solar storm. Would we really want to give up mapping? Wouldn’t we just come up with some other nifty hack? Radio beacons, let’s just say. Atomic clock timers in towns. Or maybe just little QR codes on lampposts that give you the exact location of that lamppost, and just click the thing and have it calculate where you are.

Tish Shute: Yes the NextHope OpenAMD project had a clever way of triangulating location indoors.

Bruce Sterling: Well, GPS is there and people all want to use it. It’s got good API, so of course you want to. And the guys who are good at doing it are real geolocative freaks. But the mere fact that we can do it this way, and that you can make it pay, doesn’t mean that it’s the ultimate way to provide that service to people. It’s like saying that Egyptian hieroglyphics must be the greatest way to write, because we’ve got a lot of them and they’re hard to learn. What if somebody comes along with an alphabet? It’s going to be a little embarrassing.

Tish Shute: Yeah, that’s a very good point. Now, this is a more simple ordinary question about the event. YDreams went off the map in the Auggie voting, and walked away with The Auggies. No one doubted that that was the most…

Bruce Sterling: I don’t know. I thought those Occipital guys with the panoramic painting…. That was hairy. I would have been tempted to give them the prize myself, actually.

Tish Shute: And what did you like best about that? Because I agree. I love Occipital.

click to enlarge

Bruce Sterling: I thought it was a more technically difficult stunt than the hand registration thing. Using a hand as a 3-D cursor is hot, but not like painting a panorama in 3-D in real time. That was an impressive technical feat.

Tish Shute: And they hinted at the 2.1.1 AR, more AR version of that. What do you see coming out of that as possibilities?

Bruce Sterling: Well, I’d heard of YDreams, so I wasn’t stunned. But I’d never heard of those guys. I wonder what else the heck they’ve got in the attic.

Tish Shute: very cool stuff…

Bruce Sterling: Well, more power to them. But clearly YDreams was the popular favorite. And who couldn’t like it? It was just so AR.

Tish Shute: It was so AR and so gorgeous.

Bruce Sterling: It was pretty, actually.  Except for their ugly menu button and poor font choice.

Tish Shute: Oh, yes. You didn’t like that, did you? [laughs] But with the Occipital panorama, what do you see the next stage of that?

Bruce Sterling: Well, obviously quicker and faster. Quicker and faster and more accurate in a network. Let’s just say I’m in New York and you’re in New York and I’m calling you for help. And you say where are you? I just whirl around like this and I mail it to you on a Google Wave. And you whirl around like that, and then we compare the two panoramas and do an instant triangulation. And you say: I’m over here on this red dot of your screen.

Tish Shute: Yeah, exactly.

Bruce Sterling: We’re navigating with panoramas by having two connected panoramas and considering the difference.

Tish Shute: Yeah, very interesting…

Bruce Sterling: Not shabby, right?

Tish Shute: Not shabby at all.

Bruce Sterling: If you could do it in real time.

Tish Shute: Then the other thing I missed because I was going to meet Will was I missed the Launch Pad competition. Did you catch that?

Bruce Sterling: I didn’t see it either. I thought of another app though.

Tish Shute: Oh!

Bruce Sterling: You’ve got a panorama maker in your home office, and it just scans the office 24 hours 365 and tags anything that moves, right? OK, where’s the clipboard?  At 8:15 it was over here. Now it’s vanished. Now another object is viewed over here. So, logically, ping, you hit it with a sticky light and there it is, right?

Tish Shute: Oh, that’s cool also knowing what has changed in any environment would be a big enabler for a lot of AR visions.

Bruce Sterling: I’m sure there are many other things you could do with panoramas.

Tish Shute: My jet lag is beginning to kick in big time – so many ideas to pursue from are2010 – those panoramas are very exciting though.

Bruce Sterling: Oh, well, it’s all right. We can augment reality! I’ve got three heads and six hands!

Today, I was very excited when Thomas Wrobel sent me a draft of, “Everything Everywhere: A proposal for an Augmented Reality Network system based on existing protocols and infrastructure.”

Thomas has kindly agreed to let me publish his draft, to open a discussion on this topic. The diagram opening this post (click image to enlarge) shows, “An example of how collaborative 3D-spaces could be shared over existing IRC networks.” It is from Thomas’ proposal. The full text of his paper is included later in this post.

“Can we try to avoid a browser war this time?”

Thomas notes in the closing remark to his paper:

“I am absolutely confident in my belief AR will become at least as important as the web has, and probably a lot more so. It will also face much the same hurdles and challenges getting established as that medium did. But, speaking as a web-developer, can we try to avoid a browser war this time?”

Thomas Wrobel has consistently posted insightful comments on how existing standards could be used for creating open augmented reality networks. But he expressed concern to me that his work and this paper not be overplayed:

“I’m hardly a leader, I’m just an amateur with a load of ideas on AR-related topics, some which might be useful, others might become unworkable. I don’t want anyone to get the impression this is how I think it has to, or should be done.”

I have brought/am bringing up this topic of using existing standards and infrastructure where possible for open augmented reality networks in all my interviews with members of the AR Consortium.

And I am finding agreement on a point that Robert Rice makes, “there is no perfect, ultimate solution *now*, but we have to do *something* to work from and refine/evolve.”

Thomas Wrobel makes what I consider some crucial opening suggestions. I take my hat off to him for thinking about this early, coming up with some clear, elegant, and practical ideas, and doing the work to articulate these ideas so others can participate in evolving them.  Massive props for that, many times over.

Good ideas on standards at an early stage of  a developing industry like augmented reality are like spring sunshine and April showers for new crops. No one knows what storms and pests the growing season will bring – but water and sunshine (open standards) are always a good start. And, personally, I can’t wait to see how this new industry unfolds (see Bruce Sterling’s Layar Conference awesome keynote : “At the Dawn of the Augmented Reality Industry.”)

Thomas Wrobel is:

“a web developer working for a small, brand-new company called Lost Again, which mostly works on ARGs (That is, the alternate reality games, not the augmented reality games, although there’s probably going to be big overlap there in the future). We developed two educational ARG games for the Netherlands with a company called res-nova.”

I have been following Alternate Reality Games  through the amazing work of Elan Lee and Fourth Wall Studios. Like Thomas, I think the intersection of ARGs and augmented realities is going to be very interesting. Thomas wanted me to point out that the website for his company with Bertine van Hövell, http://www.lostagain.nl/, is just a placeholder for now.

“Probably be up fully within a week or two. And, “despite the logo, we aren’t an AR company [yet], or a travel firm. The logos supposed to represent being lost in our minds.”

Thomas has been thinking about the topic of an open augmented reality network for a while now.  He is an artist also known as DarkFlame and his ARN network is included in this augmented reality concept for 2086 he did in 2006 (click on image below to enlarge).

Beyond IRC

Both Thomas and Mez Breeze made extensive and insightful comments on my last post, “Augmented Reality – Bigger Than the Web: Second Interview with Robert Rice.” And in particular they both picked up on something I am very interested in – the potential use of the Google Wave Web of protocols in creating open augmented reality networks.

Mez in her brilliant brainsplosion on social tesseracting takes on the very definition of information:

“Tish, when you ask Robert “…what is your approach to delivering a massively shared real time [augmented reality] experience that is like Wave not confined to a walled garden?” that’s an extremely relevant question + one that needs to be addressed while considering the entirety of the Reality-Virtual Continuum. I’ve recently finished a series of articles addressing this: the framework I’ve developed is termed “Social Tesseracting.”

I have recently begun exploring the Google Wave Web of Protocols which are nicely outlined in this post by J Aaron Farr which includes the very interesting diagram below (so more on Google Wave in another post).

But, as Thomas notes, while he demonstrates his ideas using IRC (Internet Relay Chat) they reach Beyond IRC:

“As mentioned before IRC has some drawbacks, which are due to its age or method of working. As such, future systems might yet prove better alternatives for a open AR network. One example of such a system is Google Wave. It shares many of the advantages of IRC (open, anyone can create a channel of data, different permission levels can be set and its free), while avoiding some critical restrictions. (The data can be persistent). I believe some of the ideas I’ve mentioned, and possibly even the proposed protocol string could be adapted for Google Wave or other future systems. I believe overall the principles are more important then any specific implementation to get to them”

Also Thomas pointed that while he uses markers to illustrate some of his examples, they are just a method for tracking.  What he is presenting is going to be transparent to the methodology of registration/tracking.

Tish Shute: You mostly use marker based examples but there is no reason why the principles you are suggesting will not be just as relevant as we move more into using more sophisticated image recognition tools is there?

Thomas Wrobel: No reason whatsoever. I mostly choose familiar markers as something that could be used now, with a lot of coding library’s already established for them. I think for most future AR use, markers will go completely…especially outside. Either things will be done purely by gps, object recognition, or the (in the case of advertising) markers will look like normal posters.

However, I do think traditional markers might “cling on” as being used for non geographical specific stuff at home. After all, if you need some reference points for moving mesh’s about in real time…(say, when playing a board game with a friend on the other side of the world)….then there’s probably nothing that’s going to be more practical then some simple bits of paper or card.

Everything Everywhere

-  A proposal for an Augmented Reality Network system based on existing protocols and infrastructure.

by Thomas Wrobel

The following paper is my vision of a open AR Network and potential methods to implement it with existing technologies. Specifically I’ll be focusing on a potential for a global outdoor AR network, although the ideas aren’t limited to that.

Of course I call it “my” vision, but I’m obviously not the first to have many of these ideas. I have been influenced and inspired by many things…

[Some of Thomas Wrobel’s influences – watched and played. Images  from Mitsuo Iso’s Denno Coil (Click to enlarge) top,  below from the game “Metroid Prime,” and Terminator, and the last from Buffy the Vampire Slayer!]

The AR Network.

When I speak of a future AR Network, I mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.
Where people can just point their phone, webcam, or pair of AR glasses anywhere were a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.

From this point forward, I will refer to this future AR Network simple as the “Arn”.

The Arn should be an inclusive, and open platform where any number of devices can connect to, and anyone can make and host their own location-specific models or data.
It should allow people to communicate both publicly and privately, and not have their vision constantly cluttered with things they don’t want to see.

There’s two old, existing paradigms that I think can help reach this goal when they are combined.

The Internet Relay Photoshop.

IRC, or Internet Relay Chat was a chat system designed by Jarkko Oikarinen in the late 80′s.

Its a system where people meet on “channels”, they can talk in groups, or privately. Channels can be read-only, or open to all to contribute to. There is no restriction to the number of people that can participate in a given discussion, or the number of channels that can be formed. All servers are interconnected and pass messages from user to user over the network.

To me, this relatively old internet technology is a great template, or even foundation, for how the Arn could operate. Rather then text being exchanged, it would be mesh data (or links to mesh data), but other then that much of the same principles could apply.

People could join channels of information to view or contribute. Families could leave messages to each other scribbled in mid-air on private channels. Strangers can watch AR games being played between people in parks. People going into a restaurant could see the comments from recent guests hovering by the menu items.
None of this would have to be called up specially, if they are on the right channel when it was broadcast, they will see it.

The IRC paradigm becomes particularly powerful when combined with another one common to many computer users; that of a “Layer” in an art program, such as Photoshop or Paint Shop Pro.
As most of us know, layers allow us to separate out different components of a piece of art while editing, either to focus our attention on one piece, or to make future editing easier.

Now what if we simply have each “channel” of information represented as a layer?

Click to enlarge image above.

Having channels corresponding to layers is an easy and intuitive way for the Arn to operate. The user can login and contribute data to any channel, like IRC as well as adjusting the desired opacity and visual range of each layer, like they would a layer in Photoshop.

In this way they can get a custom view of the world, both with shared and personal AR elements visible at the same time.
They would not have to switch between various overlays to their world view, as they could see many at the same time.

Persistence of Data

With IRC or IRC-like system to communicate the data sent is mostly temporary data…broadcast on the fly from user to user and device to device. Retained in the users local logs, but not “hosted” anywhere.

I think for the majority of day to day purpose’s this is not so much a drawback, but actually desired for AR. Most casual communication doesn’t need to be recorded permanently in 3D space and, indeed, if it was, the cost of running such a service would increase exponentially with users and with time. Not to mention, our visual view of the world would get very cluttered very quickly. Imagine what your monitor would be like if it kept a history of every window you have ever opened and their positions!

So for most cases AR space should be treated like a 3D monitor letting us display many pieces of data from remote and local sources, and even to share them with others, but not being, by default, a permanent record for it all.

Most data will be analogous to pixels on a display, and if kept in records its only on the clients devices, not on the network itself.

However, occasionally we do want 3d data analogous to a web-page, such as (in the example above), the map layer. Data here should be persistent and visible to all that have that layered turned on.  I see no reason why hosting this data needs to use anything else but standard web-hosting with the (read only) #channel on the Arn merely providing a route to the data.

As the user logs onto the channel, the server, using a chat-bot, can send them a list of meshes with location data attached, and the Arn browser can simply pick the data to display that’s local to them. (Note 1: By doing it this way around, it allows some degree of anonymity to be possible, rather then the server knowing exactly where you are and feeding the specific correct string to you.)

We simply need to establish standards so this data can be pulled up and interpreted.

For instance, this standard could be as simple as a XML string pointing to a KML file on a server. This could then be then displayed in the users field of view at the co-ordinates specified.

In this way permanent data tied to locations, such as historical overlays or maps, could co-exist on the same protocol as temporary data such as mid-air chat’s or gaming related meshes.

There is also no reason why this shared-space/personal spaces based on channels of data has to be restricted to things given absolute co-ordinates.

(Different ways to access the same mesh)

It could work just as well with Markers and thus relative co-ordinates.

This would be mostly useful for indoor use, letting people logged onto a channel see the same meshes as everyone else on the markers. Thus allowing multi-player AR games, or AR games with observers very easily.

For example; games like Chess could be played between people with no additional code needed; You simply have a set of markers for only your own pieces, and as you move them the channel updates with the new positions, which are displayed in place in your opponents field of view.

This sort of game comes “free” with just having a generic system of shared space supporting markers.

It would also allow AR adverts down the street or in magazines to be viewed by simply logging onto the right AR channel

If markers are designed with URL data in them, this could even be a prompted or automatic process.
“There is visual data in this area on the following channel; #ABCD would you like to view this channel?”

Pros and Cons of using IRC or IRC-like systems

Pros;

• Anyone can write a IRC interface software.
• Anyone can create new IRC channels without cost
• Channels can have read and write permissions set.
• Users can easily have multiple channels open at once.
• Already established with thousands of severs worldwide.

Cons;

• 500-or-so character limit. 3D data must be linked too, not sent.
• Slow update rate. Lines of data can take a whole second or more to send.
• Non-persistent. Good for a 3d-view, not good for storage.

An example of how collaborative 3D-spaces could be shared over existing IRC networks;

Click on the image to enlarge

While in the long run I would hope for a dedicated AR network to be developed, with greater flexibility with persistence of data, there is a lot that can be done with the existing IRC system to implement the ideas mentioned above.

Below I will show an example of simple, crude, pseudo-protocol that could be fairly easily implemented to create shared AR spaces broadcast across IRC channels.

Its important to note, the goal here isn’t to exchange the mesh data itself on IRC, its to exchange links to the data.

Exchanging the mesh data directly within the 500 character IRC limit would be very hard, and liable to errors.

It’s also a waste of network bandwidth, as many people logged onto the channel might not have that object in their field of view, so their clients should not bother downloading it. (it should be up to the client browsers when to anticipate and cache mesh data).

Proposed Basic XML link exchange for AR;

Principle;
As user creates or changes an object, the clients software  posts a simple xml formatted string to
the IRC channel.
Anyone logged into that channel then sees that mesh displayed in the specified location.

This string could be formatted as follows;

<Mesh
ID=”DARKFLAME:1”
Obj=”http://www.darkflame.co.uk/mesh/church/chuch.kml”
Loc=”(49.5000123,-123.5000123)”
Permissions=”None”
LastUpdate=”12/12/0000,2012:12”
/>

This string allows other users client logged into the channel to automatically load the object from the URL and display it at the correct position in their field of view.
If the permissions are set to allow it, they could then move the object themselves, with the update being feeding back seamlessly to other users on the channel.

The objects posted are given an ID, which can be just the posters name, followed by a unique object number for that name. These unique ID’s would allow clients to track different instances of the same mesh, as well as making it easy to implement permissions. (if only the poster should be allowed to move this object, then the clients simply check if ID matches the user name posting the update. If its not, they can ignore it).

Next the objects need to be linked to a mesh.

The location of the objects mesh doesn’t have to be a fixed remotely-hosted url, it could be an IP address and port number of the user posting the mesh,hosted by the application posting the link to the channel.

Obj=”www.darkflame.co.uk/mesh/church/chuch.kml”
Obj=”123,223,14,23::3030”

The objects co-ordinates, likewise, need not be specified as absolute gps co-ordinates, but instead could refer to generic Marker.

Loc=”(49.5000123,-123.5000123)”
Loc=”Marker1”
Or relative to a marker;
Loc=”Marker4 (+0.0023,-0.0023)”
Or relative to a default plane;
Loc=”Default(+0.213,-0.123)”

The AR Browsers could then handle the association between the Markers pattern and its Name.
This way the markers are reusable, they do need unique markers to be printed for every new bit of AR they want to look at.
Users could just keep a set of generic markers handy, which they could simply assign to be Marker1,Marker2 etc for any AR use. (Note 2: As mentioned above specific makers could also contain a default ID name and channel built into their data, letting the Arn browser simply prompt the user if they want to see the model even if they aren’t in the right channel. This set up would be most useful for paper and even billboard advertising.)

The Default location could be a settable region, or marker, on the clients browser that defines a playable/user-able area in the field of view. Mostly useful for home use, this could typical be a square region on a users desk.

So, in the chess-game example, the client of the person making the moves simply updates the position relative to the Default every time they move their marker (which is tied to a chess piece mesh).
Then the (non-owners) clients software could automatically display it relative to their Default plane. This would make games like Chess, Checkers, Go or any other game involving merely moving objects about automatically very intuitive and easy to set up.

So by having meshes settable to absolute gps, marker-relative, or default-relative locations, reduces the bother necessary to experience AR content quite considerably, and makes “non-geo-specific” AR applications and games trivial to implement.

Next is permissions.

Mesh-permissions would be a simple string saying who else can update the data, if anyone.

eg;
Permissions=”None”
Permissions=”RandomPerson1, RandomPerson2”
Permissions=”All”

By default you could only update or move your own meshes. (identified by the ID of first posting). If you attempt to update anyone else’s,  their clients would just ignore it.

Thus in a game of chess, you can only move your own pieces. If you attempted to move your opponents (by reassigning  your own marker to their pieces Ids), the clients would just ignore that assignment. You’d only be fooling your own system.
Likewise, when pinning a message in mid-air for your friends to read, no one else can change that message without your permission, although copying it would be easy.  (Note 3: It’s important to note this sort of object-specific permission system is in addition to the global-permissions, or “user-modes” it’s possible to set for the IRC channels and users as a whole.)

Finally, as object data could change within all sorts of time-scales, the easiest way to keep everyone logged in up to date is to just have a time-stamp of when each model was last updated.

LastUpdate=”12/12/0000,2012:12”

This would not necessarily be the same as the XML string post date, because the models mesh might not be updated, but merely moved, and in such case the Arn browser shouldn’t redownload the mesh.

This sort of arrangement could be used as a standard today, and users wouldn’t have to constantly download special AR programs to view a single AR mesh.

In the long-term I would hope for more advanced methods to manipulate Arn-content online, analogous to Dom manipulation in web-pages. But for now, we should at least establish standard methods for devices to pull up meshes and overlay them in the correct position.

So, having a layered system could give the user a seamless blend of dynamic and static data with which to paint their world with.
I believe this is all relatively easy to achieve using modifications of existing web technology, combined with some basic graphics systems.

Local Data:

However, so far I have only talked about remote data.
What of programs originating on the device itself? This is, after all, how most AR software we have at the moment works.

I think, that just like the remote channels, local software should also be blended into the same list of layers.  People shouldn’t have to “Alt+Tab” out of one view of the world, to see another.
They should be able to see both at once, if they wish.

For instance, if your playing a AR game, why shouldn’t your chat window be viewable at the same time?

If you have skinned your environment with a custom view of the world, why shouldn’t you also see mapping or restaurant recommendations?

So local data and remote data should be blended in the same view.
How can AR software – of which I hope, there will be  thousands – seamlessly be expected to layer their graphics, not only with the real world, but with each other, and with online data too? Will games and software makers need to co-operate to allow their graphics to be integrated together with correct occlusion taken into account? A tall order, no?

I must confess though, my technology knowledge fails me here.

I can only guess special graphics drivers, or 3D APIs,  will have to be developed to let programs share their 3D world with that of a Arn browser.
Maybe programmes should simply treat themselves as a local-sever which the browser can connect too, and let the Arn handle all the rendering itself (although I imagine many games designers would find this quite limiting).
So I leave it as an exercise to the readers to discuss and propose the best methods by which this vision of a layered world could be realised..

Beyond IRC:

As mentioned before IRC has some drawbacks, which are due to its age or method of working.
As such, future systems might yet prove better alternatives for a open AR network.
One example of such a system is Google Wave.
It shares many of the advantages of IRC (open, anyone can create a channel of data, different permission levels can be set and its free), while avoiding some critical restrictions. (The data can be persistent).
I believe some of the ideas I’ve mentioned, and possibly even the proposed protocol string could be adapted for Google Wave or other future systems.
I believe overall the principles are more important then any specific implementation to get to them.

Summary;

⁃    In order for AR to flourish the user shouldn’t need to download a separate application for each mesh they want to see.
⁃     Having url’s embedded into QRCoded markers which point to standard mesh files like dxf or kml would be a way to do this right now.  The QR code would only have to be seen precisely  in shot once, then its borders could be used like a standard marker.

⁃    An augmented view of the world needs to support visual multitasking, and having  layers of information is the best way to do that.
⁃
⁃    Methods need to be devised to allow drastically different software to contribute to these layers, without restricting either the software’s rendering ability’s, or the users ability to pick and choose what layers of information he wants to see.

Last point;

I am absolutely confident in my belief AR will become at least as important as the web has, and probably a lot more so. It will also face much the same hurdles and challenges getting established as that medium did.
But, speaking as a web-developer, can we try to avoid a browser war this time?

Everything Everywhere , draft.
by Thomas Wrobel
Darkflame a t gmail