“The game is about the world not dragons,” Will Wright, Founder and Chief Executive  Stupid Fun Club, Creator of Spore and The Sims.

I had a brief chat with Will Wright after his talk at Engage!, and I was delighted to hear that augmented reality is high on his agenda at the moment:

“a lot of our stuff is kind of in the experimental format right now, but definitely one of our strong interests is AR.”

Will Wright will be coming to speak at Augmented Reality Event, Santa Clara, CA., June 2nd, 3rd,  2010.  But, for now, here are a few hints at some of the directions that are intriguing him, e.g., the game potential of 3D mapping like Blaise Aguera y Arcas’s  demo of augmented reality maps at TED -  see the full conversation below.

There has been a vital shift, Will Wright points out.  Before the Wii,  immersive was understood as  how much we were pulled into the world of the game.  Now immersive is how much the game pulls us deeper into our world, e.g., our relationship with the people we are playing with as in Rock Band, or engaging with other people’s crazy antics when playing Wii games.

“Computers are imagination amplifiers and toys are imagination constructors.”

The slide above is from Will Wright’s talk at Engage!

Will Wright’s talk was extraordinary, dense, layered, and deeply thought provoking.

I have picked out a few samples from Will Wright’s vast tome of slides here.  They are just a glimpse of the many insights he offered.  If you are still wondering what will transform augmented reality into a mainstream experience, I suggest studying this talk carefully (I think the audio will be posted on the Engage! web site).  Also watch Will Wright’s, Games For Learning Institute talk at NYU, February 17th, 2010, archived here.

Will Wright and Stupid Fun Club are getting ready to takes us to the next level of imagination amplification and construction.

“Smart” things can make us dumber by overriding our instincts

Just one of the many wonderful anecdotes Will Wright told  was the story of his experiences with a new “smart” car (he bought this car with the intent of exploring the pinnacle of the “smart” car experience). The slide above is from Will Wright’s talk at Engage!

Increasingly, artifacts are being designed to send us more and more data, and this car was endowed with an array of sensors supplying data aimed at assisting parallel parking – a notoriously challenging aspect of driving.  But the car  failed miserably in helping.   While parallel parking had been easy for him prior to being deluged with all this data, Will Wright pointed out, ironically, he had to learn to ignore this stuff to park the “smart” car.

Instinctively, we learn to filter the information necessary for parking to the relevant stuff.  This kind of pre-conscious filtering is a key challenge for augmented reality, and one that Will Wright, as a game designer, has given great deal of thought to.

As Will Wright pointed out, a  lot of our ideas about augmented reality, and sensor enabled artifacts, are rooted in trying to give us more data, to “take over our instincts.”   Not only do these artifacts attempt to give us more data, which as in the case of the HUDs for parallel parking can get in the way of our own highly effective intuitive instincts.  But, as Will Wright also noted, these artifacts also have more data which they can deploy independently to override our instincts, e.g., the car detecting your head has turned back to talk to a passenger and applying the brakes!

“Toys Encourage Agency”

Toys can be the antidote to instinct blocking “smart things.”  In contrast to “smart” data spitting cars that “take over” our instincts, toys encourage agency.  Will Wright gave the example of  high dynamic lighting photographs that make the world “toy like” and encourage us want to reach in and play with it (photo above from HDRCreme).

“What Computers are really good at is harvesting human intelligence”

Another key insight that Will Wright explored in depth in his talk was the significance of  crowd sourced intelligence (the slide above is from Will Wright’s talk at Engage!).  If the crowd is training the filter, he suggested to me, this might build the kind of context we need to build meaningful augmented reality experiences (for more on this see the conversation below).

Talking with Will Wright at Engage!, NYC, 2010

Tish Shute: I was very interested by the idea you put out that this deluge of information gathered by sensors is not necessarily a kind of nirvana for augmented reality, in fact it can be just the opposite.  In the embryonic world of augmented reality, we have two streams it seems at the moment – one is the idea of a kind of like hyper local nirvana imagined for AR, in which we get information relevant to us, when and where we need it.   But you talked about some of the problems in realizing this, didn’t you?  The other strand  is the emerging stream of play which you are exploring..

Will Wright: Right.  I think part of it is like what I was talking about-the way our senses are set up to know how to filter out 99% of what is coming into them.  That is why they work, and that is what is beneficial.  I think that is why AR needs to focus on…

You look at what I can find out on Google or whatever, the amount of information is just astronomical.  The hard part, the intelligent part, is how do you figure out that one tenth of 1% that I actually care about at this given second?

Tish Shute: Yes.  Have you seen any examples of AR beginning to do that?

Will Wright: No, not at all.  I think that you have to have a contextual understanding of where I am at, where my mindset is, what my situation is, what my goal state is in a moment by moment basis.  And then it is still a complex task.  But the very first thing we need is more context for building a filter.  See, that filter is changing every few minutes, you know, what I am filtering into my senses is changing, and my context is changing moment to moment.

Tish Shute: I really liked your emphasis on crowd sourced intelligence as the key power of a networked world, is this the seed..?

Will Wright: Well, you can imagine crowd sourcing that filter…it would affect a million people and get a sense of what mental context that they were in and what filter they turned on.  And so, in a sense, the crowd is training the filter.

Tish Shute: Yes.  The problem with projects like SIRI, that is driven by the big DARPA AI project, CALO, is it is centralized – although I am not sure what they intend to do in terms of crowd source corrections?  But if it was all open and we could crowd source as well that would be interesting.  But in the end we need a framework for AR that is as open as the internet, don’t we?

Will Wright: Right.  I think the technological infrastructure needs to be much lighter so that it can be grounded in more like a Twitter feed or something.

Tish Shute: Yeah.  I’m actually working on a project using the Wave Federation protocol as the basis for an open communications framework for augmented reality, AR Wave – not the Wave user interface, just the real-time federation protocol.  But, of course,  for it to become an open framework that could be a vehicle for crowd trained augmented reality it would need good take-up!

Will Wright: Right.  You really want a million people involved.

Tish Shute: Yes our dream is that the creation of augmented reality content will be as open, accessible and  simple as making an html page, or contributing to a wiki.

So in terms of AR games what is interesting on the horizon, presumably games also have to solve the problems of  delivering a hyper local experience.  The car that you described in your talk tried hard to use augmented reality to solve the problem of parallel parking and ended up making it harder.  So giving us the information we need, where we need it, when we need it, and specific to who we are is going to be a big challenge.  But I mean in terms of games, what kinds of hyper local experiences will be most fun and what have you seen that is interesting in terms of augmented reality games up to now?

Will Wright: I’ve not actually seen much at all.  I’ve seen people doing interesting stuff with like Google Maps.  They aren’t really entertainment oriented, but I think you can start thinking about…

I mean I think for a lot of people, Google Street View is entertainment.  But I haven’t really seen something that was really leaning into an entertainment application using existing technology and data that is already out there.

I mean I have seen some cool experiments-people playing Pac-Man in Washington Square and stuff like that, but nothing really serious.

Tish Shute: Yeah.  of course I think one of the missing links is that the barrier of entry is way to high for creating social augmented experiences for smart phones, and as you point out in your talk it is the social implications of the game is what makes it compelling.

Will Wright: Also, I think using them [smart phones] as data aggregation devices rather than just data consumption devices…so that people out there are using their phone, cameras, microphones, or whatever to gather data and get an experience where they are rewarded for gathering data.

Tish Shute: Like foursquare where you get the badges, and people can become the mayor of like a cafe or something.

Will Wright: Right.  Yeah, you can imagine people using their phones to actually kind of pull information…

Tish Shute: A Dutch developer/artist/game designer, Thomas Wrobel,  Lost Again, came up with the original concept for the AR framework we are building on the Wave Federation protocol.  Thomas and his partner Bertine van Hovell design alternate reality games, amongst other things they do…so they are deeply immersed in the potential of the world as game.

Will Wright: Yeah, one of my programmers actually works in Amsterdam….there is a whole sub-community…
Well, yeah.  The possibilities are tremendous.  And Wii is actually training us that way [to be as much engaged with the other players in the physical space as the virtual game], so it is going to happen.

Tish Shute: What are the most exciting things you see at the moment, and for the next 12 months for augmented reality?

Will Wright: Gosh.  I mean I just think there is cool stuff happening in mapping, in general.

Tish Shute: Like Blaise Aguera y Arcas’s  demo of augmented reality maps at TED?

Will Wright: Yeah, I thought the 3-D mapping with Microsoft…I think like the next level of that is going to be really compelling.

Tish Shute: You see game potentials in that?

Will Wright: Yeah.  You start overlaying really cool game potential on top of that.

Tish Shute: Might you get interested and do something?

Will Wright: Oh, yeah.  I mean in terms of games, that is one of my biggest interests, is AR.

Tish Shute: Are you allowed to talk about anything specific at all?

Will Wright: Not yet, no.  I mean a lot of our stuff is kind of in the experimental format right now, but definitely one of our strong interests is AR.

Tish Shute: Yeah, absolutely.  We are over being tied to our desks to use computers -we want to be doing it anywhere, anytime, with anything…

Will Wright: Now the game is about the world instead of about dragons.  I love that.

Visual search is heating up, and with it a key stage of turning the physical world into a platform is underway as images become hyperlinks to the world in applications like Google Goggles, Point and Find, and SnapTell – see this post by Katie Boehret.  And while there may be no truly game changing augmented reality goggles for a while, make no mistake, key aspects of our augmented view, factors that will have a lot to do with what we will actually see when an augmented vision of the world is a commonplace, are already in the works.  And, as Anselm Hook (pic above from @caseorganic’s flickr) notes:

“There is a real risk of our augmented reality world being owned by interests which are not our own. There is a real question of when you hold up that AR goggle, what are you going to see?”

Cooperating services, e.g., Google Earth, Maps, Streetview, Google Goggles, and leader in local search like Yelp (see here) would have an enormous ability to filter and control a mobile, social, context aware view of the physical world, and Google themselves see an ethical quandary.

“A Google spokesperson says this app has the ability to use facial recognition with Goggles, but hasn’t launched this feature because it hasn’t been built into an app that would provide real value for users. The spokesperson also cites “some important transparency and consumer-choice issues we need to think through” (quote from Wall Street Journal Column by Katie Boehret).

Anselm Hook and Paige Saez, with great prescience, have been advocating a social commons for the placemarks and imagemarks to our physical world platform through a number of pioneering projects, including imagewiki.   I have interviewed both Anselm and Paige (upcoming) in depth, recently.  My talk with Anselm was nearly three hours long!  So I am publishing the transcript in two parts.

Understanding what it means to have a social commons for  our physical world platform, and augmented reality, are key questions for all of us to think about, but especially important for those of us involved in the emerging industry of augmented reality.

Anselm notes :

“The placemarks and imagemarks in our reality are about to undergo that same politicization and ownership that already affects DNS and content. Creative Commons, Electronic Frontier Foundation and other organizations try to protect our social commons. When an image becomes a kind of hyperlink – there’s really a question of what it will resolve to. Will your heads up display of McDonalds show tasty treats at low prices or will it show alternative nearby places where you can get a local, organic, healthy meal quickly? Clearly there’s about to be a huge ownership battle for the emerging imageDNS”

The mobile internet is moving beyond the internet in your pocket phase of mobility with mobile, social, proximity-based, context aware networks like FourSquare, Gowalla, Brightkite and GraffitiGeo (see Smart Data Collective) likely, soon, to start to take precedence over other forms of social network.

Regardless of the timeline for true augmented reality – 3D images & graphics tightly registered to the physical world,  proximity-based social networking and real time search are already taking us into a hyper-local mode and the realm of augmented reality which is “inherently about who you are, where you are, what you are doing, and what is around you” (Robert Rice – see here). The ground is being prepared for augmented reality now.

If you have been reading Ugotrade, you will know I have been actively involved in developing  an open, distributed AR platform/mobile social interaction utility for geolocated data based on the Wave Federation Protocol – AR Wave a.k.a Muku – “crest of a wave” (see my posts here, here and here for more on this project, and the AR Wave Wiki here).  Federation is, I believe, one vital aspect to developing a social commons for augmented reality and the physical world platform.

Also, a bit of news, I am co-chairing the upcoming Augmented Reality Event (are2010) with Ori Inbar of Games Alfresco and Ogmento, whurley.  Sean Lowery, Prospera, is the event organizer, and are2010 has the support of the AR Consortium.   The are2010 web site is live and there is an Open Call For Speakers.  You can submit your proposals and demos for one of the three tracks, business, technology, or production on the web site here.

Bruce Sterling “prophet” of  augmented reality and more, “will deliver the most anticipated Augmented Reality keynote of the year.”

It didn’t surprise me when Anselm mentioned that Bruce Sterling was a key influence for his work on the geospatial web and augmented reality.  Anselm explained:

“I’d seen a talk by Bruce Sterling at an event called Planetwork [May, 2000]. And that event was, for me, a turning point where I decided to focus full time on exactly what I cared about instead of doing things that were kind of similar to what I cared about. So, his influences is a pretty significant one to me at that exact moment.”

For more see viridiandesign.org -  seems it is time for a “Neo-Viridian,” revival!

This post by Bruce Sterling on Pachube Feeds, and Thomas Wrobel’s prototype design for open distributed augmented reality on IRC, were key inspirations for me when I began thinking about the potential of Google Wave Federation protocol for augmented reality.  I had been exploring Pachube and deeply interested in the vision of Usman Haque, but I had a real aha moment when I read this :

“(((Extra credit for eager ubicomp hackers: combine this [pachube feeds] with Googlewave, then describe it in microsyntax. Hello, 2015!)))”

I think the AR Wave group will earn the extra credit and more very soon!  Davide Carnovale, need2revolt, and Thomas Wrobel have been leading the coding charge, and there will be a very early AR Wave demo soon, perhaps as soon as the Feb 16th ARNY Meetup. 

Open access to the creation of view that will eventually find its way into AR goggles, will depend not only on the power of  an open distributed platform for collaboration like the AR Wave project.  Our augmented reality view will be constructed through complex “hybrid tracking and sensor fusion techniques” (Jarell Pair), cooperating cloud data services, powerful search and computer vision algorithms, and apps that learn by context accumulation will drive our augmented experiences, and at the moment, these kind of resources, at least at scale, are for the most part in private hands.

In the interview below, Anselm’s discusses  how trust filters, and being able to publicly permission your searches so that other people can respond and so that people can reach out to you, and the democratization of data in general, are even more of a concern with augmented reality and hyper local search. The task of understanding what it means to have  a social commons for the outernet remains an open, and pressing question.

Anselm explains (see full interview below):

“as we move towards a physical internet where there’s no clicking and there’s no interface and the computer’s just telling you what it thinks you’re looking at, translating, you know, an image of a billboard to the name of the rock star who’s on that billboard, or translating the list of ingredients on a can of soup to the source outlets where it thinks that, those ingredients came from. When you have that kind of automated mediation, the question of trust definitely arises.

And we haven’t seen the Clay Shirkys or the Larry Lessigs of the world start to talk about this yet.  Although I suspect that in the next four or five years that the zero click interface will become the primary interface, that we’ll have…we’ll come to assume that what we see with the extra enhanced data we get projected onto our view is the truth. Yet, at the same time, there is just no structure or mechanism even being considered for a democratic ownership of it.”

Augmented Reality will emerge through sensor fusion techniques & cooperating cloud services

In 2010, sensor fusion techniques, computer vision technology in conjunction with GPS and compass data will create data linking that can enable the kind of augmented reality that has been the stuff of imagination for nearly four decades (see Jarrell Pair’s post).

Putting stuff in the world in 3D is of course key to the original vision of augmented reality, and one of its biggest challenges.  Augmented reality is going to be implicated in a real time mapping of the world at an unprecedented scale and granularity.  We have barely an inkling of the implications of this now.

Anselm and Paige have been working in the heart of the social cartography movement for nearly a decade.  The vision and experience of this community is vital to understanding how augmented reality and the world as a physical platform can evolve into something that benefits people and allows them “to have a better understanding of the opportunities around them.”

We have been hacking maps for millenia –  “from conceptual story mapping, to colloquial mapping in European development and the cartographic renaissance created by the global voyages and rediscovery of Ptolemy’s maps” (Andrew Turner).  And, recently, initiatives on a public-provided GIS, like OpenGeo, have led the way toward more open, interoperable, geospatial data.

Mapping takes on a new an crucial role to augmented reality.  Nokia’s ImageSpace is beginning to do what many thought Microsoft would do with photosynth two years ago.

And, if we see these kind of projects developed into a “photo-based positioning systems” -  “3d models of the environment to cover every possible angle, and then software that can work out in reverse based on a picture precisely where you are and where your facing” (Thomas Wrobel), we would find augmented reality leap forward over night.

It is time to take very seriously the vast opportunities and potential pitfalls of an augmented world.

“when you are mediating the translation layer between the image and the data, then there is an opportunity for you to control it, and that opportunity is hard to resist.  It is hard to choose not to own that opportunity. It is an advertising opportunity. It is a revenue opportunity. It is a chance to send a message and a tone.

I know that Google and companies like that are keenly aware of the kinds of roles they don’t want to hold, but it is sometimes seductive to think about them. And I am afraid that we, as a community, need to assert an ownership, kind of a commons, over how computers will translate what they see to information that we perceive.”

There are some initiatives emerging.  Tonchidot (who closed on $4 million of VC for augmented reality last December) has helped create the AR Commons in Japan.  CFO of Tonchidot, Ken Inoue explained in an interview with me in September 2009.

“We feel that public data, such as landmarks, government facilities, and public transport should be shared. We see an AR world where people can readily and easily access information by just seeing – quick, easy, and efficient.  And because of this ease and intuitiveness, children, the elderly and handicapped will surely benefit.  AR could help create a safer society.  Warnings, alerts, and safety information could save lives and avoid disasters.  These are what we, and AR Commons would like to tackle in the not so distant future.”

But the task of building a social commons for the physical world platform has only just begun.

Interview with Anselm Hook

photo from Anselm’s Flickr stream here

Tish Shute: We first met last year at Wherecamp. The start of 2009 was I think the “OMG finally” moment for augmented reality and in less than a year AR, at least in proto forms, AR is breaking into the mainstream now! You are one of the founding visionaries/philosophers/hackers of the geo web and you have been thinking about geo web and AR for a long time – all the way back to the legendary Head Map Manifesto, and before.  Mostly recently you led the way in the very successful ARDevCamp in Mountain View. Could you start by telling me a little bit about the history of your pioneering work with geolocated data?

Anselm Hook: I am a long time Geo fanatic. I’m really interested in social cartography and what some people call public-provided GIS, that’s some language that people use. Anyway, my personal interest, when I talk to people who are non-technical (and it’s been a long term interest in the way I phrase it) is that I want to help people see through walls. So, the goal is very simple. I want people to have a better understanding of opportunities around them, the landscape around them. I always get frustrated when people make bad decisions because of a lack of information, especially when it’s related to their community and related to their environment. But, plainly put, I really just want “to help people see through walls”. It’s a very simple goal.

Tish Shute: I know you worked on Platial, which is really one of my favorite social mapping applications. It really broke new ground. What was the history of that? How did you get involved with Platial?

Anselm Hook: That’s an interesting question. It actually started at around 2000 when I saw Bruce Sterling talk. I had been writing video games for many years, and I was quite good at it, and I enjoyed it. But, the reasons I was doing it diverged from why the industry was doing it. I was making video games because I like to make shared spaces for my friends to play in and to share experience. I really enjoyed making shared environments. I worked on BBS’s and my friends and I were always making these collaborative shared environments.

Once the video game industry kind of started to take off, I started to do high performance, 3D interactive video games and making compelling shared spaces, and it was a lot of fun. But, the frustration for me was that there was a huge industry growing around it and became very commercial. Although it paid well, it started to diverge from my values which were more centered around community environments, and shared understanding.

Tish Shute: Yes very rapidly, the big games kind of devolved from the social aspects and became more and more into single player really, didn’t they?

Anselm Hook: It was the way, actually, because even though often you were in a many player world, you weren’t collaborating, everything else became just a target.  I liked the idea of deep collaboration that calls the kind of playful space you see in IRC, or in the real world, where people are solving real world problems.

And I grew up in the Rockies, and I was always had a lot of access to the outside. So, I saw shared spaces and collaboration as a way to protect our environment. [ To step back ] I think people used different metrics for measuring their choices in the world and many people have a value system centered around minimization of harm: making sure that the people are not hurt. But, my value system is different. I personally believe that protecting the planet is more important: to maximize biodiversity. I feel like protecting people around me comes from protecting the ecosystems they live in.

Tish Shute: That’s interesting, isn’t it, because the history of Keyhole was really that, wasn’t it.  Keyhole later became Google Earth, but I mean it began out of a project to look at what was going on in the ecosystem over Africa at that time, didn’t it?

Anselm Hook: Yes, in fact many people’s projects are stemming from an environmental concern. Mikel Maron’s works for example – he’s doing Map Kiberia, and he also worked on OpenStreetMaps.

Tish Shute: Map Kiberia – that is the new project?

Anselm Hook: Oh, yes his project is called Map Kiberia. He’s mapping a city in Africa.
[For more see Map Kiberia’s YouTube Channel – photo below from ricajimarie ]

Tish Shute: Right, great!

Anselm Hook: When I started to look at GIS and mapping I started to meet people who had a very similar background. What happened to me is I kind of stepped away from games around the year 2000. I’d seen a talk by Bruce Sterling at an event called PlaNetwork. And that event was, for me, a turning point where I decided to focus full time on exactly what I cared about instead of doing things that were kind of similar to what I cared about. So, his influences is a pretty significant one to me at that exact moment.

[For more see viridiandesign.org – seems that it is time for a “Neo-Viridian,” revival.]

Tish Shute: It’s interesting because now your paths are crossing again with augmented reality. You are on the same wavelength again.

Anselm Hook: It’s funny, actually, I’ve had a couple of brief overlaps in that way.  Well, so in 2000 I went to see this talk and I did a small project called — well, I called it SpinnyGlobe. What I did is I mapped protests from a number of websites onto a globe to show the level of community opposition to the pending war in Iraq. It was the first time there had been a protest before a war. So, it was very interesting to me. [ See http://hook.org/headmap ]

Tish Shute: That’s really fascinating. Do you have any pictures of that you could send me?

photo from anselm’s flickrstream

Tish Shute: Yes, I’ll definitely look SpinnyGlobe up. It sounds very interesting.  One of the aspects of your work on geo-located data projects like this and Platial is that you really started to develop this idea of a culture of place, about how people make place. This was the wake up call to me regarding the power of networks combined with geo-data.

We are hoping to extend this idea into augmented reality with the an open distributed platform for AR so that we can collaboratively map our worlds from the perspective of who we are, where we are, and what we are doing.  I know you’ve just done some work recently in augmented reality.  I know you put the code up already.

By the way, I love the way you take your philosophy into the way you make code – the practice of making some code, trying some things out, making it all public and publishing your findings, you know, your comments on that experience.  Perhaps you could recap sort of how you picked up recently on the state of play with augmented reality and what aspects you looked at, and what came out of that experience?

Anselm Hook: So, it’s a very simple trajectory. Coming out of the work I had done, Platial, among other projects and I started to just look at the hyper-local and I suddenly realize that even those services weren’t really speaking to living, and how to really see and solve local problems. What was missing was a sense of context.

The map doesn’t know how you’re feeling, it doesn’t know if you’re in a hurry, it doesn’t know what you want, it’s very static. Even the web maps are very static. And augmented reality for me I started to recognize as a combination of — well — it’s probably collision of many forces, many forces that we’re all a part of. We’ve also didn’t realize that the real-time web is really important, it’s part of what AR is about.

We have all started to realize that the context is important. You know, your personal disposition, your needs, if you want to be interrupted or not. That is the kind of thing that the ubiquitous computing crowd has talked about. We started to recognize that there are sensors everywhere, and the ambient sensing communities talked about that. So what is funny for me about augmented reality is I started realizing it is just a collision of many other trends into something bigger.

Everything else we thought was a separate thing is actually just part of this thing. Even things like Google Maps or mapping systems we think are so great are really just kind of almost an aspect of a hyper-local view. You actually don’t really care what is happening 10 blocks away or 100 blocks away. If you could satisfy those same interests and needs within a single block, one block away, you would probably be really happy. You really just want to satisfy needs and interests, find ways to contribute, or get yourself fed, or whatever it is you want. And AR seemed to be the playground to really explore the human condition.

Tish Shute: Anyway, I think one of the things that has been very amazing this year is we to have the good mediating devices that, for the first time, give us compasses, GPS, and accelerometers. But one sort of missing pieces with AR at the moment is [tracking, mapping, and registration] – the kind of things colloquial mappings of the world could be of great help with.

We have seen mapping coming out of the Flickr data, e.g., the University of Washington, put the maps together from the geo-tagged Flickr photos. Now if we could have that linked up with AR, then we have the kind of mapping we need to kind of really hook the geo-data onto the world in a way that goes beyond…you know, what compass and GPS can really deliver is pretty minimal at the moment.

Anselm Hook: There is a real risk of our augmented reality world being owned by interests which are not our own. There is a real question of when you hold up that AR goggle, what are you going to see? Are you going to see corporate advertising? Are you going to see your friends’ comments or criticisms? It is going to be an Iran or a democracy, right? It is unclear.

Right now there are some disturbing trends I have noticed. I am a big fan of Google Goggles. I think it is a great project. But when you are mediating the translation layer between the image and the data, then there is an opportunity for you to control it, and that opportunity is hard to resist. It is hard to choose not to own that opportunity. It is an advertising opportunity. It is a revenue opportunity. It is a chance to send a message and a tone.

I know that Google and companies like that are keenly aware of the kinds of roles they don’t want to hold, but it is sometimes seductive to think about them. And I am afraid that we, as a community, need to assert an ownership, kind of a commons, over how computers will translate what they see to information that we perceive.

Tish Shute: Yes. And this is how we met, again, recently [over the project to create an open, distributed platform for AR using the Wave Federation Protocol]…

This is something I feel really deeply is that, you know, basically we need the physical internet to be as open as, as the, as the internet, as the end-to-end internet has been. Or more so, actually, because the end-to-end internet has seen the trend has been to walled gardens.  Basically Facebook became enormous, an enormous walled garden which, I think, was despite, our predictions about them, [walled gardens] are the social experience really on the web.  It’s very much in walled gardens still and I, and I really feel that with the physical internet, we need to make great efforts not for it not just to be a series of small pockets of privately funded walled gardens.

There needs to be some kind of communications infrastructure that keeps it open so that was when I got interested in looking at the Wave Federation Protocol because it was a real time, you know, an open real time protocol that could possibly be a basis for that. But I think the point you’ve talked to just now, the mapping of the world and who has the “goggles”, i.e., the image data, image databases, that make the world meaningful is really, that’s still a, it’s still a BIG question [i.e. who controls the view?].

When I saw ImageWiki, [I realized] that is a piece that is vital for, for augmented reality. We need to have a huge social effort to be involved in this,  linking in and creating the  physical internet, in creating the image hyperlinks that will make that meaningful.

Anselm Hook: I think that’s a great point. The search interface, the kind of Internet that we’re used to, the way we talk to the network now, is fundamentally open end to end. Yes, you can have your oligarchies inside of it, as we see with Facebook, but you can always start your own venture up and you can do a search on something, and you can find that, that website and you can join it or you can put up your own webpage and people can find it.

The translation layer, the idea of text search and the ability to discovery power and the serendipity and the openness of that discovery, it’s pretty open right now. We do have some serious boundaries of language, which is one of the reasons I was working at the Meedan.org [hybrid distributed, natural language translation] for a couple of years, trying to bridge that issue.

But here, as we move towards a physical internet where there’s no clicking and there’s no interface and the computer’s just telling you what it thinks you’re looking at, translating, you know, an image of a billboard to the name of the rock star who’s on that billboard, or translating the list of ingredients on a can of soup to the source outlets where it thinks that, those ingredients came from. When you have that kind of automated mediation, the question of trust definitely arises.

And we haven’t seen the Clay Shirkys or the Larry Lessigs of the world start to talk about this yet.  Although I suspect that in the next four or five years that the zero click interface will become the primary interface, that we’ll have…we’ll come to assume that what we see with the extra enhanced data we get projected onto our view is the truth. Yet, at the same time, there is just no structure or mechanism even being considered for a democratic ownership of it.

We have with DNS, for example, the idea that you can register the domain name and people can search for it, and find it, and go to it. There’s no such thing as an Image DNS, or an image translation to DNS right now. What does it mean when everything is just “magic”, when there’s no way for you to be a part of the conversation, where you’re just a consumer of what people tell you, or of what one company right now, tells you, is reality? That’s a real concern.

Tish Shute: This, to me is the most important question at the moment. I mean, it’s the big one and it’s the place to put energy if you love the Internet [and what it can now become] right. You’ve got to put a lot of energy into this because this [a democratized view of the physical world as a platform] won’t just happen, because there’s a lot of momentum already for it to be heavily privatized, partly because, one reason is, some of the computer vision algorithms that, say, make sense of things like the geotag photographs are not open.  I mean, for example, the beautiful maps that have been made from the University of Washington [from Flickr geotagged photo sets], that isn’t in the public domain.

Anselm Hook: Right. Tish, and in fact you’re referring to [with the maps from the Flickr photos] to ordinary maps and the fact we’ve already seen that maps lie, we’ve already, seen how much maps are reflecting a certain truth that becomes the normative truth. Google maps reflects roads, because this is roads and cars, right? Only recently have they thought about buses and walking. So the normative view that people assume is the reality, is showing off you know Starbucks, and roads, and cars, that becomes the default, those prejudices are just assumed, you know, the truth. But they’re not the truth at all.

I was talking to a friend of mine in Montreal, [Renee Sieber], and she said that their Indian portage routes are a bridge across land and water, they don’t think of a piece of land and a piece of water as being different things, they think of them as one thing: a route. It’s already a different kind of language we can’t even reflect it.

So not only is there this kind of formal, anthropological lie, in a sense, but there’s this way that we deceive ourselves because of our own prejudices.

Tish Shute: Yes I agree and that’s why I think when I saw some of the things you had written on the ImageWiki point clearly to the need to create a social commons. We need a social commons for the real-time physical internet, we need it for the image hyperlinks that make sense of that.

And it’s a complicated thing in a sense, though, because we don’t actually have a good distributed infrastructure for AR yet, and I found exploring AR Wave, that at last we have the suggestion of an open, federated protocol for real-time communication – the wave federation protocol. [Real time communications is a very important part of AR].  It isn’t an actuality yet where lots of people are able to use it, set up their own servers, and there’s not a standard all the way through  [there is not a standard for how data is sent between the client and the server].

But Wave Federation Protocol does make possible truly distributed social AR.  I started thinking when I saw ImageWiki that to bring ImageWiki together with the social collaborative power of distributed AR.  This really would be the basis of creating a social commons for augmented reality and the physical world as a platform – the start of a bottom up with deep social collaboration on how we create augmented reality colloquial maps that can inform a hyper-local of the world.

Anselm Hook: Yes. When Paige Saez, John Wiseman, and myself, and a few other folks… You know, Benjamin Foote, Marlin Pohlmann, and a couple other people started to play with this, we quickly found that… We started to realize, “Oh, this kind of thing will be at least as popular as IRC. There will be at least as many people doing this as chatting in little virtual spaces. There’ll be at least as many people decorating the world with augmented reality markup, and maybe using the real world as a kind of barcode for translating what you’re looking at into an artifact, a digital artifact.

And that the size of that space was going to be huge, basically. Maybe not quite as commodifiable as Twitter, but certainly very energetic.

Many of the projects we did were just kind of looking at these kinds of issues sort of from an artistic, technical, and political point of view. We weren’t so much posing complete solutions, but simply using a praxis to explore the idea with an implementation, as a foundation for this discussion. So I think we sort of opened that can of worms for sure.

Tish Shute: Did you actually set up ImageWiki to be working as a location based app yet?

Anselm Hook: It is a location based app. It collects your longitude, latitude, and the image and stores it. And then it uses that as a way to translate that image to anything else. It could be a piece of text or a URL.

Tish Shute: So there is a smartphone app, but you didn’t take it as far as an AR app yet?

Anselm Hook: No. We didn’t do a heads-up view. There are apps on the iPhone store that do that, but they don’t do the brute force image recognition that we were using. We used a third party off the shelf algorithm that we found on Wikipedia and downloaded the source code, and threw it on the server. And John Wiseman in LA wrote the scalable database backend so that we could scale the actual…

Tish Shute: So how did you set the iphone app up to work?

Anselm Hook: The iPhone side was very simple. You take a picture of something and it tells you what it is. That is all it did. We would take the location, but the client side, the iPhone side, just rendered, returned to you…It said, “Someone said that this picture of a barking dog is an advertisement for a local band.”

Tish Shute: Right. So basically it was a geo-tagged?

Anslem Hook: Yes. We are just collecting the geo information. Actually, there were a whole lot of technical challenges. The whole idea of ImageWiki is actually kind of beyond our technical ability for a small team like us. It really does take a team, a group like Google, to do this kind of thing in a scalable way.

Tish Shute: Why is that?

Anslem Hook: There are two sides. There is the curating the images. I think that is the job of groups like us – open source groups who can curate images that are owned by the community. And then the searching side, the algorithm side, where you are actually matching the fingerprint of one image to images in your database, that takes a much more…that is much more industrial.  We get both sides, ours is not a scalable solution. It is mostly…proving that it could be done was important.

Tish Shute: In terms of hooking Imagewiki up to the collaborative possibilities of AR Wave wouldn’t federation pose some interesting possibilities for scaling search algorithms and all that?

Anselm Hook: Yes. And what is funny also, incidentally, is that, nevertheless, we did look for some financial support for it, but we couldn’t…we just didn’t find the investors to scale it. Now, other companies like SnapTell took a shot at it. And they have an app in the iPhone store where you can point at a beer bottle and get back the name of the beer bottle.

The classic example everyone uses is a book. Amazon has all the image jackets of all their books. You can point SnapTell at almost any book and get back links to buy that at Amazon, the price of the book, and user comments on the book. So they are treating Amazon as the canonical voice of the book, for better or worse. That is the state of the art so far, up until Google Goggles came out a little while ago, which actually blows it out of the water. But, that is where we are now.

Tish Shute: Right. But the point you raise about how when something like Amazon comes canonical of what is book, right, this is the whole point, isn’t it?

Anselm Hook: Is Amazon truth? It’s not bad. Jeff Bezos seems like a nice guy, but, you know.

Tish Shute: And this is the point of having these open infrastructures for this.  And this should be obvious in a way, but it comes back to the thing about what made the Internet great was the fact that even though as you note, you get an oligarchy like Facebook, but people always could just go off and do something else, right? Because the fundamental infrastructure was basically open and designed to be available for everyone. And many people have championed that and fought for it hard [to maintain this openness] haven’t they? They have devoted their lives to keeping it that way, even if the oligarchies have done their thing.

Anselm Hook: Yes. There are really some things that are underneath all of this that haven’t been solved yet.

One is that the trust in social networks has not been built yet, so we can’t do peer based recommendations very well. We can’t filter noise by peers. Twitter kind of is moving there, but I don’t just want to listen to my Twitter friends. I want to listen to my friends of friends. If I am getting truth from somebody, I want to get that truth from people my friends say that they trust.

Then the second problem is that there is a search business. My friend Ed Bice, who owns Meedan, always says that a search itself, a search request, is an opportunity to make…is a publishing moment. It is an opportunity to say what you think. In the real world, if you are just hanging out with humans and you look somewhere, other people might look at your gaze and they might look at what you are looking at. Your gaze itself is a public act.

Gaze is a soft act, but it is one that is visible. With Google, the gaze of four billion people is invisible. We don’t what people are looking at, there is no opportunity to participate. Let me give you a real example.  I have taken a image of something of the bust of figure or a statue.  Why can’t the museum in Cairo look at my request and tell me oh yeah that is Tutankhamen, or that is Nefertiti right? Why can’t they have a chance to participate in the search and respond to me?

Right now the the only person that responds is Google when I do a search. We need to invert the search pyramid and open up search, so that search is a democratic act, so that you can publicly permission your searches so that other people can respond and so that people can reach out to you, not just you having to do a dialogue.

The common example of this.. and we see this everywhere: I am looking for a slice of pizza right, now I am hungry I want some pizza. I have to ask Google, look find twelve websites, call twelve phone numbers, and talk to each of the twelve stores, and ask them are they open late, is the food organic, is the food in any good, do my friends like it.

Whereas what I should be able to do is just say it’s a search moment and I am interested in pizza. If those pizza places my criteria like you know my friend’s like them and they are organic, they are open, then that pizza place can call me. I have the money why should I do the search? So the whole business of search, the whole structure of search is predicated around a revenue model, but its a really short-sighted revenue model, its not a brokerage.

Search isn’t search, search is hand waving.  These should be moments for us to have a discourse. So problem we are seeing in AR with communication of the right information is actually underneath AR, at the level of the whole infrastructure.

Search needs to be inverted, trust filters need to be built. We need to democratically own our data institutions.  We don’t right now.  That will be more of a concern, especially with AR.

Tish Shute: Yes, especially with AR, which is this why got all excited about federation.  Do you think federation has the potential, an opportunity to create [the new infrastructure you describe?]

Anselm Hook: Absolutely,  its absolutely what we must do. It is much harder to do. It is absolutely critical.

Tish Shute: And why is it much harder to do? Could you explain that?

Anselm Hook: Well, it’s very easy for a bunch of hackers to build a service that you log into and fetch some data, it’s a single thing. They don’t have to talk anybody, they can use their own protocols, they can hack it, it’s a big black box, behind the scenes. There’s running back and forth in a giant Chinese room delivering manuscripts and scrolls to you. Whatever is behind the black box, you don’t care, it just works.  But when you federate, you need to actually publish and have standards, and then you’re talk about semantic, everyone starts getting really excited and wave some hands. It becomes a disaster. It’s, at least, another power order, more difficult than DIY, build it yourself.

Tish Shute: So, in terms of what Google Wave have done with their approach to federation, what do you think have been their achievements and what do you think is their obstacles? What do you think are the failings of the Wave? Because it’s the first big public major player backed approach to something federated, isn’t it? In real time.

Anselm Hook: Yes. I think the most important non-federated service on the planet today is Twitter.  Identi.ca it’s not getting any traction with respect to Twitter. [ Even though ] Identi.ca is a federated version of Twitter and is very good. [ Identica is now Status.net ] . So, we see already there that small players aren’t being competitive. Then look at other services like IRC. IRC is the secret backbone of the Net. All the open source projects, all the teams, all the people that work on opensource projects are all on IRC. It’s the only way they get anything done.

With Google Wave, and the protocols underneath Google Wave, we see an attempt to build a similar kind of real time, but distributed protocol. I think it’s the right direction. I think, people should pick up the offering and make their own servers. I think that protocol is really great, I think the fact that is compressed, its high performance, it is small, real-time of blobs of data flying around, all exactly the way it should be done. It is getting close to this kind of rewrite of the Internet that people keep talking about, because, you know, the net protocols are so bad, it is starting to treat the idea of intermittent exchanges being more transitory, volatile, and not heavy.

….to be continued.  Part 2 coming soon!

It is now nearly two weeks since the Google Wave preview launch and I am happy to say we have some AR Wave news. The diagram above shows Thomas Wrobel’s basic concept for a distributed, multi-user, open augmented reality framework based on the Google Wave Federation Protocol and servers (click on the image to see the dynamic annotated sketch or here).

Even in the short time we have had to explore Wave, some very exciting possibilities are becoming clear. Thomas puts some of the virtues of Wave as an AR enabler succinctly when he writes:

“Wave allows the advantages of both real-time communication, as well as the advantages of persistent hosting of data. It is both like IRC, and like a Wiki. It allows anyone to create a Wave, and share it with anyone else. It allows Waves to be edited at the same time by many people, or used as a private reference for just one person.

These are all incredibly useful properties for any AR experience, more so Wave is open. Anyone can make a server or client for Wave. Better yet, these servers will exchange data with each other, providing a seamless world for the user…..a single login will let you browse the whole world of public waves, regardless of who’s providing or hosting the data. Wave is also quite scalable and secure…data is only exchanged when necessary, and will stay local if no one else needs to view it.

Wave allows bots to run on it…allowing blips in a waves to be automatically updated, created or destroyed based on any criteria the coders choose. Wave even allows the playback of all edits since the wave was created.

For all these reasons and more, Wave makes a great platform for AR.”

There will be much more coming soon on Wave enabled AR because the Google Wave invites have begun to flow out to a wider community now. This week, many of our small ad-hoc group looking at the development challenges and implications of Google Wave for AR actually got into Wave for the first time.

Many thanks to all the people who have contributed to this discussion so far including: Thomas Wrobel, Thomas K. Carpenter, Jeremy Hight, Joe Lamantia, Clayton Lilly, Gene Becker and many others.

We will be setting up some public AR Framework Development Waves this week.  If you have any trouble finding them, or adding yourself to it, please add Thomas and I to your contact list.  I am tishshute@googlewave.com  Thomas is darkflame@googlewave.com  The first two are currently called:

AR Wave: Augmented Reality Wave Framework Development (developer forum)

AR Wave: Augmented Reality Wave Development (for general discussion)

The discussion so far has been in two areas. On the one hand, it is gear-heady and focused on the Google Wave Federation Protocol, code, development challenges, and interfacing to mobile, while on the other hand people have been looking at use cases and questions of user experience.

Distributed, “shared augmented realities,” or “social augmented experiences” – that not only allow mashups, & multisource data flows, but dynamic overlays (not limited to 3d), created by users, linked to location/place/time, and distributed to other users who wish to engage with the experience by viewing and co-creating elements for their own goals and benefit – are something very new for us to think about.

As, Joe Lamantia, puts it, now:

“there’s a feedback loop between which interactions are made easy by any given combo of device;/ hardware / software / connectivity, and the ways that people really work in real life (without any mediation / permeation by tech).”

Joe Lamantia whose term, “social augmented experiences” I borrow for this post title, has done some thinking about “concepts and models for understanding and contributing to shared augmented experiences, such as the social scales for interaction, and the challenges attendant to designing such interactions.” Check out Joe Lamantia’s blog for more on this later this week.

It is very helpful, as Joe points out, to shift the focus  back and forth between the experience and the medium.

It is super exciting to have clear evidence that shared augmented realities are no longer merely possible, but highly probable and actually do-able now.

I should  be absolutely clear about what Google Wave does to enable AR because obviously Wave plays no role in solving image recognition and tracking/registrations issues.  But, for example, Wave protocols and servers do provide a means to exchange, edit, and read data, and that enables distributed, social augmented realities.

Thomas explains how the newly named “AR Blip” works as:

“An AR Blip is simply a Blip in wave containing AR data. Typically this would be the positional and url data telling a AR browser to position a 3d object at a location in space.

In more generic terms, an AR Blip allows data of various forms (meshes,text,sound) to be given a real-world position.”

I have mentioned in other posts (here and here) that Wave can be used for AR as precise or as loose as the current generation devices can handle. And as the hardware and software for the kind of AR that can put media out in the world to truly immerse you in a mixed space, the framework  should  be able to handle this too.

(a note on the Wave playback feature – this opens up a whole new world of possibilities.  Check out this post on some of the implications of playback for writing!)

The use cases we have been coming up with are too numerous to go into in detail this post.  The open nature of an AR framework/Wave standard will lead to many new applications we have barely begun to imagine.  As Thomas points out, different client software can be made for browsing, potentially allowing for various specialist browsers, as well as more generic ones for typical  use. The multitudes of different kinds of data in/output that could be integrated in an open AR framework as it evolves are mind boggling.

But, for now, some  obvious use cases do come to mind:
eg.

- Historical environmental overlays showing how a city used to be/and how this vision may be constructed differently by different communities

- Proposed building work showing future changes to a structure/and the negotiations of this future (both the public and professionals could submit their own comments to the plans in context), seeing pipes, cables and other invisible elements that can help builders and engineers collaborate and do their work.

- Skinning the world with interactive fantasies

I asked Thomas to help people understand how Wave enables new interactions to data by explaining how Wave could enable citi sensing and citizen sensing projects (e.g. this one being pioneered by Griswold):

“Sensors, both mobile and static could contribute environmental data into city overlays;

Having these invisible aspects of the world made visible would create ways to improve sustainability, social equity, urban management, energy efficiency, public health, and allow communities to understand and become active participants in the ecosystems and infrastructure of their neighborhoods.

The key is reflecting this  kind of data back to people “making it not back story but fore story,” right where we are, right where it happens, as well as having it available for analysis.

As well as  creating new opportunities to interact/respond to/and enhance data, making visible the invisible as Natalie Jeremijenko’s work on Amphibious Architecture and Usman Haque’s project Natural Fuse shows, can also create new connections/understandings between humans and the non human’s that share our world, e.g. fish, plants, waterways.

At a more prosaic level  potential buyers of property could see more clearly what they are buying, city planners could see better what needs to be worked on, and environmental researchers could see more clearly the impact people are having on an area.

Also Wave can provide some of the framework necessary to begin to begin to address tricky problems of privacy. Sensitive data can be stored on private waves, e.g. medical data for doctors and researchers, but the analysis of the  data could still be of benefit to all, e.g., if it’s tied disease occurrences to locations and  relationships between the environmental data and health were…quite literally…made visible.

“The publication of energy consumption and making it visible as overlays, could help influence the public into supporting more energy efficiency companies and businesses. It could also help citizens to try to keep their own energy usage down, to try to keep their street in “the green.”

Thomas notes:

“With all of the above, it becomes fairly trivial to write persistent Wave-bots that automatically send notice when certain criteria are met (pollutants over a certain level, for example). On publicly readable waves, anyone can use the data in their local computers, process it, and contribute results back on a new wave. Alternatively, persistent remote severs could run Cron jobs, or other automated processing, using services such as App Engine to run wave robots.

All these possibilities become “free” when using Wave as a platform for geographically tied data.”

But of course this is just the beginning!

Recently, I talked at length with Jeremy Hight who has been thinking about, designing and creating shared augmented realities, that anticipate the kind of dynamic, real time, large scale architecture we now have available through Wave,  for quite some time now.   This is exciting stuff.

Modulated Mapping: Talking with Jeremy Hight about Layers, Channels and  Social Augmented Experiences

image from Volume Magazine (Hight/Wehby)

Tish Shute: I know you have been involved in locative media from its early days. Perhaps we can talk about how AR continues the locative media journey?

Blair MacIntyre gave me this distinction, recently: “AR is about systems that put media out in the world, and immerse you in a mixed space.  Even the current “not really registered” mobile phone AR systems are still “sort of” AR (e.g., Layar, etc).

Locative media/ubicomp/etc are very different, in that they tend to display media on a device (phone screen) that is relevant to your context, but does not attempt to merge it with the world.
The difference is significant, and making it clear helps people think about what they do and what they want to do, with their work. The locative media space though points toward future AR systems (when the technology catches up!).”

Jeremy Hight: The need is to finish the arc that locative media and early AR have started and to now truly return to the map itself, but as an internet of data, interactivity, channels of data , end user options like analog machines once were but in high end tools, a smart AI-ish ability for it to cull data for the user, and to allow social networking to be in real world places on the map both in building augmentation and in using and appreciating it..not hacks..which have their place…but a rhizome, a branched system with shared root,end user adjustable and variable..this is the key.

This takes AR and mapping and makes a possible world of channels in space and this eventually can be a kind of net we see in our field of vision with a selected percentage of visual field and placement so a geo-spatial net, a local to world wide fusion of lm into a tool and educational tool

VR[virtual reality] has greatly advanced, but in nodes as it has limitations…LM [locative media] is the same…AR [augmented reality] is the way.. it now has locative elements and aspects of VR integrated into its functionality and nodes…it is the best option with all of these elements, greater hybridity and data level potential a well as end user and community sourcing potential

I wrote an essay for Archis’ Volume, the architecture magazine on a near future sense of some of this….a visual net on the lens like ar but with smart objects and social networking and dissent.

I also wrote of these things for immersive graphic design, spatially aware museum  augmentation, education through ar and lm and nod to the base interface of eye to cerebral cortex in layered and malleable augmentation in my essay “Immersive Sight” a few years back

image [above] is simple illustration of a possible example on a screen or in front of eye where in a mondrian show..the graphic design of information actually builds as one moves

(key is calibrated spatial intervals and related layers of further augmentation which is logical due to location and proximity)

from immersive sight on immersive graphic design: “The design can work with this in a way that creates an interactive supplemental set of information that is malleable, shifts based on location, builds and peels away as one moves closer to a work and plays with the forms of the works and the elements of the space itself. The sequence can contain many different elements and their interplay (both in the field of vision and in terms of context and layers of information). This is the model of sections of augmentation turning on and off at key points as individual spatial and concepts moments and nodes.

Another interesting possibility is that individual points of augmentation don’t turn off, but instead are designed to build as one moves in a direction toward a specific part of the exhibit. The design can work in a sequence both content wise and visually in terms of a delay powered compositional development and style in which each discreet layer of text and image does not fade out, but builds on each other into a final composition. This can form paintings similar to Mondrian perhaps if it is a show of similar works of that era or it can form something much more metaphorical and open interpretation of the space and content but utilizing a sense of emergence spatially in terms of the composition (pieces laid bare until final approach for effect).

Each section will be well designed, but they build in layers as one moves until finally forming the final composition both visually and in terms of scope of information or building immediacy. The effect can be akin to taking a painting and slicing it into onion skin layers laid out in the air at intervals, each the same dimensions, but only one section compositionally of the greater whole. This has many semiotic applications beyond its potential aesthetically and as spatialized information possessing a sense of inter-relationship as one moves.“

Tish Shute: One of the things I found very inspiring when I read your papers was that your ideas are not all dependent on a model of AR that would necessarily require goggles, back packs and lots of CPU/GPU – not that that wouldn’t be nice, but that even using “magic lens” AR of the kind smart phones has enabled in an open distributed framework would open up a lot of new possibilities for what you call modulated mapping wouldn’t it?  What kind of social augmented realities might be enabled by a distributed infrastructure like this [AR Wave]?

Jeremy Hight: right….I see that as wayyy down the road…most important is the one you talk about as it is more immediate and thus more essential and needed. Eventually the goggles will be like a contact lens and a deep immersive ar version of  this will come, that to me is certain, but a ways down the road.  An incredible amount is possible now, and this is a more pragmatic move as opposed to the more theoretical of what is a few steps from here. Thus it is more important and essential now. Tools like Google Wave are taking what even 2 years ago was more theoretical discussions of what may be and instead introducing key elements to a more immediate, powerful, flexible level of augmentation. What have been hacks and isolated elements are to be integrated and social networking, task completion, shared tools and graphics building and geo-location.

Tish Shute: I think some people question what augmented reality has to bring to the continuum of location based experiences that other forms of interface/mapping do not?

Jeremy Hight: right….and the schism between its commercial flat self and tests with physics etc and in between …there are a lot of unfortunate assumptions it seems as to where ar and lm cross and how ar can be many things beyond deep immersion or the opposite pole of a hockey puck having a magic purple line etc….like lm is seen as either car directions or situationist experiments with deep data…..the progression to me is deeply organic….and now augmentation can be more malleable, variable and end user controlled.

Tish Shute: Yes, it is really exciting time for AR.  Historically AR research has gone after the hard problems of image recognition, tracking and registration because we have had available to us these dynamic, real time, large scale architectures like Wave available (until now!),  so less work has been done on exploring the possibilities for distributed AR fully integrated with the internet and WWW hasn’t it?

A distributed augmented reality framework such as we have envisaged on Wave would  allow people to see many layers from many different people at the same time. ‬And this kind of model has been part of your thinking and fundamental to your work for a while, hasn’t it? But it is a very new idea to most people to think about collaboratively editing layers on the world, and to be able to view  augmented space through channels and networked communities?  Could you explain some of the ways you have explored these ideas and how they could be explored further now to create meaningful experiences for people?

Jeremy Hight: right..exactly…modulated mapping to me can be an amazing tool for students…back end searching data visualizations and augmentations based on their needs…while they do something else on their computer or iphone…that can be amazing..and not deep immersive..The map can be active, malleable, open source fed, and even, in a sense, intelligent and able to adapt. The possibility also exists for this map to have a function that based on key words will search databases on-line to find maps, animations, histories and stories etc to place within it for your study and engagement. The map is thus a platform and yet is active. Community is possible as people can communicate graphically in works placed on the map and in building mode in the tool. All the tropes of locative media are to be in a mapping system of channels of augmentation and a spatial net. The software by design will allow development on the map and communication like programs such as second life but in mapping itself.

image from Parsons Journal of Information Mapping Volume 2 (Hight/Wehby)

I wrote an essay a few years ago for the Sarai reader questioning the traditional map and its semiotics and need to reconsider – then did work looking into it and what those dynamics were and they got into 2 group shows in museums in Russia…so it actually was my arc toward modulated mapping…an interesting way to it! But yes the map itself..this is a huge area of potential and non screen based alone navigation etc. I see now that my 2 dozen or so essays in lm,ar, interface design and augmentation have all also been leading in this direction for about 10 years now

Tish Shute: I  love immersive visualization but can we “return to the map – the internet of data” as you mentioned earlier and produce interesting augmentation experiences that go beyond locative media’s device display mode without having the goggles, for example, through the magic lens of or smart phones?

Jeremy Hight: yes, absolutely.  the map in the older paradigm is an artifice born often of war and border dispute and not of the earth itself and its processes…the new mapping like google maps is malleable, can be open source, can read spaces and can be layers of info in the related space not plucked from it as in the past..this is amazing. the old map also was born of false semiotics/semantics like “discovery of new lands” or ” pioneer”  while the places were there already and names often were of empire…now this is no longer the case

Tish Shute: So geoAR is an a better way to express a new social relationship to mapping? And how does this fit into the evolving arc of locative media that evolves into augmented reality?

Jeremy Hight:…early lm was mostly geocaching and drawing with gps..it took new paradigms to invigorate the field  a lot of folks focus on tools and what already is, cross pollination can ground ideas that are more radical…a metaphor in a sense to place what can be in a familiar context.

Tish Shute: one of the great disappointments in VR has been its isolation from networked computing and also, up to now, augmented reality – to achieve an immersive experience with  tight registration of media/graphics have to create separate system isolated from the internet and power of the web.

Jeremy Hight: yes….this will change. vr is to me an island but ar takes a part of it and shifts the paradigm and new things open this way. Do you know the project “life clipper”? friends of mine..doing interesting things..they are a clear bridge betwen lm and ar….and from vr

in ar augmentation and what is being augmented become fused or in collision or in complex interactions as a means to a larger contextualization and exploration of what is being augmented..this is true in immersive or non ar….huge potential

vr is a space, now can be surgery which is amazing. but not layered interaction, thus an island and graphic iconography on a location can use symbolic icons which opens up even more layers (graphic designer/information designer in me talking there I suppose..)

Tish Shute: Yes !  talk to me more about layers and channels I think this is one of the most interesting questions for me  in augmented reality at the moment – what can we do with layers and channels and the new possibilities on connections between people and environments that these can create?

The ability for anyone to post something is critical to the distributed idea but one of the reasons I am so excited by Google Wave is I am fascinated by the playback function. How do you think this will enable new forms of collaborative locative narratives (nice post on Wave playback here ).

Jeremy Hight: We are in an age of cartographic awareness unseen in hundreds of years. When was the last time that new mapping tools were sold in chain stores and installed in most vehicles? When was the last time that also the augmentation of maps was done by millions (Google map hacks, etc)? The ubiquitous gps maps run in automobiles while people post pictures and graphic pins to denote specific places on on-line maps.

The need is for a tool that combines all of these new elements into an open source, intuitive layered and rhizomatic map that is porous (like pumice, organic in form yet with “breathing room” ),ventilated (i.e: adjustable, a flow in and out), and open (open source,open access,open spatialized dialog).

I wrote of this in my essay “Revising the Map: Modulated Mapping and the Spatial Interface .”( http://piim.newschool.edu/journal/issues/2009/02/pdfs/ParsonsJournalForInformationMapping_Hight-Jeremy.pdf )

image from Parsons Journal of Information Mapping (Hight/Wehby)

Tish Shute: One mapping project I really like is Mannahatta.  How could distributed AR contribute to a project like Mannahatta?

Jeremy Hight: that is a good example..imagine taking manhattan and having channels of options to overlay, that being an excellent option, and imagine being able to even run a few at once with deliniating icons..you can augment a space with history, data, erasure, narrative, scientific analysis, time line of architecture, infrastructure, archaeological record etc….endless possibilities, and this agitates place and place on a map into an active field of information with end user control…and open options for new layers

Tish Shute: and do you think we could do interesting things with AR on a project like Mannahatta even with the current mediating devices we have available – i.e. our smart phones as obviously the rich pc experience of Mannhatta has built for it’s web interface would not be available as AR at this point?

Jeremy Hight: yes….k.i.s.s right?   these projects do not have to only be immersive and graphic intensive……take how people upload photos onto google maps….just make that on a menu of options, there are some pretty cool hacks already..
…options is key, a space can have a community as well, building on it in software, and others navigating it, i see it near future and down the road..always have with ar really

image from Volume Magazine (Hight/Wehby)

Jeremy Hight: and yes, a lot of people focus on ar as its limitations and processing power needs as a major road block

Tish Shute: so do you see AR on smart phones adding any value to a project like Mannahatta?

Jeremy Hight: yes…that it can be integrated into other similar works and even disparate but cloud linked ones…so a place can be “read” in diff ways on the iphone….beyond its map location, and more can be possible if you are there…others away, so it becomes channels of augmentation

Tish Shute: AR like locative media puts who you are, where you are, what you are doing, what is around you center stage in online experience but it also “puts media out in the world” – people I think understand this well as a single user experience but we are only just beginning to think about how this will manifest as a social experience – could explain more about modulated mapping as an experience of social augmentation?

Jeremy Hight: Modulated Mapping is a tool that will allow channels to be run along the map itself. This will allow one to view different icons and augmentations both as systems on the map and in deeper layers of information (photos, videos, animations,  visualizations, etc) that can be turned on and off as desired. The different layers of icons and data may be history, dissent, artworks, spatialized narratives, and annotations developed that are communally based on shared interests, placed spatially and far beyond. The use of chat functionality in text or audio will be open in building mode and in mapping navigation/usage as desired. This also allows a community to develop or augment in the spaces on the earth. These nodes can be larger and open or small and set by groups in their channel. The end result is an open source sense of mapping that will also have a needed sense of user control as one can select which layers of augmentation they wish to see and interact with at any time. It also will incorporate all the functionality of locative media in mapping software and mapping. In building mode and in map mode, icons will be coded to represent within channels (remember that the person using it has selected channels of augmentation from many based on their current interests and needs). Icons will be coded as active to show work in progress in cities and the globe to both invite participation and to further agitate the map from the sense of the static as action is visible even with its icons as people are working and community is formed in common interest/need .

locative media got a buzz for “reading” places…when I helped create locative narrative that was what blew me away back in 2001…that we could give places a voice by placing data from research and icons on a map……this meant lost history or augmentation was possible as kind of voices of a place and its layers…….I called it “narrative archaeology.” We now have tools that can push these ideas and concepts farther..much farther…and with a range beyond what was before, and then the map was just a tool….but now we are returning to the map itself…..and this as place as much as marker..this is where ar takes the ball to use a bad metaphor

also that project could only work if you came to our spot of a 4 block augmentation and with us there to lend you our gear…we are far beyond that now but it had its place

Tish Shute: How do you see “in context” AR and something we might call “context aware” cloud computing models interacting?

Jeremy Hight: sure…and I must add that I have issues with cloud computing as much as it is a good idea...

Tish Shute: because of loss of autonomy?

Jeremy Hight: tivo is simply a hard drive…but it keyword reads and gives  suggestions..that is the is cro magnon link to what can be

Tish Shute: The nice thing about Wave is because of the Federation model, the cloud model and local store ur own data models should work together.

Jeremy Hight: yes..that is better…..loss of autonomy also opens up the arbitrary which is the flaw of search engines as we know it…even Bing fails to me in that sense

Tish Shute: how do you mean, could you explain?

Jeremy Hight: spiders  cull from words  but cull like trawlers at sea …. tested Bing with very specific requests.. it spat out the same mass of mostly off topic results….
I wonder if there is a way to cull from key words and topics from a user…not Orwellian back end of course…but from their preferences, their searches etc..

Tish Shute: did you see the discussion on search in the AR Framework doc? AR search will be a massively important thing that will take a lot of intelligence and all sorts of algorithm development won’t it?

Jeremy Hight:It also has one area of key functionality that moves into more intuitive software. Upon continued usage, the mapping software will “learn” and search based on key words used and spheres of interest the user is mapping or observing as mapped and will integrate deeper data and types of animations, etc. into the map or will have them waiting to be integrated upon user approval as desired. Over time the level of sophistication of additions and of search intuition will increase dramatically. The search can also, if the user wishes, run in the back end while working in the mapping program, or in off time as selected while doing other tasks. It also can never be used if one is not interested. One of the key elements of this mapping is that it is not composed of a closed set or needs user hacks to augment, but instead is to evolve and deepen by user controls and desired as designed. Pre-existing data,visualizations and augmentations can be integrated with relative ease.

Tish Shute: One of the things that Joe Lamantia points out about social augmented experiences is that they will operate across a number of different scales – conversation > product design & build team > neighborhood / town fixing potholes > global community for causes. How do designs for channels and layers change across these different social scales?

Jeremy Hight: quote myself …”The “frontier” is often defined as the space just ahead of the known edge and limit, and where it may be pushed out deeper into the previously unknown. The frontier in the world of ideas is not the warm comfort of what has been long assimilated; and the frontier in the landscape is not of maps, but of places beyond and before them⁄

The border along what has been claimed is not only that of maps – it is of concepts, functions, inventions and related emergent industries. Ideas and innovations are like the cloud shape that briefly forms around a jet breaking the sound barrier, tangible yet not fully mapped into measure. It is when things are nailed down into specific entities, calibrated and assessed, that the dangers may inflict themselves – greed, competition, imitation, anger, jealously, a provincial sense of ownership either possessed or demanded”. (from essay in Sarai reader). Otherwise channels and augmentation do not have to be socio-economically stratifying or defined by them. We built 34n  for almost nothing on older tools.

image from 34north 118west  (Spellman/Hight/Knowlton)

The ar that is not deep immersion can be more readily available and channels can be what end users need like the diversity of chat rooms or range of Facebook users among us.

I had two moments yesterday that totally fit what we talked about.  I went to west hollywood book fair and traditional directions off of mapping for driving directions were wrong and we got lost…our friend could only get a wireless signal to map on itouch and we had to roam neighborhoods then we called a friend who google mapped it and we found we were a block away….so a fast geomapping overlay with an icon for the book fair on some optional grid service or community would have made it immediate.  Then at the book fair talked to a small press publisher who is trying to map works about los angeles by los angeles authors on a map..she was stunned when I told her it could be a kind of google map feature option

it also has great potential to publish and place writing and art in places..both for commentary and access. imagine reading joyce in chapters where it was written about and then another similar experience but with writers who published on a service into their city.

Tish Shute: The challenge of shared augmented realities is not just a matter of shipping bits around, but also of how it we will use channels and layars – to create and negotiate different, distributed perspectives, understand a shared common core/or expressions of dissent (this came up in an email conversation with Simon St Laurent).

Jeremy Hight: well my example earlier could have been communal in a way too..a tribe sort of augmentation channeling ….like subscribing to list servs back in the day but of augmentation communities/channels, and for folks to build and use in shared live form, coordinating too

Tish Shute: one good thing though about building an open AR Framework is that as bandwidth/CPU/hardware gets better shared high def immersive experiences could be supported by the same framework..

Jeremy Hight: excellent

Tish Shute: were you thinking of the image recognition and tracking with this example?

Jeremy Hight: yeah….like scanning across a multi channeled google map augmentation with diff icons and their connected data…and poss social networking and fle sharing even in that mode…and rastering etc….could be cool with google wave - on the map..then zooming in a la powers of ten..(eames film).

-I have pictured variations of this for a few years now in my head like the example of my friends and I yesterday…we could have correlated a destination by icons in diff channels..one being lit events within lit channel in l.a map…maybe things streaming on it too…remote info and video etc… that would be awesome

Tish Shute: So many of the ideas in you paper on modulated mapping (see here) are brilliant use cases for shared augmented realities. Perhaps you could talk more your ideas about locative narrative because this is something I think is at the core of the kinds of experiences that a distributed AR Framework would make possible?

Jeremy Hight: on the project “34 north 118 west” we mapped out a 4 block area for augmentation of sound files triggered by latitude and longitude on the gps grid and map and the map on the screen had pink rectangles that were the “hot spots” where the augmentation had been placed.

image of interactive map with map based augmentation connected to audio augmentation on site for 34north 118west (Spellman/Hight/Knowlton)

We researched the history of the area and placed moments in time of what had been there at specific locations ….I called this “narrative archaeology” as it allowed places to be “read” by their augmentations…info that was of the place beyond the immediate experience (diff types of info) that otherwise would be lost or only found in books or web sites elsewhere. there now are locative narratives around the world but they need to be linked.  from humble origins “narrative archaeology” went on to be recently named of the 4 primary texts in locative media which is pretty amazing to me…but it is growing

- the limitations then were what I called the “bowling alley connundrum” – the specifc data had to reset like pins…..and was isolated….this led me to think about ar back then and up to now.  How these could lead to much more from that point, data that would be more layered, variable , fluid..yet still augmented place and sense of place and social networking within data and software

lifeclipper to me is a bridge

Tish Shute: But Life Clipper is isolated from the internet currently is it?

Jeremy Hight: yes…ours was too.. that is what google wave makes possible.. our project only ran on our gear..in 4 blocks…with additional auxiliary info online, and not malleable..but hey 2001 and all..

Tish Shute: so the sites for 34 north 118 west are still active though?

Jeremy Hight: oh yeah!

Tish Shute: nice I really like sound augmentation – have you seen Soundwalk?

Jeremy Hight: yes, very cool.. we chose sound only as it fought the power of image..instead caused a person to be in a sense of two places and times at once

Tish Shute: and in 2001 that was definitely a visionary project!

You must be very excited that finally the pieces are coming together to make this stuff scale!

Jeremy Hight: I can’t even tell you!! it is funny..i have known that this would come..just waited and waited…

..knew it needed the right people and tools..

..so the bowling alley connundrum led me to develop my project shortlisted for the iss (international space station)  as I thought a lot about how points and works are not to be isolated…but connected  and should be flowing in diff parts of a map….to open up perspective and connected augmentations , but also to think about the map again…not as a base only. then moved into my work with new ways to visualize time and it all really began to gell.  The ideas first were published as an essay (http://www.fylkingen.se/hz/n8/hight.html) and later my project blog (http://floatingpointsspace.blogspot.com/)

Tish Shute: One thing I noticed when I was reading your paper is how you have been exploring non-euclidian geometries.  Could you explain how this is part of your idea of modulated mapping?

Jeremy Hight: Yes, this first came to me when my wife was reading to me from a book on the Poincare Conjecture and I was hit with a new way to measure events in time and after months of sketches, schematics and research came to see how it could also be connected to a geo-spatial web of projects and augmentations.  It was published in the inaugural issue of Parsons School of Design’s Journal of Information Mapping which was an exciting fit. I call it “Immersive Event Time”(http://piim.newschool.edu/journal/issues/2009/01/pdfs/ParsonsJournalForInformationMapping_Hight-Jeremy.pdf)

so the last 3 years I have been working on how it could all work as channels of augmentation, and building and navigation as open and community in a sense as well as ai capability that was the time work especially. how time as experienced within an event is not a time “line”  but points on and within a form….and how this model is better for visualizing events in time and documenting them. it actually sprang form reading a book on the poincare conjecture, popped a bunch of other stuff together so one could visualize an event in time as like being in the belly of a whale..with time as the ribs..and our measure of time as the skin…and moving within it….hoping this will be used as educational tool

and this also can be tied to ar and map again…how documentation of important events can be kept within icons on a google map..then download varying visualizations based on bandwidth and desired format

Tish Shute: I have been thinking about is the new forms of social interaction/agency that these kinds of augmentations of space/place/time will create.  it seems there are two poles – one is the area Natalie Jeremijenko explores of shifting social relations from institutions/statistics to real time/location based/interactions and new forms of social agency.  The other pole completely is more like the cloud based AI and perhaps crowd sourced machine learning.

Your ideas explore the possibilities of both these poles.  And certainly one of the big deals of distributed AR integrated with would be the possibilities it opened up both for new forms of networked social relationships and for new ways to draw on network effects.

Jeremy Hight: and cross pollinations within …that is what my mind goes to

Tish Shute: The other night I met Assaf Biderman, MIT, from the Trash Track team. Trash Track doesn’t utilize AR but I could see that there are possibilites there.
What do you think?

Jeremy Hight: yes, absolutely, there can sort of skins on locations that user end selection can yield …like channels of place….and can range from pragmatic core to art and play and places between….how this recalibrates the semiotics of map…more than just augmentation as seen as a kind of piggy back on map..map becomes interface and defanged platform if you wil, interestingly my more poetic/philosophic writing led me here too

Tish Shute: I know they are at very different poles of the system but I do wonder how AR can bring some of the level of social agency/interaction that Natalie Jeremijenko works on into a productive interaction with the kind of innovations in Machine learning that Dolores Lab style machine learning!!and others are pioneering?

Jeremy Hight: Natalie’s genius to me is in practical functional tech that also opens deeper questions and even new openings of what is needed..amazing layers in her work that way.. succint yet deep..very deep

Tish Shute: Yes – I a just writing a post about her work – I find it deeply moving the way she has delved into the possibilities to using technology to open us up to our world.  One of the reasons I find distributed AR so interesting is because it will make it possible for all kinds of people to create and use augmentation in their lives and communities.

So to return to how a distributed AR framework could contribute to a project like Trash Track?

Jeremy Hight: what about using it for community, dissent and awareness raising then?  like Natalie’s work but building like a communal work of multiple points, like the old adage of the elephant and the blind men  sorry..metaphor – like one of my points in immersive sight was how one could take augmentation as multiple works sort of turning the faces of a thing or place…and how this would make a larger work even in such a flow so people moving in a space could also build..

what of ar traces left as people move calibrated to user traffic and trash as estimated in an urban space…like it goes back to chris burden in the 70′s making you know that as you turn the turnstyle you are drilling into the foundation and may be the one that collapses the building?

so their movements leave trash. Natalie is all about raising awareness to cause and effect and data , space and ecology. love that.  so maybe …
a feedback loop , artifact and user end responsibility can leave traces …trash…

.. cybernetics vs ecology and human waste

Tish Shute: could you elaborate?

Jeremy Hight: brain fart…that the mass of trash people leave is a piece at a tiime….and how like the space shuttle mission when it was argued first true cybernaut occured….one cord to air for astronaut..one for computer on their back to fix broken bay arm…if there is a way to build on that and in relation to the topic…..how this can go further, that machines do not waste as much…as ar is a means to cybernetic raise awareness..eh..In a sense it is like  the space shuttle mission when arguably the first true cybernaut occurred….one cord to air for astronaut..one for computer on their back to fix broken bay arm…if there is a way to build on that and in relation to the topic…..how this can go further, that machines do not waste as much…as ar is a means to cybernetic raise awareness..eh.. hmmm... sensors etc…wearables too – could be eco awareness with data and machine and human

what about a cloud computing system with a slight ai in the sense of intuitive word cloud and interest scans…..so as one moves through say new york they can be offered new ai data and services as they move ? could also be of eco interests? concerns about urban farming, eco waste, air pollution etc….perhaps with (jeremijenko element here)  sensors placed in locations and these also giving data reads in public areas  with no input but hard data itself……hmm..could be interesting

it can also give info of the carbon footprints (estimated prob unless data is public record somehow) of chain businesses  and data on which are more eco friendly as well as an iconography color coded and icon coded to the best places to go to support greening and eco friendly business?  and the companies could promote themselves on this service to attract eco aware customers who would be seeing them as kindred spirits and helping the
larger effort?

kind of eco mapping..and ar on mobile app

what about sensors that read air pollution levels, levels of solar radiation (to aid with skin protection in shifting light values in a city space..ie put on some skin cream now…), light sensors that detect density and over density in public spaces…to use the old trope in art of reading crowds in a space..but instead could indicate overcrowding, failing infrastructure in public spaces (which is a congestion that leads to greater pollution levels as well as flaws in city planning over time..), and perhaps a tie in to wearables……worn sensors  on smart clothes….this could form a node network of people in the crowds ….and also send data within moving in a space…

here is a kooky thought… what of taking the computing power and data of people moving in a space..and not only get eco data and make available to them levels of
data..but make possibly a roving super computer…crunching the deeper data of people open to this……a hive crunching deeper analysis of the space, scan properties from sensors, and even a game theory esque algorithm of meta data if say 40 people out of 50 hit on a certain spike or reading…and even their input…..I worked in game theory for paleontology in this manner for a time as a teen….a private project……   the reading can lead to a sort of meta read by what hits most consistently..as well as in their input..text of what they experienced, observed,postulated,analyzed even…. this could be really interesting…even if just the last part from collected data and not from any complex branching of servers..

I thought at 19 or so that the flaw in paleontology was in how so many larger theories were shifting exhibitions and larger senses of things like were there pre-historic birds that were mistaken for amphibean and then back again….so why not make a computer program and feed all the papers published into it and see what hits were counted in terms of an emerging meta theory…and landscape of key points being agreed upon…this data would be in a sense both algorithmic and a sort of unspoken dialogue …came from a lot of study of game theory one summer…

hope this makes some sense…I forgot to mention that I originally planned to be a research meteorologist and my plan in middle school or so was to get a phd and develop new software to have a global map and then run models of hypothetical storms across it in real time animations of cloud forms, radar and wind analysis/fields, barometric pressure spaghetti charts etc….and to also do 3d cut away models of storm architectures…so been into visualizations of complex data and mapping for a long time!

Tish Shute: Wow let me think about this one!

I first started talking to Robert Rice, CEO of Neogence Enterprises, Chairman of the AR Consortium, in 2008.  Robert was already actively working on creating the world’s first global augmented reality network.  But it took a few months before what Robert had said to me about impending explosion of  augmented reality into our lives really sunk in – “this is going to be much bigger than the Web!,” he extolled.

By January, 2009 I was convinced and I posted my first interview with Robert, “Is it OMG Finally for Augmented Reality?..” As I mentioned in the intro, I had recently tried out Wikitude and Nathan Freitas’s grafitti app on the streets of New York City and I was impressed.  Now, 7 months later, Augmented Reality has  not disappointed and there is an explosion of new applications, and the arrival of some of first commercial and practical toolsets, SDKs, and APIs for aspiring developers.

For more on this see my previous post, Augmented Reality’s Growth is Exponential: Ogmento – “Reality Reinvented,” talking with Ori Inbar, which is an introduction to my series of interviews with the key players in augmented reality and founding members of the ARConsortium – Int13, Metaio, Mobilizy, Neogence Enterprises, Ogmento, SPRXmobile, Tonchidot, and Total Immersion.

As I mentioned before, Maarten Lens-FitzGerald of SPRXmobile told me the other day that my first Interview with Robert Rice, in January of this year, was a key inspiration for SPRXmobile to get started on the development of Layar – a Mobile Augmented Reality Browser. Much more on Layar and Wikitude – world browser in my upcoming interviews with Maarten Lens-FitzGerald and Mark A. M. Kramer, respectively.

Recently, both Layar and Wikitude earned a mention in the white paper by Tim O’Reilly and John Battelle, Web Squared: Web 2.0 Five Years On. Web Squared is essential reading not only because it covers the underlying technological shifts of “Web Meets World,” which augmented reality is a vital part of;  but, crucially, Web Squared focuses on how there is a new opportunity for us all:

“The new direction for the Web, its collision course with the physical world, opens enormous new possibilities for business, and enormous new possibilities to make a difference on the world’s most pressing problems.”

I am currently working on a post on Green Tech AR which is one of the areas augmented reality can play an important role “in solving the world’s most pressing problems.” Augmented Reality has a lot to offer Green Tech development.  As Gavin Starks of AMEE said at Euro Foo in 2006, “climate change would be much easier to solve if you could see CO2.”

But really useful Green Tech AR requires still hard to do markerless object recognition (going beyond feature tracking and modified marker recognition), and a tight alignment of media/graphics with physical objects, in addition to a quite a high level of instrumentation of the physical world.  And for Green Tech AR to really shine, we are going to need innovators like Robert Rice who are working on, and solving, multiple really hard problems like:

“privacy, media persistence, spam, creating UI conventions, security, tagging and annotation standards, contextual search, intelligent agents, seamless integration and access of external sensors or data sources, telecom fragmentation, privilege and trust systems, and a variety of others.”

Recently Robert Rice presented at MoMo Amsterdam. Here is a drawing of him in action (picture below from wilgengebroed‘s Flickr Stream).

In his Twitter feed Robert Rice ( @RobertRice ) Robert reminds us: “By the way folks, what you see out there now as “augmented reality” is not what it is going to be in two years.”   Robert plans to show the first public demo of his “platform for platforms” at  ISMAR 2009.

Robert is writing up a series of White Papers currently.  I got a preview of the first, “The Future of Mobile – Ubiquitous Computing and Augmented Reality.”  Robert points out, “AR through the lens of the mobile industry and ubiquitous computing is almost overwhelming compared to AR as marker based marketing campaign.”

I asked Robert, “What are the key take-aways for investors interested in the augmented reality field at the moment:

“First, Mobile AR is going to be bigger than the web. Second, it is going to affect nearly every industry and aspect of life. Third, the emerging sector needs aggressive investment with long term returns. Get rich quick start ups in this space will blow through money and ultimately fail. We need smart VCs to jump in now and do it right. Fourth, AR has the potential to create a few hundred thousand jobs and entirely new professions. You want to kick start the economy or relive the golden days of 1990s innovation? Mobile AR is it.

Don’t be misguided by the gimmicky marketing applications now. Look ahead, and pay attention to what the visionaries are talking about right now. Find the right idea, help build the team, fund them, and then sit back and watch the world change. Also, AR has long term implications for smart cities, green tech, education, entertainment, and global industry. This is serious business, but it has to be done right. I’m more than happy to talk to any venture capitalist, angel investor, or company executive that wants to get a handle on what is out there, what is coming, and what the potential is. Understanding these is the first step to leveraging them for a competitive edge and building a new industry. Lastly, AR is not the same as last decade’s VR.”

Talking with Robert Rice

Picture of Robert Rice at MoMo from Guido van Nispen‘s Flickr Stream

Tish Shute: So perhaps we better start with an update on state of play with Neogence?

Robert Rice: Neogence is doing well actually. We don’t talk much about the fact that we are still a small startup and we face a lot of the usual obstacles related to that and being a small team. Fundraising has been extra difficult, mostly because people are just now beginning to see the potential in AR, but that is still colored by perceptions based on a lot of the gimmicky AR ad campaigns out there. Still, it is better than it was two years ago the idea of an AR startup was a bit of a joke to a lot of VCs we talked to. However, we do have an agreement from a new venture fund in Europe (which we can’t talk about yet) for our first round of funding, but we don’t expect to close that for several months.

If all goes well, we hope to debut our first public demo at ISMAR 2009 in Orlando to select individuals and a few press folks. We might release a few viral videos before then that are conceptual and about what we are building in the long run, but that depends on how things go over the next several weeks.

We are also very active in looking for and building strategic partnerships and relationships with other companies, and this is not restricted to the augmented reality or mobile sector. As I have said before, we are looking at this as a long term business venture and the industry as something that will be bigger than the web itself within ten years. We are doing typical contract work and custom AR solutions to keep the cash flow going and build up the corporate resume a bit. So, if you want something done, and better than the stuff you are seeing now with all of the generic “look at our brand in AR with markers and a webcam” you should definitely give us a call.

Tish Shute: Just to clarify because most of the recent press has been about browser type AR like Wikitude and Layar which are not in the purist sense AR ‘cos they do not have graphics tightly linked to physical world. Neogence, if I am correct, is focused on building a true AR platform in the sense I just described?

Robert Rice: Hrm, I have argued with a few others about the actual definition of AR. Some people prefer a narrow and limiting view (3D overlaid on video), but I think in terms of the market and the end-user, it is better to have a wider definition. In that sense, AR is purely the blend of real and virtual, with or without full 3D overlaid on video. If we go with that, then Wikitude, Layar, Sekai, NRU, and others all fit into the AR definition.

Anyway, you are correct. We are building a true platform for AR, and this is quite different from what others are marketing as AR browser “platforms.”

There are a few problems with the “AR Browsers” approach that no one seems to be noticing. One is that they are all trying to get people to build new applications for their browsers, when they should be trying to get people to create content that they can share and browse.

Second, someone using Layar is not going to see anything that is designed for Sekai or Wikitude.

Third the experiences are generally for one user. While I love all of these guys and think each of the teams has some real talent on it, the model is flawed until someone using Wikitude can see the same thing that someone using Layar or Sekai camera is seeing (provided they are in the same physical location).

While we are working on our own client side technologies that we hope will be useful and integrated with every mobile device and AR browser out there, our core focus is on connecting everything and everyone together, and facilitating the growth of the industry with the tools to create content, applications, and so forth. We want to solve the really difficult technical problems (some of which most people haven’t even considered yet, because of the perspective they are looking at the potential of AR with), and make it easy for everyone else to do the cool stuff. We want to be the facilitators.

If you really want an idea of where we are going or some of what has inspired us, you have GOT to read Dream Park, Rainbows End, and The Diamond Age. If you have heard me speak anywhere or read my blog, you know that I am continually suggesting these and others.

Anyway, short answer, yes, we are building a true platform for ubiquitous mobile augmented reality, and we are absolutely the first to be doing so. I hope to demo some of this in October at ISMAR, with a full commercial launch next year (10/10/10 at 1010am Hehe, seriously). We will probably launch a website soon for people to start signing up and building a community now (especially if you want in on the beta testing of the whole kibosh).

Tish: So just to clarify,  how will Neogence’s approach differ and fit into the  growing world of Augmented Reality tools that we have now, e.g.,  ARTookit, Imagination, Unifeye?

Robert: I guess you could say that we are trying to build the infrastructure for the global augmented reality network. This could be viewed as a service, or even a platform for platforms. If Neogence does its job right, anything you create using ARtoolkit, Unifeye, or Imagination would be applications you could ultimately link to, integrate with, or deploy on or through, what we are building, and not be tied to a specific set of hardware, browser, or walled garden.

Tish: You mention Neogence is going to provide a platform for platforms. Without knowing the details that sounds like a lot of centralization which prompts the inevitable question: “Who owns the data?” Do you think other AR applications or providers would resist a “Platform for Platforms?” I know the potential centralization power of Google Wave has already got people talking about these issues (one of the comments in my recent blog post was about how Google Wave protocol may be interesting for a least some parts of augmented reality communication).

Robert: It really depends on perception and how we end up building it. We aren’t talking about creating a closed system. As far as who owns the data, it depends on what data we are talking about. For the most part, I think that if the end-user creates something, they should own it and have control over it. They should also be able to do what they want with it, independent of everything else.

This is one thing that proponents of the smart cloud and the thin/dumb client don’t like to talk about. It sounds great on paper, but when you start thinking about it, all that does is strip away power from the end user. Case in point…Amazon recently wiped every copy of George Orwell’s 1984 from all Kindle devices. They claimed they didn’t have rights to distribute/publish it and it was available on accident. The scary thing though, is that they literally went into every kindle out there, found copies, and deleted them.

How would you like it if Microsoft suddenly decided to delete every copy of Microsoft Office? Or every file that had a .doc extension? That is a huge violation…we feel like we own what is on our computers. But with the whole cloud thing, your data is at the mercy of whoever is running the cloud servers. No privacy, no ownership, no control. And if the system breaks, all you will have is a pretty dumb device that can’t do much on its own. Now, that isn’t to say that the technical merits and benefits of a cloud model aren’t worth pursuing, they are.

But I think there needs to be some hybrid model. Don’t dumb down my computer or my smart phone, let’s keep pushing how much these devices can do. We should take full advantage of centralized and distributed systems, but in a hybrid mashup sense. That is what we are pursuing with our AR platform, while trying to protect ownership and intellectual property rights of the end user.

Tish: Earlier today I was telling you how impressed I was by Google Wave – it is quite mind blowing to experience massively multiplayer real time interaction on what will be an open internet wide platform – Wave is breaking new ground here and more than one person has mentioned its potential role in AR to me (see the comments to my recent post on Ogmento).

I know you are a strong advocate of this kind of real time shared experience being part of AR.  But we are only just beginning to see it emerge via Wave on the existing web – what will it take to have this kind of real time shared experience in AR!  We got briefly into the thick client, thin client, cloud versus P2P discussions – what is your approach to delivering a massively shared real time experience that is like Wave not confined to a walled garden?

Robert: I’m not a fan of any of those models as being stand alone or mutually exclusive. Again, the hybrid model with the best of both worlds is key. In the early stages of the emerging industry, you are likely to see some walled gardens (or perhaps a walled garden of walled gardens…).

No one knows how things are going to turn out in the next five to ten years and few people are thinking about it actively. For us though, I favor Alan Kay’s quote (pardon the paraphrasing): “To accurately predict the future, invent it”. That’s what we are doing. In the short term, there will be plenty of experimentation in the industry and a lot of model testing.

Tish: Do you think though Wave protocols might be useful as at least part of the picture for AR standards?  As you point out open standards and open protocols are going to be vital for shared experiences of AR.  Is it important to build off existing protocols to get the ball rolling and what do you see as being the important early protocols for AR?

Robert: I think for now, we will use a lot of existing protocols for communications and whatnot, as well as the usual standards for things like 3D models, animation, and so forth. This is only natural. However, as the industry and technology evolves, we will need entirely new ones. As far as I know there is no existing market standard for anything like the Holographic Doctor from Star Trek Voyager, and that type of thing is definitely in the pipeline for the future (sooner than you would think).

Tish: All the excitement at the arrival of the browser like mobile reality developments has been really great – I feel people are getting a taste for what it means to compute with anyone/anything, anywhere and and anytime.

Wikitude started the ball rolling. And with Wikitude.me it is the first to support user generated content. Now there is Layar, Sekai Camera also. But as you mentioned to me in an earlier chat, with Layar and Wikitude opening up “their are probably half dozen other apps coming out in short order with similar functionality (even the AR twitter thing has some similarities).”

What has been most exciting to you about these developments up to this point? What will these apps/platforms need to do to stand out in a crowd.  Up to now, these browser like AR experiences do nothing with close by objects. Do you see “world browsers” with near object recognition coming out in the near future. Could Wikitude do this with an integration of SRengine or Imagination?

Robert: Yes, Wikitude or Layar could do this (integrate with something else for “near” AR) and it would be a step in the right direction. Tagging things in the real world is the basic functionality that will grow from text tags to photos, videos, 3D objects, and all sorts of other types of data and meta data. This gets really fun when that data is generated by the object itself. First is just giving people the ability to tag something and share that tag with their friends, everything else grows from that. This sort of functionality is probably the most exciting in terms of near future advancement.

However, I think the idea of a stand-alone browser platform is a bit awkward…unless you also consider firefox a website browser platform. After all, you can create widgets (applications) for it. Anyway, the point is having access to the same data…if you put three people in a room, one for each browser, they should see and experience the same content, although the interface might be different (based on what browser and of course which hardware they are using). This means there needs to be some communication between whatever servers they are storing their data on (meaning, user tags) and some standard for how those tags are created.

Of course, if all they are doing is grabbing the GPS coordinates of the nearest subway station and telling you how far it is and in what direction, then they should all be able to see the same thing, regardless of the platform. But then, that isn’t really interesting is it? I could get the same info on a laptop with google maps.

This is part of the problem right now though…no one seems to be thinking about the bigger picture much. All of the effort is either on making the next cool ad campaign for a car or a movie, or creating a tool to tell you where the nearest thingamajig is, but in a really cool fashion on a mobile device.

No one is talking much about filtering data, privilege systems, standards, third party tools, interoperability, and so on. There is also little conversation about where hardware is going. Right now everyone is developing software based on what hardware is available. This needs to change where hardware is being developed to take advantage of new software coming out (this happened in the PC industry a while back and growth accelerated dramatically).

These are some of the reasons why I led the effort to start the AR Consortium. We brought CEOs from 8 different AR companies and startups together to start talking about these issues. We are still getting organized and have plans to expand the membership to other companies, but we want to do this right and we aren’t rushing things. The important thing is that we have started and there is at least a line of communication open now, where there wasn’t before.

I would expect to see the early movers expanding what they offer very soon, and they will probably lead the way in the short term. Definitely keep an eye on the companies involved in the AR Consortium. There are lots of very smart and motivated people there, and they are far ahead of all the experimental dabbling in AR we are beginning to see on youtube, twitter, and elsewhere.

Tish: When we had a discussion about what were the basics for an AR platform and an AR browser earlier, you talked about the difference between tools, a platform, and a AR browser – like Wikitude and Layar which should be about features/functionality e.g. to create treasure hunts AR geocaching, invisible AR yellow sticky notes you can leave at restaurants you don’t like, etc. Also you noted it should let you explore (browse) multiple formats, and open content content for AR – any data, information, or media that is linked to something in the real world and the visualization/interaction with the same.

Wikitude is a stepping stone to a true browser by your definition. But are we also seeing what you would define as an AR platform emerging – Unifeye, Wikitude (you can recap your definition if you like too)?

I think Wikitude hopes to provide the lego blocks for  augmented reality readers, browsers, applications, tools, and  platforms?

Robert: I expect some segmentation among the various AR companies that are out now, as they find their individual strengths and focus on them. Some will emphasize the client software (the browser), others will develop robust tools for creating content, SDKs/APIs will advance and facilitate rapid development of applications, etc. Neogence is ultimately working on the glue in the middle that ties everything together, makes it massively multiuser, persistent, and ubiquitous. Things like Unity3D have the potential to fill a need in the middleware space.

Tish: I know Blair McIntyre (see my interview with Blair here) and others are using Unity3D as an AR client, Could Unity3D become increasingly important?

Robert: It has the potential to become a favored middleware for providing the rendering layer. It already works nicely in regular browsers, and on several mobile platforms. Why code all the graphics rendering stuff from scratch when you can just license something and extend its features with AR functionality?

Tish: Now to ask your own question back to you! There seems to be a lot of reason to think that, eventually, there will be the kind of access to the iphone video API that augmented reality really requires and by that I mean more than we will get with OS 3.1 which is rumored to deliver only about half of what we really need for AR on the iphone – “not truly useful when you want to align video. with graphics.”  So:

“The iphone…future or failure? Seemingly anti-developer stance regarding augmented reality, and only a sliver of the global market share. Are we letting the short term glitz of Apple and the iPhone fad pull us in the wrong direction? Shouldnt we be focusing on symbian devices that have the lion’s share of the market? or should we be looking more at either other OSs (winmobile, android) or not at all and trying to create a new platform that is more MID and less smart phone with a hardware partner?”

Robert: Apple and the iphone are a bit problematic right now. There is no way I can go to a venture capitalist (at least in North America) and say hey we are building awesome AR applications for winmobile or symbian…they would either laugh or they simply wouldn’t get it. There is this false perception that the iphone is the ultimate mobile device, it is the sexiest, and the only thing that people want. Everyone wants a demo on the iphone, the media is mostly interested in iphone developments, and the apple fanatic market could give a fig about other devices. Other devices may have a larger market share or even better hardware, but we have to focus on the iphone right now at least in the demo stage to get any market attention and traction worth the time and effort.

In the future though, unless Apple changes its stance with their SDK and APIs, and starts adding hardware that is key for mobile AR (beyond what is there now), the market will move on without them. This is a really easy decision to make given Apple’s draconian policies and the fact that their percentage of the global market is miniscule. The smart companies are looking at the whole picture and not putting all of their eggs in the Apple basket.

Of course, once the wearable displays are commercially viable everything changes. Wearable computers with small screens or even no screens are going to be what everyone wants. The interface will go from handheld touch screens to virtual holographic interfaces that you interact with using your bare hands.

So for now, (the immediate short term), its all about the iphone. Taking mobile ubiquitous AR to the global market and building for the future will be based on something else. Hardware risks becoming a commodity or a closed platform. Do you really want to buy the Apple iGlasses and only see AR content that is compatible, where your best friend has a pair of WinGlasses and sees something entirely different? No. The hardware, and the client software (what people are calling the ar browser now) will become common and it won’t matter what brand you use, they will all be accessing the same content.

But at least for the forseeable future, we are building software for specific hardware, and the sexiest mobile on the block is the iphone. The second someone comes out with something much better and the paradigm shifts (software driving hardware instead of vice versa) everything changes.

Tish: How is the quest for sexy AR eyewear going.  I know we were checking out the Japanese eyewear with Adam Johnson from Genkii just now.  For the Neogence project – as you are going for a fully developed model of AR doesn’t this necessitate going beyond the iphone and getting the hardware companies moving on the eyewear?

Robert: The guys making wearable displays really need to get off the pot and stop paying lip service to mobile AR. If they don’t do something quick, I,  and others, are going to be scouring the planet looking for someone capable of building the lightweight stylish wearable displays with transparent lenses we are begging for. We aren’t going to be waiting around for hardware anymore. The AR Pandora’s box has been opened. I should note that many of us (AR Consortium members) have had less than pleasant experiences or communications with the half dozen companies or so that are making wearable displays. Either their visual design is terrible, the materials feel flimsy, the field of view is limited, or the companies are preoccupied with other business and government contracts. Any attention to the growing AR market is an afterthought and in a few cases condescending. AR is going to be a billion dollar industry in a very short time, and these guys are just leaving money on the table. If they were smart, they would be begging the CEOs from the AR Consortium to fly out to their offices and collaborate on building a pair of wicked sick glasses. The smart phone manufacturers should be doing the same thing, but I have to say that they at least seem to have some ambition and zeal to create better devices, so I can’t really complain too much there.

Anyway, to answer the rest of your question, we have to assume that the hardware guys, especially regarding the eyewear, is going to take a long time to develop and release the things we need for the ultimate AR experience. So, our goal is to start building things now for what is available. That means scaling things down and handicapping what AR can do, so it works on the “sexy” iphone. The important thing though is to start creating applications -now- so when the glasses are commercially available, there will be a wealth of content for people to access and use on day one.

As long as Apple isn’t playing nice, it is going to hurt everyone. Is it any surprise that they shut down Google Voice? There is a huge opportunity for someone to step up and leapfrog the rest of the industry. Give us the hardware and we will create amazing software for it. Don’t compete with the iphone, surpass it.

Tish: What is the state of play of current AR technology and toolkits?

Robert: The current crop of AR technology and toolkits is absolutely critical for this stage of the industry, and everyone should be leveraging it as much as possible. I talk down marker and image based tracking a lot, but I also like to point out that it is the necessary baseline that the industry is going to be built on. The problem is that there is only so much you can do with marker driven apps, and as creative people and marketing types start conceptualizing about all sorts of cool stuff for the future, they risk setting the expectations too high. It is one thing to show someone the future, it is another to say this is the future and its happening right now. This is why I cringe everytime I see a conceptual video presented as “our product DOES this” instead of “our product WILL DO this.” Something that simple can still cause the butterfly effect of raising expectations too high and contribute to overhyping.

Tish: One of the things that seems very exciting about the new Ogmento partnership is that experienced content producers  Brad Foxhoven and Brian Selzer from Blockade are now taking a leading role in AR.  What are the most exciting directions for content that you see emerging for AR in the next 12 months?

Robert: Virtual (well, augmented) pets, and multiuser mobile AR games (2-4 people) are probably going to lead in the next 12 months for content. Easy, accessible, engaging.

Tish: And are you at Neogence also involved in content partnerships?

Robert: Yes, we are in the process of finalizing some content partnerships with an eye for long term relationships. We are specifically looking for partners that want to find substantive ways to leverage AR technology, and not use it as a superficial gimmick or attraction that wears off after five minutes. I’m still cringing over the Proctor & Gamble Always campaign with AR.

Tish: So back to your observation about some of the tricky problems re creating a true global massively multiuser, ubiquitous, mobile AR platform – what are some of the main obstacles to this mission in our view? (aside from getting investment!)

Robert: Trying to explain it to people. The technical problems we can handle or have already solved. But trying to communicate what exactly we are doing is still tough. Not because it is overly complicated, but rather because it is so new and different. People are having a hard time grasping augmented reality beyond marker/webcam.

Tish: Which AR tools are most important right now?

Robert: Content is critical right now to show what the technology is capable of and to continue building the presence of augmented reality in the public mind the big benefit to integrated / unified platforms now is speed of development for content. I think that the flash artoolkit = papervision is rocking the planet right now. It is accessible, easy to learn, and lets people create something very quickly. More tools and middleware are coming out and this increases options for designers and developers.

Tish: What are your favorite papervision apps?

Robert: Hrm, I don’t have a favorite papervision app just yet, although I think the tech is solid. I expect to see a lot of stuff built on that platform in the near future. Especially as more ad agencies get on the bandwagon and start telling their IT guys to learn how to program flash so they can make something. Have you seen www.ronaldchevalier.com Not so much for the actual AR stuff, but because the whole thing is just brilliant. Its exactly like some cult figure spiritual guru would do with AR. I wish I had thought of it first actually. This is probably one of the best -seamless- implementations of AR in marketing where it fits…it isn’t just jammed in there for the sake of saying they used AR.

Tish: Do you think Apple is going open the iphone to the full potential of augmented reality anytime soon – a lot of expectations have been raised?

Robert: Apple is like that guy has a party at his house and owns this really awesome state of the art home theater in his basement, but makes everyone watch a movie in the living room on a regular TV with a VCR.

They need to get over themselves and quit being a wet blanket. Otherwise, we are taking the beer and pizza we brought, and going to someone else’s house. Sorry, the Apple thing is a bit of a sore point with me.

Tish: But will people leave all that candy and soda at the appstore?

Robert: I tell you what though, there is an opportunity for certain mobile phone manufacturers to give me a call and start talking to Neogence and the other members of the Consortium. We have some ideas and specs that could have a radical impact on the mobile market and stuff the IPhone in a box. Hint hint.

Tish: So what is your vision for the ARconsortium.  I know it kicked off with a letter to Apple about the video API.  What is the next step? There was a lot of hope that this year would be big for MIDs but this really hasn’t happened yet – do you think there is hope for a MID take off despite the lousy economy?)

Robert: MIDs? No, not yet. smart phones are too lucrative and too hot. It isn’t time yet for the MID to go mainstream. For that to happen, there needs to be a driving need (cough ubiquitous AR cough)

The AR consortium is mostly an informal affiliation. I expect that representatives from each member will probably meet at every significant conference to catch up over drinks. We are also going to be planning for our own members conference at least once a year. That will happen after we expand the membership though.

The main idea behind the consortium though was to open up a channel of communication between the CEOs so we could work together on standards, solving problems, collaborating, forming some partnerships, and using the collective to bang on the doors of companies like Apple and others. There is power in a group.

Tish: You mentioned there is a whole long conversation we can have about getting the eyewear.  As you point out true AR eyewear changes everything.  Can give a little road map of where this has to go?

Robert: There are essentially four or five main approaches, depending on whether or not you make the lenses special or if they are just plain. You would normally want them to be plain so people with prescription lenses wouldn’t have problems and would have the option to switch them out. Some types use a more prismatic approach for top down projection, or a corner piece mounts lasers and bounces them off the lens into the eye.  Another approach is embedding OLEDs or something else into the lenses themselves.

I really like the Lumus approach, but their product design isn’t quite there yet. If the wearables don’t look cool, people won’t use them. To be honest, if I had the money, I’d probably ask the Art Lebedev guys to design them based on someone else’s optical engineering. They designed the optimus maximus old keyboard…   brilliant industrial designers, loaded with engineers too. If these guys couldn’t build the glasses and make them look damn bad ass, I’d be shocked. Heck, I bet they could build the next gen MID while they were at it.

Tish: Getting the hardware innovation and software innovation feeding into each other would be really great.

Robert: Absolutely.

Tish: That would push the eyewear forward too wouldn’t it?

Robert: All it takes is one, and then the competitive landscape would fire right up.

Tish: What applications would the accurate gps enable?

Robert: Everything. for example, you know exactly where the phone is and where it is facing, that means you can put it on a table and hit a button, then move it somewhere else and do the same thing in a few minutes, you have a nearly accurate “mental” model of the whole place now you go back and start dropping virtual flower pots everywhere.

This is one area where I think the smart phone guys are missing the boat and taking the cheap route. It is possible to have very accurate GPS (down to a six inch area) with better chips and firmware, but it is cheaper to stick in old tech. Most apps today dont need that hyper accuracy, so they aren’t bothering. Mobile AR though, thats a different story.

With that level of accuracy, you would know exactly where the mobile device is, so all you would need to know is the direction it is facing (orientation), and you could solve one of the problems with registering exactly where 3D objects and augmented media is (it is more complicated than I am describing it, but we don’t need to get into that much detail here). You wouldn’t need markers anymore.

Tish: Isn’t Wikitude doing this with Wikitude.me their tagging app.?

Robert: Not really. That type of approach is on a very large scale using the accelerometers compass and GPS to determine where you are and what is in the distance. They (and others like Layar) don’t handle “near” AR. They effectively poll your GPS and then check a database to see what is nearby and what degree/distance it is and then they draw a representation on the screen. They don’t even need a mobile device’s camera at all.

Even if they did things up close, its still based on finding landmarks or on things that are broadcasting their location. For example, if they were standing near me, they might get “robert, 37 degrees, 15 meters away” but they wouldn’t be tracking me exactly as I walk around or have the ability to overlay graphics on ME.

Tish: I retweeted your #ar marketing using ARToolkit + flash (markers/webcams) = Photoshop pagecurl <six months. Bad design kills innovation. I know you like Dr Chevalier though!  What are some of the other AR marketing projects that you like. What would you like to see in terms of innovation in the next 6 months?

Robert: The marker/webcam approach is already becoming overused and cliche (tremendously fast). Older readers will remember the ubiquitous photoshop page curl that adorned nearly every website and graphic on the internet back in the day. It was horrible. Yes, the Dr. Chevalier stuff cracks me up.

I want to see some big companies or ad agencies really try to do something different with AR, preferably mobile. Take some risks, do something different. Don’t follow the crowd. Innovation? I want to see some wearable displays with transparent lenses, I want a mobile device specifically designed for ubiquitous AR, I want to see some experimenting with AR in the green tech sector, and I’d like to see someone get that GiFi wireless technology from that researcher in Australia and jam it into a smart mobile. I would also like my flying car and lunar vacation now, thank you. It is almost 2010 and no one has found that black obelisk yet.

Tish: So a few closing thoughts! What do you see as the next big thing? Hopes for the ar consortium?  Biggest bstacle for commercial AR?  And what is the coolest thing you have seen this year?!

Robert: The next big thing is what I’m working on hahaha. I hope the AR Consortium will grow and be the active catalyst in making AR mainstream, practical, and world changing.

The biggest obstacle is making sure that the right funding finds the right developers to develop the right technology and create kick ass applications.

The coolest thing I’ve seen this year would probably be the facade projection stuff (see below): Now, imagine that, but without the projector. Thats part of what I envision for AR in the future.

Neogence is on stealth mode with an immersive mobile augmented reality platform – “tools, sdk, and infrastructure plus some applications.” They are probably six months away from YouTubing anything according to CEO, Robert Rice.  But Robert rustled up this pic for me – a Google street view of Neogence R&D labs: “the patio on the lower left is where I do a lot of pacing and smoking my pipe and the porch and office upstairs is where  a lot of  meetings have been held.”

Robert Rice (@RobertRice ), CEO of Neogence, recently tweeted:

I’m changing my name to Robert Mobile Ubiquitous Geospatial Augmented Rice. I’m betting on radical changes in next 18 months.

Although Robert’s new AR platform is still under wraps, I think you will get a good idea of what direction he is going in from this interview (full text at end of  this post). Robert is the author of “MMO Evolution” and is a key developer and thought leader in persistent immersive environments, simulations, virtual worlds and massively multiplayer games as well as large scale communities and social networking.

It is OMG finally, at least, for minimally immersive but truly useful AR.

Since the launch of Android a new generation of useful augmented reality applications like Wikitude are emerging.

After the last NYC Tech Meetup, my  friend Nathan Freitas,  (@NatDefreitas),  or rather Nathan Mobile Meets Social Freitas, demoed for me a cool graffiti app  he has developed on Android.   You leave a marker for your graffiti so other people can find view/add their own – a nice primal experience like pissing on the lamp post to let your pack know where you’ve been.  Also the graffiti app taps into a long history of  NYC street culture around tagging and graffiti art.  For more cool mobile projects Nathan is working on – Vote Report and data collection for mass events, a guide to pubs and nightlife in New York City, and more, see his blog, “Nathan’s OpenIdeals. With Camera, GPS, compass, and accelerometer, and APIs on Android for temperature, light meters, (no hardware yet), Nathan says Android:

“seems to be the platform most likely to socialize the idea that sensor data could be a piece of every application.”

As Nathan is fond of saying:

The compass is a killer app enabler!

Also see OpenIntents for some interesting Android Sensor projects.

Wikitude was one of Thomas Wrobel’s two top AR milestones for 2008 (see Gamesalfreso):

Wikitude I think. Seems the first released, useful, AR software.

AR Geisha doll is also a remarkable breakout for AR – but useful, nah.

I asked Robert if he also thought Wikitude and AR Geisha doll as  significant breakthroughs:

Yes,  these are among the first attempts to get away from the novelty of simply rendering a 3D object based on a marker and making it interesting.

Remember, one of the biggest risks that AR has, is being branded as “novelty”, which means “cool for five minutes but ultimately a waste of time.” I think we have a ways to go before something is truly useful, but as 2009 progresses we should start seeing some effort here. I’d guess 2010 before something really useful comes out…at least something practical.

Now, having said that, I should say that I expect entertainment and games to take the lead (as usual), although there are a few companies really trying to leverage AR and video/graphics compositing for marketing (brochures) and location based methods (kiosks, large screen projections, etc.)

So when is it “OMG finally!” for massively multiuser augmented reality?

The picture above is from benjapo’s portfolio on istockphoto – also see the istock video here.

Alex Soojung-Kim Pang (who weighed in recently on the twitter-baby debates – see my KickBee Posterous blog) challenges design assumptions for augmented reality that take as a given the user’s desire for numerous private enhancements to their reality.

Alex points out less will probably be more so that enhancements do not impinge on shared experience.  See his write up of a talk he gave at the Royal College of Art, “and the end of my own private Shibuya.” Photo below by Stéfan, “Karaoke in Shibuya“

Part of the pleasure of these streetscapes is precisely that they’re collectively experienced, rather than individual visions: for even a brief period, we share with other postmodern, globe-hopping flaneurs and expatriates and temporary natives the light of the ABC-Mart sign and storefront.

It is collective experience of enhanced, augmented, virtual or real experiences that interests me too. This is one of the reasons I find Pachube and the EEML project of Haque Design and Research so interesting.

Extended Environments Markup Language (EEML), a protocol for sharing sensor data between remote responsive environments, both physical and virtual. It can be used to facilitate direct connections between any two environments; it can also be used to facilitate many-to-many connections as implemented by the web service Pachube, which enables people to tag and share real time sensor data from objects, devices and spaces around the world.

“Distinctions between virtual and real are as quaint and outmoded as distinctions between mind and body” (Usman Haque)

Usman Haque (founder of Pachube and Haque Design and Research) points out this is an underlying premise of his work – and augmented reality (full interview coming up soon!).

The pictures above show the Haque Design project, Remote:

‘Remote’ connects together two spaces, one in Boston the other in Second Life, and treats them as a single contiguous environment, bound together by the internet so that things that occur in one space affect things that happen in the other and vice versa – remotely controlling each other.

There was a discussion in twitter recently about how the terms like Second Life, Exit Reality, Virtual Worlds are misleading and outmoded. As Robert pointed out we need:

one word please…that sums up virtual and/or augmented reality, interactive, immersive, virtual worlds, mmorpgs, simulations, etc… also, I really don’t like the term “augmented reality” or “mixed reality”. Neither is all that great. And NO “matrix” or “metaverse” please

Robert argues strongly that there is a stultification both in virtual world technology – much of what we call virtual world technology was already, basically, where it is now in the mid 90’s. And MMOGs have devolved into gameplay design “that emphasizes the single player experience and does nothing to take advantage of the potential of the massively connected internet.”

Robert suggested I take a cruise through a new Virtual Space -  CoolIris to find some good pictures for this post (note the partnership between CoolIris and Seesmic to Streamline Video Blogging. I added the Cooliris Plugin to Firefox and typed Augmented Reality into search and soon I was cruising a highway of images and links. The Road Map image grabbed my attention (see below). It shows the continua that the Metaverse RoadMap authors thought are likely to influence the ways in which the Metaverse unfolds. It is “a map of the spectrum of technologies and applications ranging from augmentation to simulation; and the spectrum ranging from intimate (identity-focused)external (world-focused)”

Quite to my surprise, when I clicked out of CoolIris to the source for the image, I found it had been drawn from a post I wrote in May 2007, Hybridized Digital/Physical Worlds: Where Pop and Corporate Cultures Mingle. My post talks about a number of hybridization experiments that were bringing together lifelogging, sensors everywhere, simulation, virtual worlds, and augmentation.

The striking difference from 2007 to now is that we have definitely moved on from mere experimentation. And the poles of the continua intimate/extimate, augmentation/simulation as expressed in the Metaverse Roadmap are now becoming entwined (note the picture above seems to be slightly different to the one used in the road map as published here – perhaps I had an early version?)

“Augmented Reality is not just about overlaying data…” (Robert Rice)

The screenshot above is from TotalImmersions video demoing Augmented Reality with 3D Cell Phones. Also see video of their immersive games, and FutureScope kiosks here and here.

Gamesalfreso noted that Will Wright, delivered the best augmented reality quote of the year. When describing AR as the way of the future for games, Will Wright said:

“Games could increase our awareness of our immediate environment, rather than distract us from it”.

Robert points out in this interview the term Augmented Reality itself has become associated with a very limited understanding of what “enhancing your specific reality,” is really about. Robert notes:

it is inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.

When I talk about AR, I try to expand the definition a little bit. Usually, when you talk to someone about augmented reality, the first thing that comes to mind is overlaying 3D graphics on a video stream. I think though, that it should more properly be any media that is specific to your location and the context of what you are doing (or want to do)…augmenting or enhancing your specific reality.

In this sense, anything that at least knows who you are (your ID, mobile phone #, etc.), where you are (GPS coord or a specific place like a cafe), and gives you relevant data, information, or media = augmented reality. Sure, you can make things more interactive or immersive, but that is the minimum.

So, in this case, yes, I think there will be networked applications in the next 18 months…mostly things that are enhanced by friends lists (you are here, your friend is over there). These will be *application specific*. My team at Neogence is already going beyond this, building a platform and infrastructure for other applications to be developed on…all networked through the same backbone. Now, in this context (the science fiction AR that we all dream about), no I do not see anyone else trying to leap a generation or two ahead of the industry to build a massively multiuser shared AR space. Expect to see things like multi-user AR games, virtual pets, kiosk marketing, magic book, “gee whiz” presentations (tradeshow booths, entertainment parks, etc.), and so forth.

Goggle’s Are Not The Secret Sauce…

AR Cat left and Robert Rice right

What has come to be associated with the term Augmented Reality, in the popular imagination – an idea of 3D graphics projected over markers that has been forever waiting for the advent of “wicked next generation transparent wearable displays” – nirvana for augmented reality. While such displays may be nirvana for AR (and they could be with us in less than twenty four months), Goggles are not the “secret sauce” of AR as Robert points out.

All the glasses are, is another display device. At the end of the day, it doesn’t matter if you are looking at an LCD monitor, an IPhone, a head mounted display, or a pair of wicked next generation transparent wearable displays that magically draw directly on your retinas.

The real tricky stuff is what happens on the backend…making it all persistent, massively multiuser, intelligent, interoperable, realistic, etc. etc.

There has been quite a buzz going around about the new Vuzix Eyewear, and recently Robert talked with Vuzix and checked The Wrap 920AV eyewear out:

Vuzix is not alone in pursuing the ultimate in hardware, at least as far as wearable displays. However, I think they are much farther than the rest of the pack in vision, roadmap, and execution. They have put together a team that has a sense of urgency and ambition that will blow the industry away. After talking to them, I got the feeling that they really know what they are doing and there is a lot of mind blowing stuff in their pipeline. I’m sure they are one of the few companies that really gets it and has a clear vision of the future. Definitely my first choice to work with.

Hybrid Augmented/Virtual Reality

Cory Ondrejka posted this picture of the anybots telepresence robot and “congrats to Trevor Blackwell and the rest of the Anybots team on the launch of QA at CES.”  Cory (one of the founders and former CTO of Second Life) also made some predictions for Virtual Worlds, some optimistic and some less so, including “the increasing need to be able to diversify the Second Life product offering to begin truly rebuilding the code base.”

Robert is unabashedly irritated with the state of play in Virtual Worlds and MMOGS:

Unless both industries (Virtual Worlds and MMOGs) have some serious upheaval or radical new approaches, they will quickly be eclipsed by AR, which will eventually evolve into something hybrid..AR/VR depending on your level of access and hardware.

I’d like to see someone grab an engine like Offset, Crytek, HERO, or Unreal 3, and smack on a fat MMO server infrastructure (Eve or Bigworld)…toss in the right tools, and you would see a revolution and renaissance occur at the same time in the virtual world space. All the puzzle pieces are there, just no one is putting them together the right way.

I did just find out that Nortel’s WebAlive is powered by the Unreal 3 engine. You can try WebAlive out here.

Robert points out how rare it has become to see people really push virtual worlds technology and MMOGs into entirely new directions.  Although, of course, there are exceptions.  I managed to engage some interest from Robert in the possibilities the opensource modular architecture of OpenSim opens up, and the augmented reality experiments from Georgia Tech with Second Life (screenshot below) got praise from Robert for trying to do something new. (Georgia tech have also put out a virtual pet app for the iphone ).

But while Robert clearly has zero patience for virtual world technology which he sees stuck in the mid nineties, he notes:

the innovative and wonderful stuff about SL isn’t SL, it is what people are doing and creating on their own with terrible tools *IN* SL [Second Life].

The immersive mobile augmented reality platform Robert is building, he hopes, will generate this kind of user creativity but with 21st century tools.

So is it “OMG” finally for the Augmented Reality we have dreamed about?

According to Robert:

It really boils down to a markerless solution and a good application.

In the interview below we cover a number of topics including business models for Augmented Reality, e.g., how business models based on micro-transactions and virtual goods will translate to Augmented Reality.

Many of the challenges to becoming mainstream faced by virtual worlds  are similar to the challenges AR must overcome. Robert discusses these including the interface/gui that is a critical element for AR, solving the riddle of one world or many, patent wars in Virtual Worlds and Augmented Reality, the role of Augmented Reality in the future of sustainable computing, and what interoperability is about.

The Back Story for AR/VR…

In case you want to get up to speed on the required background reading for  Augmented Reality. This is Robert’s required reading list and Denno Coil is an absolute must see (feel free to add to this list in the comments, please).

“If you want to see the things that have inspired our vision of what we want to build, check out:

* Dream Park by Larry Niven and Steven Barnes
* Rainbows End by Vernor Vinge
* Spook Country by William Gibson
* Halting State by Charles Stross
* The Diamond Age by Neal Stephenson
* Donnerjack by Roger Zelazny and Jane Lindskold
* Otherland by Tad Williams
* Neuromancer by William Gibson
* Idoru by Wiliam Gibson
* Cryptonomicon by Neal Stephenson

and watch the whole anime of Denno Coil (subbed NOT dubbed!)”

Screenshot from Denno Coil from Concrete Badger.

Interview With Robert Rice

Tish Shute: I am glad to hear that you are working on this [an immersive mobile augmented reality platform]!

Robert Rice: We switched gears from MMO stuff about a year ago and we are finally getting some traction. It is very hard doing anything in this economy right now, but we found an opportunity to take AR to a new level beyond what you see on youtube. AR is still too “cute” and novelty. We don’t want to play around.

Tish Shute: I like Wikitude ‘cos it even manages to do something useful!

Robert Rice: Yeah, useful = traction. Now that we are getting near a prototype we are starting to get a lot of interest even though we are still technically way under the radar.

Tish Shute: r u funded?

Robert Rice: privately funded, some revenues from an early license, and ongoing discussions with several institutional investors. So, we have some funding, but nothing spectacular just yet.

Tish Shute: are you just developing an AR platform?

Robert Rice: hrm, sort of, but not just that. By platform I mean tools, sdk, and infrastructure plus some applications. The idea is to build something that facilitates everyone else making cool things and useful applications for different industries/sectors

Tish Shute: Yes that is the cool thing to do but isn’t that hard to fund!

(Robert grins) Well, that depends on the business model. We’ve got that figured out. I’d be absolutely happy if everyone and their brother were making applications on our stuff that gives us an edge on market penetration/saturation. There are plenty examples that prove the model. If you give people free and easy to use tools, they will run with it. ARtoolkit for example, has tons of people making nifty things and posting videos on youtube that has pushed them to the forefront as THE AR middleware to use right now, or heck, look at youtube free service, and they dominate video sharing.  Sure there will be a lot of “noise”, but there will also be a lot of “signal” that will rise to the top, facilitating and enabling is creating value in its own right.

Tish Shute: But how do you expect to monetize?

Robert Rice: There are a good half a dozen ways to monetize AR or an AR platform.

Tish Shute: What are your top 3?

Robert Rice: hrm, microtransactions, localized mobile advertising, and enterprise solutions (visualization)

Tish Shute: Do you think the consumer market will give the lead?

Robert Rice: I’m not sure. We are getting people from academia, intelligence, defense, border security, and some corporate types knocking on our door already, and pretty aggressively. It may be that those sectors push AR before consumer entertainment really kicks off.

But going back to a discussion we had earlier – yes working with “no markers” is a big deal.

Tish Shute: Can you talk about what you are doing there or is it still under wraps?

Robert Rice: I can say that between some university tech transfer and some of our own proprietary stuff, we are using some fairly common visual tracking technology. if you are really plugged into the AR scene, you will know there are probably half a dozen visual tracking methods out there. We just looked for the best one, licensed it for commercial use, and then started working our magic. This is a very small piece of the overall effort, but worth noting.

The downside with working with university tech is that it is usually based on research, incomplete, and not wrapped up in a nice commercial package on the upside, it can be a good start to build on.

Tish Shute: As you know I am very interested in “technology that matters” in particular tech that can help us accomplish the urgent goal of sustainable living.

Robert Rice:: oh, I’m pretty keen on sustainable living as well…after I sell off a few companies and have money of my own, I’m going to get into arcologies
❦
Robert grins

The interesting thing with the visual stuff combined with our other tech, is that we can make things multiuser, persistent, dynamic, and mobile.
The markers (fiducials) are really really limiting outside of basic applications. You can’t really plaster everyone and everything with a marker.  And they are, by nature, static (even if they are animated or whatever).

Also… our stuff works indoors and outdoors even without a GPS connection.
❦
Robert grins

Tish Shute: Now that does sound interesting!

Robert Rice: Yeah, with visual, you don’t need a compass or accelerometers either. Less hardware : )

You start with wifi triangulation or gps coord to get a “brute” location, and then you use the visual stuff for down to the meter accuracy and that by nature, gives you your orientation and positioning.

Tish Shute: Wow this is beginning to sound very interesting!

Robert Rice: Once you have that, it doesn’t matter where you go, it continues to track and continually refines areas you have been before. We’ve spent the last year figuring all this out. There are so many problems and obstacles that are going to be developing in the future for anyone trying to do what we are, but we have already discovered solutions.

oh, visual tracking = gesture based interfaces too that’s going to take some work, but its doable.  The real pain in the ass there isn’t the actual tracking, it is in the interface design.

That’s something that almost every AR company, venture, and research program is missing out on entirely. They are so focused on making cute things with markers.  They are missing the larger problems of AR Spam, interface, iconography, GUI, metaphor, interoperability, privacy, identity.

Tish Shute: So how are you dealing with all that!!

Robert Rice: We took the backwards approach of trying to think where we want things to be in ten years (and we read all the cool books…Vinge, Stephenson, Gibson, etc.) and then we spent time trying to think of what the potential problems are….like AR spam. Its bad enough when a giant penis flies by in second life, we don’t want that to happen in a global wireless AR platform.

Tish Shute: Do you have a prototype yet?

Robert Rice: hrm, 6 months away from youtubing something. Problem has been slow funding, which equals slow development. We also don’t want to show our cards too soon…too many potential competitors out there.

❦
Robert grins

Tish Shute: when you say microtransactions what is the business potential there?

Robert Rice: hrm last year I think, $1.5B was spent on virtual items. That’s games and virtual worlds. That should hit $5B in a couple of years. That’s basically people buying and selling things like WoW gold or items in SL or whatever. microtransactions, is basically the same thing, but in AR space.

Why couldn’t a 3D artist make a wicked animated 3D dragon, and then sell it to someone else? With AR, you could sit it on your shoulder. With a good scripting engine, you could train it to do stuff. Thats what I want to enable.

tools + sdk + platform = enabling people to make and create. Add in a commerce level (microtransactions) and wala.

Tish Shute: At the moment all of these virtual goods are very platform specific, is that a problem for you?

Robert Rice: Not at all. This is at a higher level. You have to switch mental models when you talk about what AR could or should be. For example, lets contrast the web and virtual worlds. For every virtual world you go to, you have to download a whole new client. Imagine if that model was applied to the web… you would need a brand new browser for every website you went to. That is just so…wrong.

It’s the same thing for AR…people are thinking about it with the same mental and business models and development philosophies as virtual worlds or web.  There are some things and aspects that work fine, but not everything.

Virtual worlds, are, by nature, necessarily different and walled gardens. The idea of 100% open and interoperable virtual worlds is a red herring…it sounds good but in practice it is a really dumb idea.

Tish Shute: I was wondering if you had a way to leverage all the 3D content already created ‘cos that would jump start things in AR wouldn’t it?

Robert Rice: Oh yeah, that’s easy. They all use the same polygons. Any virtual item in any game or virtual world is likely created with 3D studio or maya or something similar would be easy to convert and use.

Tish Shute: So people could bring their WoW weapons into your system?

Robert Rice: Not legally, but sure. Its just a 3D model with a texture.  It doesn’t matter if you use corel draw or photoshop or paintshop pro….or one screwdriver or another. Part of my team’s advantage, is that we are all experienced in MMORPG and virtual world design and development. We know the tools, the tech, and what works and what doesn’t.

Tish Shute: But some of the 3D content created in the social worlds is what has most value to people.

Robert Rice: Right, and that can be exported out easily.

Tish Shute: But back to “real” life applications. Is you platform really markerless?

Robert Rice: Yes.  marker = printed icon or glyph, also known as a fiducial

Tish Shute: But u must have some marker?

Robert Rice: hrm, more accurately, you need a point of reference.

Visual tracking has been around for more than a decade.  Lots of work for robots and other sectors.

Robert Rice: But isn’t the specificity of reference n terms of RL applications a vital key, for example, for a database of things?

Robert grin That is a different problem…tracking, registration, mapping, positioning, etc. That question has to do with mapping which is related to visual tracking, but not the same thing. We have a rather unique approach to some of this that I can’t discuss (patent pending).

Tish Shute: But for example, to create an augmented natural history of food – say I want to point at the slab of meat on my plate and know where that cow came from, what feed lot how it was treated etc

Robert Rice: That is not possible without ubiquitous nanotechnology. Shall I explain?

Tish Shute: Yes please!

Robert Rice: Ok, lets step back a minute and turn that burger back into a cow… the first problem (of this particular situation) is differentiating from one cow to another since most cows look alike, you can either attempt to discriminate visually (cow patterns) or use a much simpler option, like giving each cow a rfid chip in their bell, or hoof

Now, most people would try to figure out how to jam all sorts of info in the rfid chip, which sounds like a good idea, but isn’t, the trick would be to simply use the rfid to store a unique identifier with is then linked to a database elsewhere, or hoof.

That database should continually be updated with whatever relevant information you need so as you get close with your AR laptop, wearable displays, or embedded brain chip, you get the identifier broadcast, then you get the info downloaded to you, and it “sticks” to the cow with the generic visual tracking (object following, even simple bounding box is sufficient for a slow moving cow)

So, up to that point, you can get tons of information about that specific cow, that cow population (remember, AR is not just about overlaying data…it is inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.) Tie in data visualisation and some farmer tools and all sorts of other things happen. Now, lets move the timeline ahead a bit.

The butcher gets the cow and does his handiwork…because we know all the info about the cow, all of the meat can be properly labeled and marked. Ideally, with a UPC code or a unique glyph (somewhat problematic depending on how many unique glyphs you can create) so, while you are in the grocery store, you can access the relevant shopping data…age of cow, state of origin, type of feed, how many spots, how much body fat, which butcher, whatever not because of what is inside the package, but the package itself.

Getting back to your hamburger, the problem is that it is a burger…there is nothing to distinguish that burger from another one at the table…unless you stuck a rfid chip in it or splattered it with ink and a unique glyph, or maybe a special one of a kind plate.

However, a properly designed AR system could say “hey! that/s a hamburger! and I know I am at Fat Daddy’s Burger Joint in Raleigh North Carolina on Glenwood Avenue, and I know that they cook their burgers this particular way, and their meat supplier is those guys over there, and they usually get their cow meat from a farm out in Utah”

With ubiquitous nanomites or whatever, then its not that far out to consider edible nanos that are in the meat and that broad cast info so a slab of meat can tell you about itself and broadcast that to the general public.

Tish Shute: What useful scenarios can we create without the nanomites?

Robert Rice: If it wasn’t a burger or a consumable organic, the scenario changes.

Tish Shute: What is the time scale on nanomites?

Robert Rice: ehhhhhhh 20 years minimum if we are lucky. They sound good on paper, but there is a whole book worth of problems and why they are so far off…as consumer grade, all over the place, type of stuff.

Tish Shute: Did you see the Nokia Home Control center?

Robert Rice: Yes, I saw the Nokia stuff.

AR for sensors, like security systems, temperature control, etc. all become “sources of data” that a AR system can visualize. So yes, thats easily doable. You could do that in a short period of time with some half decent engineers.

The trick of what Nokia is doing is aggregating sensor data from a building/home/facility, mashing it together, and sending the mobile device alerts and data visualization conceptually rather simple, but no one has done it right or well yet.

It wouldn’t surprise me if Nokia pulled it off.

Tish Shute: yes and if they do and someone does an AR interface to it that would be an inflection point for AR?

Robert Rice: In a roundabout way, yes. You could get data directly from your house, or get it through your mobile device and in either case, use the AR for visualization and control.

The interface/gui is a critical element for AR. That is one of the areas where it, as an industry, risks doing a bad job and turning into just a fad or another novelty like VR.  Virtual worlds have been struggling with that for a while, but MMORPGs have had the effect of extending their life cycle

Tish Shute: Yes VWs have not solved the interface problem.

Robert Rice: The interface is one of their problems yes. Most virtual worlds are stuck in 1996/98

Tish Shute: If AR  is inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc. seems that it is the ideal interface for home control?

Robert Rice: Well for home control, you must know:

1) Who am I? Am I authorized to know this information? Am I a guest?

2) Where am I? Is this my house? or someone elses?

3) What am I doing? Do I want to make all the doors lock? Turn on or off lights? Open the garage door? Trigger the security alarm?

So the same questions apply

I’d say that all virtual worlds are stuck in the mid 90s. They are at least a decade behind the game worlds…in technology, design, implementation, architecture, etc. etc. In my opinion, things like Second Life are shameful in how they are presented as state of the art, innovative, ground breaking, new, wonderful, and world changing.

But thats another topic of conversation : )

Tish Shute: Well for me the contribution of VWs is the presence enabled real time interaction with application (as 3D info machine) and context with other people.

Robert Rice: Oh,there is no doubt that they are greatly useful and have a phenomenal amount of potential.

They *could* be all those things I just said that SL isn’t…the problem is that they are either just existing, or they are meandering around without any real focus or direction. They aren’t evolving.

Even MMORPGs are losing their way and beginning to stagnate terribly

Tish Shute: yes I agree

Robert Rice: But, AR has the potential to change a lot of things.

Im sure you have seen the yellowbook commercials? The technologies you are seeing here are doable in hrm, a year or less maybe. The tricky part is the interactivity and AI…that is, the content. Everything else isn’t a problem. The avatar there could be photorealistic or stylized like a WoW character.

You could do that to some degree with markers for registration but dynamically changing the content linked to those markers is a little weird

(by the way, for the record, I like markers just fine, I just don’t think they are useful for real-world mobile applications)

I also think that the guys that want to dust the planet with miniature rfid chips are on crack and are going about it the wrong way

Tish Shute: A high level of interactivity is hard though. Isn’t it? Even in VWs it is very limited.

Robert Rice: it depends if you can track what the user is doing, and interpret that properly. Interactive is also a very lose term.

Clicking a button and making a light blink could be considered interactive.

Tish Shute: In VWs a high level of interactivity would  be to wield a virtual hammer and have a real nail go in! is physics part of the problem?

Robert Rice: physics aren’t difficult, plenty of middleware out there for it. The problem with that isnt so much the physics as much as it is the scale and purpose

Tish Shute: well for robotics?

Robert Rice: that gets into a conversation about meshes, textures, and volumetric collision detection and stuff

Tish Shute: virtual robotics?

Robert Rice: You mean teleremote/telepresence of real robots?

Tish Shute: yes!

Robert Rice: ah, for that, you need some tactile feedback and some other stuff – doable, but insanely difficult. That’s why you don’t see a whole lot of remote controlled surgery robots all over the place.

They do exist…

Tish Shute: Will AR contribute to sustainable living by freeing us from some of our energy hogging devices?

Robert Rice: AR will ultimately encourage energy saving and recycling. where did I leave a light on at? where is the nearest trash can? what is the UV index outside today?

Yes, computers are energy hogs, but as we start seeing larger SSD drives, more efficient CPUs (even if the number of cores increases in multiples), and so on, the power will go down.

Also, think about this…wearable displays potentially use less energy than LCD monitors on your desk.

Tish Shute: Yes I should pick the brains of my intel chums on energy saving!

Robert Rice: Getting rid of the monitor and switching to solid state drives will save an assload of power. Yes, I said assload.

Tell your intel chums to quit screwing around with single core mobile CPUs. We need multiple cores, that are smaller, faster, and use less power.

Tish Shute: Is AR is the sustainable future of VW and MMOGs?

Robert Rice: The fun stuff will happen when they are both integrated in some fashion.

Tish Shute: So perhaps this is why the Georgia guys are thinking in trying to combine AR and SL (see video here).

Robert Rice: That first video was pretty damn cool. It just pains me that they are using SL for it. And omg, all those markers on the table.

Although, I could care less about seeing my SL avatar on my coffee table. I would rather see an avatar representing ME in the real world, moving around in a virtual world that is a “to scale” replica of the real world. That is MUCH more interesting and innovative.

But even if I don’t like where they are going, or that they are using SL, the important thing is that they are doing something and forging ahead. I have a massive amount of respect for anyone, private, government, or academic, that is doing that.

And yes, the door (or window, or looking glass) has to work both ways for maximum potential, at least, that’s what Id like to see. They don’t *have* to, but it would be rather cool.

And going back to sustainability, AR has the potential to make monitors generally obsolete, laptops too. That’s a lot of power hungry devices with all sorts of metals and batteries inside.

But, even if the tech was absolutely crazy awesome right this minute, it would take a little while for consumer adoption.

Tish Shute: But AR unleashes the mobile device?

Robert Rice: Yes, AR is going to be built on powerful mobile devices for the near future, eventually embedded comps in clothing and whatnot. But that is a ways off

Entertainment is going to be the first huge driver.

Tish Shute: So people will get used to having a pet virtual dragon on their shoulder first?

Robert Rice: Yes, virtual dragon is way cool, easy tech for games, and can eventually be leveraged into a smart agent which becomes a practical application…agent based contextual search, etc. Yes, entertainment will also drive people to get used to the tech

Tish Shute: Oh thanks for turning me on to gamesalfresco!

Robert Rice: Ive noticed that the good stuff usually gets linked to there. They don’t list my blog, but that’s what I get for staying under the radar and not posting often. But anyway, gamesalfresco is the first place I send people that need a crash course in AR. Great site, great owner.

Tish Shute: So are you in agreement with Thomas Wrobel’s positioning of  Wikitude and AR Geisha doll as being significant milestones for AR?

Robert Rice: Yes,  these are among the first attempts to get away from the novelty of simply rendering a 3D object based on a marker and making it interesting.

Remember, one of the biggest risks that AR has, is being branded as “novelty”, which means “cool for five minutes but ultimately a waste of time.” I think we have a ways to go before something is truly useful, but as 2009 progresses we should start seeing some effort here. I’d guess 2010 before something really useful comes out…at least something practical.

Now, having said that, I should say that I expect entertainment and games to take the lead (as usual), although there are a few companies really trying to leverage AR and video/graphics compositing for marketing (brochures) and location based methods (kiosks, large screen projections, etc.)

Tish Shute: Many people would say SnowCrash (metaverse) is now and Halting State (AR) is ten years from now. But you are seeing a development timeline for some popular AR apps in the next 18 months?

Robert Rice: Anyone that says SnowCrash is -now- is living in a box. Virtual Worlds, Virtual Reality, and immersive tech in general stopped innovating in the mid 90s. I’m continually flabbergasted at the number of people that think that things like Second Life are state-of-the-art or innovative. You might as well try to market a walkman as cutting edge, even though we have IPods out there.

Id like to see someone grab an engine like offset, crytek, hero, or unreal 3, and smack on a fat mmo server infrastructure (eve or big world)…toss in the right tools, and you would see a revolution and renaissance occur at the same time in the virtual world space. All the puzzle pieces are there, just no one is putting them together the right way.

Tish Shute: Why doesn’t anyone do that?

Robert Rice: Its not cheap, people will only fund a copy of something that exists already, people fear change and innovation, etc, The list goes on. The right money goes to the wrong people all the time.

Alternatively stated, there is a lot of “right idea, wrong implementation”

MMORPGs carried the torch and have made huge strides on the technology front, but have devolved in design. More often than not the gameplay emphasizes the single player experience and does nothing to take advantage of the potential of the massively connected internet.

Unless both industries have some serious upheaval or radical new approaches, they will quickly be eclipsed by AR, which will eventually evolve into something hybrid..AR/VR depending on your level of access and hardware.

But yes, I’d say that the next 18 months are going to be very interesting with a lot of money being thrown around, new ventures, and plenty of content/applications. I expect most of this will be centered on single user AR experienced through a mobile device with a screen (iphone, android, etc.). I expect that there will be a significant boost after Vuzix releases some of their wearable *transparent* displays, putting Microvision back into the “has potential but is too quiet” position.

Tish Shute: AR conjurs an image in many people’s minds of dreadful head gear!

Robert Rice: Yes, it is either transparent wearable displays (in eyeglass formfactor) or nothing. HMDs with miniature LCD or OLED displays are good for streaming video, but for the mobile ubiquitous AR we all dream about, it has to be something that looks and feels like a pair of Oakleys.

I should also mention that several different types and modes of AR are going to find themselves being defined and refined over the next two years as we continue to blaze new trails, establish a lexicon (we keep borrowing terms from games, VR, virtual worlds, mmorpgs), and really work out the how as well as the why.

Even though the idea of AR has been around for a long time, the technology is just beginning to emerge, and very few people are even looking far enough ahead to figure out the problems and solutions that the tech creates. Really, who is thinking about how to deal with AR spam right now?

Tish Shute: Do you see any successful networked AR applications emerging in the next 18 months?

Robert Rice: Yes and no.

When I talk about AR, I try to expand the definition a little bit. Usually, when you talk to someone about augmented reality, the first thing that comes to mind is overlaying 3D graphics on a video stream. I think though, that it should more properly be any media that is specific to your location and the context of what you are doing (or want to do)…augmenting or enhancing your specific reality.

In this sense, anything that at least knows who you are (your ID, mobile phone #, etc.), where you are (GPS coord or a specific place like a cafe), and gives you relevant data, information, or media = augmented reality. Sure, you can make things more interactive or immersive, but that is the minimum.

So, in this case, yes, I think there will be networked applications in the next 18 months…mostly things that are enhanced by friends lists (you are here, your friend is over there). These will be *application specific*. My team at Neogence is already going beyond this, building a platform and infrastructure for other applications to be developed on…all networked through the same backbone. Now, in this context (the science fiction AR that we all dream about), no I do not see anyone else trying to leap a generation or two ahead of the industry to build a massively multiuser shared AR space. Expect to see things like multi-user AR games, virtual pets, kiosk marketing, magic book, “gee whiz” presentations (tradeshow booths, entertainment parks, etc.), and so forth.

The big thing I’m worried about is AR becoming the next silicon valley trend…once they realize the potential, an enormous amount of capital will flow to a bunch of startups with half baked ideas, weak business models, ten year old tech, and a lot of overhyped marketing. That is the very thing that will kill this technology as something that has true power and potential to literally change the way we interact with each other, our surroundings, information, and media.

Tish Shute: Do you think AR has value for a project like Pachube that helps us connect dtat from lots of different environments and sensor actuator data?

Robert Rice: I think that AR has value as an interface to this data (essentially data visualization based on information streaming from a sensor or source that is interpreted in some dynamic graphical manner that has meaning). This is one of the “big areas” where ubiquitous augmented reality and wearable computing will really shine. I’ll definitely be keeping an eye on Pachube .

Tish Shute: I can’t help it! I am really interested to hear more about the Vuzix glasses?

Robert Rice: Yeah, everyone is getting hung up on the glasses as the end-all be all and having markers everywhere too.

All the glasses are, is another display device. At the end of the day, it doesnt matter if you are looking at a lcd monitor, a iphone, a head mounted display, or a pair of wicked next generation transparent wearable displays that magically draw directly on your retinas.

The real tricky stuff is what happens on the backend…making it all persistent, massively multiuser, intelligent, interoperable, realistic, etc. etc.

I think that we are within 24 months of the magic wearables (these new ones by vuzix are probably the real first generation attempt at doing it right). They wont be perfect, but I expect they will be functional…and once we have functional, we can start doing the good stuff.

Tish Shute: You mentioned you disappointement with VWs and MMORPGs earlier.  Could you tell me more about that?

Robert Rice: Yeah, there was an evolutionary divergence between virtual worlds and mmorpgs a while back. One stagnated almost completely, and the other leapt ahead in one sense and devolved horribly in the other sense. Neither is where the state of the art should be.  That is a whole other conversation, and probably a second book.

Tish Shute: So making AR persistent, massively multiuser, intelligent, interoperable, realistic, etc. etc. that is where your efforts are going?

Robert Rice: Yes. I fully expect that the hardware is almost ready for it. You can cobble together some amazing things in the lab right now, and I think commercial viability is imminent. The real value (as far as I’m concerned) is in making it mobile, wireless, persistent, and massively multiuser. You could argue that augmented reality will take over where virtual reality failed and become internet 3, internet one being the internet, internet two being the web…

mmorpgs are nothing more than single player games in a multiuser environment these days. I’m more than a bit bitter about it. All the right money went to the wrong people, and the best games we have are barely shadows of what we could have had by now.

Tish Shute: Are there any open source AR platform dev projects?

Robert Rice: open source? hrm, Im sure there are multiple ones out there

if not entirely open source, there are plenty of things to experiment with that are generally free if you aren’t trying to sell something, DART and ARTOOLKIT come to mind as very accessible applications.

Marker based AR is very important right now…it is easy, low tech, understandable, highly customizable, and most importantly, accessible to the average joe. Ultimately though, we need a method of pure tracking…no markers glued to everything on the planet, no “billions of RFIDs” embedded in every square inch of every object on the planet, etc.

Tish Shute: What do you mean by interoperability in AR? And what do you think about the development of standards?

Robert Rice: Ooh, good question.

Ok, so the internet is basically computers communicating with computers, and the web is mostly pages linking to other pages (I’m greatly oversimplifying here). Hold this thought for a minute.

Switch over to MMORPGs. If you want to play in one (or a virtual world), you need to download a client that is specific to that world. One client does not work with another world. There are plenty of efforts to change this, but they are all barking up the wrong tree. The specific uniqueness of each world defeats the need and purpose of true interoperability, unless you completely reinvent the whole thing with a common backbone, features, functionality, etc. The very nature of virtual worlds and mmorpgs rebels against this.You absolutely do not want an avatar from second life running around in world of warcraft (for reasons that should be obvious).

On the other hand, with the web, you can use just about any client (browser) to access nearly any website (some requiring plugins or whatever).

The thing with augmented reality, is how do we go about making this? I’ve seen a few people thinking about this from the wrong perspective. There was a question at the last techcrunch to the Sekai Camera guys (a conceptual AR application for the iphone) where someone on the panel wanted to know how website owners would convert their content for augmented reality. BZZZZZT! That is a fundamental misunderstanding of what AR is, or could be, and it falls into the same trap I see a lot of people doing…and that is looking at AR through the web 2.0 lens or the virtual world lens. It is absolutely fundamentally different at the core…sure there are similarities: it has social networking/media applications and properties, and it has 3D graphics, but it stops there.

Ubiquitous augmented reality will be dramatically different depending on which standards, approaches, and philosophies get the most traction first. Will you walk down the street with your AR glasses and have a pop up every 30 feet asking you if you want to access the AR content on another server? Will you then have to register, subscribe, or whatever?

Or will all AR content be mediated by one sole master control server deep in the bowels of google? What about some other option? Will you need different sets of glasses to access different features and content from multiple sources?

At the end of the day, it should not matter what brand of glasses you are wearing, you should never have to deal with AR server popups to join/subscribe, and so forth.

Interoperability, in the context of what I was saying earlier, is the sense of how to build the infrastructure so all of this is seamless to the end user, but still maintaining the features/functionality necessary for all of what augmented reality promises us…I dont want to see everything in AR space, I want to be able to tune in or filter out some things, and I want to customize the snot out of what I see (perhaps changing metaphors or “holoscapes”), and so on. It all has to work together and simplify the end-user experience or it won’t get anywhere

Tish Shute: So what caused the stagnation of new development and devolution of MMOGs in you opinion?

Robert Rice: yes, look at all the hope and hype for the mmorpgs released in the last 12 months really, what is different or better? Now, what is worse?

I bet any decent mmorpg gamer could give you a list of 2 or 3 things for the first question and 20-30 things for the second.

And, VWs seem to be stuck in a feedback loop

Tish Shute: feedback loop?

Robert Rice: Imagine nailing one of your feet to the ground and then trying to run ’round and ’round and ’round.

Tish Shute: Why do you think this happened to VWs?

Robert Rice: Men in suits and flashy watches.

actually, hang on…..

I saw a video clip the other day from a conference about using various virtual and game technologies for simulations and other real world applications several people were talking about “avatar technology” and how theirs was better than their competitions and what not.

Now, can you tell me what “avatar technology” is? Avatar technology is a red herring. Avatar technology is the same thing as calling a toaster a new “fire technology.”

Robert Rice: The problem is that a lot of people that don’t have a clue about what they are doing are selling the tech to other people that have no clue what they are buying, but they feel like the should for some unknown reason.

That is happening all over the government, academic, and industrial sectors now with a few companies selling virtual worlds (again, mid 90-s tech) as the ultimate solution to all problems.

Anyway, getting back to your question

Once virtual reality started getting some buzz, some people got greedy and jumped into the avatar/virtual world thing and tried making it commercial too soon half of the 3D chat worlds were being jammed into platforms for virtual shopping malls.

Most of the money funding tech R&D started funneling towards VRML, and doing 3D in web pages, etc.

Tish Shute: yes horrible idea trying make web pages 3D IMHO

Robert Rice: The money people got involved too soon, and then the greedy people jumped in and tried patenting everything possible. Take a look at the worlds.com patent for 3D worlds.

They filed it back in 2000 or so and it was awarded in 07 (it shouldnt have been in my opinion) now they are suing everyone they can.

Tish Shute: Will there be patent wars in AR?

Robert Rice: Yes, the AR patent wars will be legendary once people start waking up to the real potential here.

The only solution is for everyone to band together and pre-emptively patent or make public domain every possible patentable concept, technology, or implementation for AR otherwise, you haven’t seen anything yet.

Tish Shute: Is the AR community organized enough to do that yet?

Robert Rice: That depends on how my company fares in the next six months.

Tish Shute: Will you patent or make your tech public domain?

Robert Rice: I plan on patenting the snot out of everything we can possibly think of, and then giving away our content creation tools and SDK stuff for free. The whole goal of what we are trying to build is to empower the end user and facilitate the creation of a wonderful world of augmented reality.

There are some things we will make public domain for sure, on top of that

Tish Shute: So back to my question on networked real time experience. Will we have networked Real time AR experiences in the next 18 months

Robert Rice: It is possible, yes. Other than what we are doing, I am not aware of anyone else taking the same approach we are, but the potential for an “under the radar venture” (much like my own company) is definitely there.

Tish Shute: Will you use cloud computing?

Robert Rice: I think that’s overrated and probably another attempt at the whole “thin client” model that some companies have been pushing for the last 20 years.

It sounds good on paper, but ultimately takes power and control away from the end user.

Tish Shute: cloud computing?

Robert Rice: Yes. You know, we aren’t playing around, We are totally building “THE AR” that everyone keeps dreaming about. None of this cute stuff you see on youtube. Actually, if you want to see the things that have inspired our vision of what we want to build, check out:

* Dream Park by Larry Niven and Steven Barnes
* Rainbows End by Vernor Vinge
* Spook Country by William Gibson
* Halting State by Charles Stross
* The Diamond Age by Neal Stephenson
* Donnerjack by Roger Zelazny and Jane Lindskold
* Otherland by Tad Williams
* Neuromancer by William Gibson
* Idoru by Wiliam Gibson
* Cryptonomicon by Neal Stephenson

and watch the whole anime of Denno Coil (subbed NOT dubbed!).

Tish Shute: So scaling the real time experience won’t be a problem in your project hehe

Cos no sharding allowed in AR right

And if you have lots of API calls?

Robert Rice:: haha, sharding is one of the dumbest things to happen to the VW/MMO industry

It is a solution to a technical problem that was relevant 15 years ago.

Tish Shute: so why did it stick (i know men in suits)

Robert Rice: it stuck because “thats what the other guys did” and the mmo designers are too lazy to reconcile gameplay for PvP and RP gamers

However, there is a curious problem between dealing with “one world” and “anyone can start their own custom AR server”

Tish Shute: Now that is a very interesting problem the one world and own AR server

Robert Rice: It took me a few weeks of not sleeping to figure that one out. It gets back to the interoperability issue

Tish Shute: What did you come up with?

Robert Rice: a solution. Thats all I can say for now on that.

Tish Shute: eeextra seeekrit!

Well I will definitely have to bug you on that.

The problem has produced some creativity in OpenSim with people coming up with hybrids of p2p and oneworld

Robert Rice: As far as virtual worlds are concerned, they need to look at the problem from a different perspective. They are trying to make all virtual worlds interoperable intead of creating a new model for interoperable worlds that new ones will be created to adhere to.

Tish Shute: well some people are. I would say most OpenSim developers see their modular approach doing this.  And you choose to interoperate based on what modules you have activated and then social agreements…

Robert Rice: hrm, thats a start, but that only works on a functional and social level – doesn’t account for content (story, mythos, game rules), unique data (my +3 sword), or the concepts of commerce, inherent value, and intellectual property

Enabling my WoW avatar to run around in SL and vice versa creates more problems than it solves.

Its like two alien races working hard to make sure that their two spaceships can dock but no one is paying any attention to the fact that race A breathes nitrogen and race B breathes sulpher.

Its technically possible, but they are missing the boat on the content side of the problem.

Tish Shute: Yes but don’t you think when a modular open source tech for virtual worlds  becomes pervasive, what will happen is that those interested in a similar genre will increasingly use the module in ways that allows their content to interoperate if they want it too

Robert Rice: everyone has to use the same backend tech, and the front end clients need to adhere to the same standards. Bu I have to admit, I haven’t been paying much attention to the vw space in the last 9 months or so.

Oh I have to run now.  But download and install cooliris. I promise you will be blown away and will start using it to search for images and videos

Its frigging awesome.

Tish Shute: Will do!  Thanks so much great talking to you. I can’t wait for your launch.

Google Street View Hacking

This Google Street View Hack (via @timoreilly) will get my nomination for a Hacking the World Award this year, if there is such an award.

A parade (the screenshot opening this post), a marathon, a mad-scientists laboratory, a sword fight, and more (see The Infonaut Blog) were staged all along the route of the Google Street View truck by artists Robin Hewlett and Ben Kinsley working in conjunction with the local community and Google Street View.

The Google Street View Hack suggests at a myriad of possibilities for anyone with their eye on the prize for a great world hack for 2009.  In my mind’s eye, I imagine the Google Street View truck’s trek across the planet triggering local environmental street action carnivals wherever it goes.

Local energy conservationists, “passive house” architects, retrofitters, could turn the arrival of  Google Street View into an occasion to create projects for a sustainable future – a traveling StreetCamp (see my post on HomeCamp ’08 here).  As Google Street View intends, surely, to go everywhere,  this would be a global hack for sustainable living that crossed the bounds of the physical and the virtual.  And the vast public record of Google Street View would became a generative engine and global resource for sustainable living.

Working together on the noble aim of sustainable living

- this is my (and many other people’s) big theme for 2009.

A Hacking the World award should also go to  Pachube – “patching the planet” – for demonstrating that instrumenting the world is not merely a Sci Fi  fantasy anymore.  By facilitating “interaction between remote environments, both physical and virtual,”  Pachube demonstrates (see diagram here) how we have only just begun to dip our toes into the many new opportunities we have to work together to save energy, rethink our culture of consumption, and to reboot our failing economy under a new sustainable operating system.

Energy awareness unlike the glut of information we have in entertainment and games suffers from a dearth of information. We really have very little idea about what we are consuming and the waste we are producing.  So more Hacking the World Awards should go to projects like AMEE – creating the world’s energy meter, and Wattzon – your personal energy meter, for giving us new ways to understand and work with energy data.

Many people and organizations, given the information, will change their behaviours. But the cultural changes necessary for sustainable living are deep and old habits die hard (see this disturbing report on the recent return to SUV buying in November as soon as gas prices fell!).

A  Small Community of Volunteers Can Bring Change on a Global Scale

Picture above by Benjamin Ellis, “HomeCamp – The Throng,” from his Flickr stream.

One of my favorite “instrumenting the world” projects to date and another top contender for a Hacking the World Award is HomeCamp ‘08 (see my previous post).  HomeCamp brings together a community of creators and enthusiasts of  “smart stuff,” creating a wikiculture for the noble cause of sustainable living.

The key to whether “instrumenting the world” empowers people and changes our lives for the better will be the capacity our systems of instrumentation have for what Jonathan Zittrain in “The Future of the Internet: And How To Stop It:,” defines as generativity, i.e.:  “the system’s capacity to produce unanticipated change through unfiltered contributions from broad and varied audiences” ( Zittrain, 2008).

Generativity is the “secret sauce” that makes the difference between, for example, Wikipedia and its all but forgotten predecessor – the “written by experts” Nupedia.

Jonathan Zittrain writes:

Wikipedia stands for more than the ability of people to craft their own knowledge and culture.  It stands for the idea that people of diverse backgrounds can work together on a common project with, whatever its other weaknesses, a noble aim - bringing such knowledge to the world. (p.147)

At Web 2.0 Summit, Jonathan Hochman (Known as Jehochman on Wikipedia), shared with me his insider perspective as a Wikipedia administrator. The full interview with Jonathan is later in this post.

Jonathan comments on the role of wikiculture in sustainable living:

“Sustainable Living requires everything to become more efficient. Incentives need to line up with conservation priorities. This requires a radical change to the way we govern ourselves. Command economies, whether commanded by politicians or capital, lead to huge inefficiencies.”

And surely, if we have learned anything in 2008, we have learned that very bad things happen when the complex systems of modern life are left in the hands of a few people motivated solely by the urge to make profit.

Hacking Design and Planning Processes for Real Estate and Transportation with Virtual Worlds

This great machinima by Azwaldo Vilotta shows the progress so far on the Wikitecture 4.0 project, ‘Re-Inventing the Virtual Classroom’ for the University of Alabama.

Though still a niche market Virtual Worlds are growing at a steady pace.  As I mentioned in my previous post, energy hungry avatars themselves will be a target for optimization in 2009.  But as my personal power usage breakdown from Wattzon shows, cutting down the amount of flying I do in 2009 would be far more effective in reducing my carbon footprint than deciding not to log into Virtual Worlds!

Note: Read Write Web’s recent post, “Report Enterprise Virtual Worlds More Effective Than Web Conferencing.  Also check out Web.Alive, and Immersive WorkSpaces and Dusan Writer’s post on “ThinkBalm,The Immersive Internet and Collaborative culture,”

My friend Melanie Swan points out in her Top Ten Computing Trends for 2009, that Virtual Worlds not only have the power of the 3 Cs (communication, collaboration and commerce) but they are fast expanding into rapid prototyping, simulation and data visualization.

My Hacking the World, 2008, Awards for Virtual World innovation would go to three potentially world changing projects for sustainable living:

1) Studio Wikitecture, (see “Reinventing the Virtual Classroom” for The University of Alabama).

2) Oliver Goh’s work on “The Path to Sustainable Real Estate.”

3) Encitra, a company recently co-founded by Crista Lopes and Christer Lindstrom focused on improving urban planning processes, starting with transportation, using virtual worlds (see my previous post here for more).

The latter two projects are being developed in OpenSim – the open source project that should also get a Hacking The World Award for creating an open modular architecture for virtual worlds that is unleashing all these new possibilites for integrating physical and virtual worlds.

The 2008 code contributions to OpenSim of special note re world hacking are Crista Lopes’ OpenSim Hypergrid – see Justin CC’s blog for full details on “What is the hypergrid?,” and David Levine’s work (IBM), in collaboration with Linden Lab (see Architecture Working Group), on interoperability (see my earlier post here).

Both these projects expand the frontiers of interoperability for virtual worlds although they “slice the problem from different ends,” as David Levine put it.  The emphasis in the LL/IBM approach is on security so assets are not moving yet.  In Crista’s solution you can have assets but the security issues are not addressed yet. But this work is vital to expanding the usefulness of virtual worlds and both projects should get Hacking the World Awards IMHO.

I asked Jon Brouchoud (full interview upcoming) what he thought were Studio Wikitecture’s most important successes to date:

“I think the greatest success has been in proving, on some level, that everyone has important knowledge that can inform and improve the design of a building, not just architects.  If we can continue building on that success, I hope we can eventually start to hack the traditional design process, and find ways to harness the wealth of knowledge held by the general public, instead of ignoring or avoiding it, as is most often the case.”

Harnessing the “Smart Stuff” to the Noble Cause of Sustainable Living

Robert Scoble’s, The Interview of the Year: Tim O’Reilly, is not to be missed. Tim O’Reilly discusses the key trends for 2009 that are bubbling up at O’Reilly Media.  And, Yes, Tim O’Reilly, as the guru of Hacking the World, gets the “Distinguished Thinker – Hacking The World Award of 2008!”

Tim O’Reilly’s trend list includes:

1) big data- vast peer produced data bases in the cloud accessible by mobile devices

2) “smart stuff” – sensors and robotics and hacking on stuff for fun and not for profit

3) Green Tech

4) Advances in Biological/Life Sciences.

And, in Robert Scoble’s interview, there is a nice titbit of history re his attendance of early Foo Camps.  Foo Camp is the wiki of O’Reilly conferences and a lineage holder to my favorite Hacking the World event of 2008, HomeCamp ‘08.

But what will be the “secret sauce” for these big ideas  – the generative engines that harness to the noble cause of sustainable living these vast peer produced data bases and all the creative “smart stuff” hackers across the globe are creating?  What will motivate the mass adoption of Green Tech and sustainable living?

What can Wikipedia teach us about how generative systems and bottom up approaches can change the world?

Jimmy Wales (interview coming soon!)  writes in his recent personal appeal for support for Wikipedia.

At its core, Wikipedia is driven by a global community of more than 150,000 volunteers – all dedicated to sharing knowledge freely. Over almost eight years, these volunteers have contributed more than 11 million articles in 265 languages. More than 275 million people come to our website every month to access information, free of charge and free of advertising.

To answer questions on a how to create a successful wikiculture for sustainable living, an insider’s view of Wikipedia may be a good place to start.

Interview With Jonathan Hochman on Wikipedia.

The picture on the left is from the Wikipedia article, Gamma-ray Burst, that Jonathan Hochman is currently working on.  It is a drawing of a massive star collapsing to form a black hole. Energy released as jets along the axis of rotation forms a gamma-ray burst. Credit: Nicolle Rager Fuller/NSF

The picture on the right, Jonathan at Web 2.0 Summit, is taken by me. Jonathan was part of the, Defending Web 2.0 from Virtual Blight, panel.

Known as Jehochman on Wikipedia, he serves as an administrator and as a leader in addressing online harassment, disruption and sock puppetry. He is also the founder of Hochman Consultants, an Internet marketing consultancy, and the director of Search Engine Marketing New England, a regional conference series.

Tish: Second Life and Wikipedia are the two great experiments in collaborative co-creation what do they have to teach us about the future of the internet?

Jonathan: Yes, Wikipedia and Second Life are key social spaces.  Some people have been seeing Second Life as the beginning of Web 3.0 – a wrap around environment where you can almost experience another life. Wikipedia is sort of another example of this.

All the problems that exist in the real world are mirrored right into that little universe.  For example, the Armenians and the Turks are at each others throats and the Japanese and the Koreans are going at it, the Palestinians and the Israelis, and the “Troubles”  … all the conflicts are imported into Wikipedia.  People are fighting over the content of these articles. They want to have it their way because these are first ranked in Google and they have a big impact in public opinion.

There was a huge fight on the waterboarding article a while back. Some guys from Little Green Footballs – they are a very conservative reactionary type of media. They are trying to change the article to say that water boarding might not be torture – change it to say it is probably not so bad.  Crazy stuff. They were trying to water it down.  And it is very clear, from every source out there, that waterboarding is torture.  We did a study and there are 115 sources that say waterboarding is torture. You simulate drowning – you simulate killing someone – that is a violation of the Geneva Convention and everything else. People were fighting, fighting, fighting!

One of the things I did was to try and clear people out who were being disruptive.  We actually had to go to arbitration over that article. It is like the supreme court of Wikipedia. There is a panel of 15 arbitrators.  They hear the case. There is evidence, arguments and decisions. It is really like a simulated law suit. You get all the experience of a simulated law suit with the real threat that you could be banned. If they don’t like what you are doing they can actually ban you or restrict you from topics.

So it is really fascinating how this social space Wikipedia becomes a very real platform though it is in a virtual world for real world disputes.  Most disputes are over the definition of things.  If you have a you suit most disputes are about how things are defined. And Wikipedia has become the defacto definition of things in the real world.  People want to know what are “The Troubles.”  If you go to Wikipedia you find out  The Troubles are a dispute over Northern Ireland.  What the article says has a profound impact on public opinion.

Tish: So who is on the court of Wikipedia?

Jonathan: They are volunteers. these people work two or three hours a day to run this court.  There are all kinds of projects.  There is a WikiProject Spam which has people who can write computer programs to statistically analyze Wikipedia projects – not only Wikipedia. But all of them are looking at the links and reporting them and banning those people who are abusing or gaming the system.

Tish: You were on the Stopping Virtual Blight Panel at Web 2.0 Summit – what are the most important things to think about on this topic?

Jonathan: Yes we were talking about how to defend the web against virtual blight. The thing I find interesting about Wikipedia is that because it is the eighth largest web site and possibly the second largest web site comprised of user generated content after YouTube. The problems that exist in Wikipedia are larger and more detailed than any other site.  For whatever problem someone has for their social media site or their Web 2.0 site these problems already exist in Wikipedia and the solutions are there and they are transparent. You can actually see the history of what’s been done.

If there is, for example, a problem on Digg – some problem with sock puppetry or vote stacking – it happens, it goes away.  You don’t get full disclosure.  With Wikipedia you can actually go in and look at a dispute and watch it unfold.  You can watch the arbitration cases that are filed, the arguments, the decisions, the logic, the rationale.  You can see the successes and the failures and the different things people have tried to control blight. For example, we tried to resolve this dispute one way but it was a disaster, so we have tried something else and that worked.

Wikipedia is a large laboratory for social media. Wikipedia and the large universe around it Wiki and WikiMedia projects that individuals, enterprises and put together like Commons.  Wikimedia Commons is a repository of publicly licensed images that anyone can take and reuse. They have sound and they have video, and all of this stuff is being stitched together now.

So if you go to the article on Obama  you can probably now hear his acceptance speech because that is public domain – its been stitched into the article.  If you go to the article on Richard Nixon – his resignation speech – you may even hear his conversation with the astronauts when they landed on the moon.  So this becomes a giant repository of all our culture and knowledge.  When I design a website, a lot of times I go to Commons to find images I use for free.  I don’t want to pay for an image I can get for free. 

Tish: And the Commons images get contextualized in Wikipedia too.

Jonathan: Some of these articles are fascinatingly detailed. If you want a quick summary of the Dr. Strangelove, the article is fantastic.  It is enjoyable, a pleasure to read.  I was reading about S.A. Andree’s North Pole balloon expedition of 1897. Some guys from Sweden decided to fly a balloon over the North pole.  They managed to get aloft then they flew over the icepack for 24 hrs then they crashed.

They unloaded their stuff and hiked back across the ice toward the island they had launched from. They ended up being on the ice pack for three months before they finally holed up in an ice cave and starved to death.  There weren’t found until thirty years later!  There was a camera with these guys and the frozen pictures taken 30 yrs earlier.  They developed the film and those pictures are now on Wikipedia.  It is just a fascinating thing!

Tish: Do you see real time collaboration beginning to play more of a role in Wikipedia – whether virtual worlds or just voice/IM — how could real time collaboration change the wikipedia editing process?

Jonathan:  The Presidential candidate articles were being edited very rapidly yesterday. There are certain real time problems.  Some of the more interesting problems are when you get two administrators who “get into it.” One administrator says I am blocking this user and the other one says I am unblocking him, and the other one “NO I am blocking him!” And so on…… And everyone says, “Stop fighting. You are not allowed to do that!” And they both get their powers stripped. People do get very heated over the silliest things. Wikipedia does have some mailing lists attached and there are some IRC channels. So there are some real time elements.

Tish: What is the role of avatars in Wikipedia?

Jonathan: In Wikipedia you have a user page and many users are anonymous.  They create an avatar and they personalize it and show themselves in ways they want to show themselves through an avatar. In many ways it is a lot like Second Life.

Some users have created second accounts – or a humerous second account. Bishzilla – a Swedish lady who is in tremendous command of the English language and has a razor sharp wit.  She has created this secondary account – almost like in a baby language.  Her avatar is a dinosaur that is not very bright that goes around frying people. Bizarre what people do! People may be editing a topic like an interest they have – e.g. Pokemon that they don’t want associated with their professional avatar. Or people may be editing a topic about hot political issues.  There have actually been some death threats issued to people over stuff they have been putting into the encyclopedia.

Tish: So avatars are important in Wikipedia.

Jonathan: Absolutely because people may be going in and editing articles that they may not want their friends and family to know they are editing.  One editor may say to another, “Stop putting stuff in or I will come and kill you!” Well then we have to ban them.  We have to call the police.

Tish: Can you build reputations on multiple avatars?

Jonathan: You are allowed to use multiple avatars as long as they don’t cross paths.  You can’t have two avatars editing in the same area beacuse you are going to be giving yourself double weight commenting on a discussion.


Tish: How do you know when this is happening?

Jonathan: You can watch the style of a users editing.  You have to watch behavior.  And if you have enough evidence through behavior that suggests accounts are controlled by one person you can go and request a technical check.

There are some uses who are called Checkusers who are able to access information desired from the server logs and check the technical characteristics of these accounts to see if they are using the same IP address.

Tish: So if you want to understand avatar interaction on the web it helps to understand Wikipedia.

Jonathan: Yes it is a fantastic way to understand how avatars work in some aspects, and also how to deal with community dynamics.  We have some very strong willed people – people in their 40s, 50s, and 60s – who are very successful in business.  They have plenty of money and spare time and they are doing this as a hobby. And some of these people can really butt heads.  You can have a problem when you have an editor who has been writing fantastic articles but also happens to be rude and chew other people out and tell them to f**k off if they are not behaving. What do you do?

Tish: Sounds a bit like Second Life!

Jonathan: The person is a great contributor to the community but they are telling noobies to f**k off, so you can’t allow that.

What do you do?  Vested contributors are a major problem to some of these sites. They are vested in the community but they start misbehaving. You can’t block them, because if you block them there is a huge upsroar from all their friends and it causes a cataclysm.  It requires very careful diplomacy to deal with some of these situations.

Tish: How many Wikipedia volunteers are there now?

Jonathan: Think of a Venn Diagram – a big circle. The total number of contributors are about one million different people that contribute.  But there are probably about 5,000 active editors  that are consistently and regularly contributing.  And within that kernel there are fifteen hundred people that have administrator access and probably only eight hundred of them are active.  People have a natural life span with the community.  People come an typically stay for 6 months to 3 years.  Usually after that they become bored, disillusioned or get into a conflict with someone.  There is a natural tendency for people to stay for a while and move on. Some people stay longer, a few, but the majority will move on at some point.  So it is a lot of fresh faces moving in.

Tish: What lessons of trust does Wkipedia have to teach us about new projects like AMEE that aims to aggregate the world’s energy data?

Jonathan: Well you have to know who is releasing the data. Who is creating the data? The beauty of Wikipedia is that you have an edit history so you can see exactly who has done what.  So you can judge whether this person is trustworthy or not.  That’s a huge problem on the web today.  We don’t have enough identification information.  When you see a web page you don’t necessarily know when that page was created and by whom, or how many revisions it has had.  Sometimes you can glean information by checking it.  If you see typos and errors you may decide that that page probably didn’t receive as much attention as it should have, and probably it is not that good.

Typos are an interesting thing.  People always try to figure out how Google ranks web pages. Matt Cutts was here from Google today.  And he was talking about spam.  But Matt also did a blog post about how he was in an airport once, and how he has a policy – when you are reading a document as soon as you come to the first error just stop because if the author hasn’t taken the care to make everything correct, you don’t need to read it. So he was in the airport, there was a sign, he came to a typo and stopped reading it. Somehow he got in trouble for not reading the sign and not having the information.  But it is interesting to think whether Goggle is looking for for typos, misspellings, broken links and using that as a signal of quality to rank pages.

Tish: Aaaagh typos might bring down your page rank!!!  That certainly is a scary thought for a blogger like me who likes to write impossibly long posts that are hard to check………

OpenSim is designed for interoperability innovation. Adam Frisby explains:

By allowing easy customization and extension, we can test and refine interoperability protocols very quickly and efficiently. The interconnect with Second Life (TM) was developed by David Levine, IBM, in only a number of days (David Levine on the right, Adam Frisby, left).

Please join us on Thursday, September, 4th, 4pm to 5pm at the Virtual Worlds Conference and Expo, LA for our panel, “Open-Source, Interoperable Virtual Worlds,” which will be part of the Future of Virtual Worlds track.

Support for standardisation in Virtual World technologies has been growing steadily in recent times, join the developers of OpenSim and industry commentators as they discuss where open-source virtual worlds are heading and the progress made towards standard protocols for interoperability.

- Adam Frisby, Director, DeepThink (and OpenSim Developer)
- Tish Shute, Writer/Virtual World Evangelist, Ugotrade.com
- Mic Bowman, Principal Engineer, Intel
- Justin Clark-Casey, OpenSim Developer, IBM

Also, special panel guest Mike Mazur, OpenSim Developer, 3Di. We hope David Levine, IBM, will be back from Europe and join us too!

Adam Frisby, Deep Think, is one of the founders and leading developers of OpenSim. Adam has been behind so many important steps forward in the open metaverse that I cannot list them all here. But a notable recent project of Adam’s is a new “Lively style” viewer for OpenSim and Second Life (TM), Xenki (see Adam’s blog for more).

Mic Bowman, Intel

Mic Bowman presented at the Intel Developer’s Forum earlier this month. He outlined a road map for how virtual worlds will move into the fabric of everyday computing with open source software playing a key role (the slide of OpenSim architecture above is from this presentation). Virtual Worlds are an important part of Intel’s strategy for developing connected visual computing experiences. For more about Intel’s CVC initiative (see this press coverage of IDF, The Inquirer, bit-tech.net, hexus, Trusted Reviews).

Basically… my message for the panel is this: To achieve a thriving, growing, broadly adopted CVC ecosystem, we believe the industry must come to some agreement on common building block technologies. Open source technologies represent a critical element in the discovery and development of these technologies, and foster innovative usages that drive adoption.

Justin Clark-Casey, Fashion Research Institute, Inc.

Another interesting member of our panel and important OpenSim developer is Justin Clark-Casey. Justin, formerly of IBM, is now with the Fashion Reasearch Institute, Inc as a full time OpenSim developer/architect. Fashion Research Institute is considered by many one of the most advanced business cases on OpenSim (more about Shenlei Winkler, CEO of FRI, in my next post, “Meet the Rising Stars of the Open Metaverse at VW 2008.”

Recently Justin has been working on something called a region archive in OpenSim.

Basically, this is a way of saving a sim to a single file (currently a tar.gz) and reloading it into another OpenSimulator. This file contains all the necessary data (prim xml and assets such as textures and scripts) necessary to restore the entire region.

It’s currently experimental so still has some bugs. But, it can be used via the load-oar/save-oar commands on the OpenSim region console.

The ability to save and load archives is something that people developing whole applications using OpenSim will be very interested in, and will contribute to the business value of OpenSim.

OpenSim and Linden Lab are at the Center of Interoperability Innovation

Since the launch of Open Grid (beta) the interoperability initiative from OpenSim and Linden Lab (for more see here), cross world interoperability has begun with avatar hopping. Now the thorny issues of trust management, economy, and IP that are the major part of asset interoperability are on the table.

The picture above of Gridnaut, Lawson English, (Saijanai Kuhn in SL) is from Lynn Cullens (Bjorlyn Loon in SL), Director of Communications for Metanomics.
Open Grid has expanded to 31 regions in less than a month and there is now OGP support in OpenSim trunk. Vivaty has expressed interest in joining interoperability efforts so we may see some of the new browser based worlds become part of this initiative soon

Hamilton Linden, who is leading the Open Platform Product Group (OPPG) as Director, Engineering for Linden Lab, and, Tess Linden, Technical Director, leading design and Implementation for the OPPG (Open Platform Product Group), will be holding office hours at the Linden Lab booth at VW2008, LA to discuss Open Grid. We hope that both Hamilton and Tess will special guests at the panel also!

The video above (see here on YouTube) of an OpenSim integration with Facebook was posted today by Stefan Andersson of Tribal Media also see his blog here. This is the third in a series of videos that introduces Tribal’s new concept for 3D/web integration.

The picture above shows the in the left pane fetched pictures from Stefan’s Facebook photos. As Stefan explains a hybrid web app is talking to the region to change the picture accordingly and pull the photos into frames on the wall (for a more detailed technical explanation see here).

These videos (see the first here) from Stefan and his partner at Tribal Media, Darren Guard, Stefan explained, demonstrate as proof of concept how a “3D web architecture” could look and feel.

This concept has quite far reaching implications to many or our current notions of how inventory, virtual economy and content will work in virtual worlds. Stefan explained:

Sufficient to say is that Tribal One was an experiment, showing some concepts – much like realXtend is introducing some innovative new concepts. For example: Who creates your ‘inventory’ and what are you supposed to do with that? In our concept, the ‘inventory’ is a set of services that I can use to interact with the world, i.e, I have a “pictureFrame” in my inventory that, when I drag it onto a wall, the act of dragging it onto the wall instructs the region with the wall to use that url to fetch the definition of that pictureFrame, as well as showing that pictureFrames application web page to me. And, when I interact with that web page, that web page interacts with the region to change the picture in the frame. Contrast that with the notion of an ‘inventory’ that contains a set list of stored objects that I can ‘rez’.

The next video Stefan noted will demo a ‘friends’ tab and you can choose to ‘join’ or ‘visit’ them. “The point there being not having to run, fly or teleport just click. Also, the clip shows ‘snapping.’ the picture frames ‘snap’ to compatible surfaces something you’d like in Second Life.”

Interview with Stefan Andersson

Tish Shute: What are the goals of Tribal One?

Stefan Andersson: One of the central goals with the Tribal Server platform is actually about cost-efficiently adding 3D to existing communities or intranets. We did the Tribal One to:

1) show how easy it is to build what we call a “Community Provider” which basically is a connector between an OpenSim-based server and any given community or intranet – the ‘provider’ is just a small piece of code that provides authentication and profile data for the 3D region.

2) show how a hybrid web/2d interface could look, and address those very questions that we discussed would be interesting to ask Avi Bar-Zeev (see full interview here).

3) show our concept of 3D web applications, where a user works with an application trhu a seamless mix of 2D and 3D interactions, all resulting in http methods on a simple web application in the background.

We wanted to show how third-parties could create 3D-aware applications without coding anything but web services, for example in php. This has two great benefits; a) you can let web-coders (of which there are plenty) do the work of 3d-coders (of which there are few) and b) you can utilize existing web service contracts and security solutions when passing over trust boundaries, which would let a 3D-host securely interact with public web services, instead of letting a hosted region have direct database access.

Tish: Is it correct to say while your experiment here provides a nice solution to identity management between 2D and 3D spaces but it does not approach the issues that OGP and AWG (Architecture Working Group) are dealing with re trust management now, where the real meat on the table is now inventory and permissions and economy?

Stefan: That would be correct; in our model, the inventory actually consists of url references to web applicatons; for example – the picture frame that is visible in the clip I’m uploading, is in the inventory as a url to a web application that creates the xml definition for the 3D object

We are aiming for something a bit more ‘active’ than a static collection of statically stored assets.

We wanted to proof-of-concept an approach where you had an application generating the inventory and the objects in the inventory, as opposed to having a static databse listing of statically stored items.

So, the inventory is generated on-the-fly for the user, depending on what services he or she has subscribed to, and how those services are configured – not just by how stuff has been given/taken.

So, if you had an Phat wardrobe node in your inventory, for example, Phat could add items to that node depending on what you did on their web site

Tish: And that would be assuming they offered virtual clothing right?

Stefan: Yes, exactly.

Okay, take some virtual clothing company – the point is that the inventory, in this case, could be modified ‘from the outside’. Of course, in an SL-centric world view, that’s outrageous, but if you thought of this from a service-centric view, it would make sense.

Tish: But protocols could be worked out to extend this beyond Tribal to other OpenSim worlds and even Second Life assuming those other worlds wanted to participate?

Stefan:Yes, definitively – we did this to show how a 3D web architecture could look and feel.

Tish: Server centric is like Tribal where you are basically running your sim on your own PC

Stefan: Yes – we actually based all this on very simple web protocols; the ‘web viewport’ concept is just an extension of the existing OpenSim object exchange xml ‘standard’.

Tish: And while this idea would have no place in the old SL grid – would it be more feasable to integrate it in the new Linden Lab Open Grid with the agent domains?

Could agent domains say choose to opt in and opt out of these kind of web services?

Stefan: To be honest, I’m waiting for the AWG (Architecture Working Group) to solve the ‘inventory’ issue before I would venture using the term ‘agent domain’.

Tish: So when there is inventory sharing that is when the agent domain comes of age in your view?

Stefan: Well the question is: Who creates your ‘inventory’ and what are you supposed to do with that?

In our concept, the ‘inventory’ is a set of services that I can use to interact with the world.

Ie, I have a “pictureFrame” in my inventory that, when I drag it onto a wall, the act of dragging it onto the wall instructs the region with the wall to use that url to fetch the definition of that pictureFrame, as well as showing that pictureFrames application web page to me.

And, when I interact with that web page, that web page interacts with the region to change the picture in the frame.

Contrast that with the notion of an ‘inventory’ that contains a set list of stored objects that I can ‘rez’.

Tish: And at the moment in AWG discussions ideas are limited to Second Life models of inventory?

Stefan: That is my understanding

Again, Tribal One is not a ready-made product – it’s a vision of how future 3D/Web applications could function.

We just need somebody to pay us to finish it! *laughs*

Tish: Is there any reason why assuming AWG figure out how to manage the conventional notion on inventory this concept couldn’t be layered into extended protocols and options?

Stefan: I would assume the AWG solution would be able to fit our concepts – since we’ve already done this with the existing SL protocol.

Tish: Do you see this new notion of inventory offering new oppoertunities to content creators to be rewarded for their work?

Stefan: Well, with this prototype we were more looking at how the web works; we were more concerned with generating and distributing content from a service point of view.

This concept is more oriented towards users utilizing a business service, like a web site.

In our scenario, content is created by server-side services written in for example php.

imagine, if you will, if the SL model was based on scripts that were able to create objects, instead of objects having scripts.

Tish: Won’t this involve the development of lots of new tools for content creation?

Stefan: Well, yes and no; it’s like with the web you need people to do the designing, and then people to do the scripting, but you also need people to do the server side coding.

We are not addressing the current SL paradigms of content creation we are addressing what we felt was missing; a platform for organisations to make dynamically created content, like the difference between a static html page and a php page. Both can have client-side scripting on them, but the php page can pull data out of databases – any kind of databases – existing databses.

Take the example with the “where I’ve been” application in Facebook: You add that app, then you can fill in where you’ve been, and you get a nice map of where you been, and you can compare that to that of your friends

Now, that’s a database creating those images out of the data that you and your friends entered.

What we did, was an api, so that the author of that facebook app could add a set of php pages, which would let users share in inventory item, that was an url to an web app that returned the definition for a map of where I’ve been, so I could have that same map on the wall in my 3d living room.

Tish: So this implies a completely different model of content creation and economy to the one Second Life uses?

Stefan: Well, in these kinds of business scenarios, the content in itself would probably have little value, but the service that generated the customized content would have great value.

Tish: So this will raise the bar for people making a living from content creation? right?

Stefan: Most definitively. But that is how it should be.

Tish: I’m not sure everyone will agree with that!

Stefan: Oh? That’s the back side of progress; the ‘advanced’ part of ‘advancement’ if you will. That’s why it has value.

Tish: Well one of the things that made SL popular was the fact that a non professional developer, just someone with talent, could make a few Lindens creating content.

But, don’t you think the two models of content production will work together?

Stefan: They still will be able to. Most definitively in SL.

Again, it’s like with the web; in the beginning, people could make money out of just being able to notepad some html and ftp it to a server.

Today we have .NET server side database programmers doing the server bits, and educated professional graphical designers doing the ui and design bits.

Tish: Won’t the web services model have to rest in some way on a template understanding of content?

Stefan: Of course it has. You’re very right about that.

But then again, it’s not an ‘either-or’ – it’s a ‘both’.

And with that, a whole new spectrum of utility.

Tish: Perhaps, the two models will find interesting ways to interact in the end? They already do to some degree?

Stefan: Yes, you know these malls with 10 packs of skins, where the only difference is the combination of skin and makeup?

Or hair variations?

A good designer teams up with a good programmer and creates one good app that just delivers that exact combination.

There is good design and bad design, good design will always be coveted.

Tish: So do you have any summarizing remarks?

Stefan: Social games and technical innovation aside, what is needed to bring forth the 3D web revolution are good examples of business and user value – applications reaching beyond social networking bubbles and into the intranets and databases of organizations – and I believe Tribal Media has shown that we are very well suited to help bring those applications to life.

Open Grid Public Beta opened today (see Second Life blog) marking the beginning of a new era of interoperable virtual worlds and a new architecture for Second Life TM. The magic of “running code and consensus” is here and, at least between OpenSim and Second Life TM, avatars are jumping back and forth. Hamilton Linden, who is leading the Open Platform Product Group (OPPG) as Director, Engineering for Linden Lab said:

The Public Open Grid Beta is an important step towards opening up the Second Life Grid to become interoperable with other virtual worlds. Having successfully demonstrated interoperability with IBM, we’re excited to begin interoperability testing with the entire OpenSim community.

In the picture opening this post Adam Frisby (avatar Adam Zaius) one of the founders of OpenSim, David Levine, IBM, (avatar Zha Ewry) who wrote the interoperability code. and myself are about to teleport from the Ugotrade OGP (Open Grid protocol) enabled OpenSim to the Linden Lab Open Grid. The teleport to an external region option is in a pull down menu that brings up the box you see on the left. If you join the Beta and want to visit, my region URL is http://ugotrade.net:9000

As these teleports are about moving identity, at the moment, and no digital assets are moved, we are all Ruths.

You must join Gridnauts in Second Life TM if you want to participate. The download and instructions for the OGP (Open Grid Protocol) Open Grid Viewer will be on the Wiki. And, to get a zipped binary package to set up an OGP enabled OpenSim you can go to the OpenSim forge site. Thanks to Mono and .NET using the same bytecode format, the same package will work just fine for .NET and Linux/Mono. Mike Ortman, DeepThink has generously created the zip package which he will keep updated.

In the screenshot below, Adam Zaius, Zha Ewry and Tara5 Oh are preparing to teleport back from Open Grid to the Ugotrade OGP OpenSim.

Linden Lab’s New Architecture

But along with interoperability the Open Grid Beta marks the debut of Linden Lab’s new architecture that has been incubated in the Architectural Working Group (AWG) spearheaded by Zero Linden. As Zero Linden explained:

A key component of virtual worlds that sets them apart from web sites, is that you interact with them with your chosen identity. Separating out the Agent Domain enables your identity to be held and hosted by a organization of your choice, and enables your identity to be truly independent of the many organizations that will eventually host regions. The web can’t do this – your identity on a web site is tied up with that web site. You have an account at each web site. In virtual worlds, independent persistent identity is key to the experience – and Agent Domains are just the technical mechanism that enables them in an open virtual world.

The interop protocols developed in AWG and used in David Levine’s, IBM, (Zha Ewry in Second Life) interoperability patch not only play an important part in enabling virtual world interoperability, they will be a key component of the new Linden Lab architecture and eventually part of their main production grid Agni, that is the grid we call Second Life. Zero explained:

The plan is, that once this is shown to work, that this code base will eventually be rolled into Agni, probably even before Agni is opened up to outside grids. TPing, and Login will be done on Agni using these interop protocols as the standard method. Of course, there are legacy viewers to support – so the existing stuff isn’t going away for some time. And we’ll proceed very cautiously onto Agni, with “kill switches” that allows to revert all viewers, even new, back to the old pathways.

Zero Linden and Zha Ewry will be speaking on “OpenSim and the Future” – the progress they have made, and the implications of their work at Metanomics, Noon PST on Monday, August 4th. Dusan Writer will also be announcing the follow-up to his much-lauded competition to create a better Second Life client viewer at the start of the show.

The picture below shows how the Open Grid client which, in addition to the teleport option after login, allows you to select an external region even before you log in

The Open Grid – a community of developers, “playing with shiny things”

There is a strong team of Linden’s working with Hamilton in the Open Platform Product Group. Tess Linden, Technical Director, leads design and Implementation for the OPPG, and Layla Linden has been getting the agent domain ready. Periapse Linden is project manager for OPPG. Whump Linden is managing the Open Grid Public Beta. Whump is also a very interesting contributor, I think, to the evolution of the Open Grid. He has an enormous amount of web experience and has been a blogger since 1998. Whump came to Linden Lab from Apple’s MobileMe group. He is the point person for the Open Grid Beta which is organized through the Second Life TM Gridnauts group.

Enus Linden and Infinity Linden are working on testing tools known as the PyOGP test harness. These testing tools are a very interesting project themselves. Tao Takashi who was the prime mover in the PyOGP project before it became part of the Open Grid Beta explained to me:

My vision was always to create something like libsecondlife but for plain Python instead of .NET. The old protocol was just too undocumented to really get something like this done quickly so when OGP was getting born I though of trying again but with a better protocol and by coincidence Linden Lab need a test harness for testing all those components out there so PyOGP was born, as the library can now serve as backend for the tests. But in the long run of course more is possible. It can also become a full implementation of client and server, web service interface and more. I am working on an agent domain implementation for pyogp right now and I have some ideas for some text based or maybe even 2d gfx client.

Something worth noting about the Interoperability effort between Linden lab and OpenSim, the Architectural Working Group, and the PyOGP initiative is the large number of experienced and talented developers that are putting extraordinary amounts of time and effort into these projects.

The meetings are packed. I had my first God-mode teleport into a full sim in Second Life TM from Zero Linden today so I could get into the AWG meeting to ask some questions for this post. Yes, God-mode is truly the finest way to travel! I hope to devote a series of posts to the pioneering developers that are creating the future of open source virtual worlds. Their dedication and brilliance is quite extraordinary.

Hey but for starters a tip of the hat to the indefatigable and omnipresent Saijanai Kuhn (Lawson English in RL) – “a 20+ year script kiddie programmer who always wanted to get into game programming.” Saijanai says: “This is my chance to do something kool on a significant scale, so I’m excited about the whole AWG OGP thing.”

And, If you want a little example of how quickly some of this developing brilliance produces results in this community check out this prototype for a “Livelyâ„¢”-style viewer for OpenSim+SL. that Adam Frisby (OpenSim/Deep Think) whipped up in a few hours! There is a currently a petition to release llmath/llvolume.cpp under a more liberal license which Adam pointed out to me is “somewhat required to do accurate rendering in alternate clients.”

I spent this weekend jumping around Second Life and OpenSims with Whump Linden and Zha Ewry. The picture below shows Zha, Whump and I arriving on the LL Open Grid from Zha’s laptop sim. There is a bug Zha told me that is making us arrive at (0,0,0) on the sim.

Managing Assets and Identity in an Interoperable Open Grid

Linden Lab is NOT throwing the baby (the Second Life economy) out with the bath water (the old Second Life Architecture). Linden Lab have made this very clear many times but Zero reiterated for me when I asked this question:

Absolutely — after all, I love babies — we positively need to build an architecture that supports the economy of SL — while at the same time allowing the virtual world to be open to a wider variety of experiences.

And, if you have already watched the video of the NYC Metaverse Meetup you will know that interoperability of assets and managing identity in open virtual worlds is what’s on everyone’s minds. But as David Levine (Zha Ewry in SL) pointed out several times: “These teleports are just about moving identity for the moment they do not bring a single digital asset with them for a moment.”

There was a long but very interesting discussion about some of the issues of managing and federating identity and moving assets between multiple virtual worlds at the Meetup. And, Adam Frisby and David Levine outlined some of technical and social steps to full interoperability in that discussion.

David Levine has also asserted several times that a big priority for him is looking at how the interoperability of assets can be implemented without detriment to “creators” whom he describes are “the secret sauce” that makes Second Life a compelling place and the ingredient that makes a virtual world either work or not work. But, interoperability, regardless of how particular virtual worlds decide to handle it, will force virtual worlds to rethink the way they do or don’t help their content creators and users to relate outside of the little puddle of their own particular terms of service. But, David pointed out, if we want to do something that spans not just one or two applications, this discussion, which is social as much as technical, has to be done in a broader community

For now, the goals of the OGP Beta are narrow. As Whump pointed out:

The matter of inventory is not in scope for this part of the beta. Figuring out inventory is a combination of technical and community work. Some of this will be figuring out a common vocabulary for talking about these issues. We want to figure out the basics of protocols for teleport, find the bugs, and refine these issues. We want to have running code and test suites, because that will bring interested parties.

But, while the beta has begun with a simple version of OpenSim trunk the next step will be to work on interop with projects like realXtend and Tribal Net. Both these initiatives are bringing a lot of innovation to OpenSim. Both realXtend and Tribal see interoperability as a key project and are looking forward to joining the Beta soon.

Roadmap for Open Grid

I asked Zero Linden what the roadmap for the next few months would be:

Zero Linden: Well, now that we’ve demonstrated some technical work, and are going into a public beta, August is going to find much of the LL side hunkered down and fleshing out much architectural detail. For some areas, especially inventory and identity, we’ll be putting together some concrete frameworks so those more complex discussions can make progress in the Fall. So the next step is to pave the way for clear progress on them. They are big issues and deserve the time and background work to make them be successful discussions and eventually successful desgins.

Tara5 Oh: So when you say concrete framework you mean code and architecture?

Zero Linden: I mean more of a specific set of issues, use cases and design options to have a discussion about. We’ve been talking about identity and inventory in largely general terms for almost a year. I think we as a whole have a common sense of what we are talking about. Now we need some specific points to answer, and a guide for the design. Then, the code will follow.

Tara5 Oh: so when you say uses cases do u have a wish list yet?

Zero Linden: Well, I have may personal pet use cases — who doesn’t — what we will be developing in August is a more rational set. So, in short, nothing yet. I’m trying to stay purposly “zen mind” about it — since it can be such an explosive topic.

In the picture below Whump Linden gazes out at the open horizon.

Here is the video of our last Metaverse Meetup: OpenSim & Virtual Worlds Interoperability 7.23.08 (from Vimeo). The video of this landmark event was produced thanks to the awesome Annie Ok, Artist, Creative Director, Curator, Video Director, Metaverse Evangelist/Consultant, Co-Organizer of Metaverse Meetup.

While Annie’s first love is art, she has been involved in an extraordinary number of projects (see her bio here). Notably, Annie Ok, with Jeff Crouse, & Stephanie Rothenberg made the documentary and helped with the amazing Invisible Threads project which shows how excellent Second Life is for such innovative mixed reality installations. The documentary premiered at Synthetic Times. Annie also created the interactive, collaborative Timeline of Virtual Worlds that the whole community can help with.

Photos of the meetup are now posted here on Flickr and some nice portraits here on Facebook.

With the video Annie sent out a great write up about the meetup.

Annie noted:

“Adam Frisby and David Levine gave us incredible insight into OpenSim and shared compelling details that really expanded on what has previously been known about its amazing potential and revolutionary role in the future of the metaverse.”

And she was very kind about my really minor supporting role!

“Tish Shute was great as the guest moderator, asking key questions and adding salient commentary.”

And I really agree with Annie’s synopsis about what is at the heart of our metaverse meetups!

“It was so nice to see all the familiar regulars as well as meet the new ones. In true Metaverse Meetup style, we migrated en mass to a local bar where we continued the conversation about all things metaversal and had fun hanging out with fellow avatars until the late hours.”

Thanks to everyone and especially Global Kids for making the meetup possible.

Please be sure to check out the list of Metaverse Meetup links on the new About page. There are now Metaverse Meetup group on LinkedIn, Flickr and FriendFeed, as well as a list of Metaverse Meetup chapters in other cities for those of you who are not based in NYC.

Looking forward to seeing you at the next meetup!