Will Wright’s talk at Where 2012 is brilliant. It elucidates a turning point for mobile gaming.

Will describes an important shift:

gaming has primarily up to this point been about simulating parts of reality, now I think it’s moving toward the idea that maybe we can start parsing actual reality, and incorporating that into our play experiences.

He unfolds a vision for a new genre of personally aware mobile games that move away from a “presumption that reality sucks and we want to get away from it.”

I’m super excited to be working with Will Wright and Stupid Fun Club to create a new genre of mobile experiences that express this vision.

Will notes:

as a designer nowadays, I don’t feel like there’s any meaningful limitation that I have. The amount of technology that used to be applied to NORAD, tracking incoming missiles, is basically now in my pocket, helping me find frappuccinos. Actually quite a bit more technology than NORAD had back then.

It is a rich, dense talk so enjoy the video and explore the CaptionBox too!

Also I hope you can join us at Augmented Reality Event, 2012, Santa Clara, CA on May 8th and 9th. Will Wright will be judging the Auggies with Bruce Sterling, Daniel Suarez and others. I will be talking about “Augmented Awareness & Life Based Games” in the AR Games session, along with Brian Selzer, Ogmento and Richard Wetzel, Fraunhofer

And please do use my discount count TISH375AR to register!

The prophets of Augmented Reality, Bruce Sterling (best know for his science fiction works, his non-fiction includes “The Hacker Crackdown,” “Tomorrow Now,” and “Shaping Things”), and  Vernor Vinge (author of “Rainbows End” and “The Coming Technological Singularity”) are joining  Will Wright (Legendary game designer of SimCity, The Sims, and Spore), and Jaron Lanier (a computer scientist, composer, visual artist, legend of virtual reality research, and the author of You Are Not A Gadget: A Manifesto) to judge the Auggies at the Second Annual Augmented Reality Event, 2011, Santa Clara, Ca., May 17th, May 18th!  Pictures top row, Bruce Sterling, Vernor Vinge, Will Wright, 2nd row, Jaron Lanier, Blaise Agüera y Arcas, and Frank Cooper III.

Augmented Reality has been prototyped in science fiction writing and films for many years now.  But Augmented Reality Event, the first global event dedicated to the emerging industry of AR,  is your chance to be part of a momentous transformation of science fiction into science fact.   Bruce gave a seminal keynote last year at ARE2010, “At the 9am of the Augmented Reality Industry…” But we are edging closer to high noon this year – see John Markoff’s recent New York Times post, “Augmented Reality Comes Closer to Reality,” and Bruce Sterling’s opening keynote at ARE2011 will raise the bar for us all again.

Bruce Sterling  and Vernor Vinge will wrap up the show with fireside chat.  Jaron Lanier will be keynoting at are2011 for the first time and Blaise Agüera y Arcas (a leader in Microsoft’s mobile strategy), will return to top his Ted talk demonstrating innovations in Bing Maps and Augmented Reality.  Frank Cooper III, who serves as Senior Vice President and Chief Consumer Engagement Officer of PepsiCo, will give, an up to now, missing perspective of  the big brands’ on Augmented Reality (AR).

“The main stage at ARE 2011 will be blessed with science fiction visionaries, ground breaking scientists, mind blowing technologists, and legendary game designers and more!”

Come prepared to Augmented Reality Event.  Read as much Bruce Sterling and Vernor Vinge as you can, and follow @bruces blog on Wired, Beyond the Beyond.  With only a few weeks to go until Augmented Reality Event, myself and co-chairs Ori Inbar, and Chris Grayson, have been working hard on a preliminary schedule ( by the time you click on this link it should be updated again).  You can follow us on twitter for updates as they come in – @tishshute, @comogard, @chrisgrayson,  and please follow @arealityevent and join our ARE facebook posse.  And, of course, don’t forget to register soon while we are still offering discount codes.

Readers of this post can use the code TISH295 for $100 of the already sweet price if $395.

If you joined us for Augmented Reality Event last year, you will know that the”Auggies” – an award for the best AR demo presented live with commentary American idol style,  is a chance to join the best and brightest in AR as they write the next chapter for AR in the sharp and often amusing reparte between judges and contestants.  Submit your demo proposal here under the “Auggies” track.

The picture below is the Auggie Award for the best AR demo designed by Sigal Arad Inbar.  (See, Ivan Franco recounts the team’s ARE 2010 experience, and winning the event’s first-ever Auggie Award, and the video shot at the YDreams booth by Bruce Sterling. “The Hotness” – YDreams rocking it at ARE2010 from brucesflickr)

ARE2011 will include more than 90(!) speakers in 30 sessions organized into 3 tracks: business, technology & programming, and production & design.

It will feature special activities such as a Startup launch pad (submit your proposal here under track: “Startup Launch Pad”), and an ARt Gala with live performances.  There will be displays by top AR artists of AR art projects and live performance art by the world’s best AR artists featuring: Helen Papagiannis, Amir Baradaran, Sander Veenhof, Ina Centaur and more to be announced.

The exhibition area will be open throughout the 2 days of the event and will include the latest product demos by the leading AR companies, as well as a career fair to help grow the fledging industry.  A small number of booths is still available in the exhibition hall ($995 for 10′x10′).Grab them while they last!

A press conference will kick start the event, as a vehicle for AR companies to launch new products and services. Submit your proposal here and add: “Press Conference” in the title. The event wrap up will feature Marco Tempest with a live AR Magic show

I have been so busy working on are2011 (and another project yet to be named -  hint I am focused on social augmentation and the world as a platform for a geo-situated social interest graph), that I have had no time to blog since I was on The Potential for Augmented Reality in Education Panel at SXSW, 2011. But I have posted my slide deck here for my talk, “Enchanted Objects and People: Data Driven AR.”

We are entering a new era of collective engagement possibilities for augmented reality, consumer to consumer brokerages which could unleash the the visions of the collaborative consumption movement and tap into our collective energies in totally new ways.  As Seth Praebatsch, Scavngr, suggested in his SXSW keynote, The Game Layer on Top of the World, the power of communal game play may even help us address even the most intractable problems like failure in education and global warming!!

One of my favorite new AR ventures exploring the potential of new forms of social engagement through AR is Suwappu.  We are working on how to get them across the pond and a few time zones to are2011.

“Dentsu London are developing an original product called Suwappu. Suwappu are woodland creatures that swap pants, toys that come to life in augmented reality. BERG have been brought in as consultant inventors, and we’ve made this film. Have a look!”

Click here or on the image above to watch the video.

“Suwappu is a range of toys, animal characters that live in little digital worlds. The physical toys are canvasses upon which we can paint worlds, through a phone (or tablet) lens we can see into the narratives, games and media in which they live.”

Many new players in Augmented Reality will join the pioneering AR companies, Layar, Metaio, Occipital, Ogmento, Total Immersion, YDreams, and renowned innovators, Mark Billinghurst, Steven Feiner, Blair MacIntyre and the Georgia Tech team and many others will return to surprise and amaze us this year – see the still growing list of speakers here.   Qualcomm, who announced an SDK for vision based augmented reality, and $200,000 developer challenge last year, will also be back with several of their team at are2011!

Visual Search and OCR augmented reality apps have flourished in the last year and they will have a strong presence at are2011.  Long anticipated and oft fretted over face recognition apps are coming to a phone near you soon.  Viewdle, working locally on the phone, is taking a new approach to face recognition.  Google has so far said that they will not use or release the face recognition technology that they apparently already have.  But Shailesh Nalawadi, Google Goggles, will return to are2011 to discuss building out a computer vision search database and 3D Classifiers to solve AR challenges and show us what Google Goggles has been working on recently.  Aurasma, out next month, is perhaps the next big thing in this space, so we are all eager to try it out and see what they bring to ARE2011.

My interview with Anselm Hook, Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, is still a great reference point for important questions in this space.

Wordlens, wowed us all with OCR done locally on the phone, and they will be  showing off new features at ARE2011. Pulling out WordLens to translate at subway poster for the first time is a magic moment, and guaranteed to impress the person sitting next to you too!    Tactical Information Systems, who Fred Wilson said had the potential to be a Shazam for faces at Web 2.0 Expo, NYC, will show us what they have been up to since then. 

Data driven AR is still in the early stages, but it is data that will unlock augmented experiences of the world around us in many powerful ways.   Also, we are seeing exciting new efforts to put people at the center of their data.  Jeremie Miller who ushered in the Twitter era with his invention of XMPP has brought us a new real time protocol, TeleHash and the Locker Project, that will unleash the full potential of communication at the edge of the network.   And don’t miss Ben Cerveny and Bloom,  at ARE2011, with “pop-cultural instruments for data expression and exploration” and “where data visualization meets game design,”  I highly recommend Ben’s talk!

Also I am very excited that Mike Kuniavsky will be speaking at  ARE2011 for the first time.  He  pointed out in 2007 that Magic is a powerful core interaction metphor for Ubiquitous computing [and AR].  Offices. libraries, and the desktop provided us with common metaphors that unleashed the power of the PC through files, desktops and trash cans, browsing, etc.,  but metaphors of magic – “enchanted objects and people,” are keys to augmented reality experiences.   And, if Magic is a core interaction metaphor for AR, kinect hacks have shown us that gesture will be the soul of the AR experience.  Jaron Lanier’s keynote will be a must see and make sure you catch Chris Arkenberg’s presentation on the Future of Hands Free AR.

This post is just a glimpse at some of what will be going on at Augmented Reality Event.  I cannot do justice to everything in a single post, so I hope to see you there!  And, if by any chance you have a few moments to daydream between now and then, feel free to ponder your own script for a Green Lantern movie:  “In brightest day and darkest night, no evil will escape my sight.”  I will too.  As @brendanscully noted, “Green lantern’s power is Augmented Reality!”

The Points of Control map is interactive, so please click here or on the image above for the full experience.

Today at 4pm EST, 1pm PDT John Battelle and Tim O’Reilly will discuss the Points of Control map and The Battle for the Internet Economy in a Free Webcast:

“More than any time in the history of the Web, incumbents in the network economy are consolidating their power and staking new claims to key points of control. It’s clear that the internet industry has moved into a battle to dominate the Internet Economy.

John Battelle and Tim O’Reilly will debate and discuss these shifting points of control as the board becomes increasingly crowded. They’ll map critical inflection points and identify key players who are clashing to control services and infrastructure as they attempt to expand their territories. They’ll also explore the effect these chokepoints could have on people, government, and the future of technology innovation.”

I’ve been wanting to start a discussion on the  Points of Control map in the Augmented Reality community for a while now, and Chris’ recent post on the latest edition of the Gartner Hype Cycle, “Is AR Ready for the Trough of Disillusionment?” and this post by Mac Slocum, “How Augmented Reality Apps Can Catch On,” and the conversation in the comments between Mac, Raimo (one of the founders of Layar), and Chris, all prompted me to get a conversation started…(see below for all that followed!).  Chris put me on the hot seat back in June when he did this very generous interview with me on Boing Boing, so it was time to turn the tables.

Tim O’Reilly, in his  keynote for Web 2.0 Expo, pointed out there is both a fun and a dark side to the Points of Control map.  There are companies on this map, he noted, that rather than “growing the pie,” are trying to divide up the pie, and they are forgetting to think about creating a sustainable ecosystem. I expect the conversation between Tim O’Reilly and John Battelle to dig deep into this Battle for the Internet Economy.  If, like me, you have another engagement at the time of the webcast, you can register on the site to receive the recording.

AR is still too young to figure in the battles of the giants, but there will be a lot to be learned from this conversation.  And, The Points of Control map is good to think with from the POV of AR in many ways.  As Chris Arkenberg observed:

“When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?”   You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development.  And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.”

Chris Arkenberg is a technology professional with a focus on product strategy & development, specializing in 3D, augmented reality, ubicomp and the social web. He uses research, scenario planning, and foresight methodologies to help organizations anticipate change and adopt a resilient and forward-looking posture in the face of unprecedented uncertainty. His personal work is collected at urbeingrecorded, and his professional profile is here.

He is also one of the founder/organizers of AR DevCamp which is currently scheduled for Dec. 4th (somewhere in SF or The Valley!)  Chris said, “No further details atm (still trying to find a venue and get sponsors) but please direct people to http://ardevcamp.org for upcoming information.”

Talking with Chris Arkenberg

Tish Shute: I know some people thought the positioning of AR by Gartner near the peak of the hype cycle was misguided, and based on a very narrow understanding of AR as used in marketing apps. But reading your post I thought you made a lot of good points.

Chris Arkenberg: It’s tracking hype, right? It’s not necessarily tracking the growth of the technologies or their maturation so much as it’s tracking the general attention level. And what’s interesting to me is that tends to affect the amount of money that goes into those technologies.

Tish Shute: I was particularly interested in your post because I have been writing a post about two recent O’Reilly events in NYC, Maker Faire, Web 2.0 Expo, and then Hadoop World, where Tim gave a very interesting 45 minute keynote.   AR was pretty low profile at all three events.   But the NVidia augmented reality demo attracted a lot of attention at the sponsors expo, and Usman Haque, Founder of Pachube announced in his presentation, they are working on an augmented reality interface for Pachube called Porthole, its designed for facilities management and, “as a consumer-oriented application that extends the universe of Pachube data into the context of AR – a ‘porthole’ into Pachube’s data environments.. “  Usman also mentioned, when I talked to him, that he is contributing to the AR standards discussion and on the program committee now for the W3C group on augmented reality.  For more on this standards discussion and the Pachube AR interface, see Chris Burman’s paper for the W3C, Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual.”

I think pioneers in the augmented reality commmunity should pay attention to these wider conversations about the Battle for the Internet Economy, and the exploration of the  “Platforms for Growth” theme at Web 2.0 Expo is very important- this is a course also a nudge to read my upcoming post on these O’Reilly events!

Also I have another project I have been chewing on that I would like to talk to you about.   I want to start an AR conversation about the wonderful Points of Control map produce for Web 2.0 summit by John Battelle. [ Note there will be, "Battle for the Internet Economy" free Web2Summit webcast w/ @johnbattelle & @timoreilly Wed 10/27 at 1pm PT http://bit.ly/b46cmb #w2s]

Up to this point, understandably given the immaturity of the technology, AR has little role in the “Battle for the Internet Economy.”  But this doesn’t mean that the map isn’t good for AR visionaries, enthusiasts, entrepreneurs, and developers to think with.   And both you and Tim have pointed out the potential for AR to leverage the giant data subsystems in the sky.   I have to say the positioning of Cloud Computing on the brink of heading down into the trough of disillusionment in this recent rendition of the Gartner Hype Cycle seems ridiculous!

Cloud Computing is already ubiquitous hardly seems credible that it is headed for a trough of disillusionment!

Chris Arkenberg: Yeah, it’s ubiquitous so why even talk about it when it’s your fundamental infrastructure?

Tish Shute: Yeah and I seriously doubt it is imminently headed for a trough of disillusionment….and this brings me back to the Points of Control Map which as John Batelle points out, “aims to identify key players who are battling to control the services and infrastructure of a websquared world” in which the “Web and the world intertwine through mobile and sensor platforms.”  This instrumented world, of course, creates a great deal of opportunity for augmented reality. Have you seen that, that points of control map?

Chris Arkenberg: I think I have, actually.

Tish Shute: There has been much debate about how this intertwining of the web and the world will play out in augmented reality.   Chris Burman points out in his position paper for W3C,  Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual”, that “trying to draw parallels between a browser based web and the possibilities of AR may solve issues of information distribution in the short-term,”  but it must not have a limiting effect in the long-term.   But now we at least have one web standards-based browser for AR thanks to the work of Blair MacIntyre and the Georgia Tech team.  But I think the discussion in the comments of Mac Slocum’s recent post, “How Augmented Reality Apps Can Catch On” is an interesting starting point from which to think about platforms of growth for AR.  I am not sure if I am stretching his meaning but I think Raimo, Layar, is suggesting that what the Point of Control map call the the Plains of Media content is very important to the growth of the fledgling AR industry right now. And I would agree with this, and add that the neighboring terrain of gaming levels will be pretty key as one of my other favorite AR start ups Ogmento hopes to reveal in the near future! But what do you think was most important in this brief but pithy dialogue between you Raimo and Mac?

[The screenshot above is a teaser video the Gary Hayes from MUVEdesign for his upcoming (2011 release date), game called Time Treasure.  See Gary’s blog for more and Gary’s  post from over a year ago on AR Business models.  Thomas K. Carpenter, on Games Alfresco notes, “I think this is a terrific idea and I find it interesting he’s planning this on a tablet rather than a smartphone.”

Chris Arkenberg: The way I took it…And to give a little bit of context, I came from sort of this apprehension of augmented reality as an expression of the existing Internet. So as sort of a visualization layer that allows you to kind of draw out data, and then, with all the affordances of being able to anchor it to real world things.

And my own sort of path has led me to want to really try to understand that and refine it, particularly with respect to the sort of Internet of things and the smarter planet idea of just having embedded systems everywhere. And specifically, what is the value-add for augmented reality as a visualization layer of an instrumented world?

And so that’s caused me to be a bit biased towards that side of AR. And the way I took Raimo’s comment was that he was saying that, “You know, really what we’re interested in is media.” That he was effectively saying that AR for them is really just about that space between the screen and the the world, or between your eyes and the world, and what you can do there.

Certainly I had considered it in the past, but I hadn’t really focused on it or assumed that it was a priority as a business model. And so he kind of reminded me that, actually, there’s a lot of entertainment applications. There’s a lot of, obviously, advertising and marketing applications.
And so I felt that I was being a little narrow in my focus…

Tish Shute: Yes this comes to the heart of what I am interested in about the role AR can play in opening up new relationships to the world of data that we live, not just making it more accessible and useful to us when and where we need it, but AR as a road to reimagining  it..

Have you seen any interesting work yet to explore these great data economies in the cloud through AR. I mean can you think of any others – there is planefinder.net but others?

Chris Arkenberg: I’ve seen a few just sort of skunk works type applications that people have been playing around with, again, to try and reveal things. One of them was similar to the aircraft, but it was more for military use and being able to identify things of interest in the sky. I’ve seen a couple other for navigation, so being able to identify mountain peaks on a visual plane, for example, but this isn’t so much about revealing an instrumented world.

Tish Shute: Yeah, I think that was from the Imagination right? I know that’s an interesting one. Usman at Web 2.0 Expo, in his presentation, mentioned the work Pachube is doing on an Augmented Reality interface. I interviewed Usman again as my last long interview with him was nearly 18 months ago now and Pachube is well on the way to becoming the Facebook of Data or the analogy that Usman prefers – the Twitter of sensors!

Chris Arkenberg: Hmm, interesting.

Tish Shute: And to go back to your comments on Augmented Reality not getting caught in some of the traps that have made virtual worlds lose relevancy I think that is vital that AR developers understand the strategic possibilities of key points of control in the internet economy because the isolation and Balkanization of virtual worlds were certainly a factor in their rapid slide into the trough of disillusionment – although many would argue that a fundamental flaw in the kind of virtual experience that Second Life and other virtual worlds constructed was really the fatal flaw (see James Turner’s interview with Kevin Slavin Reality has a gaming layer).

But Second Life’s isolation from the other great network economies of the internet was certainly a limiting factor.

Chris Arkenberg: And that’s been exactly my sense, and I’ve, over the years, tried to encourage development in that direction for virtual worlds. I did work, through Adobe, to help develop Atmosphere 3D back in the the early 2000’s. And we did a lot of work to try and understand the marketplace and the specific value-add of doing things in 3D over 2D.

And this is kind of why I keep referring back to VR and VW’s with respect to augmented reality, is that with immersive worlds, there was this idea…there was this big rush. Everybody was so excited about it. It was obviously the next cool thing. And everybody wanted to try to do everything in it. You could do your shopping in virtual worlds. You could have meetings in virtual worlds.

Tish Shute: and shopping, yes ..that didn’t work out so well!

Chris Arkenberg: And everybody was very excited in developing these things. And what it really came down it is, “Yeah, you can, but it’s actually a lot better to do those things on a flat plane or in person.” Meeting Place, WebEx, TelePresence – those tools generally do a much better job at facilitating TelePresence meetings than a virtual world does. The same with TelePresent Education. There are only very specific things that both VR and AR are really good at.

And that’s where I find myself with augmented reality right now, trying to really pick through that and critically look at which uses are really appropriate for an AR overlay. And again, I think that’s why the hype cycle is important, because it reflects back this desire that AR is going to be the next big thing – the be-all, end-all of interacting with data in the cloud – and forces us all to take a critical look at why we should do things in AR instead of on a screen.

AR is not going to work well for most things but it’s going to be very good for certain uses. Right now I’m very keen at trying to understand what those things might be.

Tish Shute: I had this wonderful conversation (more in an upcoming post) with Kevin Slavin one of the founders of Area/Code at Web 2.0 Expo and I think some of what he describes about the data brokerages of High Frequency trading have some interesting implications for AR’s role, say, in ubiquitous computing. The trading markets are now pretty much dominated by machine to machine intelligence; machine to machine brokerages. They are basically game economies on the scale that we can barely wrap our heads around where the speed that bots and algo traders can access the network is the key. We really have no clue what is going on until we lose our house…

Kevin was also interviewed by James Turner on O’Reilly Radar. He talked about how much of the interesting work in location based mobile social apps is defined in opposition to the model of Second Life. He also talked to me about how we are seeing “first life” take on the qualities of “second life.” What goes on the trading floor is largely a performance secondary to a more important world of machine intelligence with giant co-located servers and bots fighting for trading advantages measured in fractions of seconds.

He pointed out how we draw on all these tropes from sci-fi movies, these HUDs based on ideas of machine intelligence where the robot talks to the other robot in English through an English HUD!  Many of our current visual tropes for AR are perhaps just as inadequate for the kind of data driven world we live in.

Of course, when you are thinking of having fun with dinosaurs, or illustrated books, or whatever, this is not, perhaps, an issue.  But if you are thinking of augmented reality interfaces as being important in a battle for network economy, and platforms for growth,  how this new interface helps us live better in a world of data is an important issue.

Chris Arkenberg: Now, does that indicate that the UI just needs more overhaul and innovation, or more that the visual interface for those experiences shouldn’t really leave the screen? It shouldn’t move on to the view plane?

Tish Shute: Yes we have a few concept videos that try and explore this ..

Chris Arkenberg: Well, and I think this will happen at the level of human-computer interface. I mean that’s always been its role, in making coherent the sort of machine mind, for lack of a better term, making it coherent to the human mind. So I mean there is a lot of this sort of machine intelligence, the semantic Web 3.0 revolution, where it really is about enabling machines, and agents, and bots to understand the content that we’re feeding them.

But at the end of the day, they, for now, need to be providing value to us human operators. So there’s always going to be a role for human-computer interface and user experience design to make this stuff meaningful.

I mean, if you look at the revolution in visualization & data viz, this is of incredible value because it takes a tremendous amount of data and collates it into a glanceable graphic that you can look at and immediately comprehend massive amounts of data because it’s delivered in a handy, visual way.

So I see that as a fascinating design challenge, how the user experience of the data world can be translated into meaningful human interaction.

Tish Shute: Yeah. And when we see Stamen Design pursuing a big idea in AR, that’s when we might start to rock and roll, right?

Chris Arkenberg: Yeah. In my article, I sort of jokingly suggested that Apple will create the iShades. But, they’ve got the track record of being way ahead of the curve and delivering the future in very bold forms.

Tish Shute: A key part for the battle for the network economy is to bring the complexity of data into the human realm in a way that increases human agency. Kevin suggests that the giant robot casinos of markets should actually lift off into total abstractions as theses machine-driven trades get back into the human realm in ways that are so damaging to our lives – a lost house or job! The notion of a counterveillence society where people have more agency over the important aspects of their lives, health, housing, job (which I discussed with Kevin – interview upcoming) has gotten pretty tricky!

But I think we will begin to see AR eyewear for specific applications (gaming and industrial) get more common fairly soon – possibly as smart phone accessories.

And it is clear that AR is going to be, increasingly,  a part of our entertainment smorgesborg in coming months. Itouch has a camera (although lower resolution), Nintendo’s are AR-ready and many aspects of the AR vision of hands-free spatial interfaces will go mainstream through Natal.

But we are yet to see an app/platform emerge for mobile. Social AR games that turn every bar and cafe and ultimately the whole city into a gaming venue  -although I think Ogmento and MUVE aim to lead the way here! Will an AR company achieve Zynga level success by using the Foursquare, for example?

My feeling is that the lesson of Zynga is pretty important for mobile social AR games. Could Flash social gaming have taken off without Facebook?

Chris Arkenberg: And that’s the real driver. And again, as you mentioned with Second Life, and this was exactly my own sense, is that they stuck to the closed garden model and didn’t get the power of social and collaboration. They attempted to add some of those affordances within the world, but, you know, ultimately most people aren’t in virtual worlds, and most people aren’t using augmented reality. So leveraging the really predominate platforms like Twitter and Facebook and Foursquare, being able to leverage those affordances, that connectivity, into a platform like augmented reality, I think, is really critical. Because again, you get nothing unless you have the masses, unless you have people present.

Tish Shute: In AR research there is a long history of the notion of powerful AR-dedicated devices, but smart phones and tablets are good enough,  and can launch augmented reality into the heart of the internet economy. I think  the elusive AR eyewear will come to us initially as a smart phone accessory for specific apps.  But, for the moment, most AR apps make little attempt to play in the wider internet economy.

Chris Arkenberg: And I think it’s actually much lower hanging fruit, really, to do gaming, marketing, transmedia. Because then you don’t really care about the cloud, or maybe you only really care about a little part of it that your gaming property is addressing. Then it becomes much more about entertainment, and much more about persuasion, and sensationalism. And if you’ve got dancing dinosaurs on your street, great! It’s entertaining, it’s cool, it’s new. That stuff is fairly straightforward.

I keep coming back to this idea of, you know, the instrumented city. What sort of data trails do you get out of a fully instrumented city? So maybe you get traffic patterns, maybe you get geo-local movements of masses, maybe you get energy usage, that sort of thing, all the, sort of heat maps you can generate from a city. But then what good does it do to be able to have that on an augmented reality layer versus just looking at it on a mobile device or looking at it on your laptop?

Tish Shute: Of course the use cases for “magic lens” AR are different from the kind of hands free, 360 view with tightly registered media, that a full vision of AR has always promised. The 360 view is quite a different metaphor from the web and mobile rectangular screens.

Chris Arkenberg: Yes, yes.

Tish Shute: Did you see that great parody of Michael Jackson’s “Beat It” with the iPads versus the iPhones, right?

Chris Arkenberg: Oh, really?

Tish Shute: I tweeted it cos i thought it was quite funny and a little close to the bone!
[laughter]

“ur wanna an ipatch 2 b the new fad?” #AR gets cameo in Twitter, iPads & iPhone’s Michael Jackson-Inspired Parody via @mashable

It is hard to get away from the importance of eyewear when discussing AR!

Chris Arkenberg: Yes, so the hardware, to me, is a big stumbling point right now, or it’s a large gating factor, I think, for realizing what an augmented reality vision could really be like. That it really does need to be heads up. This holding the phone up in front of you is fun to demonstrate that it’s possible, and it’s valuable in some ways…

Tish Shute: And it’s particularly nice in some applications like the planes app, the Acrossair subway app where you hold the phone down and get the arrow, right?

Chris Arkenberg: Yeah, the way-finding stuff I think is really valuable…

Tish Shute: Sixth Sense really caught people’s imagination because it managed to deliver the gesture interface with cheap hardware, even if projection has limited uses (no brightly lit spaces or privacy for example!).

The other important and as yet unrealized part of the AR dream is real-time communications. Many interesting uses cases would require this. As you know that is my chief excitement, along with federation, in the Google Wave Servers for (which should soon be released at Wave in a Box) for ARWave.

Chris Arkenberg: Well my sense of Wave is that it was a ChromeOS protocol that they instantiated, or that they exhibited in the public deployment of Google Wave. That that was a proof of their sort of low level architectural solution. Because, you know, they’ve been rumored to be working on this cloud OS for some time. And so my sense is that Wave is actually one of their core components of that cloud OS, and that it just happened to incarnate for the public in a test run as Google Wave.

Tish Shute: I do hope that Wave In the Box will lower the barriers to entry to people experimenting with this technology. The FedOne server was just way too hard for most people to take the time to set up. Of course, it is the brilliance of the Wave Operational Transform work that also poses problems in terms of ease of use. But Wave Federation Protocol is pretty innovative. And could even play an important role in a real time communications for AR eyewear connected to smartphones. The challenges that Wave takes on re real-time communications, federation, permissions and filters are pretty important ones for AR…

Chris Arkenberg: Especially when you’re trying to federate a lot of permissions and filter a lot of data, which all of that gets even more important when you have a visual layer between you and the real world.

Tish Shute: You got it. Yeah!

Chris Arkenberg: I think that’s really valuable real estate, both for third parties that want to get access to your eyes, as well as for you, as the user, who still needs to navigate through the phenomenal world and not be occluded by massive amounts of overhead data.

Tish Shute: Yes, I am sure Google has big plans for the next level of cloud computing and Wave looks at some key challenges. I suppose federation poses some key business problems. I think it was Michael Jones who said to me that it was a bit like socialism in that you have to be willing to give something up for the greater good.

Perhaps federation does not present enough appeal because of its challenges re business models?

Chris Arkenberg: Well, I wonder. I mean there’s got to be some value for their ad platform as ads are moving more towards this personalized experience. Advertising is becoming less of a shotgun blast and more of a very precise, surgical strike. So being able to track user data to such a fine degree to mobilize the appropriate ads around them wherever they are, on any platform, is certainly very valuable to Google and their ad ecology.

Tish Shute: Many people have high hopes that HTML 5 by lowering the barrier of entry for  browser style AR could also pave the way for some interesting AR work..

Chris Arkenberg: Well, as much as I would hope that all the different players are going to come together and establish some shared set of standards, really, what’s happening is it’s a rush to the finish line to be the first…to get the most penetration in the marketplace so that Layar, for example, can say, “It’s official. We’re the platform.” And then the consolidation that will follow, where the Googles and the other big players like Qualcomm say, “OK, it’s mature enough. We’ll start buying up all the smaller companies.”

And that’s where the real challenge is right now is that there are no standards. It’s such an immature technology that you have a lot of different players trying to establish the ground rules. And again, this is one of the challenges that faced public virtual worlds, is that you had a lot of different virtual worlds that weren’t talking to each other in any particular way, and that they each had their own development platform. And so you end up with a very fractured ecosystem or set of competing ecosystems, which is kind of what’s happening with AR right now, where a developer has to choose between a number of different new platforms or hedge by deploying across multiple platforms. Basically, the web browser wars are set to be recapitulated by the AR browsers.

Among them, Layar and Metaio seem to be getting the most traction. But there’s still not a really strong case for a unified development ecosystem to emerge.

Tish Shute: So a discussion of ecosystem development brings us back to the Points of Control Map I think. So what do you see as key points of interest for AR developers to watch in the Points of Control Map? And where do you want to sort of put your bets, right? We are still really waiting for mobile social AR to emerge into the mainstream.

Chris Arkenberg: Yes. And that’s primarily the shortcoming of the hardware itself, but also of the accuracy of current GPS technology. That’s another kind of gating factor, because again, AR wants to be able to express the data within a distinct place or object.

So in a lot of ways, other than kind of what we’ve allowed for the broader entertainment purposes, for AR to really work, there needs to be more resolution in GPS location. So for it to be truly locative…because it’s OK to tell Foursquare that you’re in Bar X. But if you want to be able to draw data directly on a wall within that bar, or do advertising over the marquee on the front, you need more factors to accurately register those images on a discreet location. So that’s another, sort of, aspect of the immaturity of AR, is that it’s still very hard to register things on discreet locations without employing a number of diverse triangulation methods.

Tish Shute: Right. The mobile AR games we see at the moment are really just faking a relationship to the physical world unless they rely on markers or some limited form of natural feature recognition which is really just a more sophisticated form of markers. But the Qualcomm SDK does offer some opportunities to tie AR media to the world more tightly as does the Metaio SDK. But in terms of a mobile social AR game that could be like the Cape of Zynga to FourSquare in Location Basin [see the Points of Control map]… We haven’t seen anything close yet.

AR should be able to bring the check-in mode to any object in our environment.

Chris Arkenberg: Yes, yes. And that’s actually one of the early interests I had in the notion of social augmented reality. I wanted a way to tag my community with invisible annotations that only certain people could read, and found pretty quickly that that’s very difficult to do. I mean you can kind of do some regional tagging, like on a beach, for example, but if you wanted to tag the bench that was on the cliff above the beach, it’s very difficult to do that using strictly locative reckoning.

There’s all sorts of really cool social engagement that can be revealed when people are allowed to attach things to the world around them, to the streets they normally pass through, or the points of interest that they normally engage in. To be able to author on the fly on the streets and attach it discreetly to an object effectively.

Tish Shute: And yes we do have all kinds of markers and QR codes. But Erick Schonfeld of Tech Crunch made a good point that QR codes: “Until QR code scanners become a default feature of most smartphones and they start to become actually useful enough for people to go through the trouble to scan them, they will remain a gee-whiz feature nobody uses.”

Chris Arkenberg: So again, this gets back to competing standards and who gets access to the phone stack, the bundle. Who gets the OEM deal…?

Tish Shute: Yes, the battles for the networks on the Handset Plains are pretty important for AR!
[laughter] I think Layar have made some smart moves on The Handset Plains.

And there are a lot of acquisitions of nearfield technology to look at.  If I remember rightly Ebay bought the Red Laser tech from Occipital – now there’s any interesting company. Their panorama stuff rocks!

Chris Arkenberg: Right. There’s a lot of nearfield stuff that’s supposed to hit all of the major mobile platforms in the next year or so.

I mean I think where this is heading, in my mind, is basically smart motes. You know, little nearfield wide-range RFID’s that are the size of a small, tiny square that you could attach to just about anything and then program it to be a representative of your establishment or of an object, that then you can start to tag just about anything. I mean you can’t rely on geo to do it, but if you have a Nearfield chip there that costs maybe like two cents to buy in bulk, and you can flash program it, then you can start to attach data to just about anything.

Tish Shute: Yes ‘cos some things still remain very difficult for near field image recognition technologies like Google Goggles.

Chris Arkenberg: Well, if your phone can interrogate for Nearfield devices, and it detects a chip in its near field, it can then interrogate that chip. The chip may contain flash data on itself, or it may contain the local server in the establishment, or it may go to the cloud and get that data back.

Tish Shute: Yes there is moverment from the top and open source hardware like Arduino has created an opportunity for all sorts of creativity with instrumented environments.  And the handheld sensors in our pockets – our smart phones create a lot of opportunity for bottom up innovation too.

Chris Arkenberg: I mean that’s my guess. If you look at what IBM is doing with their Smarter Planet initiative, they’re partnering with a lot of municipalities, and obviously with a lot of businesses and their global supply chains.

But they’re basically working with municipalities and all these stakeholders to instrument their territory, their business, or their city, as it were. So they’re working to provide embedded sensors and the software necessary to read them out and run reports & viz. And presumably that software can extend to include some sort of mobile device to interrogate the sensors and read the data.

That’s kind of a top-down approach of a very large global company working with top-down governance bodies to do this. Simultaneously you have the maker crowd experimenting with Arduino and such to build from the grassroots, the bottom up approach.

And that’s primarily gated by the amount of learning it takes to be able to program these devices, to be able to hack them. Typically, the grassroots creators who make these devices don’t have the luxury of very large budgets to make things highly usable and Wizywig.

So the bottom up community is a sandbox to create tremendous amounts of innovation, because they are unconstrained by the very real financial needs of the top down innovators. And so you get a lot of fascinating innovation, a very rich ecology from the bottom-up approach, but you don’t get a lot of wide distribution. But that does filter up to and inform the top down approach that has a lot more money to put into this stuff. And it ultimately has to respond to the needs of the marketplace.

I mean if there’s an answer to the question of whether something like AR will succeed through the bottom-up grassroots approach or the top-down industry approach, I would say it would be both. That handsets will be hacked to read the bottom up innovations of the maker community, and handsets will be preprogrammed to read the top down efforts of the IBMs of the world.

Tish Shute: Yes but i have to say it is very time-consuming hacking phones (I have just seen a few days suck up in this myself so that I could upgrade my G1 to try out the new ARWave client!). I mean Android has obviously been the platform of choice because of openness but the business model of iPhone and its market share in the US sure make it important for developers.  It’s like you don’t exist if you don’t have an iphone app for what you are doing.

Chris Arkenberg: Yeah, and that’s the challenge, because at the end of the day developers prefer not to work for free and a solid, reliable mechanism to monetize their efforts becomes very appealing.

When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?” You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development. And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.

Tish Shute: Some of the important platforms for AR don’t appear to have spots on the map like Google Street View and other mapping technologies that hold out so much hope for AR, or am I missing something?

Chris Arkenberg: You mean on the map?

Tish Shute: Yes for the full vision of AR we need sensor integration, computer vision and cool mapping technologies to come together. Do you see where Google Maps and Google Street View… Where would they be?

Chris Arkenberg: Yeah, I mean it’s certainly content, it’s location…

Are you familiar with Earthmine?

Tish Shute: Yes, yes I am, definitely. Earth Mine, Simple Geo, Google Street View, user generated internet photo sets like Flickr all of these could be very important to AR, potentially.

Chris Arkenberg: Well, and the interesting thing about Earthmine is that they’re effectively trying to do an extremely precise pixel to pixel location mapping. So they’re taking pictures of cities just like Street View, except they’re using the Z axis to interrogate depth and then using very precise geolocation to attach a GPS signature to each pixel that they’re registering in their images. Effectively, you get a one-to-one data set between pixels and locations. And so you can look at something like Google Street View, and if you point to the side of a building, in theory, it should know exactly where that is.

They’re rolling this out with the idea of being able to tag augmented reality objects in layers directly to surfaces in the real world. So that’s another approach to trying to get accurate registration and to try and create what are essentially mirror worlds. Then your Google Street View becomes a canvas for authoring the blended world, because if you plop a 3D object into Street View on your desktop, and then you go out to that location with your AR headset, you’ll see that 3D object on the actual street.

Tish Shute: There was some experimental work with Google Earth as a platform for a kind of simulated AR but I suppose Google Earth doesn’t figure in the battle for the network economy as it never got developed as a platform.

Chris Arkenberg: It hasn’t tried to become a platform, to my knowledge. I mean I know some people are doing stuff with it, but as far as I know, Google owns it, they did it the best because they have the best maps, and there’s not a huge ecosystem of development that’s based around it other than content layers.

And my sense of everything else on the Points of Control map is they’re looking more at these sort of platform technologies that…

Tish Shute: Yes, re platforms for growth for AR. Gaming consoles will probably emerge as a significant platform for AR this year.

Chris Arkenberg: There will be much more of a blended reality experience in the living room for sure, and with interactive billboards. Digital mirrors are another area. So I mean if we kind of extend AR to include just blended reality in general, you know, this is moving into our culture through a number of different points. As you mentioned, it will be in the living room, it will be in our department stores where you can preview different outfits in their mirror. We’re already seeing these giant interactive digital billboards in Times Square and other areas.

It’s funny. I mean for me, the sort of blended reality aside, the augmented reality, to me, is actually a very simple proposition in some respects. When I look at this map, augmented reality is just an interface layer to this map in my mind, just as it’s an interface layer to the cloud and it’s an interface layer to the instrumented world. It’s a way to get information out of our devices and onto the world.

Tish Shute: The importance of leveraging existing platforms has become pretty clear but it is interesting Facebook definitely gave Zynga the opportunity but would Facebook be so big without Zinga’s social gaming boost?

Chris Arkenberg: I feel that Zynga has definitely helped its growth…But I think Zynga has benefited a lot more from Facebook than Facebook has from Zynga.

Tish Shute: Zynga certainly proved you could build a profitable business on Facebook’s API!

Chris Arkenberg: They did. And they also really validated the Facebook ecosystem and the platform. They really extended it… Zynga benefited from the massive social affordances that Facebook had already architected and developed. They brought gaming directly into Facebook, and particularly, this emerging brand of lightweight social gaming that when you sit it on top of a massive global social network like Facebook, it suddenly lights up.

Tish Shute: AR pioneers should quite carefully go through this map. There is so much to think about here. I’m a kind of fanatic about Streams of Activity in AR. Real time brokerages and their potential for AR is something I am fascinated by. That is one reason I love the ARWave project.

Anselm Hook, to me, is one of the great thinkers in this area of real time brokerages – with his project Angel, and the work of Ushahidi, which is now the platform for augmented foraging (see here). Anselm is now working on AR at PARC which is exciting.

Chris Arkenberg: Well, there are some challenges working with data streams. Presentation and filtering I think is a big challenge with any sort of stream. Because obviously, you have a lot of potential data to manage, to parse, and to make valuable and comprehensible. So I think this is bound very closely to being able to personalize experiences, or having very discreet valuable experiences. Disaster relief, for example, I think is an interesting idea that ties into the Pachube type of work. Where, if you had the headset and you were a relief worker, and you had immediate lightweight, non-intrusive, heads up alpha channel overlay, waypoint markers showing you all of the disaster locations or points of need, AR becomes extremely valuable, because it’s a primarily hands-free environment. This is why the military stuff is so interesting.

Tish Shute: Ha! We are running into the eye patch/shades/goggles/sexy specs thing again. But filtering and making streams of activity relevant will be very interesting for AR.  Again that it why I love the Wave Federation Protocol work because what they have built into their XMPP extensions. You can have your real-time personal data streams, or community streams, or broadcast publicly – the permissions are built.

And Thomas Wrobel’s original vision of these layers and channels is only fully expressed if you have the eyewear.

Chris Arkenberg: Well, and it becomes redundant if it’s on a mobile. To use a very basic example, Twitter, obviously there’s an app you can view those streams of activity on the camera stream. But you can view that real time data on the screen. Why do you need to see it heads up?

The reason I really pay attention to what the military is investing in, one, because they have a ton of money, but also because they tend to represent the core bio survival needs of the species…So, when I look at computing, I see this very obvious trend of computers getting smaller and smaller and closer and closer to us because they’re so valuable to our success. They give us so much valuable information for engaging our world on a moment by moment basis. So, of course now we have these tiny little handheld devices that give us access to the global knowledge depositories of human history, because it’s so useful to have that stuff right at hand.

The only impediment now is that it takes one of our hands, if not both of them, to access it. So if you are in the natural world, which we are all always in the natural world, ultimately, you want your hands free in order to engage with the world on a physical level.

I see computation, or rather, our access to computation is just going to get thinner and thinner, and we’ll very soon move into eyewear, and inevitably, we’ll move into brain computer interface in some capacity.

So when you’re the disaster worker, or a deployed soldier, or the extreme mountain biker, or the heli-skier, or just an adventurer, there are a lot of very practical reasons to have access to information on a heads-up plane. I see AR as being so profound and so valuable, but we’re getting a glimpse of it in its infancy, and it’s got a ways to go to be able to really contain what it is we’re reaching for.

Tish Shute: I agree.

Chris Arkenberg: And that’s been a big criticism I’ve had with all the existing AR implementations that I’ve seen, is that the UI really needs a revolution. It’s very heavy handed. It is not dynamic, even though it’s supposed to be. It does not take advantage of transparencies. It treats the screen like a screen. It doesn’t treat the screen like a window onto the real world. When you’re looking on the real world, you don’t want a lot of occlusion. You want very soft-touch indicators of a data shadow behind something that you can then address and then have it call out the information that’s important to you.

Tish Shute: Now, that’s a very nice kind of image you’ve conjured for me there. Do you see that more could be done on the smartphone than is being done within that? Or are we like waiting for the old ishades?

Chris Arkenberg: I think there’s definitely a lot of room for improvement on the smartphone UI. Nobody’s really played around with it much. And again, I think that’s in part that there hasn’t been a really established platform with enough money to fund interesting UI work. We see it in some of the concept demos that float around every now and then.

I guess it’s both a blessing and curse that I’m always five steps ahead of where I’m trying to get to.

Tish Shute: Yeah, I am familiar with that feeling!

Chris Arkenberg: So I’m always trying to reach for the vision even though it’s a bit distant. I think there’s going to be a lot of development on the handsets. But again, I think we need a lot of refinement. We need a lot of real critical analysis of why this is a good thing.

To get back to the original point of Raimo’s comment, it struck me. And I knew it, but I just had set it aside as gimmickry. But he’s right. Content is a huge driver for this. Just stuff that’s engaging, and fun, and cool, and shows off the technology so they can get enough money to make it through whatever Trough of Disappointment may be waiting.

Tish Shute: Yeah, don’t underestimate the Planes of Content!  They are a great place to get interest and money to keep AR technology moving on, right?

Chris Arkenberg: Yeah, yeah. Because, you know, there’s a lot of freedom there. And you can piggyback on all the rest of the content that’s out there and jump on memes and marketing objectives, etc…

And there’s a lot of stuff…I’m blanking on some of the names, but some of these historical recreations of city streets. There’s a street in London where they overlaid historical photos in a really compelling experience. [Museum of London - http://www.museumoflondon.org.uk/] Again, I’m completely forgetting the attributions, but hose are the type of things that can really be pursued on the existing platforms. There is stuff that’s really compelling and really cool.

I heard of another interesting use case – and I should say that I can’t find attributions to this anywhere on the web and I may be paraphrasing or mis-representing the actual work, but I think the concept is worth exploring anyway. But the idea was that you could take the locations of border checkpoints and conflict sites in Palestine and Israel and visually overlay them on an AR layer in San Francisco. And it would do some sort of transposition where you could virtually view these things in San Francisco with the same locational mapping superimposed. So you could see where the checkpoints where. You could see where the wall was. You could see where suicide bombings were and where there had been conflicts. [I cannot find any citations for this!]

Tish Shute: But with an AR view? But why would you use an AR view if you are in San Francisco, then?

Chris Arkenberg: Because it superimposes two realities, translating the Gaza conflict into San Francisco as you are walking around. You can interrogate the world. There’s a discoverability aspect where you’re using the headset to reveal things, or the handset rather, to reveal things that you could not see otherwise in your city. It was done as an art piece, but as a provocative, obviously political art piece.

Tish Shute: Very interesting. I’d love to see that. Because that’s interesting to get away from this idea that you actually have to sort of have this one to one relationship between the data and the world is kinda nice, isn’t it? Well, not one to one, but a very literal…getting away from that literalness is kind of good.

Chris Arkenberg: And that’s a possibility of virtual reality and augmented reality merging, that maybe virtual reality is actually going to do best by coming out of the box and writing itself over our reality, so that as you are walking around, you are no longer seeing San Francisco, but you are seeing part of Everquest or World of Warcraft.

Tish Shute: Well this is where Bruce Sterling gets to that point he made in his keynote for are2010, that if we actually have viable AR eyewear, then you get the gothic stepsister of AR, VR rising from the grave!  He asks whether the very charm of augmented reality, is in fact that it adds rather than subtracts from your engagement with the world and that getting get sucked back into the black hole of VR might not be so great.

Chris Arkenberg: And then you get all sorts of interesting challenges to social cohesion if you have a lot of different people experiencing very different worlds, effectively. That if there is no real consensual reality and a majority of your local populous is, in fact, experiencing very different and unique versions of the world, what does that do to social cohesion? How does that reinforce tribalism, for example, when only you and certain others get to opt in to a particular layer view of the world?

Tish Shute: Yes Jamais Cascio wrote an interesting piece on that issue on AR and social cohesion a while back.

An eye patch is a more logical vision than the goggles in many ways but I suppose the loss is stereo vision?

Chris Arkenberg: And actually, there were developments in military helicopter technology many years ago that used a single pane square of glass over the eye mounted to the helmets of pilots. And then they drew various bits of heads-up information on it. So that ensures that you’re having a real strong engagement with the real world, which, obviously, when you’re a helicopter pilot is quite important. But you still have access to the data layer of the invisible world.

Tish Shute: I just went to Hadoop World and I have to say, I was awestruck about how big that’s got. I mean Hadoop has gone from like zero to huge in just a few years. I mean it’s just like now everyone has the power of the Google big table at their fingertips.

What’s the play for AR in the land of search?

I could imagine Hadoop being very powerful tool for AR analytics?

Have you got any thoughts on the land of search and AR? Of course visual search is proceeding at a fast pace and there is a lot of promise for integrations with AR in the future but the latency for visual search is still pretty high?

Chris Arkenberg: In the near term, not a lot. In the medium term, there’s a larger trend towards virtual agents that you can program or teach to keep watch over things for you as an effort to scale down the data overload. So search is something that’s going to become more personalized and more active. There’s a movement to make it so people can essentially deputize these agents to be always searching for them; to be out there looking for the things that they have told these agents are important to them.

So active search for AR I think presents some challenges, obviously because you need to do text input, typically, or voice input. Voice input, I think, is much more achievable than text input for AR. But I can certainly imagine an AR layer that is being serviced by these agents that we have roaming around the web for us reconciling their visual view of the world with our personalizations. AR apps are contextually aware so it knows that if you’re downtown, it’s not going to be giving you a ton of information about Software as a Service infrastructure, or what have you. But that, instead, it’s going to be handing you little tidbits about a particular clothing brand you’ve opted in to follow and information about music venues & schedules, for example. Or perhaps you’ll be on the lookout for other users that have opted in to publicly tag themselves as a member of this or that affinity.

I keep coming back to this idea of AR as really just a simple visualization layer that all of these other technologies can potentially feed into. So in that sense, search becomes a passive thing that AR is just simply presenting to you in a heads-up, hands-free, or potentially hands-free environment.

Tish Shute: Yes, the big challenge is the stepping stones to that point! Small steps that keep interest going into developing the underlying technology (and not just in research labs!) that will bring us that interface.  We have seen some movement already with Qualcomm.

Chris Arkenberg: And there are bandwidth issues as well, as we can see with the Google Goggles, which is a great idea of visual search. But you have to take a picture and send it to the cloud and wait for your results. It’s not a real-time dynamic interrogation of the world.

Tish Shute: Yes we are really only at the very beginning of AR being ready for prime time.. it would be interesting to ask AR developers how many of them use AR on a daily basis.

Chris Arkenberg: I think a lot of us, we’re just informed by the sci-fi myths and fascinated with the potential now that’s it’s starting to become real. But I think we all kinda get that it’s still extraordinarily young. I mean the web is extraordinarily young. And AR is itself far younger in a lot of ways in its implementations.

Everybody has a lot of excitement about all of the great potentials that are being unleashed by this great wave of the Internet and the web and ubiquitous mobile computing. So that’s why, you know, you look at that map and we talk about AR and you can’t talk about any of the stuff without talking about all of it, in a lot of ways, particularly with something like AR where it’s so ultimately agnostic and could be completely pervasive across all of these layers.

So my fascination is with the future, and I measure our progress towards it by the young nascent offerings from the platform players and the developers. And yeah, a lot of it is…it’s akin to getting that first triangle on the screen in 3D. You know, when the renderer finally works and you get a triangle on the screen, and you go, “Oh my God, it renders.” And then you can start to really build polygons and build objects, and start doing boolian operations, and get light and rendering in there, and textures, and on, and on, and on.
So I’m fascinated by the Layars and the Metaio’s…
[laughter]

Tish Shute: Yes and hats off to all the players in the emerging industry, Layar, Metaio, Ogmento, Total Immsersion, and all the others who are finding clever ways to bring fun aspects of AR into the mainstream, and fuel interest to take the technology to the next level.

Chris Arkenberg: Absolutely. And the hype cycle is very valuable. It has really helped launch the AR industry. It’s brought a lot of eyes, and it’s brought a lot of money into the industry. And it’s forcing people like us to have these conversations to understand how to refine its growth and really focus on the potential in all these different venues, whether it’s trying to save lives, or better understand your city, or have really compelling entertainment experiences.

Everybody’s excited, and everybody’s sharing, and everybody’s trying to move it forward in a way that’s the most productive.

Today Metaio is holding Inside AR in Munich, Germany.   Metaio (the picture above shows Metaio co-founders Thomas Alt and Peter Meier), is behind some of the best known commercial and industrial AR experiences of recent years.  But as important as the many AR projects they have executed are the AR tools that Metaio has made available to developers.  Metaio’s AR products and tools have played an important role in bringing AR to a wider public, and given many developers the opportunity to explore AR.

Inside AR is a great opportunity to see what these AR pioneers  will be up to in the coming months.   I could not make it to Munich this year.  But, fortunately, I had the opportunity to talk with Thomas Alt, recently.   In this conversation – see below, I got a chance to discuss what was going on inside AR with Metaio.

The Fall season is always jam packed with great events, and I wish I could be in two places at once.  But this week, I will be in my home town, NYC, attending Web 2.0 Expo which, reflecting the heat in the NYC tech community, is a sold out event with a very exciting schedule this year (more on some of the presentations that I will be attending later in this post).  If you missed out on tickets to Web 2.0 Expo, all Keynotes will be Streamed Live: TUES 9/28 to THURS 9/30, and keep your eye on @w2e and #w2e on twitter.

Meanwhile, I am missing Inside AR, which had some great speakers lined up, including fellow New Yorker, John Swords, partner and Ringleader at Circ.us.  Hopefully, Swords will share his experiences at next month’s ARNY Meetup which will be “joining forces with another vibrant community – NY Gaming – for an unforgettable night of Augmented Reality Games” on Tuesday, Oct 19th, 6:30 PM at AOL Ventures in New York, NY.

At the most recent ARNY @swords gave a brilliant talk on the possibilities for AR Game development on the newly available opensource Parrot ARDrone platform.  It was great to hear from social game guru @swords on his plans for Parrot ARDrone games, and more.  The picture below of an ARDrone camera view is from John Swords Flickr set.  Swords was flying it inside his garage because the winds outside were too strong.

Also, I kicked off what will hopefully be an ongoing discussion on, “Story Telling with AR and the Big ARNY a collaborative AR Game for NY,” with a few slides.  I have opened up the presentation document for collaboration, so please ping me if you would like to be added as a contributor/editor, and are interested in getting involved.

Ori Inbar, CEO and co-founder of Ogmento, Games Alfresco, ARNY and my co-chair on Augmented Reality Event 2010, suggested The Big ARNY – A Collaborative AR Game Development Project modelled after A Swarm of Angels last year at the First ARNY Meetup – so let’s make it happen!  I will be catching up with Ori in October about what Ogmento has been up to since they became the first VC backed AR Game company!

“Games allow us to see each other, for a moment, in a way that living in a city prevents” – Kevin Slavin

I believe that, AR, to get beyond the stage of “interface du jour” needs to offer us new ways to relate to each other and the world around us so that we can actually improve  and deepen our engagement with reality not just create experiences that are primarily optical  (see James Turner’s interview with Kevin Slavin “Reality has a gaming layer” on not letting “the pleasure of a game and the meaning of a game and the experience of a game rest primarily in the optics.“  And see my recent post, Urban Augmented Realities and Social Augmentations that Matter: Talking with Bruce Sterling, Part 2).

Two of the most inspired creators of urban games,  Kevin Slavin and Kati London of Area/Code will be speaking at Web 2.0. Expo tomorrow.   And you can be sure I will be at both these sessions. Loitering on the Motherboard, Kevin Slavin, is 2:35pm Tuesday, 09/28/2010, and Games that Know Where you Live, Kati London – is a keynote that will also be live streamed – 4:55pm Tuesday, 09/28/2010

Recently Kevin Slavin was interviewed by James Turner, on O’Reilly Radar.   This, Reality has a gaming layar, is a must read piece about a “world where games shape life and life shapes games”  (see @timoreilly). 

Interview with Thomas Alt

Tish Shute: Perhaps you could just start with your background Thomas because I think there’s a lot of newcomers to AR but you are really one of the first movers in commercial AR. How long you’ve been involved in this?

Thomas Alt: Actually I’m an ex-researcher in augmented reality. I started with me actually after getting my master’s work in engineering from the Technical University of Munich working for a big company called Volkswagen. And at that time,1999, we got a research grant for researching how augmented reality could change manufacturing processes in the automobile industry.

And from the research work there, I basically went back to school, did my PhD about augmented reality. And while speaking at a conference, I met Peter Meier who is the co-founder of the company who was also a master’s student writing his thesis about augmented reality. That was in 2002.

And so it really was in the very early days of augmented reality. And both Peter and myself we got really excited about the technology; we saw endless possibilities. We said, ‘OK. Let’s just found a company. We actually founded the company in early 2003 with virtually no money. As a matter of fact the founding capital of the company was 25,000 Euros and this 25,000 Euros were won in a case competition in Germany – a business plan competition.

Tish Shute: So you won 25,000 Euros on this case competition and that’s where Metaio started….

Thomas Alt: Exactly. And to legally found a company in Germany it takes exactly 25,000 Euros so that was the founding capital. We started pretty much like good old SAP started.  It wasn’t in a garage though it was a very small office and we basically built up the business through work, so we don’t have any investors or whatever. Right now we are 66 people located in Munich where our headquarters have been for five years. We have some presence in the US, and we have a venture company in Seoul, South Korea.

Tish Shute: Awesome. I just noticed how fast you’ve been growing. So right now, I’m going to ask a couple of questions about where you see the technology and the emerging industry going.

First, what are the platform of choice for Mobile Augmented Reality at the moment?

Thomas Alt: Obviously in the cellphone hardware space there’s a fierce competition going on. It’s yet to be defined what will be the prevailing platform right now, obviously it’s the iPhone is big now, right? But Android is catching on very, very fast.

Tish Shute: You have pioneered bringing a cross platform SDK for vision assisted AR to a wide community of developers with Junaio and with your partnership with Kooaba – a visual search company from Europe.

Thomas Alt: Yes, yes, and this is how we would, also in the future like to position ourselves with Junaio.  Junaio will be a platform, a technology platform, which will allow users to do whatever they want to do in augmented reality. The API of Junaio is huge in the sense you can do anything from outdoor gaming, to visual search, to normal, uh, lay out style, you know, find the next burger king a mile away kind of super impositions.

Tish Shute: The only licensing you pay is for unifeye right? When you want to use your tool kit right?

Thomas Alt: Yes and this is how we’re distinguishing it. Junaio is our consumer brand targeting newbie AR developers, with limited programming skills, while the Unifeye platform is really our B to B platform where B to B customers can create their individual AR experiences.

Tish Shute: Yes which is what my friend Patrick O’Shaughnessey, Patched Reality, did for the Ben and Jerry’s app he created using Unifeye.

Thomas Alt: exactly…

Tish Shute: It is a lot of work developing for so different mobile platforms isn’t it. Junaio is on Android and iphone but you havent moved Junaio to Symbian?

Thomas Alt: To be honest with you right now its a matter of priorities we have other things we want to do first. And from analyzing the user base, iphone was a big step Android was a big step and now we are pretty much seeing what is happening next. As you know Nokia going into different directions as far as their smart phone operating system goes, and so on and so forth.  There are also capacity constraints. And right now obviously the most – potentially not the most possible users, but the users most inclined to do AR on a day to day basis are the ones using the iphone and android devices. But obviously there are a lot bigger cellphone manufacturers out there.  But just you know even the mobile web users aren’t as strong as the users on the iphone and android devices?

Tish Shute: So what do you think the iphone 4 has that brought to the AR picture?

Thomas Alt: Very fast camera access, very good for marker recognition. If you go to the Metaio site you’ll find a movie where we show on the iphone 4 app for a real augmented reality Leggo peice – this is something which is very nice

Tish Shute: Yes I see that, yes that is nice, yes, yes very nice. The Unifeye SDK is really putting markerless AR into the mainstream.

Thomas Alt: Yesterday we launched the first, a err very nice shopping… shopping solution for , but that’s completely external.

Tish Shute: Oh yes – the augmented reality shopping for seventeen.com, i was going to ask you about that, because it is the first augmented reality online shopping using natural feature tracking.

Also I am very excited to see the gestural interface, awesome!

The seventeen augmented reality shopping app is a PC experience but are you working on developing gestural interfaces for mobile AR?

Thomas Alt: We are continually pushing the envelope of what’s possible with AR. Gestural interfaces for mobile AR is certainly the next step in taking what we’ve done on the PC and making it more portable by using the mobile platform. One thing to keep in mind here is the limitations of mobile platforms and size of the screen needs to fit and make sense for the user experience.

Tish Shute: I know you started off as an AR researcher (although as you mentioned earlier you have been working in commercial AR and building Metaio for a long while now.

So in addition to how we are progressing towards gestural interfaces for augmented reality, I would like to ask some questions about AR eyewear. We won’t really have hands free AR without eyewear. What is your projection on when we will see consumer AR eyewear? And, Do you have a any comments on those speculating that we will not see AR eyewear go mainstream for 20years?! What is Metaio doing to move eyewear technology along?

Thomas Alt: Well as you know, it’s always, you know on the technological roadmap, and we’re still doing research projects, in our AR research department. We have worked on things like calibrating eyewear for augmented reality. There is some nice development there.

But really, commercially, the whole thing with eyewear has never caught on on a level which would make it a valuable avenue, business avenue, at least for Metaio. So, I guess as an ex-researcher, it’s still a very interesting, a very good technology. And it would definitely change the marketplace radically when available. But as per right now, there are very few commercial applications.

Tish Shute: Are the obstacles to AR eyewear technological obstacles or is it just a question of a a business model. I mean is it realistic to see eyewear in the next 3 to 5 years at a price point affordable to consumers, where you really, truly can have eye tracking? You know, the whole problem there was with virtual reality and eyewear giving people a headache. How far have we come in terms of the technology?

Thomas Alt: Well, it’s not so much technological factors ’cause all fundamental problems are solved. It’s more a rather large corporation, I guess, would have to step up to the plate and say okay, do, let’s get all the state of the art in electronics and develop just a perfect HMD.

Tish Shute: Something Yohan Baillot’s company Simulation 3D is doing at is looking at is hooking up eyewear to smart phones.

Thomas Alt: Yes exactly that would be even better. Metaio has done a strategic move into this HMD space for augmented reality about a year ago by acquiring a bankrupt company. I mean, we had considerable IP around it from our research base but in the long term we still believe in it and we did a move about a year ago in buying what was left over from a bankrupt company including a lot of IP, which basically goes into the direction of mobile augmented reality but also mobile augmented reality in connection with head mounted displays.

There’s actually a press release about this but that’s about a year ago…

I know that the whole HMD thing… I mean, I’ve seen companies come and go. Metaio has worked previously, very closely, with Microvision of Seattle. We have worked with a German company, doing HMDs and we have worked with Vuzix.  We are still working with Vuzix, so we’re still consider it very valuable.  But right now, I mean, it’s just not a big part of our commercial pipeline, to put it that way.

Tish Shute: It was interesting what Bruce Sterling said in his keynote at ARE 2010.  He actually made a strong case for why smart phone augmented reality may be more interesting because it’s less immersive. I mean, he raised the question of the fact that if you really truly had AR eyewear and HMDs you’d re-enter the world of virtual reality or as he called it AR’s Gothic step sister VR would rise from the grave….

Thomas Alt: Yeah, well, that’s a cultural or even a philosophical question and we have discussed it a lot, especially in the industrial domain. Also will the deployment of HMDs come about from end consumers using it in their spare time, or from professional users using the idea in their work time?

Tish Shute: Do you think it surprised people who have been working in augmented reality research how much people have engaged with the idea of smart phones as the mediating device for AR, and that rather than having the always on experience that eyewear would give us, we use smart phones as a magic lens of a smart phone when we need to or want to. Some people were skeptical that anyone would want to hold up a little window to look at augmentations of the world – a magic lens. I mean, it wasn’t self-evident that that would be an experience people enjoyed, and it turns out that it was.

Thomas Alt: That’s actually a very good analogy. And also in my view, I mean, certain behaviors just change also, right? I mean, this is exactly what Apple’s trying with the iPad, right? You’re taking the iPad, and all of the sudden you’re not constrained to a laptop or whatever. And it’s truly a companion of the couch, in-bed Web, in the kitchen, and so on and so forth. So digital usage with the iPad, which is a different market, and I’m aware of that but as an example, the iPad has changed our behavior. And obviously, the augmented reality guys are hoping that something similar happens with AR.

Tish Shute: Which of course brings up the question, I’m assuming that some of the next generation of slates/ipads are going to have front and back cameras, GPS, and compass, right?

Thomas Alt: Actually we know that.

Tish Shute: You know that, yes. I assume that you know that, because are you working on some prototypes, and have you got some plans?

Thomas Alt: You have to understand that I cannot talk as I’ve talked as a researcher. It’s the rules, so I have to be a little bit careful about what I say. We very much think that a webpad, or whatever pad, you would want to call it is on some occasions very good device for AR.

Tish Shute: Yeah. But it is an interesting point with holding up a larger device, because you hands aren’t free, but the neat thing about the phone for augmented reality is that you really can do a lot with your thumb, as we’ve found and just the position of the phone. But, how will this work it with the two hands on the larger device?

Thomas Alt: Keep in mind, everyone’s talking about mobile augmented reality, but really where the case for augmented reality, at this point, is the strongest is in the installation business, it’s in the web business… Not necessarily only commercially, but also use case-wise. There are tons of museums out there which are using our augmented reality system in an installation fashion, and to communicate products better, and more efficiently, and so on, and so forth.

So, I know that the hype is clearly on the mobile augmented reality side, but there are many examples augmented reality experiences where holding up a larger device is not a big obstacle.

Tish Shute: Well this brings me to some questions about the future of mobile AR. My interview with Jay Wright focused on how we are now in a new period for AR bringing together computer vision, visual search into a mobile stack that is really optimized for AR. What do you see emerging in mobile AR as we move beyond compass, GPS, camera, accelerometer based AR into markless image-based AR. What will the new use cases and where will we see mainstream users getting in AR. Will AR games be the first mainstream AR experiences?

Thomas Alt: My partner is actually, first of all, one of my best friends, second of all, very emotional, third of all, very intelligent, and he said the other day something I think very valuable in this area. He said, basically think about Mobile Augmented Reality, Thomas. There’s really a very limited number of use cases which you can do if you look at these classical Point and Find applications, ok? But there are almost unlimited number of use cases when Augmented Reality becomes a day to day companion, ok? So what he meant is, ok, I’m looking at my normal day… I’m looking at the city, I’m walking throughout this, I’m coming home, I’m having dinner basically. I can deploy Augmented Reality in a pure POI search fashion perhaps not even once. Ok when I’m travelling it’s a different story, but in an ordinary day I might use a POI search not even once.

But where this ultimately leads, you know, is even in the 15 minutes I’m having breakfast, I’m using AR – looking at the cereal box with my cell phone, I’m taking part in a sweepstakes or whatever. So from that we draw the conclusion that as a general strategy for Junaio, we should basically throw as much technology as possible into Junaio, make it halfway self-explanatory, and just give people the possibility to come up with ideas on how to deploy augmented reality continuously.

We have actually got a creative team from an Art School working on that, and just, you know, with very little programming skills coming up with things you can do with Augmented Reality on a day-to-day level. And it could be a scavenger hunt game, in the city, with monsters flying around, it could be the normal POI routine, it could be marketing purposes, and so on and so forth. And I think that’s really the roadmap, and this is a little bit similar on a more technical level, to what Qualcomm is doing, ’cause they’re floating out possibilities or capabilities I want to call them, and Metaio is doing that, but on a higher level [re the tools] meaning on a Junaio level.

Junaio is a capability platform. It is also a way for Metaio to demonstrate the capabilities of our technology. We will offer all the possibilities for AR and more that we have already demonstrated on PC augmented reality.

Tish Shute: What is the business model for Junaio? Are you encouraging developers to develop business on your platform ?

Thomas Alt: Junaio is our end consumer platform and our business model is similar to the way Google structures its business model. We work with OEMs, content partners, brands, and developers to offer free content to our end users. Where we do charge is on the advertising side, more specifically contextual and location based advertising. At the current stage, we are focused on building the content base, fostering our developer community, but in the near future, we will be introducing advertising channels.

First of all you have to have very good use cases in the platform basically. And then to put a business model on top of that from a technology stand point is not hard – its a pay channel. Its all prepared for this.

Tish Shute: You have quite a broad base as a company don’t you – you do everything from industrial AR, marketing to technology licensing and more?

Thomas Alt: Basically, there’s a lot of things people don’t see us. There is also the Unifeye PC SDK and we have a client base and partners who are sourcing software from us, and we are doing great pieces. I mean the hype of augmented reality is really coming to a peak. There are lots of pieces that are not even talked about any more. China’s GQ magazine launched with AR from Metaio, the biggest AR campaign anywhere – there are a lot of potential readers in China. And um, so that’s our business model… we have our IP, our patents and so on. And on this we can move on onto the mobile platform whenever it’s advisable or feasible.

Tish Shute: Right. Yeah. I mean uh, you’re very fortunate to have this base built on uh, years of developing IP. What are the most important areas of AR that Metaio holds IP and patents in, in your view?

Thomas Alt: there’s sleepless nights in that too…

So far we’re extremely excited about what’s going on with Junaio, it’s one of our big, big success stories. But we are sensible and trying to experiment because, you know, analogies from the past won’t really work in my view for Augmented Reality in a sense, that, you know, you better bring for a new system to fly, for a new technology to fly, you better bring a very concrete use case to the table, ok? And a well-defined use case. And we are, right now, with Junaio, in a state where we are checking out what could be such a very defined use case.

Tish Shute: So how many users does Junaio have now?

Thomas Alt: Let’s put it that way, we are, especially in the last 2 months, we are very satisfied. But we are not disclosing that, because users, and we’re seeing that from the competitive landscape, always needs 1 page of description what exactly a user is.

You understand what I’m saying? So this is why, ’cause we don’t want to up or downplay things, we are very careful saying, with users. Because I mean we have people who are actually also commercially very interested in Junaio…  We go through with them and discuss what exactly a user is. Cause there’s more then…a download is not a user. An app or something on your phone is not a user, basically, in my opinion.

Tish Shute: I am still waiting to see someone do something with AR and the Four Square API, or now the Facebook Places API. Do you see an interesting potential in the marriage of the rapidly emerging location based social networking space and augmented reality?

Thomas Alt: Definitely. Augmented reality offers a way for users to find information around them easily. Adding in the social networking component such as geo-tagging, rating, commenting can enhance the user experience and create engagement beyond just viewing the information. For example, within junaio, an average user can create their own personal channel and geo-tag photos or leave text messages at different locations. They can create a virtual tour of San Francisco and share it with their friends. By connecting the social side with good content, the augmented reality experience becomes more fun and interactive.

Tish Shute: And, of course, there’s the Junaio API. Are you beginning to see developers use that?

Thomas Alt: Yeah exactly, I mean if you go to Junaio.com, you can get a login, you have an API description. And the way it works is, that you bascially set up a call, which contains the information you would like to have in your individual channel. You submit it to us, it will get checked for profanity and other things, basically. And then we admit it to Junaio basically. The API is huge.   You can also use Junaio indoors.

This is very relevant. And there’s a tool chain for that, and so on and so forth. You can do mission-based search with Junaio. It’s in there, it’s called Junaio Glue. And there will be another very interesting feature coming up in a couple of weeks. And you can just do it, you can do a scavenger hunt, a game, normal POI search, and so on and so forth. And it’s all active. And that’s, what’s sometimes difficult for us to communicate, is it’s really a capabilities platform, but on the other hand it’s obviously very good to developers. And I mean on the developers side there’s huge interest.

Recently, Qualcomm announced an SDK for vision based augmented reality – currently in private beta and open to the public this fall. The Qualcomm augmented reality (AR) bonanza will launch with a $200,000 developer challenge and a SDK that will put vision based augmented reality into the hands of developers without licensing fees.

This is a big step forward for augmented reality and a very important move made by an industry giant to support the rapidly evolving AR industry. Innovation at all levels of the AR stack, particularly at the hardware level (CPU/GPU optimization) is vital for the full vision of augmented reality – media tightly registered to physical space, to take center stage. Vision based AR takes mobile AR beyond compass/GPS based AR post-its, which are only loosely connected to the world (but the staple of most current AR apps), towards the holy grail of AR – markerless tracking with the whole world as the platform.

Click on the image above or see here for a video demo of an AR version of Rock’em Sock’em Robots game.  Mattel, one of the first companies working with the SDK demoed AR Rock’em Sock’em, at the Uplinq 2010 conference (see Chris Cameron’s ReadWriteWeb write-up on Uplinq 2010).

The Qualcomm AR stack, which reaches from the metal to developer APIs, will give Android developers an important edge in AR development. And, when vision based AR starts getting integrated with visual search capabilities, and combined with cool tools like Unity, we will start to see the augmented world get really interesting.

Visual search is already an area of AR getting a lot of attention, with Google Goggles, Point and Find, Japan’s NTT DoCoMo set to launch “chokkan nabi,” or “intuitive navigation,” in September, and the recent partnership between Layar and Kooaba. Metaio’s mobile augmented reality platform Junaio is already integrated with Kooaba’s computer vision capabilities.

And, of course, I am particularly excited about including open distributed real time communications for AR in this stack, which is why I asked a group of developers who have been inputting into the ARWave project if they had questions for Jay Wright, Qualcomm.  Thank you Yohan Baillot, Gene Becker, Anselm Hook, Patrick O’Shaughnessey, Thomas Wrobel, Markus Strickler, and Davide Carnovale for your input.  [Note: see my upcoming post, about the future of ARWave and real time distributed communications for AR following this Google announcement.]

Jay Wright, “is responsible for developing and driving Qualcomm’s augmented reality commercialization strategy.” He “handles partnerships with leading innovators in industry and academia and leads Qualcomm’s efforts in enabling augmented reality within the mobile ecosystem.” In the interview below, Jay very generously answers our questions in detail.

A key contributor of questions for this interview is Yohan Baillot. Yohan is working on a full vision of AR – integrating computer vision, visual search, open distributed real time communications and AR eyewear. Yohan Baillot is founder of Simulation3D, a consulting and system integration company specializing in interactive visualization systems and eyewear-based AR systems. (I hope to bring you an interview with Yohan soon!).

Qualcomm was the title sponsor for are2010, Augmented Reality Event, and played a vital role in making this event an historic gathering of the talent and creative minds at the heart of the emerging AR industry. Watch out for the videos of the are2010 sessions to be posted at the end of August. My are2010 co-chair, Ori Inbar, is preparing them to go online while kicking his newly funded start up, Ogmento, into high gear! Ogmento is also one of the start ups pioneering vision based AR.

Metaio, (with Total Immersion, they are one of the first augmented reality companies), has played a key role in bringing a vision component to smart phone augmented reality apps with their Unifeye mobile SDK.  Junaio, Metaio’s own mobile augmented reality platform has gone beyond location based AR with “junaio glue” – “the camera’s eye is now able to identify objects and “glue” object specific real-time, dynamic, social and 3D information onto the object itself,” (see my upcoming interview with Metaio founder, Thomas Alt).  Also, recently, Layar – who continue to innovate at a breathtaking pace, announced a partnership with the computer vision company Kooaba.

Both Maarten Lens-FitzGerald, Layar, and Thomas Alt, Metaio, when I spoke to them recently, saw the Qualcomm SDK as a very positive development for AR, and they look forward to exploring its capabilities and integrating it where appropriate with their AR tools.  See more about Layar’s upcoming visit, to the US here – August 10th NYC, and August 12th SF.  Also save the date, Sept 27th, Munich, for InsideAR, Metaio’s upcoming conference.

It is clear that vision based AR will be driving the next wave of AR apps. And, as Maarten and Thomas both pointed out, it will be interesting to see which use cases capture the imagination of users the most. Having more tools freely available to AR developers will certainly be a boost to creativity. And, Qualcomm’s SDK is going to give Android developers, in particular, a big opportunity to take the lead.

Interview with Jay Wright, Director, Business Development, Qualcomm

Tish Shute: Before I start with questions on the new Qualcomm vision based augmented reality SDK, I want to briefly look ahead to what many people feel is vital for the full realization of augmented reality – head mounted displays, or more specifically, comfortable, sexy AR eye wear. Is Qualcomm going to be involved in the development of augmented eye wear and wearable displays?

Jay Wright: I think there’s some core technology that needs to come together so we can have what we think needs to be a see-through head mounted display with a decent field of view. And that looks like something that is quite possibly further than a three to five year horizon.

Tish Shute: Gene Becker asked some interesting general questions about the Qualcomm AR initiatives. He said, “I’m unclear exactly what Qualcomm’s goal is.” It would be interesting to hear from you the Qualcomm view, from the top down.

Jay Wright: Our largest revenue stream comes from sales of chipsets. And we see augmented reality as a technology that drives demand for increasing amounts of processing power. So we want to create demand for chips, higher-end chips, and augmented reality does that. Specifically vision based augmented reality because it is so computationally intensive.

Tish Shute: Yes. And I think that is why people are very excited by the Qualcomm SDK. It is not only the first free toolkit for developers to build vision apps from, isn’t it? There’s been nothing freely available before this, has there? But also Qualcomm is paying attention to the complete AR stack to support vision based AR development, from the chips to game/app development tools like Unity.

Jay Wright: That’s really the goal. We’re not here to be in the augmented reality applications business. Qualcomm’s role in the ecosystem has been to serve as an enabler. And that’s what we want to do with augmented reality: provide the enabling technology that allows the entire ecosystem to flourish.

“Augmented Reality has a number of attributes that make it a great fit for Qualcomm’s core competencies”

Augmented Reality has a number of attributes that make it a great fit for Qualcomm’s core competencies. It’s very computationally intensive, algorithmically complex, requires tight integration of hardware and software, and benefits from tight integration of multiple hardware components. And that’s the kind of problem we like here, where we can apply our core competence of really optimizing complex systems for performance, while at the same time minimizing power consumption.

And as you know Tish, mobile AR is really extremely power sensitive. We sometimes talk about it as a battery’s worst nightmare. It’s roughly equivalent to playing a 3D game and recording a video all at the same time.

Whenever there is something that takes a lot of power, that’s a definite opportunity for us to optimize it.

Tish Shute: Right. One of the core business is chips right, but for Qualcomm there’s basically a lot of profit in licensing. When I talked to the developer community about the Qualcomm SDK developers first question was, “What’s the licensing? What’s this going to cost us in the long run to develop on this SDK re licensing?” And they had all different takes on this. So everyone had different ideas about what your approach to licensing might or might not be. Could you clarify the approach to licensing, as I think this is a core concern for developers.

Jay Wright: Anytime you see something for free, you kind of say, “Hey, what’s the hook?” So yes, it’s definitely a logical question. Our intent is not to generate licensing revenue from application developers using the SDK. So the SDK will be made available free of charge for development, and it will also be free of charge for developers to deploy applications.

Tish Shute: Now, this is another question. You also include not just image recognition capabilities but Unity in the package you are offering developers. Unity products usually involve a license. They do have some free products too, I think. But how does this work? And how do you separate your part from their part, or don’t you?

Jay Wright: That’s a good question. What we’re trying to do with the platform is incorporate it into tools that people already know how to use. So we’re actually going to have the SDK support two different tool chains. One of them is the Android SDK and NDK. And then the other one, is Unity.

We’re working with Unity to create an extension to the Unity environment that will be available as part of the Unity installer when you install Unity from the Unity website. Developers will still be paying whatever license fees are associated with Unity’s products on their existing pricing schedule.

Tish Shute: One of Thomas Wrobel’s question is whether developers can just use the image recognition without Unity? Your answer is yes, you can work with the computer vision component of the SDK separate from Unity?

Jay Wright: Yes, you can.

Tish Shute: Good because we would like to build a completely open Android client for ARWave, and not tie it to Unity unless people choose to. He’s using the open Android JPCT 3D engine, which he’s adapting for AR. So he could actually use the part of the SDK that does image recognition and association with that, right?

Jay Wright: That’s correct. You are not required to use Unity. Unity is just one option for building the application.

Tish Shute: Great! That’s very good. But I’m sure many developers are going to jump on the chance to use Unity. But I mean it’s nice to be flexible because it’s so early for AR that people have different ideas and new use cases coming up all the time. I think it’s excellent you’ve divided that.

Another of Thomas’s questions was, “Can developers use their own positioning data sharing solution?” He’s really talking about AR blips.

Jay Wright: With data sharing solutions, I am assuming that by data he means referring to augmentation data or graphics?

Tish Shute: Yes, and I’ll ask him to elaborate. But, at the moment, everyone is using different ideas for POI, aren’t they?

“The goal with our platform is to make it just as easy for a developer to create 3D content for the real world as it is for a game world or a virtual world.”

Jay Wright: Yes. So let me answer it this way, Tish. The goal with our platform is to make it just as easy for a developer to create 3D content for the real world as it is for a game world or a virtual world. So all we’re really trying to do is provide the computer vision piece that makes the real world look like a bunch of geometric surfaces and potentially some meta data that is associated with this so you know what you are looking at.

So that means from a developer’s perspective, you are still doing all of the 3D content, all of the animations, all of the game logic, all of the rendering. You are still doing that all yourself. So if you think about doing an AR game, you are doing everything you used to do, except you are not creating a virtual terrain. You are just going to map it in the real world.

So if you want to do a browser that is doing POI’s, your POI data, or augmentation, or meta data, or whatever it is, that can be in your application, it can be in the cloud, it can be wherever you want to put it. We’re not putting any constraints on what that content is or where it’s stored.

Tish Shute: Right, and that’s what I hoped for. And I think that does answer the question. People are interested to know how far Qualcomm is going with this. For instance, Gene Becker asked: “do they see a business at a certain level in the AR stack?” As you said AR development basically feeds into the core business of chip development, right? But does Qualcomm also see some new business models developing?

Jay Wright: I think it’s foreseeable that Qualcomm could identify other business opportunities down the line. But we’re certainly not there today. Today, our motivation for the investment in AR is to create technology that is going to advance the chipset business.

Tish Shute: When the news came out about Qualcomm’s support of a game development studio at Georgia Tech at the same time as the SDK I think I wondered what was the scope of Qualcomm’s interest [for more on using Unity for AR development see Vision-Based Augmented Reality Technical Super Session video from Uplinq 2010].  For example, I am interested to know how the Qualcomm initiative in developing an AR stack connects to the effort to introduce an AR browser based on web standards, i.e., the Kharma/Kamra KML/HTML Augmented Reality Mobile Architecture from Blair MacIntyre and the Georgia Tech team (image below)? Are you supporting the open standards based browser development too?

Jay Wright: Blair is going to continue to work on the browser effort. And it’s our expectation that he will use our SDK and technologies for vision pieces of the browser effort where appropriate. So they are certainly not mutually exclusive. I would just think about our technology as one element of what may be used in that browser, as I expect it would be an element of what any other app developer would put in their application, whether it be browser, or game, or whatever.

Tish Shute: Yes Now, this is an interesting question, which is sort of connected…I’m trying to keep some form of narrative for this! It follows from the question about Blair’s web-based standards browser. A few people have asked me why we haven’t heard more from Qualcomm in all these various standard discussions that are starting to come up. I mean is it just too early, or are you too busy, or what?

Jay Wright: No, let me explain. The type of standards that have come up so far have been around how HTML should be extended for geo-browser type applications. And while that’s interesting, I think the standards efforts that Qualcomm would be more likely to be associated with in the near term are those related to API’s that are hardware accelerated.

So one of the things that we are in the process of doing right now, Tish – because as you know, Qualcomm is a company that adheres to standards and strives to produce a leading implementation of those standards on our hardware and software – is we are in the process of determining what API set within the existing SDK should be standardized.

Tish Shute: Right.

Now, my next question is, “Who are the other players at this level of the AR stack in the standards conversation? Who else is working at that level?” Obviously, the AR Lab in Graz was, but now they are Qualcomm, right?

Jay Wright: They are still independent. Qualcomm is the exclusive industrial partner of the Christian Doppler Handheld AR LAB in Graz.

Tish Shute: Does this compete with, say, the work that other AR start ups are doing?

Jay Wright: Our intent is not to compete with companies that have done augmented reality technology. Our intent is to enable the entire ecosystem. So we would like to work with both Metaio and Total Immersion to find ways that they can benefit from our technology. That would be the hope – that our technology can kind of lift and float all boats in the ecosystem.

Tish Shute: There are not many implementations of vision based AR right now? I mean obviously Microsoft is doing stuff because they have Georg Klein now, right, and there is Google Goggles, Total Immersion, Metaio, and it will be interesting to see where Layar’s partnership with Kooaba will lead?

Jay Wright: Yes. I think there are relatively few commercial implementations of vision based AR stacks.

Tish Shute: One of Patrick O’Shaughnessey’s question is he wants to understand what features are going to be in the vision component, very specifically. Patrick O’Shaughnessy, Patched Reality, working with Circ.us, Edelman, and Metaio used the Unifeye SDK to do a vision based AR app for Ben and Jerry’s that’s been getting all the attention lately. He was a speaker at are2010.

He very specifically wants to know what features will be included in the computer vision component. He says, “I’m most interested in understanding what features are going to be in the vision component. Is it marker based?” Well I know it’s more than marker based. I saw some of it in Chris Cameron’s ReadWriteWeb write-up on Uplinq 2010. Is it “NFT? PTAM? other? Also, are you are integrating any backend services.” That is an interesting question!

Jay Wright: So let’s get to the features on the client side, the vision based features. There’s support for, what AR aficionados would know as natural feature targets, or image based targets. And we use those to represent, obviously, 2D planar surfaces.

The other thing that we are trying to do to set expectations, Tish, about where these can be used is to let people know that they work best in what we’re calling near-field environments. So the idea isn’t that you use the system to create a large scale AR system that can recognize buildings indoors and outdoors. It’s the idea where I can recreate 3D experiences that take place on surfaces that are in my immediate field of view, whether that be on the table in front of me, or on the floor, or on the wall, or on the shelf.

Also, when you talk about near field experiences, there are some other constraints that are implied. Like, if it’s in front of me and my immediate field of view is probably going to be pretty well lit. And lighting, of course, is an important requirement.

So we’ll support these natural feature targets, or image targets. And we also have support for sort of a hybrid marker image type. It’s something called a frame marker, which has kind of a black border with some dots on it.

Click on the image above or here to view Vision-Based Augmented Reality Technical Super Session video from Uplinq 2010

Jay Wright: So there’s this additional type. And the reason for this additional hybrid marker type is it has a lower computational requirement than a natural feature target. So the idea is these things can be used as game pieces or elements of play where I want to have a large number of them detected and tracked simultaneously.

So you can have, for example, one big natural feature target that serves as a game board or game surface, and you can use these other things as smaller game pieces. And when you put them out, different types of content can appear on them and do different things.

Tish Shute: Yes, that’s nice! And the other thing I noticed was the virtual buttons. How well developed is that?

Jay Wright: The idea behind virtual buttons is, in addition to supporting augmentation, we want to support interaction. And we think there are going to be different types of user interaction with augmented reality content. It may be hand tracking and finger tracking, but another compelling form we’ve identified so far is the ability for me to touch particular surfaces and have an event fire within the application..

So virtual buttons are rectangular areas on image targets that a developer can define, and they serve as buttons. So you can create a target that is a game board, for example, and define certain regions. And when the user covers that region with his hand, like pushing a button, your application can detect that event and take some action.

Tish Shute: Nice! And what is the documentation on these capabilities that is offered by Qualcomm…For example Yohan Baillot, who is interested in integrating eyewear-based AR systems with smartphones asked. How deep does this go? Will there be full documentation on Snapdragon, people who want to work at that level? Is there a chip SDK?

Jay Wright: . Qualcomm’s model is to work with providers of the operating systems and deliver functionality of the chip through the operating system. So many operating systems APIs will take advantage of functionality that’s in the chip. But there is no separate chip SDK per se.

Tish Shute: I suppose that does come up a little bit with one of Anselm Hook’s questions, because there is some overlap with Google Goggles here, isn’t there, in terms of what you’re doing, right? Are you going to work closely with Google Goggles ?

Jay Wright: Google Goggles is performing what we’ve described ‘visual search’. So the idea is you take a picture, send it to the cloud and identify it and the results come back. I think if we see Google Goggles go in a direction where there’s an AR experience that would be a good area for us to collaborate with Google.

Tish Shute: Anselm Hook is very interested in having some kind of open standard around this physical tagging of the world, right, – the physical world as a platform. But I suppose that’s down the road but is there a plan to start talking about open standards here – visual search with image recognition? That’s a very powerful combination. (see my interview with Anselm Hook here).

Jay Wright: I think it is. And we’re very interested to hear from developers and others that have ideas about how they would want to integrate with the functionality that we have to best enable those kinds of combined experiences.

Tish Shute: Well, I know Anselm has a lot of very important ideas on that.

Jay Wright: I’d be very interested in hearing those because we want to do everything we can to enable the maximum number of applications and best user experience for anything that people want to do.

Tish Shute: Let’s go back to some specific questions about the platform, right? For example Yohan Baillot asked, “Is it arbitrary image/tag recognition supported? Is the tag / image specifiable by user? Is face recognition supported?” Not yet, face recognition, right?

Jay Wright: Not yet.

Tish Shute: What are the plans with that?

Jay Wright: I think we’ve identified it as an interesting area and something that there’s some interest in, but have not made a decision on a particular technology direction.

Tish Shute: You’ve answered some of these but 3D model based vision tracking. Yohan’s question was, “Is 3D model based vision tracking supported (that is recover the pose of the camera using a known 3D model and a 2D camera view of this model)?”

Jay Wright: That’s something we’re looking at very closely, but again, don’t have a plan, or don’t have a future date for.

Tish Shute: And you said with the natural landmark tracking that’s not supported, right?

Jay Wright: I don’t know if I know what that means, Tish. But we don’t have any APIs that provide compass or GPS functionality other than already exists in the operating system. So if you want to take advantage of the compass or other sensors, you can absolutely do that, but the SDK does not currently provide anything different or anything more than already exists in the OS.

Tish Shute: This is an interesting question, “Is Snapdragon offloading some processing to the GPU, if any?”

Jay Wright: Certainly rendering functionality that utilizes OpenGL is being offloaded to the GPU. We’re currently in the process of determining multiple methods for offloading functionality between both symmetric and heterogeneous cores on Snapdragon. Which would include the GPU, the apps processor, and DSPs.

Tish Shute: No one has truly solved optimizing the GPU/CPU for mobile AR yet have they?

Jay Wright: That really gets to the heart of the optimization here. Which pieces ought to be operating on which cores and when, and why? And that’s something that we’re looking at very closely.

Tish Shute: Right. The only AR – that is truly 3D media tightly registered to the physical world has been done for military and medical (and that has often been with a locked of camera!). But to take mobile AR to the next level I think many developers would like access to the CPU/GPU, for example a developer interested in the future of eyewear like Yohan?

Jay Wright: We’re very interested in hearing what kinds of tools developers would like to see.

Tish Shute: What is the best forum for discussing feature specifics?

Jay Wright: To provide feature requests to us?

Tish Shute: Yes. And discuss them.

Jay Wright: if people go to qdev.net/AR there’s an application up there for the private beta program. So if people do have ideas about features or other things they would like to see, they’re welcome to submit [their requests and ideas] there.

Tish Shute: I also have some questions about the specifics of the competition. Some people are a little confused about some things. Yohan asked, “What is the expected form of the project? Lab demonstration? Specific capability? Complete end to end system?”

Jay Wright: The only requirement is that they submit an Android application that we can then get running on a device. So if it has a backend component or backend server that it works against, great. If it does, it does. If it doesn’t, it doesn’t. But that’s really it. There’s no limit to the application category. It can be a game, it can be a museum tour, it can be a children’s learning game or learning experience. It can really be anything. The idea is we want to find experiences for which AR delivers some unique value. We’ll be announcing more specifics about the competition in the near-future.

Tish Shute: Right, because some people weren’t sure about the Unity being separated whether it was biased towards games. And it’s not really, is it?

Jay Wright: Unity is a bias toward just rapid development for 3D, I think. It’s most commonly associated with games, but there are also a lot of Unity customers that use it for medical simulations and other types of applications that aren’t really games at all.

Tish Shute: Yes. It’s very flexible, I know. You did bring up the backend services again. Are you thinking of offering any of that?

Jay Wright: There is a backend tool that we offer. And the backend tool is what you use to generate your targets. So if you want to create or use a particular image for a target in your application, you upload it to our target management application, and then it will evaluate that target and tell you how well it will work. So as you know, certain images are more likely to be recognizable than others. And so there’s metrics in that application that will give you some feedback.

And then you can download your target resource from the website that you can then incorporate into your application project.

Tish Shute: So this is available at the moment to people who are in the private beta and not to…you know, all of this information and documentation, right?

Jay Wright: That’s correct.

Tish Shute: So that’s an incentive. Now, just to encourage people to submit to the private beta is the other thing that people seem confused about. In one part you say 25 developers. And some people have thought that meant it was limited to 25 individuals. And some people have like maybe four people on their team, so they were going, “Well, are we going to be accepted because we have four developers, or do we count as one because we are all working at the same project?”

Jay Wright: it’s just 25 companies.

Tish Shute: OK. I think we’ve gone through the questions. Just to clarify and maybe give some incentive for people to apply to the private beta…the big advantage of getting in the private beta, aside from getting a month’s start on the competition, is that you get a chance to input, right?

Jay Wright: Yes. A chance to provide feedback, get early access to the technology. And then we are also providing a free HTC phone.

Tish Shute: Oh, yes. I forgot the phone. Yes, right. In the requirements, though, you basically seem to be asking for sort of a full app…some people get reticent about delivering their full application plan, right?

Jay Wright: Yes. I understand that. People should just reveal what they are comfortable talking about. Just so you understand the constraint on this end, this is early technology and we’re trying to understand exactly what the support requirement is going to be. And we have limited supported resources at this time, so we want to make sure that we can focus the resources that we have on folks that are really going to use the technology and have a sound plan to actually build something. So that’s really the motivation behind limiting the size of the private beta.

Tish Shute: OK. Yes, it’s good to reiterate that. We’re down to the last question that I have, and then I’ll ask you if there is anything that I missed. You say you are partnering with Mattel. Who are the developers? Because I mean Mattel isn’t an augmented reality development team.

Jay Wright: Mattel used a subcontractor, Aura Interactive.

Tish Shute: Nice. But that’s your only partner that I saw, right? Why Mattel?

Jay Wright: Well, to launch a new technology, companies will often find showcase partners to demonstrate compelling uses of it. And we thought Mattel and the Rock’em Sock’em™ toy was a great example of combining augmented reality with an existing toy.

Tish Shute: And I think people agree with you on Rock’em Sock’em (see Chris Cameron’s RWW post).

Jay Wright: And there’s other showcase partners and applications that we will continue to work on to kind of spur the ecosystem and show what is possible.

Tish Shute: OK. Now, is there anything I’ve left out that you think? What’s the core of this narrative that we need to get across, and if I’ve left anything out that is a key piece?

Jay Wright: I think you’ve done an excellent job of covering all the bases, Tish.

Tish Shute: [laughs]

Jay Wright: I think the important overriding message to get across is that we really see ourselves in an enablement role here, and that we are trying to provide….we’d like to provide fundamental technology that helps all developers build content for the real world.

Shortly after Augmented Reality Event – are2010, I talked with Bruce Sterling on skype and in gdocs about his experience there.  I am posting the conversation in two parts to make it a more blog friendly length!

The picture above is the Auggie Award for the best AR demo (above) designed by Sigal Arad Inbar.  It was won by YDreams! See, Ivan Franco recounts the team’s ARE 2010 experience, and winning the event’s first-ever Auggie Award, for more.   The video below was shot at the YDreams booth by Bruce Sterling.

“The Hotness” – YDreams rocking it at ARE2010 from brucesflickr

Rudy Rucker, who was hanging out with Bruce Sterling, captured the are2010 buzz and some great images in his post, Augmented Reality, Painting, Twitter. As Rudy put it:

“AR is hoping to be a next big thing, a cozier and more commerce-driven cousin of the old VR, or virtual reality.”

Bruce Sterling’s opening key note is up, ARE 2010 Keynote by Bruce Sterling: Bake a Big Pie!, and also the ARE 2010 Keynote by Will Wright: Brilliant Inspiration for the Augmented Reality Community with more videos from are2010 on the way.  One must read post on are2010 is Chris Cameron’s post, Augmented Reality’s Next Steps: Sitting Down with the Titans of AR.

Talking with Bruce Sterling, Part 1

The Auggie panel, Bruce Sterling, Jesse Schell, and Mark Billinghurst inspect the award.

Tish Shute: In your keynote at the 9am of the augmented reality industry you asked some questions of the are2010 audience: “What’s the mission statement?  You’re the world’s first pure play experience designers, except that user experience it’s mostly futuristic hot air.  But run with that, right?  What are your tactical steps?  You should get dressed, have a coffee, have a to-do list.”

How much of that did you see going on in the next two days?

Bruce Sterling: Well, I wasn’t privy to any of the business discussions.  I didn’t think it was an accident that this standard AR enabled tag thing came up from Bruno Uzzan, Total Immersion.  That seemed to me to be a useful thing.  I was always interested in the Augmented Reality Consortium.  It struck me as remarkable that there was this group of people who clearly all knew one another and it had some kind of game plan.  I applaud them for that, because these are not the 1980’s.  [laughs]  You know, it’s just a different world for young startup companies.

Tish Shute: I think you’re right. There seem to be some VC conversations going on, we don’t know what went on in the meetings, but it was noticeable in the atmosphere of excitement, and remarked on by a few people. So I think that kind of was definitely going on.

And, of course, I was so busy I never even got to see the expo properly! You said you wanted to be surprised.

Did anyone surprise you in any of the talks, in any of the expo?

AR used as interfaces for devices

Italian augmented robot from SEAC02 from brucesflickr

Bruce Sterling: I have to say I was a little bit surprised to see Andrea Carignano demoing a robot. I happen to know him because he’s here in Torino. He’s the guy that came out of Fiat and went into AR. I am not a particularly huge robot fan, but I think it’s of great interest that AR is used as interfaces for devices, as opposed to the Jesse Schell idea that AR is all about a “man with the X-ray eyes.”

My suspicion is that a lot of surprises will come out of mashups of AR.

Tish Shute: I didn’t get to see Andrea’s robot.  So what did it do?

Bruce Sterling:  It’s basically a sister device to that little helicopter that those Parrot AR Drone guys were doing.  It’s a little autonomous robot and it runs around with a webcam on it.  You can place video into the acquisition stream coming off the robot.  You can play a game, and blow away imaginary monsters or whatever.

Tish Shute: It’s interesting, because did you notice Will Wright and Patrick O’Shaughnessey, Patched Reality, spend some time hacking the Parrot AR drone in the hallway?  Did you come across them?

Bruce Sterling: Rudy was there with them.  You know, I didn’t want to watch Will Wright hack a robot.

[laughter]

Tish Shute: They seemed to be having fun even though as it turned out the power supply was dead.

Bruce Sterling:  I’m sure Will enjoyed that.  As a game designer, you want to go out and get your hands dirty with a plastic gizmo.

[laughter]

My Swiss Army knife can’t get through airport security, so I really don’t want to strip anything down.  But yeah, what else did I see that was of particular interest?  I was pretty happy about the Korean guys because they are a difficult group to get close to.

AR companies are like mini-global micro-startups.  They’re “glocal”.

“Korean elegance at the Zenitum booth” – from brucesflickr

Tish Shute: Zenitum.  What did you like from Zenitum.  They were one of our sponsors, along with Qualcomm.

Bruce Sterling:  I know that Seoul is like the number one center for augmented reality discussion.  But it’s  difficult to get behind the scenes as a journalist there and  track what’s going on in Korea.  I’m fine with Italian “realtà aumentata.”   And I feel like I’ve got a handle on French “réalité augmentée.”   The Germans were not hard to find, and the Dutch all speak English!  But the Koreans, and whoever the hell it is in Kuala Lumpur…  I have no idea what’s going in Kuala Lumpur, and only the vaguest idea of what’s transpiring in Singapore!  But I know that people there are paying a coherent interest.

So the Koreans show up, and they had some relatively predictable anime style 3D avatar conversion stuff.  But they had a really nice display space.

“Anime figures become three-d smartphone animated avatars,” from brucesflickr

Tish Shute: Ah, So Zenitum created a hot spot at the exhibition?

Bruce Sterling:  Yeah.  The Koreans had  IKEA furniture and some nifty little woven baskets.  They’d really classed up their presentation.  Most Koreans in tech tend to be kind of muscular.  The Koreans are not known for their refined presentations.  On the contrary, they tend to undersell everybody else.  But I don’t know, maybe they’ve been hanging out with Samsung and upgrading their design chops. [laughs]

Tish Shute:  Did you take some photos you could send me?

Bruce Sterling:  I took a few, but  I don’t consider myself a photographer.  They’re all up on my Flickr set. It was interesting to see so many people from so many different nations in such a collegial atmosphere.

Tish Shute: Yes – there were many different countries represented at are2010

Bruce Sterling:  It’s the beginning… and so global at such a young stage.

Tish Shute: Yes. As you said, it was 9 AM, so everyone was actually super excited to be gathered together from across the globe to start a new day together.  As you mentioned, there was a very warm affirmative vibe – everyone sharing a passion.

Bruce Sterling:   They have an online commonality. They seem to be aware of one another’s work through the Internet.

Clearly they had all heard about one another.  That’s a departure from earlier models of tech startup, where you usually have like three hippies in a local garage.  Now you’ve got German-American-Korean outfits like Metaio, and Total Immersion has a Russian affiliate.  They’re inherently multinational, both inside the company and out.

Tish Shute: It was the multinational garage, wasn’t it?

Bruce Sterling:  Yeah.  AR companies are like mini-global micro-startups.  They’re “glocal.” There’s something quite new to me about that.  I don’t find it’s shocking, because in Europe today it’s common to find startup teams who are multinational.  But to see such intense globalism at such an early stage of an industry is really different.

Tish Shute: Yes it made for a fun atmosphere?  It was wonderful running into Iguchi Takahito, Tonchidot.  You have a great rapport with each other despite the language barrier?

Bruce Sterling:  Yeah.  That guy from Tonchidot, he’s very charismatic.  He’s punchy.  That’s reflected in the very strong graphic design from his company.

Tish Shute: Using minimal English to make the case for Sekai No Camera at the Auggies, Iguchi Takahito still got through to the audience.

Bruce Sterling:  Well, his visuals were good.

What AR means for artistic practice…

Picture of Eric Gradman’s Cloud Mirror, from James Alliban post ARE2010 – Augmented Reality utopia in Silicon Valley – see for more on the are2010 ARt Gala

Tish Shute: So before I move on to wider themes, I’m going to wrap up on some of the different aspects of the conference.  I was chairing the technology track but you were more free roaming, was there anything that went on in the sort of hallway discussions and the presentation rooms that struck you?

Bruce Sterling:  Well, I did get collared by artists.   They really wanted to talk to me.  We got into some serious discussions on  what AR means for artistic practice.  How you can do this and reach that, how can one sharpen up one’s presentation?  I mean, they really wanted some art criticism.

Tish Shute: That’s very interesting.  Did you come up with anything that you hadn’t been thinking about already through the conversations?

Bruce Sterling: I’ve seen augmented reality installations before, and I certainly know many electronic artists.  But I don’t know.  People in the AR art space, they are looking for guidance and trying to find fellow spirits.  In their own way, they have the same pioneer spirit as the business people.

Helen Papagiannis shows Iguchi Takahito, Tonchidot, her AR Wonder Turner, an exquisite corpse inspired installation

Tish Shute: Yeah, it’s interesting, because we wanted the art gala to be even bigger, but it turns out, because of the logistics of putting up art in a conference space is fabulously expensive, because it has to be all installed and hung…

Bruce Sterling:  I’m keenly aware of that.  At Share Festival in Turin we bring in six installations, and it’s very heavy work.  It really takes a lot of logistics.  It was like a Battle of the Bands.  It’s like doing a rock concert.

Tish Shute: One of the installations I was really sad to not have there was Uber geeks’  “Steve” H.E.AI.D installation that Brady Forrest & Co. took to Burning Man.

So I was very happy that we actually did get the number of artists we did.

Bruce Sterling:  Well, there aren’t a million AR artists in the world, so it’s hard to judge.   I didn’t see many business people rushing up to have me critique their business plans.

Tish Shute: [laughs]  They were all in the meeting rooms.

Bruce Sterling:  Maybe it’s for the best.

VC and AR Startup Action

The Zenitum Booth, are2010, photo from Chris Cameron’s Flickr stream

Tish Shute: Do you know that why your talk started a few moments late is because we had 50 people who arrived from the Silicon Valley neighborhood I guess!

Bruce Sterling:  Did they not preregister?

Tish Shute: No. They all stood in the line for the same day registration!

Bruce Sterling: It ‘ll be interesting to see what transpires there, if there is a little wave of startup action.  God knows they need some place to put their money, because the VC scene in the US is pretty much moribund.

Tish Shute: Ogmento is the first US AR Games startup to get VC, I think.  I think there was some VC action at are2010 for sure.  And Qualcomm obviously seems to have commercialization plans for their AR technology, and to be scouting talent  and ways to deliver new AR experiences.

Jay Wright, Qualcomm presents Joe Dunn, e23 Games, winner of the are2010 StartUp Launch Pad with a check

Bruce Sterling:  Some  people don’t need venture capital.  I mean, Google Goggles isn’t going to be hurting for VC money, obviously [ see Chris Cameron’s RWW post, Google Goggles Coming Soon to iPhone] .  AR may come up through other methods, like people allying themselves with Hollywood, or peeling off of advertising companies.   There’s a lot of outfits who might conceivably want in-house AR skills.  Then when people set up a specialty AR shop,  they  peel off the list of clients.  I don’t know.  Those old days  of Silicon Valley venture capital seem like a lost world.

Tish Shute: Yes.  I, again, didn’t see anything really of the business tracks and production tracks.  Did you get back and forth between the tracks?

Bruce Sterling:  I went to the Hollywood tracks.  I mean, to the extent that I could.

Is Hollywood stirring? Who’s going to have the first breakout AR property?

Tish Shute: So what did you see from…Is Hollywood stirring?  Is it waking up?  I mean I know Kent Demaine, Oooii,  and Brad Foxhoven, Ogmento, spoke about the Hollywood AR scene.

Bruce Sterling:  There were guys there from LA who were sort of saying, look…they are aware of us, but they just want AR to promote their properties to some particular niche.  They realize that AR is potentially a mass medium and that you could do some real AR entertainment.  So they were batting around some ideas as to where that might happen.  Like, could it come out of a console gaming scene?  Who’s going to have the first breakout AR property?  A popular hit AR property, as opposed to like a neat way to sell shoes, or whatever.   Really, anybody’s guess is as good as theirs or mine.  But at least they were actively guessing.

Tish Shute: I know the breaking the fourth wall discussion has been going on for a while and now the question is, whether AR is going to take down the fourth wall and bring interactive storytelling into the mainstream.  Did you hear any of that?

Bruce Sterling:  Well, I always shy away from discussions of that kind because I don’t think there’s any “final thing.”  Practically everything that AR is involved in right now is  a transitional technology. Also, because I am a storyteller, I get alarmed whenever people in technology start saying, “Oh well, it’s all about telling stories.”  Because obviously it isn’t.

People can tell stories perfectly well orally, and absolutely nobody does that.  AR is not at all about telling stories.  It’s about a great many other things, such as user bases, niche audiences,  media saturation, urban informatics,  convergence culture, and the language of digital media.   I could list these factors until the world looks level. It’s really becoming pretty chaotic.  As I was saying in my speech, AR companies are media startups who almost never use the old-fashioned word “media.”

Tish Shute: Oh, that’s interesting.  Yes.  So why do you think that has happened that way?

Bruce Sterling:  Well, it’s because they are trying to do a different thing than media does.  I mean, they are trying to “augment reality.”  They don’t want you to know that you are using a medium.  They don’t want you to realize that you’re watching computer animation overlaid on some video acquisition stream.  That would defeat the whole point of AR.  It’s entirely different from an analog medium like television, where you turn on the television and there’s a constant stream of station identification alerts.   That’s like: “Don’t touch that dial!  You’re on channel 13!  Stay with us!”  Then it’s like, “And now a few words from our friendly sponsors!”  That medium was engineered to keep your eyeballs locked to a single stream that they’re feeding you.

In AR, it’s much more participative, more geolocative.  I’m not particularly interested in station-identification branding from my AR provider. What I really want to see is the interactivity of the augments they’re bringing to me.  It’s like  FlickR, the photo sharing site. You don’t have any TV-style splash page for FlickR.  “Hi! We’re FlickR! FlickR, bringing your photos to you!” No, FlickR is all about “you, you, you,” your photos, your tags, your friends, your activity around you.   It’s immediately trying to be very participative.

Tish Shute: Will Wright got to that point, didn’t he. He was trying to move us into an idea of blended reality. That the game is about the world, not about the dragons or the overlays per se.

Bruce Sterling: Right. I think that’s true. But see, the world isn’t a medium. A medium is something like this interview,  where I’m connecting to you and there’s a video Skype channel between us.  Whereas AR is more about spatial 3-D,  about 3-dimensional impositions.  Pieces of media: sound, vision, information visualization, tags, floating tags, air tags, icons, arrows, warning signs, warning sounds, tactility, whatever, being brought into the environment around us.

That’s why it’s properly called “augmented reality” instead of just augmented media.   If you call your work “augmented media,” you’re really in trouble. Because if it’s all about augmenting somebody else’s media, why doesn’t that medium just buy you, and augment their own selves?    If you think that way, instead of augmenting the world, you’ll just be a modest little plug-in for old-school media.

The World as the Platform

Blaise Aguera y Arcas, Microsoft, Santa Clara, are2010, photo from Chris Cameron’s Flickr stream

Tish Shute: Yes, which is why Blaise so generously gave the technical underpinning  for augmenting reality in his tech talk – about the trellis and the grapes,  he really explained how the world can become a platform for augmented reality.

Bruce Sterling: I wish I could have seen that. I did not see Blaise’s speech.

Tish Shute: We’re going to put the videos up in better quality.  People in the front row have put it up on the web already.  He really went into some of the challenges of mapping for augmented reality.

Bruce Sterling: His visual-mapping technique is important.  Registration is super important for AR.

Tish Shute: I think it was a really generous talk actually because he went step by step on how we will do this.

Bruce Sterling: I rather imagine that Microsoft has patented those steps.

Tish Shute: Oh, yes, I guess so!

Bruce Sterling: I could be wrong. Maybe they’ll open-source it. You never know.

Tish Shute: You never know. Because the world as a platform isn’t something one company can own, or go it on their own to exploit.

Bruce Sterling: I expect there to be a thorny path, but sometimes I’m surprised. Sometimes people really do try to fertilize the tech field in the hope of getting a good corn crop before they start fighting.

Tish Shute: We’ll I keep hearing that we may even see the unlikely marriage of Apple and Microsoft  – maybe wishful thinking, but there are motivations beyond AR for this unlikely match, and certainly between them these titans have what it takes to realize the grand visions of AR ? [laughs] But who knows…

Bruce Sterling: Well, yeah, it depends on where the thing catches fire.

Tish Shute: Yes. You mean whether AR catches fire in the form of  AR and mapping..

Bruce Sterling: It’s hard to say, but I’m convinced now that there’s more going on than I once thought. I thought that Bruno Uzzan made a very good speech for his company when he talked about how he worked on AR for eleven years.  Eleven years is no flash in the pan.   He has his long list of clients and successful applications. I thought he was right in his impatience with the press for not catching on. It’s gone on for quite awhile. The mere fact that you’re not aware of it, doesn’t mean it doesn’t exist.

The Illusive AR eyewear

My are2010co-chair, Ori Inbar, CEO and co-founder of the hottest new AR game development start-up, Ogmento, donning his goggles to open are2010 – picture from Chris Cameron’s Flickr stream

Tish Shute: Yes. So, the other theme you brought up in your opening keynote and I would be interested to know if anything you saw at are2010 changed your view is the illusive AR eyewear, and  if we actually got AR Goggles that worked they would bring AR’s gothic sister, VR, back from the grave right? [laughs]

Bruce Sterling: Right.

Tish Shute: It took quite a lot of work, but we pulled together a six-company HMD panel, right?

Bruce Sterling: Yeah. I was impressed to see so many of them there.  And I was chagrined to see how prototype-like all their gadgets were. But that doesn’t surprise me, because if any of those head-mounts were remotely working, they would be hyped out the wazoo. Everybody’s been waiting for them and hoping for the best. They’re obviously not ready for prime time. [laughs] Maybe in certain limited applications. Like maybe a diving mask. [laughs]

Tish Shute: No, I think what was nice though they got inspired and they all got together on the last day. I saw them having a meeting about standards. They got inspired to actually work together.

Bruce Sterling: Yeah, well, unless they’re going to invent mechanical eyeballs that those machines can fit onto, it’s going to be tough. OK, I’m a skeptic, but I’m prepared to be surprised. I’m also a skeptic in Artificial Intelligence, but as soon as they bring me an AI that can write a decent novel, I’m going to get it and review that book. [laughs]

Tish Shute: It’s interesting. Re AI, I’m totally in agreement with you. In terms of the way computers turned out, it wasn’t AI per se that they turned out to be good for, not in the way everyone had dreamed of it, rather it was the harvesting of human intelligence that turned out to be the big thing. But what is interesting is that despite all of that, AI or machine learning, as it is now called, permeates our whole society now from the stock market to how many businesses make many of their decisions.

Bruce Sterling: Well, there’s a lot of so-called collective intelligence.  But Marvin Minsky-style hard AI, no way. Alan Turing-style AI, forget about that.

Tish Shute: Yeah. So, that’s an interesting comparison with the HMDs.

Bruce Sterling: People stretch the definitions.  It’s like, well, my car engine is Artificial Intelligence. Yeah, so is your wall transistor. No, I don’t really think so.

And AR is a similarly big tent. I mean, Uzzan had to admit that he had denied that AR was AR, unless it was using his favorite technology. And he felt embarrassed to be rubbing shoulders with people who put AR into cell phones. And I can understand his feeling there, because, gee whiz, that’s certainly not what AR pioneers had in mind. But he had to admit he’d become more ecumenical about it. Obviously, they’re  there and doing business like gangbusters. You can’t very well ignore success, right?

I had a similar feeling about the goggles. Obviously, the goggles would be great, should they work. But if they did work, I rather think virtual reality would come very strongly to the fore.  You’d see people doing all kinds of elaborate immersive-style stuff.   A truly immersive technology doesn’t need to “augment” much of anything.

Tish Shute: Yeah, you’re right.

Social Augmented Experiences

Bruce Sterling: I think many of the most interesting AI aspects are not personal in the way goggles are.  They’re not about guys walking around with personal tech. They’re about big, communal, social-media experiences, like stage shows, and urban informatics, things where large numbers of people can interact with the same augmented reality. The projection mapping, which I go on and on about. Augmented public spectacles.

Tish Shute: Yeah, projection’s our best example of a social augmented experience right now because we are yet to have an easy way to do networked social augmented experiences easily – but that is of course the thrust of my interest in ARWave [see the slides for my presentation, AR Wave:  Federation, Game Dynamics, Semantic Search, Mobile Social Communications here].

Bruce Sterling: I think of Edison’s early days, when he wanted to sell movies to people for a nickel a clip.  You had to bend over and put your eyes on this visor and turn this crank. That coin-op device was easy for Edison to monetize, as opposed to getting a bunch of people to sit in theater seats. But people laugh at movies when they’re together in the seats.   Cinema is a more social, involving experience in a crowd situation.

Tish Shute: But it started with them, didn’t it, Hollywood – the movie biz? Basically Nickelodeons, right?

Bruce Sterling: That’s right. They were Nickelodeons. They were a lot like the goggles because they isolated the user.

Tish Shute: Yeah, that’s a really important point that the goggles are not Nirvana because of this question of whether they actually detract from the social augmented experience and blended realities, by drawing us into VR experiences?

Bruce Sterling: I’m tempted to claim that they’re more a VR technology than an AR technology.

Tish Shute: That’s a very interesting point because…

[thunder]

Tish Shute: Wow! What was that?

Bruce Sterling: Thunder storm.

Tish Shute: Oh, my God, how very Gothic! [laughs]

Bruce Sterling: It can get pretty loud up here in the mountains.

Tish Shute: Oh, you live in the mountains, better still!

Bruce Sterling:  Torino is in the foothills. This is Piemonte. So the Apennines are over there. The Alps are over here. We do get some rather spectacularly unstable weather.

Tish Shute: It sounded like a bomb to my NYC ears. [laughs]

Bruce Sterling: Yeah, it didn’t hit the building, but it was maybe half a kilometer away. I saw the flash.

Tish Shute: Oh, you did?   Well, I hope you don’t lose your power midstream here.    I was really happy to hear of that connection between Rudy Rucker and Layar  [Rudy was touched when Maarten Lens-FizgGerald from Layar said that he met the Layar co-founder at a Rudy Rucker lecture].

Bruce Sterling: That was very fun, yes.

Tish Shute: Wasn’t that wonderful? What was that experience like going around the conference with Rudy?

Bruce Sterling: Well, you know, Rudy’s very into graphics. He’s a mathematician, so he understands the underpinnings of this stuff. But he’s a skeptic. He thinks they’re kid toys. He’s not a gamer. He’s a good old-fashioned computer-science hacker. So he wanted to tell me all about his new eighth-order, fifth-dimensional fractals. He showed me a great many of them. They’re psychedelic. Rudy’s fractals are considerably trippier than most apps that help you find a barber or a train station. [laughs] Rudy really is a visionary. He’s into some very weird stuff.

Gamer Guys at are2010

Brad Foxhoven, Chief Marketing Officer, Co-Founder, Ogmento at are2010

Tish Shute: At are2010 there was a lot of discussion about how game dynamics and AR are going to intersect, right? Anything that you saw of interest there?

Bruce Sterling: Well, obviously, there are gamer guys there. Ori’s a gamer. The gamer guys are getting some money. The big buzz right now in gaming is, of course, social gaming.  Farmville has kicked everybody’s ass because it’s not even a game and yet it has more users than the entire gaming industry.

Tish Shute: I know, right! [laughs]

Bruce Sterling: Obviously that’s kind of humiliating. For a long time, I’ve seen people trying to do giant multiuser games on cell phones. It’s difficult to do because the interface on cell phones is crap, right? People aren’t going to run around responding to SMSs.

I can imagine people running around with little Wii-style bats that have audio and visuals on them. It makes a very large native AR game seem more plausible.

Tish Shute: Yes. that would be cool!

Bruce Sterling: Again, it’s not very gamelike to use those little fiduciary markers.

Tish Shute: No.

Bruce Sterling: Moving little cardboard chips, around like with card games…. It would be pretty easy to set up a little AR chess game.  Star Trek style hologram chess pieces,  and so forth. But it’s just cumbersome.

Tish Shute: And also, from what we’ve seen from things like Foursquare, the proximity based social gaming doesn’t have to offer very much [a crown badge, a mayorship] to get some mind share.. the social is the primary game dynamic…

Bruce Sterling:  I’ve seen a lot of different philosophies of gaming over the years. Who’s to say that Second Life doesn’t have the best idea? They built a little scene and then slammed their gate shut behind them.  But at least they’ve got a really nicely-paying little cult stuck in there. It’s different. And it’s manageable and it’s really theirs, theirs, theirs.  They don’t have to call in outside experts to try and run the monster.   They haven’t blown it up to the scale of Yahoo! where they’ve lost control of the enterprise, and gone into a tailspin of management overhead. Second Life has a very intense, almost a cultish atmosphere among the player-slash-developers.

Tish Shute: One thing that helped them was the thing they were always criticized, that the barrier of entry was so high. But once they got people they never left, right?

Bruce Sterling:  That’s not a bug, that’s a feature.

Tish Shute: One of the best features!

Bruce Sterling: Yeah, it’s like being in Mensa. Why don’t you lower your barriers to entry and get in some interesting stupid people?

Tish Shute: [laughs]

Bruce Sterling: In Mensa, we’d rather sit here making puns about neutrinos and fourth-order quadratic equations. [laughs] OK, that’s a business model, if that’s what you want.

The Man With the X-Ray Eyes!

Jesse Schell’s during his keynote, “Seeing,” at are2010, picture from Chris Cameron’s Flickr stream

Tish Shute: Ok!  Now to unpack the man with the x-ray eyes idea, Jesse Schell’s keynote theme.  This is a root metaphor for AR – making the invisible visible, seeing through walls. To me. I think you kind of wrote the book on this because all my ideas on what radical transparency might be come from you – your idea of Amazon.org is key to how I understand this..

Bruce Sterling: Oh, really? That’s funny.   I was touched that Jesse brought up that famous Corman film, because I was a judge in a fantasy film conference in Trieste earlier this year.  And Roger Corman was there.  He was the guest of honor.   “X: the Man with the X-ray Eyes” was one of the films shown during the conference, and I saw it.  I even had dinner with Roger Corman.  I had never met him before, so that was quite amusing.  The difficulty with a film of that kind is that what we science fiction writers call a “House of Cards Ending.”  In that story structure,  you ramp the thing up until the protagonist sees God, and then he has to be destroyed by the falling pillars of the temple.  That’s a classic science fiction structure:  like Frankenstein.  For the sake of the drama, Corman evades the issue of what’s really going on. For instance, let’s just suppose “the Man with the X-ray eyes” is not in fact a psychopath.  Let’s say he gets a grant from the Department of Health and Human Services, and he acts like a real scientist, not a stock B-movie “mad scientist.” So he has, like, backup guys, and some placebos, and a large group of people to test it on, trusted colleagues, and so forth.  You wouldn’t get any of that movie’s wild activity out of that.  What you would get is like a 5% improvement to people’s vision.

Then, in a year, there would be a 10% improvement in people’s vision.  There would be a  classic industrial story.  A rising star, you know, a cash cow.   Real tech isn’t done by a single guy as a divine curse.  It’s created by classic  tech startup culture.  So a runaway technology really behaves in the way that personal computers do.

Tish Shute: The things that get me all Utopian and happy about this are the ideas like those you first outlined with the notion of Amazon.org.

Bruce Sterling:  It would be easy to do an entirely different kind of film than “Man with the X-ray Eyes.”  Something much less B movie,  much less pat.  I mean, at the end of the film,  he destroys his own hardware and blinds himself.  Why?  For what rational reason would he do that?  Why doesn’t anybody else know the big secret of what he’s doing?  Why aren’t there Koreans doing it?  Why aren’t there Austrians doing it?  Why aren’t there Italians doing it?  Why?  AR doesn’t behave like that.  It’s not one lone guy with magic eye drops.  It’s entire teams of people that have been working on stuff for 17 years.  They all approach it in different ways.

Now, they are going to get scandals in AR.  I can guarantee you that.  They are going to get into  hot water eventually.  At least some people will surely come out and accuse them of being Roger Corman B movie monsters.  But unless they accidentally discover atomic fission or destroy the Gulf of Mexico with an oil spill [laughs], I don’t think they’re going to be particularly badly off!   The trouble I imagine  for AR people is very typical new media trouble.  It’s like movies being accused of corrupting our morals, or comic books being accused of leading to violence, or Google being accused of making us stupid and warping our brains.

I’m not an alarmist in that sense, but at least I’m concerned about real threats.  Roger Corman is a B-movie director who’s trying to sew up his lost plot ends by destroying his hero and his hardware. That’s not very plausible. It’s a nice science fiction movie device, but technology isn’t a movie.

Tish Shute: Yes. Well, the other thing that you always remind us of with AR is not to be saying it’s going to be this glorious moment when it’s no longer gimmickey, no longer pop culture. You always emphasize that’s actually part of what’s good about it.

Bruce Sterling: It’s not an accident that practically everybody in that audience knew about Roger Corman.  Nobody looked surprised; not the Austrians, not the Koreans. They were all like: “Oh, yes! Roger Corman!  Love him!”

Tish Shute: There were so many Rudy Rucker fans. Were you watching Twitter? People like Eric Gradman were succumbing to fanboyz moments..

Bruce Sterling: “Yeah. Rudy Rucker, he’s the best.”

“Rudy Rucker gripping an Augmented Reality shoe” from brucesflickr

Tish Shute: [laughs]  I noticed you inspired him to join Twitter..

Bruce Sterling: Well, I’ve got 8,000 followers and, obviously, a lot of them are Rudy’s fans.  Of course he’s going to be gang-rushed on Twitter. That’s not really any more surprising than two motorcycle stunt guys at the same attraction. And I’m a big fan of his Rudy’s blog.   He’s always got interesting things to say.

Tish Shute: Yes. AR does seem to bring out some of the coolest smartest people!  This morning I had breakfast with Joshua Kauffman in Central Park.  He is an advisor and entrepreneur working on design in the public sphere.  I was feeling rather brain dead and jet lagged.  I told Joshua I was wondering how to get the cottonwool out of my brains for this interview and he suggested,  the All Souls College one-word question interview!  Have you ever heard of that? – although apparently they recently scrapped it.

Bruce Sterling: Well, I’ve heard of All Souls College there in Oxford. What was their interview question?

Tish Shute: They used to use only one word, so they would only give you one word. It’s not a question. Basically, they throw out the word and then you had to spin off from there.

Bruce Sterling: You’re supposed to free-associate on a single word?

Tish Shute: I guess so. I hadn’t heard about it, but Joshua suggested it.

Bruce Sterling:  Well, it’s possible..

Tish Shute: Joshua came up with some good words..

Bruce Sterling: Right.

Tish Shute: We were talking about these proximity-based social work networks like Foursquare and Gowalla and how they may influence the emergence of social augmented experiences.

So Joshua’s suggestion for the first word was “territorialization” e.g. how do these new mobile social experiences like Foursquare,  and the observation that actually rather than breaking down territorialization, which would be a good thing, tend to support territorialization…but perhaps new forms of territorialization?

Bruce Sterling: Yeah, they’re re-intensifying it in a very odd, electronic fashion.

Tish Shute: Yes.

Bruce Sterling: I have noticed that.  It’s not true of stuff like projection mapping or the webcam fiduciary display stuff. But with the handheld stuff, and especially the urban informatic stuff, it really can’t help but take on a local flavor. Layar is like “Augmented Dutch Reality.”

And TonchiDot really is “Augmented Japanese Reality.” It’s hard to imagine a Layar interface going gangbusters at Tokyo.  Whereas the TonchiDot interface, which is very clearly influenced by Anime and cartoon graphics…. Maybe it could find some niche of hipsters in Amsterdam hash bars…

…to be continued in Part 2

I have been so busy working on Augmented Reality Event, Jun 2nd, 3rd, Santa Clara, CA, in recent weeks that I have barely had time to post!  But it’s getting hard to contain my excitement about ARE2010.  If you haven’t already seen the sneak preview of the schedule  – see here.

Augmented Reality Event will be an unique opportunity to see how the complete vision of AR is emerging, one that will include visual recognition of real life objects, sensors to enable interaction with physical objects, and the long anticipated, comfortable, sexy, AR eyewear.  Six companies with be presenting and exhibiting AR glasses and HMDs at Augmented Reality Event.

Keynote luminaries will include: Bruce Sterling, “AR Prophet,” Will Wright (Spore, The Sims & now AR and The Stupid Fun Club), Jesse Schell (who will give the AR follow up to his now viral Dice talk), and Blaise Aguera y Arcas (Microsoft, Bing), whose talk on augmented reality maps was one of the most popular talks at TED this year.

Over 40 AR companies from all over the world are represented, and there will be speakers from Qualcomm, Google, Microsoft, Nokia, Intel, and Nvidia.  Over 80 augmented reality experts and entrepreneurs are presenting, including a AR titans, Total Immersion and Metaio.   Mobilizy, Layar, Ogmento, TagWhat, Tonchidot, and Zenitum (just to name a few of the stellar augmented reality start ups coming – for more, and a list of speakers see here) will demonstrate AR is not just a vision, but actually drives significant businesses today,

Five Augmented Reality industry start ups will be on the hot seat in a Launchpad competition with $10,000 prize for the winner, thanks to the generosity of  Qualcomm, a leading developer and innovator of advanced wireless technologies, and the featured sponsor of Augmented Reality Event.  Zenitum, a leading augmented reality company, with a technology portfolio that includes both location and advanced vision-based sensory input is the Gold sponsor for ARE2010, and Adobe are sponsoring the ARt Gala & Reception.  Many thanks to the sponsors for making ARE2010 possible.

Readers of this post can use my discount code TISH245 to register here for a $245 price for the whole 2 day event.

But just to add some balance to my total enthusiasm for the rocket like trajectory of augmented reality in recent months, this post will also look at an area that has been, to date, disappointing for AR developers.   While Nokia will have a couple of speakers at ARE2010, and certainly they have a very impressive history in augmented reality research, they have not delivered the magic brew to attract the augmented reality developer community.   Will this change with the release of the N8 (see video opening this post)?  I wanted to find out if AR developers see a light at the end of the tunnel with N8 as a platform for AR.  So I asked some AR developers and entrepreneurs, “How attractive is the N8 for the AR community?”

Nokia’s apparent nonchalance about augmented reality is in sharp contrast to the bear hug shown in this Verizon Droid Augmented Reality Ad featuring Layar!  As Bruce Sterling notes,

*Hollywood-up for Layar by cooking up some o’ those big-time sci-fi production values.
All magic, all the time, Mr. Roboto.

And it is clear that everyone has high hopes that iPhone SDK 4 delivers fully for AR developers this time! For more on this listen to @chrisgrayson’s 3rd video on Mobile AR here.

But before I get back to my question about the AR community’s thoughts on N8 as a platform for augmented reality, let me crow just a little more about the upcoming awesomeness of Augmented Reality Event.  As the chair of the technology track, ARE2010 is on my mind night and day at the moment!

The tools for the kind of augmented reality we have always dreamed of  – the zero click interface to a heads up, hyperlocal view, bringing you computing anywhere, anytime, with anything, are evolving fast.  Powerful algorithms for search, and machine learning, combined with cooperating cloud data services, will soon be bringing apps that learn by context accumulation to your AR view (see Siri).  Google Goggles, who will have a speaker at ARE2010, have already announced their intent to open APIs, putting a vital tool of visual search in developer’s hands.

A bunch of technologies are maturing these days and helping bring AR to consumers.  Mobile devices that pack all the ingredients for AR, location-based services, see thru video goggles, visual search, innovations in 3D mapping, Simple Geo’s innovative a approach to aggregating geo data, and the next generation of “Street View” such as Earthmine, and, very importantly, open standards and software for AR that will bring many new opportunities for creativity, and monetization.   These are just some of the aspects of this energetic technology convergence that will be represented at  Augmented Reality Event.

In the area of open standards, I am especially excited about the debut of Polaris at Augmented Reality Event.  Polaris is an AR browser, from Blair MacIntyre and the Georgia Tech team, based as much as possible on existing standards and tech.  More on this and ARWave in another post!  Also, I am excited to be demoing ARWave at the Com.Geo 2010 conference, June 21 – 23, in Washington.  Again more on this later.

But for now back to my question opening this post:

“Are Nokia back in the AR?”  and “How attractive is the N8 to the AR community?”

First I asked, my co-chair on Augmented Reality Event, Ori Inbar of Ogmento – one of the augmented reality start ups who’s star is on the rise, and Ori (also known for his writing on Games Alfresco) is one of the key thought leaders in the emerging augmented reality industry.

Ori Inbar:

“Well, the hardware spec is impressive – but that’s not what makes it a good AR device. What’s really important is reach, distribution, and ease of development – areas in which the N8 still has a lot to prove.

- Will the N8 become as prevailing as the iPhone or lead Android phones or will the Nokia landscape remain fragemented with too many variants of phones to support?

- Will the Ovi store pick up steam and become a lucrative chanel for selling apps?

While we wait for these questions to be answered – we’ll keep developing games for Android and the iPhone “

So there you go.

Ori is right that augmented reality developers like the hardware specs.  Here are some comments from Patrick O’Shaughnessey of Patched Reality, Peter Meier, Metaio, and whurley of Chaotic Moon and Robert Rice, Mirascape.

Patrick O’Shaughnessey:

“Looks like it has everything one would want to make both looking glass and magic mirror AR, and built-in face recognition software to boot!  I wonder if they allow you to use both cameras at once making AR video conferencing possilbe?”

“Dedicated graphics processor with OpenGL 2.0 enables 3D graphics” nice!

“Looks like it also does Flash (Flash Lite 4.0, anyway). Not sure if Flash Light provides access to the camera. Could be a nice way to do quick and dirty POC’s on the device”

Peter Meier:

“Metaio thinks the N8 is a very promising device for AR. Especially in Europe we expect a lot of reach.  Symbian is hard to develop for, but on the other hand is much more efficient in terms of real time performance than Android and less restrictive on APIs than iPhone.”

whurley:

“It looks really great and perfect for ar in many regards.”

“clarity of a 12 MP camera with Carl Zeiss optics will give the Nokia N8 some clear advantages in the AR world.  The clarity of this camera will allow for more advanced applications of augmented reality such as image/object recognition”.

“At first glance this is definitely very impressive from a hardware specification. I am surprised though, at the lack of buzz about the device. I have heard virtually nothing about it in the usual sources I monitor, and I wonder about the marketing.

Given the technical superiority and the difficulty Nokia has had penetrating at least the North American market, maybe something catchier like the GR-8 (Nokia Great) or M8 (Nokia Mate) might be a bit more user friendly and easier to market. I particularly like “Nokia M8″…take it with you wherever you go, can’t live without it, etc.

Anyway, as awesome as the specifications are, I would like to echo some of the other comments here…tools, community, ease of development, etc. are critical parts of the ecosystem that would make this ideal for AR development, above and beyond the pure hardware specs. I think Nokia has to really re-think some of its strategy to engage consumers, attract the developers, and establish a new beachhead in North America.”

Talking with Michael Halbherr at Where 2.0

I talked to Michael Halbherr briefly at Where 2.0 before the release of details on the N8, so, of course, he was unable to be as explicit as he might have been about the N8.  Below is a lightly edited transcript.  At Where 2.0, Michael Halbherr  gave a clear explanation of  Nokia’s emphasis on a global strategy.   And, as this blow by blow account on the Google Earth Blog noted, “They’re doing some cool stuff with their handheld maps. In particular, their Ovi Maps (turn-by-turn navigation) are now completely free and becoming very popular.”

But the question remains:  Where is AR is on the Nokia roadmap?

Michael Halbherr (Nokia) on stage at the O’Reilly Where 2.0 Conference 2010 in San Jose, California. Photo by James Duncan Davidson. See Where 2.0 Flickr set here.

Tish Shute: Can I ask you a few questions about Augmented Reality?

Michael Halbherr: So for me augmented reality and mixed reality is essentially a way you search.  So when you look at where we come from, as the company that acquired Gate 5, we believe the phone is essentially the gate that basically combines or connects your virtual and your real world in both ways.  So we can have virtual overlays on top of reality, and you can also record and push it back.

But when you look at the way we do it, I think it is actually an extension of mapping, right?  So you will have the map, which is an abstraction of reality, and then basically with a flick you can go to real reality.  And I think to really understand what you see, you need to actually really know what you see, and an image doesn’t do that.  But if you have compass, and GPS, and 3D models, which is what Navteq is capturing, we can actual do line of sight calculations and you can really say, “OK.  He is looking at that building.” And you can click on the building because you have that understanding.  And you can combine that.  And that leads to a completely new way for how people interact with their environment.

So the here and now, and me here and now, and what next is a key innovation in our focus hereafter.

Tish Shute: I suppose for AR enthusiasts we were all disappointed when we saw the N900 with Maemo, which was very exciting as an operating system, didn’t have a compass.

Michael Halbherr: Yeah, but you know, that is just a matter of timing.  At the end of the day, Meego is our, I think, high-end push.  You will see this Mego device have what you need it to have.  So I wouldn’t worry too much about that.  We are building all the  phones, and everything has everything.

I think it is not just Meego.  It is the whole thing around it.  It is the QT, it is the web run time, it is the whole Meego system.

Tish Shute: Yes we have used QT for some development on the ARWave project, it is very good.

Michael Halbherr: Yeah, and the web run times, it’s open…So Nokia is about open and emotional.  Others are maybe open and more utilitarian or closed, and we are more open and more emotional.  And I think that is what we want to basically do with this platform.

And we are working very hard.  And so whatever I say has been built almost a year ago.  So there is a lot that we are building that we don’t talk about before we announce.  And we try to do this as a global offer, so we can’t launch a little service in the US, which maybe is a problem for us.  But we are basically building complete solutions for global markets.

Tish Shute: Yes.  And when I heard you talk just now on Nokia’s global strategy just now, it all fits in.  But from the point of view from someone who lives here in the US, it seems Nokia is disappointing us.

So is Nokia enabling augmented reality in any ways that noone else is?   I mean if you are interested in augmented reality, the backend and data is vital, e.g. the Navteq part.  And Apple has excited us with the amazing performance of their hardware, and they have taken steps like controlling the manufacture of their chips, powerful cpu and gpu is very important for augmented reality experiences too? Is there a need for a chip set geared to augmented reality specifically?

Michael Halbherr: I’m not sure you have to go that far.  I think at the end of the day it is the data, right?  The map data and the 3D data that is actually extremely important.

Tish Shute: Yeah, I agree with you that the data is vitally important.

Michael Halbherr: Because at the end of the day, when you have a real abstract view, it is really important, because the camera or the view is only a viewer, right?  And people don’t really know what they see.  It is just they plaster on it believing that the person sees what he sees.  So I think real augmented reality or mixed reality is really basically having a real picture, but having, actually, the understanding of the 3D logic of the picture.  And for that, you need to capture the data which Navteq is doing.

Tish Shute: The original dream of augmented reality is 3D media and graphics tightly registered to the physical world.  But actually, it is the data…being able to integrate the streams of data and sensor fusion techniques that will make this interesting and possible right?

Michael Halbherr: Again, it is overlaying virtual elements on real elements.  It is like you are taking your glasses and now you can see information attached.  And of course, you need to make sure it is not overloaded, and that everybody works with it, and it is precise.  So that is basically it.  So I think that is why we are investing a lot in the precision of the maps and the 3D elements of the maps.

At this point we had to leave our seats has the set was being broken down for the next event at Where 2.0.  Check out all the videos for Where 2.0 here,  it was a watershed event for all things hyperlocal, geo, and augmented!

Visual search is heating up, and with it a key stage of turning the physical world into a platform is underway as images become hyperlinks to the world in applications like Google Goggles, Point and Find, and SnapTell – see this post by Katie Boehret.  And while there may be no truly game changing augmented reality goggles for a while, make no mistake, key aspects of our augmented view, factors that will have a lot to do with what we will actually see when an augmented vision of the world is a commonplace, are already in the works.  And, as Anselm Hook (pic above from @caseorganic’s flickr) notes:

“There is a real risk of our augmented reality world being owned by interests which are not our own. There is a real question of when you hold up that AR goggle, what are you going to see?”

Cooperating services, e.g., Google Earth, Maps, Streetview, Google Goggles, and leader in local search like Yelp (see here) would have an enormous ability to filter and control a mobile, social, context aware view of the physical world, and Google themselves see an ethical quandary.

“A Google spokesperson says this app has the ability to use facial recognition with Goggles, but hasn’t launched this feature because it hasn’t been built into an app that would provide real value for users. The spokesperson also cites “some important transparency and consumer-choice issues we need to think through” (quote from Wall Street Journal Column by Katie Boehret).

Anselm Hook and Paige Saez, with great prescience, have been advocating a social commons for the placemarks and imagemarks to our physical world platform through a number of pioneering projects, including imagewiki.   I have interviewed both Anselm and Paige (upcoming) in depth, recently.  My talk with Anselm was nearly three hours long!  So I am publishing the transcript in two parts.

Understanding what it means to have a social commons for  our physical world platform, and augmented reality, are key questions for all of us to think about, but especially important for those of us involved in the emerging industry of augmented reality.

Anselm notes :

“The placemarks and imagemarks in our reality are about to undergo that same politicization and ownership that already affects DNS and content. Creative Commons, Electronic Frontier Foundation and other organizations try to protect our social commons. When an image becomes a kind of hyperlink – there’s really a question of what it will resolve to. Will your heads up display of McDonalds show tasty treats at low prices or will it show alternative nearby places where you can get a local, organic, healthy meal quickly? Clearly there’s about to be a huge ownership battle for the emerging imageDNS”

The mobile internet is moving beyond the internet in your pocket phase of mobility with mobile, social, proximity-based, context aware networks like FourSquare, Gowalla, Brightkite and GraffitiGeo (see Smart Data Collective) likely, soon, to start to take precedence over other forms of social network.

Regardless of the timeline for true augmented reality – 3D images & graphics tightly registered to the physical world,  proximity-based social networking and real time search are already taking us into a hyper-local mode and the realm of augmented reality which is “inherently about who you are, where you are, what you are doing, and what is around you” (Robert Rice – see here). The ground is being prepared for augmented reality now.

If you have been reading Ugotrade, you will know I have been actively involved in developing  an open, distributed AR platform/mobile social interaction utility for geolocated data based on the Wave Federation Protocol – AR Wave a.k.a Muku – “crest of a wave” (see my posts here, here and here for more on this project, and the AR Wave Wiki here).  Federation is, I believe, one vital aspect to developing a social commons for augmented reality and the physical world platform.

Also, a bit of news, I am co-chairing the upcoming Augmented Reality Event (are2010) with Ori Inbar of Games Alfresco and Ogmento, whurley.  Sean Lowery, Prospera, is the event organizer, and are2010 has the support of the AR Consortium.   The are2010 web site is live and there is an Open Call For Speakers.  You can submit your proposals and demos for one of the three tracks, business, technology, or production on the web site here.

Bruce Sterling “prophet” of  augmented reality and more, “will deliver the most anticipated Augmented Reality keynote of the year.”

It didn’t surprise me when Anselm mentioned that Bruce Sterling was a key influence for his work on the geospatial web and augmented reality.  Anselm explained:

“I’d seen a talk by Bruce Sterling at an event called Planetwork [May, 2000]. And that event was, for me, a turning point where I decided to focus full time on exactly what I cared about instead of doing things that were kind of similar to what I cared about. So, his influences is a pretty significant one to me at that exact moment.”

For more see viridiandesign.org -  seems it is time for a “Neo-Viridian,” revival!

This post by Bruce Sterling on Pachube Feeds, and Thomas Wrobel’s prototype design for open distributed augmented reality on IRC, were key inspirations for me when I began thinking about the potential of Google Wave Federation protocol for augmented reality.  I had been exploring Pachube and deeply interested in the vision of Usman Haque, but I had a real aha moment when I read this :

“(((Extra credit for eager ubicomp hackers: combine this [pachube feeds] with Googlewave, then describe it in microsyntax. Hello, 2015!)))”

I think the AR Wave group will earn the extra credit and more very soon!  Davide Carnovale, need2revolt, and Thomas Wrobel have been leading the coding charge, and there will be a very early AR Wave demo soon, perhaps as soon as the Feb 16th ARNY Meetup. 

Open access to the creation of view that will eventually find its way into AR goggles, will depend not only on the power of  an open distributed platform for collaboration like the AR Wave project.  Our augmented reality view will be constructed through complex “hybrid tracking and sensor fusion techniques” (Jarell Pair), cooperating cloud data services, powerful search and computer vision algorithms, and apps that learn by context accumulation will drive our augmented experiences, and at the moment, these kind of resources, at least at scale, are for the most part in private hands.

In the interview below, Anselm’s discusses  how trust filters, and being able to publicly permission your searches so that other people can respond and so that people can reach out to you, and the democratization of data in general, are even more of a concern with augmented reality and hyper local search. The task of understanding what it means to have  a social commons for the outernet remains an open, and pressing question.

Anselm explains (see full interview below):

“as we move towards a physical internet where there’s no clicking and there’s no interface and the computer’s just telling you what it thinks you’re looking at, translating, you know, an image of a billboard to the name of the rock star who’s on that billboard, or translating the list of ingredients on a can of soup to the source outlets where it thinks that, those ingredients came from. When you have that kind of automated mediation, the question of trust definitely arises.

And we haven’t seen the Clay Shirkys or the Larry Lessigs of the world start to talk about this yet.  Although I suspect that in the next four or five years that the zero click interface will become the primary interface, that we’ll have…we’ll come to assume that what we see with the extra enhanced data we get projected onto our view is the truth. Yet, at the same time, there is just no structure or mechanism even being considered for a democratic ownership of it.”

Augmented Reality will emerge through sensor fusion techniques & cooperating cloud services

In 2010, sensor fusion techniques, computer vision technology in conjunction with GPS and compass data will create data linking that can enable the kind of augmented reality that has been the stuff of imagination for nearly four decades (see Jarrell Pair’s post).

Putting stuff in the world in 3D is of course key to the original vision of augmented reality, and one of its biggest challenges.  Augmented reality is going to be implicated in a real time mapping of the world at an unprecedented scale and granularity.  We have barely an inkling of the implications of this now.

Anselm and Paige have been working in the heart of the social cartography movement for nearly a decade.  The vision and experience of this community is vital to understanding how augmented reality and the world as a physical platform can evolve into something that benefits people and allows them “to have a better understanding of the opportunities around them.”

We have been hacking maps for millenia –  “from conceptual story mapping, to colloquial mapping in European development and the cartographic renaissance created by the global voyages and rediscovery of Ptolemy’s maps” (Andrew Turner).  And, recently, initiatives on a public-provided GIS, like OpenGeo, have led the way toward more open, interoperable, geospatial data.

Mapping takes on a new an crucial role to augmented reality.  Nokia’s ImageSpace is beginning to do what many thought Microsoft would do with photosynth two years ago.

And, if we see these kind of projects developed into a “photo-based positioning systems” -  “3d models of the environment to cover every possible angle, and then software that can work out in reverse based on a picture precisely where you are and where your facing” (Thomas Wrobel), we would find augmented reality leap forward over night.

It is time to take very seriously the vast opportunities and potential pitfalls of an augmented world.

“when you are mediating the translation layer between the image and the data, then there is an opportunity for you to control it, and that opportunity is hard to resist.  It is hard to choose not to own that opportunity. It is an advertising opportunity. It is a revenue opportunity. It is a chance to send a message and a tone.

I know that Google and companies like that are keenly aware of the kinds of roles they don’t want to hold, but it is sometimes seductive to think about them. And I am afraid that we, as a community, need to assert an ownership, kind of a commons, over how computers will translate what they see to information that we perceive.”

There are some initiatives emerging.  Tonchidot (who closed on $4 million of VC for augmented reality last December) has helped create the AR Commons in Japan.  CFO of Tonchidot, Ken Inoue explained in an interview with me in September 2009.

“We feel that public data, such as landmarks, government facilities, and public transport should be shared. We see an AR world where people can readily and easily access information by just seeing – quick, easy, and efficient.  And because of this ease and intuitiveness, children, the elderly and handicapped will surely benefit.  AR could help create a safer society.  Warnings, alerts, and safety information could save lives and avoid disasters.  These are what we, and AR Commons would like to tackle in the not so distant future.”

But the task of building a social commons for the physical world platform has only just begun.

Interview with Anselm Hook

photo from Anselm’s Flickr stream here

Tish Shute: We first met last year at Wherecamp. The start of 2009 was I think the “OMG finally” moment for augmented reality and in less than a year AR, at least in proto forms, AR is breaking into the mainstream now! You are one of the founding visionaries/philosophers/hackers of the geo web and you have been thinking about geo web and AR for a long time – all the way back to the legendary Head Map Manifesto, and before.  Mostly recently you led the way in the very successful ARDevCamp in Mountain View. Could you start by telling me a little bit about the history of your pioneering work with geolocated data?

Anselm Hook: I am a long time Geo fanatic. I’m really interested in social cartography and what some people call public-provided GIS, that’s some language that people use. Anyway, my personal interest, when I talk to people who are non-technical (and it’s been a long term interest in the way I phrase it) is that I want to help people see through walls. So, the goal is very simple. I want people to have a better understanding of opportunities around them, the landscape around them. I always get frustrated when people make bad decisions because of a lack of information, especially when it’s related to their community and related to their environment. But, plainly put, I really just want “to help people see through walls”. It’s a very simple goal.

Tish Shute: I know you worked on Platial, which is really one of my favorite social mapping applications. It really broke new ground. What was the history of that? How did you get involved with Platial?

Anselm Hook: That’s an interesting question. It actually started at around 2000 when I saw Bruce Sterling talk. I had been writing video games for many years, and I was quite good at it, and I enjoyed it. But, the reasons I was doing it diverged from why the industry was doing it. I was making video games because I like to make shared spaces for my friends to play in and to share experience. I really enjoyed making shared environments. I worked on BBS’s and my friends and I were always making these collaborative shared environments.

Once the video game industry kind of started to take off, I started to do high performance, 3D interactive video games and making compelling shared spaces, and it was a lot of fun. But, the frustration for me was that there was a huge industry growing around it and became very commercial. Although it paid well, it started to diverge from my values which were more centered around community environments, and shared understanding.

Tish Shute: Yes very rapidly, the big games kind of devolved from the social aspects and became more and more into single player really, didn’t they?

Anselm Hook: It was the way, actually, because even though often you were in a many player world, you weren’t collaborating, everything else became just a target.  I liked the idea of deep collaboration that calls the kind of playful space you see in IRC, or in the real world, where people are solving real world problems.

And I grew up in the Rockies, and I was always had a lot of access to the outside. So, I saw shared spaces and collaboration as a way to protect our environment. [ To step back ] I think people used different metrics for measuring their choices in the world and many people have a value system centered around minimization of harm: making sure that the people are not hurt. But, my value system is different. I personally believe that protecting the planet is more important: to maximize biodiversity. I feel like protecting people around me comes from protecting the ecosystems they live in.

Tish Shute: That’s interesting, isn’t it, because the history of Keyhole was really that, wasn’t it.  Keyhole later became Google Earth, but I mean it began out of a project to look at what was going on in the ecosystem over Africa at that time, didn’t it?

Anselm Hook: Yes, in fact many people’s projects are stemming from an environmental concern. Mikel Maron’s works for example – he’s doing Map Kiberia, and he also worked on OpenStreetMaps.

Tish Shute: Map Kiberia – that is the new project?

Anselm Hook: Oh, yes his project is called Map Kiberia. He’s mapping a city in Africa.
[For more see Map Kiberia’s YouTube Channel – photo below from ricajimarie ]

Tish Shute: Right, great!

Anselm Hook: When I started to look at GIS and mapping I started to meet people who had a very similar background. What happened to me is I kind of stepped away from games around the year 2000. I’d seen a talk by Bruce Sterling at an event called PlaNetwork. And that event was, for me, a turning point where I decided to focus full time on exactly what I cared about instead of doing things that were kind of similar to what I cared about. So, his influences is a pretty significant one to me at that exact moment.

[For more see viridiandesign.org – seems that it is time for a “Neo-Viridian,” revival.]

Tish Shute: It’s interesting because now your paths are crossing again with augmented reality. You are on the same wavelength again.

Anselm Hook: It’s funny, actually, I’ve had a couple of brief overlaps in that way.  Well, so in 2000 I went to see this talk and I did a small project called — well, I called it SpinnyGlobe. What I did is I mapped protests from a number of websites onto a globe to show the level of community opposition to the pending war in Iraq. It was the first time there had been a protest before a war. So, it was very interesting to me. [ See http://hook.org/headmap ]

Tish Shute: That’s really fascinating. Do you have any pictures of that you could send me?

photo from anselm’s flickrstream

Tish Shute: Yes, I’ll definitely look SpinnyGlobe up. It sounds very interesting.  One of the aspects of your work on geo-located data projects like this and Platial is that you really started to develop this idea of a culture of place, about how people make place. This was the wake up call to me regarding the power of networks combined with geo-data.

We are hoping to extend this idea into augmented reality with the an open distributed platform for AR so that we can collaboratively map our worlds from the perspective of who we are, where we are, and what we are doing.  I know you’ve just done some work recently in augmented reality.  I know you put the code up already.

By the way, I love the way you take your philosophy into the way you make code – the practice of making some code, trying some things out, making it all public and publishing your findings, you know, your comments on that experience.  Perhaps you could recap sort of how you picked up recently on the state of play with augmented reality and what aspects you looked at, and what came out of that experience?

Anselm Hook: So, it’s a very simple trajectory. Coming out of the work I had done, Platial, among other projects and I started to just look at the hyper-local and I suddenly realize that even those services weren’t really speaking to living, and how to really see and solve local problems. What was missing was a sense of context.

The map doesn’t know how you’re feeling, it doesn’t know if you’re in a hurry, it doesn’t know what you want, it’s very static. Even the web maps are very static. And augmented reality for me I started to recognize as a combination of — well — it’s probably collision of many forces, many forces that we’re all a part of. We’ve also didn’t realize that the real-time web is really important, it’s part of what AR is about.

We have all started to realize that the context is important. You know, your personal disposition, your needs, if you want to be interrupted or not. That is the kind of thing that the ubiquitous computing crowd has talked about. We started to recognize that there are sensors everywhere, and the ambient sensing communities talked about that. So what is funny for me about augmented reality is I started realizing it is just a collision of many other trends into something bigger.

Everything else we thought was a separate thing is actually just part of this thing. Even things like Google Maps or mapping systems we think are so great are really just kind of almost an aspect of a hyper-local view. You actually don’t really care what is happening 10 blocks away or 100 blocks away. If you could satisfy those same interests and needs within a single block, one block away, you would probably be really happy. You really just want to satisfy needs and interests, find ways to contribute, or get yourself fed, or whatever it is you want. And AR seemed to be the playground to really explore the human condition.

Tish Shute: Anyway, I think one of the things that has been very amazing this year is we to have the good mediating devices that, for the first time, give us compasses, GPS, and accelerometers. But one sort of missing pieces with AR at the moment is [tracking, mapping, and registration] – the kind of things colloquial mappings of the world could be of great help with.

We have seen mapping coming out of the Flickr data, e.g., the University of Washington, put the maps together from the geo-tagged Flickr photos. Now if we could have that linked up with AR, then we have the kind of mapping we need to kind of really hook the geo-data onto the world in a way that goes beyond…you know, what compass and GPS can really deliver is pretty minimal at the moment.

Anselm Hook: There is a real risk of our augmented reality world being owned by interests which are not our own. There is a real question of when you hold up that AR goggle, what are you going to see? Are you going to see corporate advertising? Are you going to see your friends’ comments or criticisms? It is going to be an Iran or a democracy, right? It is unclear.

Right now there are some disturbing trends I have noticed. I am a big fan of Google Goggles. I think it is a great project. But when you are mediating the translation layer between the image and the data, then there is an opportunity for you to control it, and that opportunity is hard to resist. It is hard to choose not to own that opportunity. It is an advertising opportunity. It is a revenue opportunity. It is a chance to send a message and a tone.

I know that Google and companies like that are keenly aware of the kinds of roles they don’t want to hold, but it is sometimes seductive to think about them. And I am afraid that we, as a community, need to assert an ownership, kind of a commons, over how computers will translate what they see to information that we perceive.

Tish Shute: Yes. And this is how we met, again, recently [over the project to create an open, distributed platform for AR using the Wave Federation Protocol]…

This is something I feel really deeply is that, you know, basically we need the physical internet to be as open as, as the, as the internet, as the end-to-end internet has been. Or more so, actually, because the end-to-end internet has seen the trend has been to walled gardens.  Basically Facebook became enormous, an enormous walled garden which, I think, was despite, our predictions about them, [walled gardens] are the social experience really on the web.  It’s very much in walled gardens still and I, and I really feel that with the physical internet, we need to make great efforts not for it not just to be a series of small pockets of privately funded walled gardens.

There needs to be some kind of communications infrastructure that keeps it open so that was when I got interested in looking at the Wave Federation Protocol because it was a real time, you know, an open real time protocol that could possibly be a basis for that. But I think the point you’ve talked to just now, the mapping of the world and who has the “goggles”, i.e., the image data, image databases, that make the world meaningful is really, that’s still a, it’s still a BIG question [i.e. who controls the view?].

When I saw ImageWiki, [I realized] that is a piece that is vital for, for augmented reality. We need to have a huge social effort to be involved in this,  linking in and creating the  physical internet, in creating the image hyperlinks that will make that meaningful.

Anselm Hook: I think that’s a great point. The search interface, the kind of Internet that we’re used to, the way we talk to the network now, is fundamentally open end to end. Yes, you can have your oligarchies inside of it, as we see with Facebook, but you can always start your own venture up and you can do a search on something, and you can find that, that website and you can join it or you can put up your own webpage and people can find it.

The translation layer, the idea of text search and the ability to discovery power and the serendipity and the openness of that discovery, it’s pretty open right now. We do have some serious boundaries of language, which is one of the reasons I was working at the Meedan.org [hybrid distributed, natural language translation] for a couple of years, trying to bridge that issue.

But here, as we move towards a physical internet where there’s no clicking and there’s no interface and the computer’s just telling you what it thinks you’re looking at, translating, you know, an image of a billboard to the name of the rock star who’s on that billboard, or translating the list of ingredients on a can of soup to the source outlets where it thinks that, those ingredients came from. When you have that kind of automated mediation, the question of trust definitely arises.

And we haven’t seen the Clay Shirkys or the Larry Lessigs of the world start to talk about this yet.  Although I suspect that in the next four or five years that the zero click interface will become the primary interface, that we’ll have…we’ll come to assume that what we see with the extra enhanced data we get projected onto our view is the truth. Yet, at the same time, there is just no structure or mechanism even being considered for a democratic ownership of it.

We have with DNS, for example, the idea that you can register the domain name and people can search for it, and find it, and go to it. There’s no such thing as an Image DNS, or an image translation to DNS right now. What does it mean when everything is just “magic”, when there’s no way for you to be a part of the conversation, where you’re just a consumer of what people tell you, or of what one company right now, tells you, is reality? That’s a real concern.

Tish Shute: This, to me is the most important question at the moment. I mean, it’s the big one and it’s the place to put energy if you love the Internet [and what it can now become] right. You’ve got to put a lot of energy into this because this [a democratized view of the physical world as a platform] won’t just happen, because there’s a lot of momentum already for it to be heavily privatized, partly because, one reason is, some of the computer vision algorithms that, say, make sense of things like the geotag photographs are not open.  I mean, for example, the beautiful maps that have been made from the University of Washington [from Flickr geotagged photo sets], that isn’t in the public domain.

Anselm Hook: Right. Tish, and in fact you’re referring to [with the maps from the Flickr photos] to ordinary maps and the fact we’ve already seen that maps lie, we’ve already, seen how much maps are reflecting a certain truth that becomes the normative truth. Google maps reflects roads, because this is roads and cars, right? Only recently have they thought about buses and walking. So the normative view that people assume is the reality, is showing off you know Starbucks, and roads, and cars, that becomes the default, those prejudices are just assumed, you know, the truth. But they’re not the truth at all.

I was talking to a friend of mine in Montreal, [Renee Sieber], and she said that their Indian portage routes are a bridge across land and water, they don’t think of a piece of land and a piece of water as being different things, they think of them as one thing: a route. It’s already a different kind of language we can’t even reflect it.

So not only is there this kind of formal, anthropological lie, in a sense, but there’s this way that we deceive ourselves because of our own prejudices.

Tish Shute: Yes I agree and that’s why I think when I saw some of the things you had written on the ImageWiki point clearly to the need to create a social commons. We need a social commons for the real-time physical internet, we need it for the image hyperlinks that make sense of that.

And it’s a complicated thing in a sense, though, because we don’t actually have a good distributed infrastructure for AR yet, and I found exploring AR Wave, that at last we have the suggestion of an open, federated protocol for real-time communication – the wave federation protocol. [Real time communications is a very important part of AR].  It isn’t an actuality yet where lots of people are able to use it, set up their own servers, and there’s not a standard all the way through  [there is not a standard for how data is sent between the client and the server].

But Wave Federation Protocol does make possible truly distributed social AR.  I started thinking when I saw ImageWiki that to bring ImageWiki together with the social collaborative power of distributed AR.  This really would be the basis of creating a social commons for augmented reality and the physical world as a platform – the start of a bottom up with deep social collaboration on how we create augmented reality colloquial maps that can inform a hyper-local of the world.

Anselm Hook: Yes. When Paige Saez, John Wiseman, and myself, and a few other folks… You know, Benjamin Foote, Marlin Pohlmann, and a couple other people started to play with this, we quickly found that… We started to realize, “Oh, this kind of thing will be at least as popular as IRC. There will be at least as many people doing this as chatting in little virtual spaces. There’ll be at least as many people decorating the world with augmented reality markup, and maybe using the real world as a kind of barcode for translating what you’re looking at into an artifact, a digital artifact.

And that the size of that space was going to be huge, basically. Maybe not quite as commodifiable as Twitter, but certainly very energetic.

Many of the projects we did were just kind of looking at these kinds of issues sort of from an artistic, technical, and political point of view. We weren’t so much posing complete solutions, but simply using a praxis to explore the idea with an implementation, as a foundation for this discussion. So I think we sort of opened that can of worms for sure.

Tish Shute: Did you actually set up ImageWiki to be working as a location based app yet?

Anselm Hook: It is a location based app. It collects your longitude, latitude, and the image and stores it. And then it uses that as a way to translate that image to anything else. It could be a piece of text or a URL.

Tish Shute: So there is a smartphone app, but you didn’t take it as far as an AR app yet?

Anselm Hook: No. We didn’t do a heads-up view. There are apps on the iPhone store that do that, but they don’t do the brute force image recognition that we were using. We used a third party off the shelf algorithm that we found on Wikipedia and downloaded the source code, and threw it on the server. And John Wiseman in LA wrote the scalable database backend so that we could scale the actual…

Tish Shute: So how did you set the iphone app up to work?

Anselm Hook: The iPhone side was very simple. You take a picture of something and it tells you what it is. That is all it did. We would take the location, but the client side, the iPhone side, just rendered, returned to you…It said, “Someone said that this picture of a barking dog is an advertisement for a local band.”

Tish Shute: Right. So basically it was a geo-tagged?

Anslem Hook: Yes. We are just collecting the geo information. Actually, there were a whole lot of technical challenges. The whole idea of ImageWiki is actually kind of beyond our technical ability for a small team like us. It really does take a team, a group like Google, to do this kind of thing in a scalable way.

Tish Shute: Why is that?

Anslem Hook: There are two sides. There is the curating the images. I think that is the job of groups like us – open source groups who can curate images that are owned by the community. And then the searching side, the algorithm side, where you are actually matching the fingerprint of one image to images in your database, that takes a much more…that is much more industrial.  We get both sides, ours is not a scalable solution. It is mostly…proving that it could be done was important.

Tish Shute: In terms of hooking Imagewiki up to the collaborative possibilities of AR Wave wouldn’t federation pose some interesting possibilities for scaling search algorithms and all that?

Anselm Hook: Yes. And what is funny also, incidentally, is that, nevertheless, we did look for some financial support for it, but we couldn’t…we just didn’t find the investors to scale it. Now, other companies like SnapTell took a shot at it. And they have an app in the iPhone store where you can point at a beer bottle and get back the name of the beer bottle.

The classic example everyone uses is a book. Amazon has all the image jackets of all their books. You can point SnapTell at almost any book and get back links to buy that at Amazon, the price of the book, and user comments on the book. So they are treating Amazon as the canonical voice of the book, for better or worse. That is the state of the art so far, up until Google Goggles came out a little while ago, which actually blows it out of the water. But, that is where we are now.

Tish Shute: Right. But the point you raise about how when something like Amazon comes canonical of what is book, right, this is the whole point, isn’t it?

Anselm Hook: Is Amazon truth? It’s not bad. Jeff Bezos seems like a nice guy, but, you know.

Tish Shute: And this is the point of having these open infrastructures for this.  And this should be obvious in a way, but it comes back to the thing about what made the Internet great was the fact that even though as you note, you get an oligarchy like Facebook, but people always could just go off and do something else, right? Because the fundamental infrastructure was basically open and designed to be available for everyone. And many people have championed that and fought for it hard [to maintain this openness] haven’t they? They have devoted their lives to keeping it that way, even if the oligarchies have done their thing.

Anselm Hook: Yes. There are really some things that are underneath all of this that haven’t been solved yet.

One is that the trust in social networks has not been built yet, so we can’t do peer based recommendations very well. We can’t filter noise by peers. Twitter kind of is moving there, but I don’t just want to listen to my Twitter friends. I want to listen to my friends of friends. If I am getting truth from somebody, I want to get that truth from people my friends say that they trust.

Then the second problem is that there is a search business. My friend Ed Bice, who owns Meedan, always says that a search itself, a search request, is an opportunity to make…is a publishing moment. It is an opportunity to say what you think. In the real world, if you are just hanging out with humans and you look somewhere, other people might look at your gaze and they might look at what you are looking at. Your gaze itself is a public act.

Gaze is a soft act, but it is one that is visible. With Google, the gaze of four billion people is invisible. We don’t what people are looking at, there is no opportunity to participate. Let me give you a real example.  I have taken a image of something of the bust of figure or a statue.  Why can’t the museum in Cairo look at my request and tell me oh yeah that is Tutankhamen, or that is Nefertiti right? Why can’t they have a chance to participate in the search and respond to me?

Right now the the only person that responds is Google when I do a search. We need to invert the search pyramid and open up search, so that search is a democratic act, so that you can publicly permission your searches so that other people can respond and so that people can reach out to you, not just you having to do a dialogue.

The common example of this.. and we see this everywhere: I am looking for a slice of pizza right, now I am hungry I want some pizza. I have to ask Google, look find twelve websites, call twelve phone numbers, and talk to each of the twelve stores, and ask them are they open late, is the food organic, is the food in any good, do my friends like it.

Whereas what I should be able to do is just say it’s a search moment and I am interested in pizza. If those pizza places my criteria like you know my friend’s like them and they are organic, they are open, then that pizza place can call me. I have the money why should I do the search? So the whole business of search, the whole structure of search is predicated around a revenue model, but its a really short-sighted revenue model, its not a brokerage.

Search isn’t search, search is hand waving.  These should be moments for us to have a discourse. So problem we are seeing in AR with communication of the right information is actually underneath AR, at the level of the whole infrastructure.

Search needs to be inverted, trust filters need to be built. We need to democratically own our data institutions.  We don’t right now.  That will be more of a concern, especially with AR.

Tish Shute: Yes, especially with AR, which is this why got all excited about federation.  Do you think federation has the potential, an opportunity to create [the new infrastructure you describe?]

Anselm Hook: Absolutely,  its absolutely what we must do. It is much harder to do. It is absolutely critical.

Tish Shute: And why is it much harder to do? Could you explain that?

Anselm Hook: Well, it’s very easy for a bunch of hackers to build a service that you log into and fetch some data, it’s a single thing. They don’t have to talk anybody, they can use their own protocols, they can hack it, it’s a big black box, behind the scenes. There’s running back and forth in a giant Chinese room delivering manuscripts and scrolls to you. Whatever is behind the black box, you don’t care, it just works.  But when you federate, you need to actually publish and have standards, and then you’re talk about semantic, everyone starts getting really excited and wave some hands. It becomes a disaster. It’s, at least, another power order, more difficult than DIY, build it yourself.

Tish Shute: So, in terms of what Google Wave have done with their approach to federation, what do you think have been their achievements and what do you think is their obstacles? What do you think are the failings of the Wave? Because it’s the first big public major player backed approach to something federated, isn’t it? In real time.

Anselm Hook: Yes. I think the most important non-federated service on the planet today is Twitter.  Identi.ca it’s not getting any traction with respect to Twitter. [ Even though ] Identi.ca is a federated version of Twitter and is very good. [ Identica is now Status.net ] . So, we see already there that small players aren’t being competitive. Then look at other services like IRC. IRC is the secret backbone of the Net. All the open source projects, all the teams, all the people that work on opensource projects are all on IRC. It’s the only way they get anything done.

With Google Wave, and the protocols underneath Google Wave, we see an attempt to build a similar kind of real time, but distributed protocol. I think it’s the right direction. I think, people should pick up the offering and make their own servers. I think that protocol is really great, I think the fact that is compressed, its high performance, it is small, real-time of blobs of data flying around, all exactly the way it should be done. It is getting close to this kind of rewrite of the Internet that people keep talking about, because, you know, the net protocols are so bad, it is starting to treat the idea of intermittent exchanges being more transitory, volatile, and not heavy.

….to be continued.  Part 2 coming soon!

ISMAR 2009 -  was an extraordinary mix of  high geek, academic eminence, gungho Dutch Cowboy entrepreneurial spirit, German engineering and industry, brilliant artistry, and invention, all fueled by a sense, and a very active presence in the case of Diamond Sponsor – Qualcomm, that the big technology players are waking up to augmented reality.

In the picture sequence above (click on photos to enlarge),  Noora Guldemond, Metaio, demonstrates Junaio (coming to an iphone near you Nov 2nd) to Dirk Groten, CTO of Layar (top left photo).  One of the nice social features of Junaio is that users can share the 3D augmented scenes they have created.  Noora is demoing this capability to Dirk, and as you can see he cracks up when he sees the  scene Noora has stored on her phone.  Dirk and I both recognize that this cute little dinosaur augmentation (close up above on bottom left) must have been created by Peter Meier, CTO of Metaio, during the Interoperability and Standards workshop earlier that day.  Metaio it seems were discussing standards while enjoying some 3D augmented back chat.

Both Dirk and I were active participants in the workshop too.  But little did we know that Peter Meier had introduced his little 3D dinosaur into our discussion while we diligently, and sometimes heatedly, debated the merits of XMPP, Wave Federation Protocol,  KML, ARML, VRML, X3D, and  more!  The photo I took is on the bottom right of the four pics above. It was probably taken very shortly after Peter’s augmented Junaio scene.  Of course there is no little dinosaur in my pic of  Dirk Groten with Joe Ludwig and Markus Tripp of Mobilizy who were discussing AR standards oblivious to Peter’s virtual pet in our midst.

I must say I had noticed an impish look on Peter Meier’s face (see photo above on the left – Peter is wearing glasses and holding a phone).  And Markus Tripp, of Mobilizy  revealed a little bit of gaming of his own, when he let out that, in part, ARML is a provocation.  But Peter was clearly unfazed and enjoying himself.  Dirk, tasked to summarize our discussion, stalwartly maintained an optimistic but serious tone fitting for a standards discussion:  “There is a willingness to standardize….,” he began (pic above on left – click to enlarge and read text).

But it was a little 3D dinosaur that, perhaps appropriately, had the last laugh. Fitting, as I am not sure whether anything anyone says about AR standards at the moment will hold up.  But, as Ori commented in his great post – an epilogue for ISMAR 2009, the vibe was “Peace and Love” in AR Browser land (although Chetan Damani of Across Air was not in the standards discussion because he attended the UX/content? workshop instead).  But as they say, “all’s fair in love and war.”  And it is my feeling the games have barely begun!  There are many players (virtual pets included) waiting in the wings. I met some at ISMAR, and they are just itching to join the frey.

Ori Inbar, Ogmento and Robert Rice, Neogence Enterprises, both founders of the AR Consortium, made great efforts to set our young industry off on the right foot -  in the  spirit of coopetition (a neologism coined to describe cooperative competition). See Curious Raven for Robert’s conference observations, and Ori’s post on Games Alfresco for more about Mobile Augmented Reality at ISMAR 2009.  The Mobile Augmented Reality Workshops were driven by an indomitable spokesperson for the new AR industry, Christine Perey.  Christine not only helped motivate discussion on the issue of oxygen to the system, i.e. business value, but also she was a very generous connector at the conference.

What’s Next From Augmented Reality’s Top Chefs?

As Ori pointed out, Kent Demaine, oooii (pic above is from the oooii web site), Minority report VFX designer was hanging out at ISMAR 2009 and he came to the panel I was on: “Augmented Reality in Sports,  Entertainment and Advertising.”  We chatted afterwords about instrumented environments and how this is such a key to development interesting augmented experiences.  Also I mentioned how back in the day I was involved in some of the early development of motion control software.  And it was great to hear Kent say they were still finding motion control cool at oooii.  As Ori notes, he is the “guy with the most enviable AR credentials in the world (the guy who designed VFX for minority report),” and oooii is busy and hiring.

One of the highlights of the Arts, Media and Humanities track for me was meeting Jarrell  Pair. He really brought the best out in panelists with his well tuned questions.  The recording of ISMAR was comprehensive and videos should be up next week.  I will post the slides on Ugotrade of my presentation:  “The Next Wave of AR: Shared Augmented Realities and Remix Culture.”.

“Mixed and Augmented Reality: ‘Scary and Wondrous’” – Vernor Vinge

“Imagine an environment where most physical objects know where they are, what they are, and can, (in principle) network with any other object. With this infrastructure, reality becomes its own database.  Multiple consensual virtual environments are possible, each oriented to the needs of its constituency.  If we also have open standards, then bottom-up social networks and even bottom up advertising become possible. Now imagine that in addition to sensors, many of these itsy-bitsy processors are equipped with effectors.  Then the physical world becomes much more like a software construct.  The possibilities are both scary and wondrous.” (Vernor Vinge -  intro to ISMAR 2009)

Vernor Vinge’s short intro to ISMAR 2009 (which can be downloaded with the ISMAR 2009 schedule here) captures the essence of the “Scary and Wondrous” dawn of the age of ubiquitous computing and mixed and augmented reality.  It is definitely worth a moment to download.  The future of augmented and mixed realities, as Vernor Vinge points out, is tied up in a “tension between centralized and distributed computing” that “will continue long into the future.” One of  my fascinations with Wave is that it offers a tantalizing opportunity to explore augmented reality in an open distributed architecture.

At ISMAR, I talked with as many people as possible about the AR Wave project – see my post here for more about Wave enabled AR.  Many people were very enthusiastic to join the AR wave and the only thing I really lacked was about 100 invites to hand out!

“Everything, Everywhere – making visible the invisible”

Some of the areas that I would have liked to see given more attention on at ISMAR were sensor networks, data curation, and user experience.  Not that these areas were entirely neglected with Pattie Maes, MIT as a keynote speaker, and Mark Billinghurst presenting on some fascinating work on social augmented experiences and user experience.  I highly recommend catching up on these and other ISMAR presentations when the videos go up.

And, I was very happy to meet and talk to Sean White whose work at Columbia University is one of my inspirations (for more about Sean’s work see here or click image above):

“the confluence of powerful connected mobile devices, advances in computer vision and sensing, and techniques such as augmented reality (AR) enables exciting new opportunities for interacting with this hidden network of dynamic information and shifts the locus of interaction from the desktop computer to the world around us”

And I had several very interesting conversations  at ISMAR about developing social augmented experiences that connect us to a physical world that is becoming “much more like a software construct” (Vernor Vinge).  Dirk Groten, CTO of Layar mentioned a few interesting projects Layar has up their sleeves, including something  Layar may be cooking up with The RoomWare Project.

The picture above is of RoomWare’s Social RFID Installation for Media Plaza in Utrecht (read more here).

Demos Galore!

In the demo rooms, Noah Zerkin (pic below left) pretty much single handedly carried the AR flag for a growing community of augmented reality Makers and Hackers.  But his presence was much appreciated, and he tirelessly demoed The Zerkin Glove. See the first of what may be several posts from Noah on ISMAR here.

And I got to try out the Vuzix goggles (picture above on right).   This was my first experience playing an AR game that was smart about real world gravity. It’s  “an augmented reality marble game that uses gravity as a game controller” – see Ori Inbar’s write up here.  It was a very compelling experience and I have to say I didn’t really notice the shortcomings of the Vuzix goggles while I was absorbed in the game. And  I turned out to be quite good at the game too. It is intuitive unlike the kind of  rule based games I never have time to learn properly.  But what is so special about this project is the tools that it is built with are open, and available for all, and affordable (see this list on Games Alfresco).

It was a great pleasure to meet Prof. Steven Feiner (picture on below the left) who heads Columbia University’s brilliant AR research team at The Columbia University Graphics and User Interfaces Lab.

Ori Inbar (pic below on right) also spent a lot of time in the demo room showing off Ogmento’s lovely AR learning game that delighted attendees, “Put a Spell: Learn to Spell with Augmented Reality.”

For a round up of  what’s next for augmented reality head mounted displays check out, Games Alfresco here, and Thomas Carpenter’s excellent review of the head mounted displays.

Ori Inbar on Games Alfresco asks is “Microsoft – the new big player to watch?”   “Georg Klein, inventor of PTAM-on-an-iPhone (and the smartest Computer Vision guy on the block)” has joined Microsoft to make Mobile AR.

The picture on the left above shows Georg trying out ARhrrr with Blair MacIntyre.   And on the right Blair is demoing his marker card pack to Senior Vice President of Cypher Entertainment, David Elmekies.  Yes ISMAR was abuzz with demos. See this post from Gail Carmichael for more video demos.

Next Year ISMAR 2010 in Korea!

At the banquet, I managed to find a seat at a table with Sean White (at left in photo above with Christine Perey to his right) and the Columbia University team.  The banquet culminated with the “Past and Future of ISMAR” Panel chaired valiantly by Jay Wright of Qualcomm.  We were asked to offer our input for ISMAR 2010.  I offered up an idea that I have been nurturing for a while now -  to stage a “Green Tech AR Competition.”  Perhaps, I suggested, we could base the competition around a conference (ISMAR 2010 in Korea?) and set up a target rich, instrumented environment for the occassion.  I think the Arduino open hardware community and AR developers have a synergy that is just waiting to be explored!  And, if we add the innovators of data curation to the mix, e.g., Pachube, AMEE, and Path Intelligence…(Markus Tripp left ISMAR to speak on a Web 2.0 Summit panel, “Humans as Sensors,” which also included Path Intelligence, Deborah Estrin on “participatory sensing,” and the brilliant work of Di-Ann Eisnor, Platial, on “Transactional Cartography”).  Anyway a big Green tech AR competition could get people working together across the broad spread of AR terrain on some of the sticky problems of user experience.  And, with a high level of support from Smart Phone companies, HMDs manufacturers and the chip makers we just might come up with some extraordinary magic.

The devil of course will be in the details.  But a competition like this could not only motivate key players to come together in the spirit of coopetition but also be an opportunity to show the world the power of AR to make visible the invisible ecosystems that are so important to the health of our planet.

One of the notable presences at ISMAR 2009 was the Qualcomm team.   Jay Wright’s presentation (an exclusive for ISMAR) not only outlined AR for 2012, but Jay also talked about some “close to the metal” innovation that we will see from Qualcomm very, very soon!  I had some time in the press room with Jay and his team prompted by MoMo’s Yuri van Geest.  When I twittered about Qualcomm’s presentation at ISMAR, Yuri replied:

vangeest “@tishshute: good stuff, hopefully you will integrate the neat new solutions and ideas in your talk in November ;)”

I will be presenting at MoMo #13 on AR, open AR, future of AR and GeoWeb,  and hopefully will bring some good news from Qualcomm too.  Anyway Jay seemed to like the idea of a Green Tech AR Competition, even though I did stress that I thought it needed some serious sponsorship and BIG prizes.

Where’s the beef? Tracking and Mapping at ISMAR 2009

On the flight from NYC to Orlando and ISMAR’o9 I dozed (I had been up late preparing my presentation) and I watched the Dew Tour Pro Skateboard competition and Top Chef on the Food Channel.  In this particular episode of Top Chef, the aspiring chefs were all given a brown bag of ingredients by an already famous chef who then judged whether the contenders managed to make a delicious meal with their allotment which was notably lacking in key ingredients of haute cusine.

This metaphor of  trying to cook up a great meal while perhaps missing the staples is apt for the current early stage of commercial augmented reality.  And when I arrived in Orlando, not only were the Dew Tour pro skateboarders staying at the same hotel as ISMAR, but ISMAR itself felt remarkably like an Augmented Reality Top Chef Coopetition.

Much of ISMAR was dedicated to the task of  providing the meat and potatoes of Augmented Reality, solutions to mobile tracking, mapping and registration, particularly in the Science and Technology track.

Industrial and Military Augmented reality solutions I found out, typically, solve the tracking problems by using fixed mounts which clearly wouldn’t translate well into the AR everywhere with everything mobile consumer culture expects.

In the picture on the left Fabian Doil stands by the VW engine that provided some of the outdoor targets for the ISMAR tracking competition.  On the right is a picture from the VW’s presentation on their research and development of AR.

I followed the tracking contest, organized by Daniel Pustka and Fabian Doil of Volkswagen, quite closely. And I learned a lot in the process. While  it is clear there has been progress in AR mapping and tracking, we still have a ways to go.

But hanging around the Tracking Competition was a good way to find out the state of play of this crucial part of the AR dream.  For example,  a little tidbit I learned is that Michael Goesele who has been reconstructing “high-quality geometry models from images collected from the internet (so called community photo collections, CPC)” is soon to be at the Institut Graphische Datenverarbeitung where top contenders in the tracking contest – Harald Wuest  and Folker Weintipper (in the foreground of the photo at the left and right respectively) are also to be found. [update Harold and Folker were the winning team see here for details of scoring and results!] Otto Korkalo and Tuomas Kantonen of VTT, Finland, Augmented Reality team are in the background. They have been working on the joint IBM, Nokia and VTT project that brings, Avatars and People Together for Virtual Meetings in Physical Spaces.

The picture on the right is another team that were doing very well. If my notes serve me well (and please forgive me if they don’t. I came back with my card wallet overflowing!) the photo on the right showsChristian Waechter (on the left) and Peter Keitler (on the right) of the Technische Universitat Munchen.

Germany is certainly leading the way in industrial AR. And I learned how small businesses like Metaio get to work with top research institutions and big companies like VW, thanks to very strong German funding program for AR and VR. The current iteration of a series of funding programs is  called Avilus.  Avilus  is putting 42 million Euros into AR and VR this year alone (click on the slide below to see more about Avilus ).

I wish we had the equivalent of Avilus here in the US.  But there is no equivalent to Arvilus for AR here, and  no AR is  being developed by the US car industry either it seems.  But look at the slide above to get a taste of some of the cool stuff Metaio and other small AR and VR businesses do for VW through the Avilus project.

I also got to meet many people from one of the world’s most important AR hubs -  The Department of Informatics, Technische Universitat Munchen, including Prof. Gudren Klinker on the far right in pic above.  And from left to right, Fabian Doil (VW, co-organizer of contest), Sebastian Lieberknecht , Selim Ben Himane (Metaio), Tobias Eble (Metaio).  Prof. Klinker is the engine behind much of German innovation in AR.

Metaio was one of the few teams to rely mainly on markerless tracking which in this contest was very challenging because of the very different light conditions (see pics below) between the windowless interior and dazzling Florida sunshine outside (pic on the right shows targets under ideal lighting conditions).  Many people in the US may be  familiar with Metaio’s consumer applications, like Junaio,  but thanks to Germany’s efforts to nurture augmented and virtual reality they are also respected software developers in industrial AR.  And I suspect that Metaio will spearhead markeless tracking in consumer AR too.

This post as usual has already expanded to something much longer than I originally attended – pretty typical for me! There is much I have not been able to cover including some of the interesting contributions by augmented reality artists at ISMAR – again I recommend the upcoming videos.

But I cannot end without a hat tip to, Oriel, Nate et al. who won the best student paper award for AR Sketch – again please see Games Alfresco for more on this (pic below from Games Alfresco). AR Sketch, Ori notes, is featured “in our top post and popular video.” And

“Their work is revolutionizing the AR world by avoiding the need to print markers – or any images whatsoever.”

Picture on the left Mirrorshades, picture on the right a Metroid Hud.

“Augmented Reality is like a Philip K Dick novel torn off its paperback rack and blasted out of iPhones,” Bruce Sterling in Beyond the Beyond

“a techno visionary dream come true – those are rare, really rare, you have to be patient,  it’s super cyberpunk”… Bruce Sterling, “At the Dawn of the Augmented Reality Industry.”

The Dawn of the Augmented Reality Industry continues to brighten, and now we have two augmented reality companies, Total Immersion and Ogmento, firmly established in Hollywood – the dream mother of so many of our augmented realities.

Ogmento is the most recent of these two pioneering augmented reality companies to set up shop in LA.  Total Immersion’s CEO Bruno Uzzan moved to LA from France two years ago, although he still has a fifty person RandD team in France.   Total Immersion began 10 years ago in the quiet, lonely, hours before the dawn of an AR industry.  But Total Immersion’s AR toys for Mattel, and augmented reality for Topps baseball cards, fired CNet writer Daniel Terdiman up enough to say, “I have seen the future of toys, and it is augmented reality” (see full post here on CNet).

Recently, I talked with  Ori Inbar, one of the founders of Ogmento and  the premier augmented reality blog Games Alfresco about his new venture in Hollywood. Bruce Sterling, @bruces, had some fun with my invention of brand new augmented reality trade jargon here!  Ori pointed out Ogmento brings two important new facets to the rapidly growing augmented reality field: firstly they are bringing leadership from veterans of the entertainment industry into augmented reality development. Brad Foxhoven and Brian Selzer from Blockade have partnered with Ori on Ogmento.  And, in an another important step forward for a young industry, Ogmento announced they will be acting as publishers for a fast growing cohort of augmented reality application developers and helping AR development teams out there bring their concepts to the market.

So I was very happy also to have the opportunity to talk with Brian Selzer.  Bruce Sterling pointed out in his seminal sermon from the flatlands at the Layar Developer Conference, AR is kind of a “Hollywood scene.” We have seen the web early adopter/developer/blogger community  embrace augmented reality browser experiences in recent weeks in an awesome wave of enthusiasm. Are Hollywood creatives equally smitten? For the answers see the full interview with Brian Selzer below.

Brian Selzer (www.brianselzer.com and twitter – brianse7en ) has an extensive involvement with emerging platforms:

“from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC [user generated content] projects (mashade.com, instafilms.com).”

I began my own career in Hollywood doing motion control photography and creating software that bridged the language of robotics and servo motors with the visions of  film directors. Eventually our little company, NPlus1, moved on to 3D vision systems and image recognition stuff.  So yes, I have been really, really patient waiting for this particular techno visionary dream.  And, while I have been waiting for augmented reality to manifest, I have grown to love the internet.  But now, how awesome, It is OMG finally for mobile AR!

Augmented reality is busting out all over – through our laptops, our phones, on the streets, toys, baseball cards, art installations, sticky light calligraphy and more.

Many of my questions to Brian were directed at at how and when we will see augmented realities with near field object recognition, image recognition and tracking and, of course, the illusive eyewear.  As Bruce Sterling points out we are just at the very, very beginning – the dawn of an industry.  I created the photomontage below on the right to compliment Tonchidot’s illustration suggesting the evolutionary inevitability of holding our phones up (below on the left).  The Evolutionary Reality of AR will not end there.  It is just a step into eyewear, hummingbirds or Nano Air Vehicles, and more…….

The Evolutionary Reality of AR

Cartoon on the left by Tonchidot on the right a collage of a stock photo and the Pentagon’s Robot Humming Birds – “Nano Air Vehicles.”

While we finally we have, an affordable mediating device with the horse power, mindshare and business model to bring AR mainstream with the iphone.  The much anticipated Apple 3.1 Beta SDK to be released in September will not, I am sure, open up the Video API at the levels that augmented realities with near field object recognition and tracking require (I would love to be proved wrong though). But the magic wand to deliver even tightly registered AR graphics/media (that require a lot of CPU and GPU) to a wide audience is in our hands, so full access to may not be far off. And others, of course, can/will/might knock the iphone off its current pedestal.  AR made it’s mobile phone debut on the Android after all.

Like everyone else who loves AR, I wish that Apple would open up faster (and I wish Android would manifest on some rocking hardware). But we will see enough of the iphone Video API open for the next generation of mobile augmented reality games and applications to emerge in the coming months.

One of these will be Ogmento’s. Although Ogmento is in stealth mode, they have released a teaser for their first game, “Put A Spell,” developed by ARBalloon – screenshot below.  Ori did reveal to me in this interview that they are doing image recognition and using the Imagination AR engine.

As Brian notes, Hollywood has had the AR bug for a long time. AR has been everywhere in Science Fiction Movies and video games. Nintendo’s SPD3 head Kensuke Tanabe, “effectively the man in charge of overseeing all the Metroid franchise underneath original co-creator Yoshio Sakamoto,” explains the story of Metroid to Brandon Boyer of Offworld here (an image of a Metroid Hud on the right opening this post) :

“the idea of the different visors you use in the Prime games to interact with the world: the scan visor, for instance, set the game apart from other first person shooters in that the player was using it to proactively collect information from the world, rather than having the story come to them passively, in the form of cut-scenes or narration. “Prime could have adventure elements with the introduction of this visor,” says Tanabe, “That’s how we came up with the genre — first person adventure, instead of shooter.”

But as Brian points out:

“the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear).”

Talking with Brian Selzer

Tish Shute: Bruce Sterling’s sermon at the Layar Developer conference, “At the Dawn of the Augmented Reality Industry,” was absolutely awesome. He spread the future feast/orgy of augmented reality before us  – and described many of the dishes we will tasting both delectable and diabolical.  One of the many things he points out is, AR is kind of a “Hollywood scene.” And, as Ogmento is one of only two augmented reality companies in Hollywood, I am interested to hear how it looks from your neck of the woods. We have seen the web early adopter/developer/blogger community  embrace augmented reality browser in recent weeks in an awesome wave of enthusiam – are Hollywood creatives catching the buzz?

Brian Selzer:  It was a thrill to hear Bruce Sterling mention Ogmento. I devoured all of his Cyberpunk books back in the 80′s, along with writers like Gibson, Rucker, Shirley… To me, sci-fi writers are the visionaries who define and influence our technological paths into the future. They make science and tech sexy enough to want to manifest those experiences in the real world. Clearly Bruce sees the AR industry as being sexy. I love that he called it “a techno-visionary dream come true… and super-cyberpunk.”  And yes, kind of a Hollywood scene.

Hollywood creatives caught the AR bug before they knew what AR was.  Look at science fiction movies and video games to see AR everywhere. Terminator, The Matrix, Minority Report, Iron Man.. the list goes on.  Look at any video game with an integrated heads-up display.  It’s clear Hollywood loves AR.  It’s only been in the past few months though that the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear). So yes, the buzz is here and it’s strong.  With that, we all have to be prepared for the good, the bad and the ugly as AR goes mainstream.

It certainly goes to show how young this industry is when Ogmento and Total Immersion are currently the only AR companies based in Los Angeles. It’s very exciting to be the only company right now demonstrating a natural feature tracking (markerless) iPhone experience in Hollywood. We are in talks to bring some very big brand and properties to the mobile AR space. The goal is to deliver experiences that create added engagement and value to the consumer.

Tish Shute: Also in his landmark sermon Bruce Sterling noted that augmented reality has been around for 17 yrs and now at last we are seeing the dawning of  an augmented reality industry. What inspired you to take up the challenge of launching an augmented reality company in Hollywood?  Oh congrats that Bruce Sterling name checked Ogmento in his list of companies that prove that this really is the dawn of an industry!

Brian Selzer: I’ve always been involved in emerging platforms… from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC projects (mashade.com, instafilms.com). Working with all these great CG game assets, I continued to think about what’s next, and that’s when I started to follow AR very closely and started engaging with those who were pioneering in the space.

I remember swapping instant messages with Robert Rice (@robertrice) right after the 2008 Super Bowl.  We were not chatting about the football game, but rather about some of the commercials that aired during the event as a sign that AR was making its way into the mainstream.  A lot of people became aware of AR for the first time when the GE SmartGrid commercial aired.  There were all these YouTube videos popping up of people blowing on holographic wind turbines.

The commercial that really got me excited though was the Coke Avatar commercial.  In that commercial people in the city were sporadically being portrayed as their digital persona’s, avatars, gaming characters, etc..  For me that spot did a great job showing how many of us already have these ‘alter egos” that live in cyberspace, and how the line between these worlds can sometimes be blurred. I remember watching that commercial and thinking that is exactly the type of experience I’d like to create with mobile AR.  I want to overlap the virtual world into our every-day reality. Why cant I bring my World of Warcraft or Second Life persona with me into the real world?

I am big on the notion of “Games and Goals.” I believe that games have the power to motivate people in a very powerful way. By challenging ourselves while playing a game we can climb mountains.  Augmented Reality is the perfect platform to bring gaming into the real world.  By mixing the virtual world with the physical world, this added layer of perception provides a very powerful experience for something like a role-playing game.

One of my earlier social-gaming projects was a website called Superdudes.  This was a “Be Your Own Superhero” concept that celebrated and motivated kids to create superhero avatar/persona’s online, and we gave members all sorts of games, challenges, and rewards, some of which carried into the real world. The site recognized members for teamwork, creativity, volunteer work and things like that. So the Superdudes were often involved in charity events and benefits to help children.  Everybody called each other by their Superhero names, and the line between fantasy and reality were being blurred.  This project really got me thinking about what happens when you take positive role-playing like this and mix it into the real world.  I started to work on a plan for location-based activist missions for points and rewards, but never got to complete that. So I have some unfinished business here.

I think it would be fantastic to be able to show up to some type of fun event with friends, and everybody could see each others alter ego personas standing before them. When you can turn the world into a playground, and use the power of gaming to make a positive impact on the planet… well, I don’t think there is anything better than that.  These are the types of projects that drive me, and I think AR is the best platform to support these types of social gaming experiences.

Tish: Does Ogmento have any RPGs under development?  I noticed in the Google Wave on RPG someone has been working on doing something with the Dungeons&Dragons API.  I am interested in exploring the web of protocols underlying Wave as a transport mechanism for multi-person, mobile, AR experiences (not requiring downloads), on an open global outdoor AR network. If not Wave, what do you see as the potential infrastrucure and protocols we could harness for an open augmented reality network?

Brian:  Ogmento has a deep background in video games and we interact regularly with most of the major game publishers. As a company we are not so much developing our own RPGs right now, but rather exploring what mobile AR extensions make sense for existing brands.  There are many limitations to location-based gaming, but a global AR network is exactly along the lines we are thinking.  Lots of discussions are taking place on protocols, platforms, API’s, and there are numerous ways to approach this.  We need to be able to use what’s available now and continue to refine and customize for AR’s specific needs and issues as we progress.

In general though, Ogmento is focused on what types of experiences can be had today and over the next couple of years. I still think we are several years out from a truly open augmented reality network.  We are certainly looking at launching our own “Ogmented Network” which would support some fun treasure hunt type experiences, or add an entertainment layer on top of traditional outdoor marketing campaigns.

Tish: I don’t know whether you have read Thomas Wrobel’s ideas for an open augmented reality network that I just published here on Ugotrade.  The principals he talks about are very important for augmented reality to become a major part of our lives – .  Considering the difficulty open networks can pose for emerging business models how can we fund the development of an open framework for augmented reality?

“a future AR Network, I mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.
Where people can just point their phone, webcam, or pair of AR glasses anywhere were a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.”

Brian: I think funding for these types of projects will definitely come from Venture Capital groups in the near future.  It’s early in AR, but the VC’s are watching and deciding which horses to bet on.  Until that time, it’s about service work, and developing AR experiences for others with what is possible today. That work will help fund internal development of original AR products, and platform development.

Tish: How did you get started with Ogmento?

Brian: My first conversation with Ori was actually about my interest in Location Based RPG concepts.   We had a long conversation about the possibilities with AR, and it was clear that we shared similar interests, but were coming from different complimentary backgrounds. The idea of collaboration was exciting, so we just kept talking until the timing felt right. Now, with Ogmento we bring a unique blend of AR development experience with a deep backgrounds in AR technology, animation, video games, entertainment, social media, etc.   I think this is a powerful mix that will allow us to do some great things.

It’s still so early, and things are just getting started in AR. There are only so many webcam magic tricks you can enjoy before you are ready for something else.  The location-based apps have the most potential in my opinion, which is why we are really focused on mobile AR.   We have some board-game type projects, which do not instantly scream location-based gaming, but if you look at something like the ARhrrr board game, you can see how much more compelling it can be when the game invites the player to be actively moving around during the experience.

Tish: I am interested in your perspective on how we can create the kind AR experiences that really embody what has always been so exciting about AR – the tight alignment of graphics and media with real world objects and ultimately a rich immersive 3D experience, so I am going to hit you with a bunch of those, “Is this really eyewear or vaporware?” questions.  The real deal eyewear changes everything!

While eyeware is a big challenge technically and aesthetically,  I am pretty sure that there are several outfits out there that can pull off the optics and projection. 
Will the entertainment industry get excited enough to put a major push into delivering the eyewear in short order instead of the 5 to 10 year project that some people still think it is?    The business development challenge is bigger perhaps than the technical obstacles perhaps? What is your view on this?

And, perhaps, the eyewear is a clear example of a need for partnerships. For example, we have seen efforts from companies like Vuzix and Lumus, and recently a Japanese Company, Masunaga.

I have no reports from people who have tried the Maunaga eyewear yet.  But,  limited by small field of view, and tethered, currently eyewear offerings, available at a reasonable price point, are not workable solutions for augmented reality experiences. But the problems are not insurmountable. What will facilitate the real deal?  


It seems that it is critical to start creating hardware relationships now. The industry is costly and slow moving and as Robert Rice put it to me in a recent conversation, “once the software cat is out of the bag, its going to go wild and if the hardware isnt there, its going to stutter.”

As Ori notes some of the hardware companies like Intel and others don’t seem to be paying enough attention to AR.  Ori points out they don’t see the demand yet.  But in order to create an awesome AR experience and demand from a mass audience, don’t we need to work in conjunction with hardware designers?

Brian: It’s fun to think about who will eventually deliver a great hardware solution for AR glasses. It will happen. It would be cool to see somebody like an Oakley or Nike partnered up with a company like Vuzix to deliver something people actually might wear in public.  Perhaps a hardware manufacturer like Apple or Nokia will bring us something like the iSight or the NGaze down the line.  I’d love to see a set of glasses designed by Ideo.  Microsoft or Sony are already playing with technologies like Project Natale and the EyeToy, so I think its only a matter of time before they deliver an eyewear solution. I would even look to the toy companies to eventually make an investment here.

Gamers will be the early adopters, and in a few years we may start to see people running around in the park wearing glasses with headsets, but it will be acceptable because it’s clear they are using them for a game.  It’s going to take a very sexy and stylish piece of hardware for everyday people to be willing to wear AR glasses in public while going about their everyday business.   It’s like the recent cover of Wired magazine where Brad Pitt is wearing a mobile headset in his ear, and the editors point out that even he can’t pull that look off, so why do you think you can.  When AR glasses come in designer frames, and you can’t tell them from non-AR glasses, to me that’s when things get really interesting from a mass-adoption perspective.   Compare how many people were carrying around a mobile phone in the 80s to now.  I think it will be the same thing with glasses.

I was in an AR pitch meeting the other week at a very significant media company, and brought up the point that today’s handheld Smartphones will eventually evolve into tomorrows Smartglasses. My comment was quickly shrugged off as sort of a sci-fi notion that was irrelevant to the business at hand.  Probably true, but I think it is important to understand where digital media and entertainment is going, so you can adapt quickly, and evolve into those spaces more naturally.  The more we see people walking around with their Smartphones in front of their face (like a camera), the sooner it will be that we make the jump to eyeglasses as a key hardware device for AR experiences.

At Ogmento, we definitely are working on AR experiences with the hardware and software available today.  We will get some product out this year, and 2010 will be a banner year for markerless mobile AR in general.  I think the entire AR community is looking forward to bringing this technology to the mainstream in the form of games, marketing campaigns, virtual docent apps, and much more.  It might not be the full experience we are all dreaming about for some time, but we can see the path and the true potential, and it’s pretty spectacular.

You mention the tight alignment of graphics and media with real world objects.  That is really our focus. A lot of well-deserved attention is going to the browser overlay “post-it” approach right now, which uses compass and GPS.  We are focused on markerless natural feature tracking, so once you identify something that is AR enhanced in your environment, you can interact with that integrated experience.  On an iPhone that can be as simple as using your touch screen to interact.  When you are wearing glasses, it becomes more about visual tracking. There are lots of smart people thinking through these issues. Many of which you have interviewed. It is my hope that there are exciting collaborative efforts to be had in the coming months to get us all there together and faster.

Tish: Bruce touched on some of the hard problems that have to be solved for augmented reality – and he noted for instance security needs to be tackled in the early stages. Robert made a nice list, “privacy, media persistence, spam, creating UI conventions, security, tagging and annotation standards, contextual search, intelligent agents, seamless integration and access of external sensors or data sources, telecom fragmentation, privilege and trust systems, and a variety of others.” Will Ogmento be leading the way in solving some of these hard problems?

And, won’t trying to solve these hard problems for networked AR in walled garden scenarios one company at a time lead to a lot of reinventing the wheel wasted energy?

Brian: These are all important issues, and again there are a lot of smart people thinking about solutions to these problems on a daily basis.  Ogmento is interested in partnering with developers and supporting their efforts as a publisher of mobile AR experiences.  While we intend to roll up our sleeves in these areas, we are currently more focused on taking AR mainstream with the hardware and software available today.  As the industry evolves, so will Ogmento. As the opportunities evolve, our ability to make a greater impact tackling these issues will be realized.

Tish: Another area of development that could really kick AR into high gear might be creating augmented reality hotspots  where we use can deliver the kind of location accuracy/instrumentation necessary to create interesting AR experiences (partnership with Starbucks, perhaps ?!).  Augmented reality hots spots, could deliver the kind of high quality AR experience that isn’t possible ubiquitously at the moment, and may be a real way to get people really exploring the potential of AR now, rather than later?

Brian:  Agreed. I see a great opportunity here with this approach.

Tish: Although there are many obstacles to Green AR – the energy hogging servers at the backend for starters! Last week I had a conversation with Gavin Starks, AMEE, and Robert Rice and Jim Purbrick about how to work with AMEE and the technology available and encourage Green Tech AR development (see an early exploration of green tech AR from Pachube here).

We came up with the idea of holding a competition perhaps centered around a targeted instrumented space. But I would really love to hear your thoughts on the topic of Green Tech AR (the energy hogging servers at the back end being the first cloud on the horizon!.)  Cool GreenTech AR imaginings, social gaming ideas, RPGs, not even necessarily even tied to the immediately practical, would be like rain in a drought!

Brian:  I go back to “Games and Goals”… If you make environmental and other activist efforts fun and rewarding, more are likely to be motivated and participate.  Can you imagine having a personal “carbon footprint stat” floating over your self at all times? Or over your home or factory?  How would that change your behavior?  We all love stats. Look at how the Nike+ campaign has used technology and gaming to motivate people to run.  I think there is a lot that can be done to make being green fun. It starts with the individual, and spreads from there.  Keep me posted on that one!

Tish: I would also like to explore further the RRW suggestion that ambient intelligence is both the Holy Grail of AR and possibly snake oil:

“The holy grail of the mobile AR industry is to find a way to deliver the right information to a user before the user needs it, and without the user having to search for it. This holy grail is likely in a ditch somewhere beside a well-traveled road in the district of the semantic Web, ambient intelligence and the Internet of things. Be wary of any hyped-up invitation to invest in a company that claims to have gotten the opportunity right. What we’ve seen in the commercial industry to date is a rather complex version of a keyboard, mouse, and monitor.”

So Holy Grail, Snake Oil, or a ditch somewhere….?

Brian:  I instantly think of Minority Report, where Tom Cruise’s character is being bombarded with holographic ads personalized with his name and to his current situation.  In the future, Spam is a nightmare, especially when it knows who you are.  I think the key thing here is delivering “the right information”, and we still dont have that down. I do see a day where we can truly customize what comes to us, how we want it, when we want it.  My future vision of ambient intelligence is the ability to “turn everything off” if I want to… block out the stimuli and replace it with images of nature, or natural surroundings, etc.  Where I live in Los Angeles, we have those digital billboards everywhere, so it’s like advertising overload wherever you look (hints of Blade Runner).  I personally don’t mind them, but I know there is great debate on there being simply too many billboards everywhere. So AR would only add to the noise of life by adding yet another digital overlay of information, right?

Perhaps the holy grail is to use technology to filter things out. AR might become a solution to leading a simpler life, or a perfectly customized life if you want that. Ultimately the control needs to be with the individual.  I guess I am talking about something like TiVo taken to the extreme.

Tish: And then that other biggy – augmented reality search! I am asking this next question of  Wikitude and Sekai camera too and now I must also ask Acrossair and several others I guess! Obviously a huge area of opportunity in this broader landscape that uses location-awareness, barcode scanners, image recognition and augmented reality is to harness the collective intelligence – a whole new field of search. There is the beginning of a discussion on this in the comments here.

What will it take, in your view, to become a leader in augmented reality search?

Brian:  I’m more of a content guy, so I tend to focus on things like UI, quality of creative, etc..  From that perspective, I am looking forward to evolving beyond the “post-it” text overlay user-experience we see now in AR search. I was impressed with the TAT Augmented ID concept and hope we start seeing more smart design solutions like that emerging in the space.  There are some great new design approaches coming out of the location-aware space that should be applied to AR search. I’ve been studying the heads-up display designs being used in video games, and re-watching movies like Iron Man for ideas. This is another example where Hollywood has painted a polished picture of what AR can and should look like, and the masses have already accepted these design approaches.  So from that perspective, from my view the leaders in search will be delivering sexy, smart and simple solutions. It’s all about the S’s.