Augmented World Expo 2013 was really an amazing experience. I’m co-founder and co-organizer of the conference, along with Ori Inbar, so it has meant a lot to me to see our event grow over the last four years, and thrilling to make such a big splash this year.  There were 1,163 attendees, and the expo show cased an ecosystem of emerging technologies – augmented reality, gesture interaction, eyewear, wearables, and connected hardware of  many stripes, that mark the beginning of natural computing entering the mainstream. It was a unique opportunity to get up close and personal with what it feels like to be an augmented human in an augmented world!

Videos of AWE 2013′s 35 hours of educational sessions and inspirational keynotes are now available on our YouTube channel. I am sharing my own talk (my slides are also up on slideshare here), and a few of my favorites in this post, but there are far to many to post here, so please browse further on the Augmented World Expo youtube channel.

One notable high point of AWE2013, for me, was the showcase sponsored by Meta, a startup developing the first device allowing visualization and interaction with 3D virtual objects in the real world using your hands. It was made possible by the generous contribution from the private collections of Paul Travers, Dan Cui, Steven Feiner, Steve Mann, and Chris Grayson, and passionate volunteers who are helping advance the industry. Sean Hollister of The Verge did this excellent report on the eyewear showcase 35 years of wearable computing history at Augmented World Expo 2013
Also for more on Meta see this article by Dan Farber.

My colleagues at Syntertainment, Will Wright, Avi Bar-Zeev, Jason Shankel, and LaurenElliott all gave great talks. Ironically, we’re not building augmented reality apps or H/W. We all just happen to continue to be very interested in the field.  

Thank you to everyone for supporting the event!

The press coverage was truly extensive:

In the shadow of Google Glass, an augmented reality industry revs its engines
The Verge, Sean Hollister, June 9, 2013, 271 Tweets

The next big thing in tech: Augmented reality
CNET, Dan Farber, June 7, 2013
Pick up on Current News Daily
350 Tweets

AWE 2013 Conference Report: Augmented Reality and Marketing
The Persuaders Marketing Podcast on Dublin City FM, June 23, 2013

AR Dirt Podcast – Ori Inbar AWE2013 Extravaganza Recap
AR Dirt by Joseph Rampolla, June 18, 2013

35 years of wearable computing history at Augmented World Expo 2013
The Verge, Sean Hollister, June 9, 2013
7 Tweets

Augmented Reality: Bruce Sterling, keynote at Augmented World Expo 2013
Wired, Bruce Sterling, June 9, 2013
9 Tweets

On the road for VR: Augmented World Expo 2013
Doc-Ok, Staff, June 7, 2013
3 Tweets

My Interview from Augmented World Expo 2013 [VIDEO] Wassom.com, Brian Wassom, June 7, 2013

Augmented World Expo
ZenFri, Staff, June 7, 2013

AWE2013: Hardware for an augmented world
FBNSantos.com, Felipe Neves Dos Santos, June 6, 2013

Augmented Reality Will Be the New Reality
InvestorPlace, Brad Moon, June 6, 2013

Wearable computing pioneer Steve Mann: Who watches the watchmen?
TechHive, Armando Rodriguez, June 6, 2013

Expo puts augmented reality in the limelight
ABC 7 News, Jonathan Bloom, June 5, 2013

These OLED microdisplays are the future of augmented reality
DVICE, Evan Ackerman, June 5, 2013

Visualized: a history of augmented and virtual reality eyewear
Engadget, Michael Gorman, June 5, 2013

Wikitude announces Wikitude Studio and in-house developed IR & Tracking engine
PapiTV, KC Leung, June 5, 2013

Augmented reality expo aims for sci-fi future today
USA Today, Marco della Cava, June 5, 2013

Augmented Reality: High Dynamic Range (HDR) Video Image Processing For Digital Glass
Wired, Bruce Sterling, June 5, 2013

Will Wright at Augmented Reality Conference: Don’t Augment Reality, Decimate It
AllThingsD, Eric Johnson, June 4, 2013

Philip Rosedale’s Second Life with High Fidelity
CNET, Dan Farber, June 4, 2013

Google Glass competitors vie for attention as industry grows
PC World, Zack Miners for IDG News Service, June 4, 2013

4D Augmented Reality Leader Daqri Announces $15 Million Financing
Press Release, June 4, 2013

CrowdOptic Powers Lancome Virtual Gallery App, Crowd-powered Heat Map
TechZone 360, Peter Bernstein, June 3, 2013

Augmented humans, enhanced happiness?
Crave Culture, Angelica Weihs, June 2, 2013

Metaio & Vuzix to Showcase AR-Ready Smart Glasses at the 2013 Augmented World Expo
Press Release, May 30, 2013

Four ways augmented reality will invade your life in 2013
Quartz, Rachel Feltman, May 30, 2013

Augmented Reality: Augmented World Expoâ„¢ is next week
Wired, Bruce Sterling, May 28, 2013

Strike it Rich with Cachetown and AWE 2013 Playing the Gold Rush 49’er Challenge In Augmented Reality
Press Release, May 24, 2013

Local Community College Student Headed to Silicon Valley to Learn More about Augmented Reality
St. Louis Post-Dispatch, Staff, May 24, 2013

Explore an intricate labyrinth with smartphone AR
CNET Australia, Michelle Starr, May 21, 2013

Dartmouth firm lands super app
Herald Business, Remo Zaccagna, May 21, 2013

Augmented World Expo 2013–The Future of Augmented Reality
Silicon Angle, Saroj Kar, May 17, 2013

The prophets of Augmented Reality, Bruce Sterling (best know for his science fiction works, his non-fiction includes “The Hacker Crackdown,” “Tomorrow Now,” and “Shaping Things”), and  Vernor Vinge (author of “Rainbows End” and “The Coming Technological Singularity”) are joining  Will Wright (Legendary game designer of SimCity, The Sims, and Spore), and Jaron Lanier (a computer scientist, composer, visual artist, legend of virtual reality research, and the author of You Are Not A Gadget: A Manifesto) to judge the Auggies at the Second Annual Augmented Reality Event, 2011, Santa Clara, Ca., May 17th, May 18th!  Pictures top row, Bruce Sterling, Vernor Vinge, Will Wright, 2nd row, Jaron Lanier, Blaise Agüera y Arcas, and Frank Cooper III.

Augmented Reality has been prototyped in science fiction writing and films for many years now.  But Augmented Reality Event, the first global event dedicated to the emerging industry of AR,  is your chance to be part of a momentous transformation of science fiction into science fact.   Bruce gave a seminal keynote last year at ARE2010, “At the 9am of the Augmented Reality Industry…” But we are edging closer to high noon this year – see John Markoff’s recent New York Times post, “Augmented Reality Comes Closer to Reality,” and Bruce Sterling’s opening keynote at ARE2011 will raise the bar for us all again.

Bruce Sterling  and Vernor Vinge will wrap up the show with fireside chat.  Jaron Lanier will be keynoting at are2011 for the first time and Blaise Agüera y Arcas (a leader in Microsoft’s mobile strategy), will return to top his Ted talk demonstrating innovations in Bing Maps and Augmented Reality.  Frank Cooper III, who serves as Senior Vice President and Chief Consumer Engagement Officer of PepsiCo, will give, an up to now, missing perspective of  the big brands’ on Augmented Reality (AR).

“The main stage at ARE 2011 will be blessed with science fiction visionaries, ground breaking scientists, mind blowing technologists, and legendary game designers and more!”

Come prepared to Augmented Reality Event.  Read as much Bruce Sterling and Vernor Vinge as you can, and follow @bruces blog on Wired, Beyond the Beyond.  With only a few weeks to go until Augmented Reality Event, myself and co-chairs Ori Inbar, and Chris Grayson, have been working hard on a preliminary schedule ( by the time you click on this link it should be updated again).  You can follow us on twitter for updates as they come in – @tishshute, @comogard, @chrisgrayson,  and please follow @arealityevent and join our ARE facebook posse.  And, of course, don’t forget to register soon while we are still offering discount codes.

Readers of this post can use the code TISH295 for $100 of the already sweet price if $395.

If you joined us for Augmented Reality Event last year, you will know that the”Auggies” – an award for the best AR demo presented live with commentary American idol style,  is a chance to join the best and brightest in AR as they write the next chapter for AR in the sharp and often amusing reparte between judges and contestants.  Submit your demo proposal here under the “Auggies” track.

The picture below is the Auggie Award for the best AR demo designed by Sigal Arad Inbar.  (See, Ivan Franco recounts the team’s ARE 2010 experience, and winning the event’s first-ever Auggie Award, and the video shot at the YDreams booth by Bruce Sterling. “The Hotness” – YDreams rocking it at ARE2010 from brucesflickr)

ARE2011 will include more than 90(!) speakers in 30 sessions organized into 3 tracks: business, technology & programming, and production & design.

It will feature special activities such as a Startup launch pad (submit your proposal here under track: “Startup Launch Pad”), and an ARt Gala with live performances.  There will be displays by top AR artists of AR art projects and live performance art by the world’s best AR artists featuring: Helen Papagiannis, Amir Baradaran, Sander Veenhof, Ina Centaur and more to be announced.

The exhibition area will be open throughout the 2 days of the event and will include the latest product demos by the leading AR companies, as well as a career fair to help grow the fledging industry.  A small number of booths is still available in the exhibition hall ($995 for 10′x10′).Grab them while they last!

A press conference will kick start the event, as a vehicle for AR companies to launch new products and services. Submit your proposal here and add: “Press Conference” in the title. The event wrap up will feature Marco Tempest with a live AR Magic show

I have been so busy working on are2011 (and another project yet to be named -  hint I am focused on social augmentation and the world as a platform for a geo-situated social interest graph), that I have had no time to blog since I was on The Potential for Augmented Reality in Education Panel at SXSW, 2011. But I have posted my slide deck here for my talk, “Enchanted Objects and People: Data Driven AR.”

We are entering a new era of collective engagement possibilities for augmented reality, consumer to consumer brokerages which could unleash the the visions of the collaborative consumption movement and tap into our collective energies in totally new ways.  As Seth Praebatsch, Scavngr, suggested in his SXSW keynote, The Game Layer on Top of the World, the power of communal game play may even help us address even the most intractable problems like failure in education and global warming!!

One of my favorite new AR ventures exploring the potential of new forms of social engagement through AR is Suwappu.  We are working on how to get them across the pond and a few time zones to are2011.

“Dentsu London are developing an original product called Suwappu. Suwappu are woodland creatures that swap pants, toys that come to life in augmented reality. BERG have been brought in as consultant inventors, and we’ve made this film. Have a look!”

Click here or on the image above to watch the video.

“Suwappu is a range of toys, animal characters that live in little digital worlds. The physical toys are canvasses upon which we can paint worlds, through a phone (or tablet) lens we can see into the narratives, games and media in which they live.”

Many new players in Augmented Reality will join the pioneering AR companies, Layar, Metaio, Occipital, Ogmento, Total Immersion, YDreams, and renowned innovators, Mark Billinghurst, Steven Feiner, Blair MacIntyre and the Georgia Tech team and many others will return to surprise and amaze us this year – see the still growing list of speakers here.   Qualcomm, who announced an SDK for vision based augmented reality, and $200,000 developer challenge last year, will also be back with several of their team at are2011!

Visual Search and OCR augmented reality apps have flourished in the last year and they will have a strong presence at are2011.  Long anticipated and oft fretted over face recognition apps are coming to a phone near you soon.  Viewdle, working locally on the phone, is taking a new approach to face recognition.  Google has so far said that they will not use or release the face recognition technology that they apparently already have.  But Shailesh Nalawadi, Google Goggles, will return to are2011 to discuss building out a computer vision search database and 3D Classifiers to solve AR challenges and show us what Google Goggles has been working on recently.  Aurasma, out next month, is perhaps the next big thing in this space, so we are all eager to try it out and see what they bring to ARE2011.

My interview with Anselm Hook, Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, is still a great reference point for important questions in this space.

Wordlens, wowed us all with OCR done locally on the phone, and they will be  showing off new features at ARE2011. Pulling out WordLens to translate at subway poster for the first time is a magic moment, and guaranteed to impress the person sitting next to you too!    Tactical Information Systems, who Fred Wilson said had the potential to be a Shazam for faces at Web 2.0 Expo, NYC, will show us what they have been up to since then. 

Data driven AR is still in the early stages, but it is data that will unlock augmented experiences of the world around us in many powerful ways.   Also, we are seeing exciting new efforts to put people at the center of their data.  Jeremie Miller who ushered in the Twitter era with his invention of XMPP has brought us a new real time protocol, TeleHash and the Locker Project, that will unleash the full potential of communication at the edge of the network.   And don’t miss Ben Cerveny and Bloom,  at ARE2011, with “pop-cultural instruments for data expression and exploration” and “where data visualization meets game design,”  I highly recommend Ben’s talk!

Also I am very excited that Mike Kuniavsky will be speaking at  ARE2011 for the first time.  He  pointed out in 2007 that Magic is a powerful core interaction metphor for Ubiquitous computing [and AR].  Offices. libraries, and the desktop provided us with common metaphors that unleashed the power of the PC through files, desktops and trash cans, browsing, etc.,  but metaphors of magic – “enchanted objects and people,” are keys to augmented reality experiences.   And, if Magic is a core interaction metaphor for AR, kinect hacks have shown us that gesture will be the soul of the AR experience.  Jaron Lanier’s keynote will be a must see and make sure you catch Chris Arkenberg’s presentation on the Future of Hands Free AR.

This post is just a glimpse at some of what will be going on at Augmented Reality Event.  I cannot do justice to everything in a single post, so I hope to see you there!  And, if by any chance you have a few moments to daydream between now and then, feel free to ponder your own script for a Green Lantern movie:  “In brightest day and darkest night, no evil will escape my sight.”  I will too.  As @brendanscully noted, “Green lantern’s power is Augmented Reality!”

Today Metaio is holding Inside AR in Munich, Germany.   Metaio (the picture above shows Metaio co-founders Thomas Alt and Peter Meier), is behind some of the best known commercial and industrial AR experiences of recent years.  But as important as the many AR projects they have executed are the AR tools that Metaio has made available to developers.  Metaio’s AR products and tools have played an important role in bringing AR to a wider public, and given many developers the opportunity to explore AR.

Inside AR is a great opportunity to see what these AR pioneers  will be up to in the coming months.   I could not make it to Munich this year.  But, fortunately, I had the opportunity to talk with Thomas Alt, recently.   In this conversation – see below, I got a chance to discuss what was going on inside AR with Metaio.

The Fall season is always jam packed with great events, and I wish I could be in two places at once.  But this week, I will be in my home town, NYC, attending Web 2.0 Expo which, reflecting the heat in the NYC tech community, is a sold out event with a very exciting schedule this year (more on some of the presentations that I will be attending later in this post).  If you missed out on tickets to Web 2.0 Expo, all Keynotes will be Streamed Live: TUES 9/28 to THURS 9/30, and keep your eye on @w2e and #w2e on twitter.

Meanwhile, I am missing Inside AR, which had some great speakers lined up, including fellow New Yorker, John Swords, partner and Ringleader at Circ.us.  Hopefully, Swords will share his experiences at next month’s ARNY Meetup which will be “joining forces with another vibrant community – NY Gaming – for an unforgettable night of Augmented Reality Games” on Tuesday, Oct 19th, 6:30 PM at AOL Ventures in New York, NY.

At the most recent ARNY @swords gave a brilliant talk on the possibilities for AR Game development on the newly available opensource Parrot ARDrone platform.  It was great to hear from social game guru @swords on his plans for Parrot ARDrone games, and more.  The picture below of an ARDrone camera view is from John Swords Flickr set.  Swords was flying it inside his garage because the winds outside were too strong.

Also, I kicked off what will hopefully be an ongoing discussion on, “Story Telling with AR and the Big ARNY a collaborative AR Game for NY,” with a few slides.  I have opened up the presentation document for collaboration, so please ping me if you would like to be added as a contributor/editor, and are interested in getting involved.

Ori Inbar, CEO and co-founder of Ogmento, Games Alfresco, ARNY and my co-chair on Augmented Reality Event 2010, suggested The Big ARNY – A Collaborative AR Game Development Project modelled after A Swarm of Angels last year at the First ARNY Meetup – so let’s make it happen!  I will be catching up with Ori in October about what Ogmento has been up to since they became the first VC backed AR Game company!

“Games allow us to see each other, for a moment, in a way that living in a city prevents” – Kevin Slavin

I believe that, AR, to get beyond the stage of “interface du jour” needs to offer us new ways to relate to each other and the world around us so that we can actually improve  and deepen our engagement with reality not just create experiences that are primarily optical  (see James Turner’s interview with Kevin Slavin “Reality has a gaming layer” on not letting “the pleasure of a game and the meaning of a game and the experience of a game rest primarily in the optics.“  And see my recent post, Urban Augmented Realities and Social Augmentations that Matter: Talking with Bruce Sterling, Part 2).

Two of the most inspired creators of urban games,  Kevin Slavin and Kati London of Area/Code will be speaking at Web 2.0. Expo tomorrow.   And you can be sure I will be at both these sessions. Loitering on the Motherboard, Kevin Slavin, is 2:35pm Tuesday, 09/28/2010, and Games that Know Where you Live, Kati London – is a keynote that will also be live streamed – 4:55pm Tuesday, 09/28/2010

Recently Kevin Slavin was interviewed by James Turner, on O’Reilly Radar.   This, Reality has a gaming layar, is a must read piece about a “world where games shape life and life shapes games”  (see @timoreilly). 

Interview with Thomas Alt

Tish Shute: Perhaps you could just start with your background Thomas because I think there’s a lot of newcomers to AR but you are really one of the first movers in commercial AR. How long you’ve been involved in this?

Thomas Alt: Actually I’m an ex-researcher in augmented reality. I started with me actually after getting my master’s work in engineering from the Technical University of Munich working for a big company called Volkswagen. And at that time,1999, we got a research grant for researching how augmented reality could change manufacturing processes in the automobile industry.

And from the research work there, I basically went back to school, did my PhD about augmented reality. And while speaking at a conference, I met Peter Meier who is the co-founder of the company who was also a master’s student writing his thesis about augmented reality. That was in 2002.

And so it really was in the very early days of augmented reality. And both Peter and myself we got really excited about the technology; we saw endless possibilities. We said, ‘OK. Let’s just found a company. We actually founded the company in early 2003 with virtually no money. As a matter of fact the founding capital of the company was 25,000 Euros and this 25,000 Euros were won in a case competition in Germany – a business plan competition.

Tish Shute: So you won 25,000 Euros on this case competition and that’s where Metaio started….

Thomas Alt: Exactly. And to legally found a company in Germany it takes exactly 25,000 Euros so that was the founding capital. We started pretty much like good old SAP started.  It wasn’t in a garage though it was a very small office and we basically built up the business through work, so we don’t have any investors or whatever. Right now we are 66 people located in Munich where our headquarters have been for five years. We have some presence in the US, and we have a venture company in Seoul, South Korea.

Tish Shute: Awesome. I just noticed how fast you’ve been growing. So right now, I’m going to ask a couple of questions about where you see the technology and the emerging industry going.

First, what are the platform of choice for Mobile Augmented Reality at the moment?

Thomas Alt: Obviously in the cellphone hardware space there’s a fierce competition going on. It’s yet to be defined what will be the prevailing platform right now, obviously it’s the iPhone is big now, right? But Android is catching on very, very fast.

Tish Shute: You have pioneered bringing a cross platform SDK for vision assisted AR to a wide community of developers with Junaio and with your partnership with Kooaba – a visual search company from Europe.

Thomas Alt: Yes, yes, and this is how we would, also in the future like to position ourselves with Junaio.  Junaio will be a platform, a technology platform, which will allow users to do whatever they want to do in augmented reality. The API of Junaio is huge in the sense you can do anything from outdoor gaming, to visual search, to normal, uh, lay out style, you know, find the next burger king a mile away kind of super impositions.

Tish Shute: The only licensing you pay is for unifeye right? When you want to use your tool kit right?

Thomas Alt: Yes and this is how we’re distinguishing it. Junaio is our consumer brand targeting newbie AR developers, with limited programming skills, while the Unifeye platform is really our B to B platform where B to B customers can create their individual AR experiences.

Tish Shute: Yes which is what my friend Patrick O’Shaughnessey, Patched Reality, did for the Ben and Jerry’s app he created using Unifeye.

Thomas Alt: exactly…

Tish Shute: It is a lot of work developing for so different mobile platforms isn’t it. Junaio is on Android and iphone but you havent moved Junaio to Symbian?

Thomas Alt: To be honest with you right now its a matter of priorities we have other things we want to do first. And from analyzing the user base, iphone was a big step Android was a big step and now we are pretty much seeing what is happening next. As you know Nokia going into different directions as far as their smart phone operating system goes, and so on and so forth.  There are also capacity constraints. And right now obviously the most – potentially not the most possible users, but the users most inclined to do AR on a day to day basis are the ones using the iphone and android devices. But obviously there are a lot bigger cellphone manufacturers out there.  But just you know even the mobile web users aren’t as strong as the users on the iphone and android devices?

Tish Shute: So what do you think the iphone 4 has that brought to the AR picture?

Thomas Alt: Very fast camera access, very good for marker recognition. If you go to the Metaio site you’ll find a movie where we show on the iphone 4 app for a real augmented reality Leggo peice – this is something which is very nice

Tish Shute: Yes I see that, yes that is nice, yes, yes very nice. The Unifeye SDK is really putting markerless AR into the mainstream.

Thomas Alt: Yesterday we launched the first, a err very nice shopping… shopping solution for , but that’s completely external.

Tish Shute: Oh yes – the augmented reality shopping for seventeen.com, i was going to ask you about that, because it is the first augmented reality online shopping using natural feature tracking.

Also I am very excited to see the gestural interface, awesome!

The seventeen augmented reality shopping app is a PC experience but are you working on developing gestural interfaces for mobile AR?

Thomas Alt: We are continually pushing the envelope of what’s possible with AR. Gestural interfaces for mobile AR is certainly the next step in taking what we’ve done on the PC and making it more portable by using the mobile platform. One thing to keep in mind here is the limitations of mobile platforms and size of the screen needs to fit and make sense for the user experience.

Tish Shute: I know you started off as an AR researcher (although as you mentioned earlier you have been working in commercial AR and building Metaio for a long while now.

So in addition to how we are progressing towards gestural interfaces for augmented reality, I would like to ask some questions about AR eyewear. We won’t really have hands free AR without eyewear. What is your projection on when we will see consumer AR eyewear? And, Do you have a any comments on those speculating that we will not see AR eyewear go mainstream for 20years?! What is Metaio doing to move eyewear technology along?

Thomas Alt: Well as you know, it’s always, you know on the technological roadmap, and we’re still doing research projects, in our AR research department. We have worked on things like calibrating eyewear for augmented reality. There is some nice development there.

But really, commercially, the whole thing with eyewear has never caught on on a level which would make it a valuable avenue, business avenue, at least for Metaio. So, I guess as an ex-researcher, it’s still a very interesting, a very good technology. And it would definitely change the marketplace radically when available. But as per right now, there are very few commercial applications.

Tish Shute: Are the obstacles to AR eyewear technological obstacles or is it just a question of a a business model. I mean is it realistic to see eyewear in the next 3 to 5 years at a price point affordable to consumers, where you really, truly can have eye tracking? You know, the whole problem there was with virtual reality and eyewear giving people a headache. How far have we come in terms of the technology?

Thomas Alt: Well, it’s not so much technological factors ’cause all fundamental problems are solved. It’s more a rather large corporation, I guess, would have to step up to the plate and say okay, do, let’s get all the state of the art in electronics and develop just a perfect HMD.

Tish Shute: Something Yohan Baillot’s company Simulation 3D is doing at is looking at is hooking up eyewear to smart phones.

Thomas Alt: Yes exactly that would be even better. Metaio has done a strategic move into this HMD space for augmented reality about a year ago by acquiring a bankrupt company. I mean, we had considerable IP around it from our research base but in the long term we still believe in it and we did a move about a year ago in buying what was left over from a bankrupt company including a lot of IP, which basically goes into the direction of mobile augmented reality but also mobile augmented reality in connection with head mounted displays.

There’s actually a press release about this but that’s about a year ago…

I know that the whole HMD thing… I mean, I’ve seen companies come and go. Metaio has worked previously, very closely, with Microvision of Seattle. We have worked with a German company, doing HMDs and we have worked with Vuzix.  We are still working with Vuzix, so we’re still consider it very valuable.  But right now, I mean, it’s just not a big part of our commercial pipeline, to put it that way.

Tish Shute: It was interesting what Bruce Sterling said in his keynote at ARE 2010.  He actually made a strong case for why smart phone augmented reality may be more interesting because it’s less immersive. I mean, he raised the question of the fact that if you really truly had AR eyewear and HMDs you’d re-enter the world of virtual reality or as he called it AR’s Gothic step sister VR would rise from the grave….

Thomas Alt: Yeah, well, that’s a cultural or even a philosophical question and we have discussed it a lot, especially in the industrial domain. Also will the deployment of HMDs come about from end consumers using it in their spare time, or from professional users using the idea in their work time?

Tish Shute: Do you think it surprised people who have been working in augmented reality research how much people have engaged with the idea of smart phones as the mediating device for AR, and that rather than having the always on experience that eyewear would give us, we use smart phones as a magic lens of a smart phone when we need to or want to. Some people were skeptical that anyone would want to hold up a little window to look at augmentations of the world – a magic lens. I mean, it wasn’t self-evident that that would be an experience people enjoyed, and it turns out that it was.

Thomas Alt: That’s actually a very good analogy. And also in my view, I mean, certain behaviors just change also, right? I mean, this is exactly what Apple’s trying with the iPad, right? You’re taking the iPad, and all of the sudden you’re not constrained to a laptop or whatever. And it’s truly a companion of the couch, in-bed Web, in the kitchen, and so on and so forth. So digital usage with the iPad, which is a different market, and I’m aware of that but as an example, the iPad has changed our behavior. And obviously, the augmented reality guys are hoping that something similar happens with AR.

Tish Shute: Which of course brings up the question, I’m assuming that some of the next generation of slates/ipads are going to have front and back cameras, GPS, and compass, right?

Thomas Alt: Actually we know that.

Tish Shute: You know that, yes. I assume that you know that, because are you working on some prototypes, and have you got some plans?

Thomas Alt: You have to understand that I cannot talk as I’ve talked as a researcher. It’s the rules, so I have to be a little bit careful about what I say. We very much think that a webpad, or whatever pad, you would want to call it is on some occasions very good device for AR.

Tish Shute: Yeah. But it is an interesting point with holding up a larger device, because you hands aren’t free, but the neat thing about the phone for augmented reality is that you really can do a lot with your thumb, as we’ve found and just the position of the phone. But, how will this work it with the two hands on the larger device?

Thomas Alt: Keep in mind, everyone’s talking about mobile augmented reality, but really where the case for augmented reality, at this point, is the strongest is in the installation business, it’s in the web business… Not necessarily only commercially, but also use case-wise. There are tons of museums out there which are using our augmented reality system in an installation fashion, and to communicate products better, and more efficiently, and so on, and so forth.

So, I know that the hype is clearly on the mobile augmented reality side, but there are many examples augmented reality experiences where holding up a larger device is not a big obstacle.

Tish Shute: Well this brings me to some questions about the future of mobile AR. My interview with Jay Wright focused on how we are now in a new period for AR bringing together computer vision, visual search into a mobile stack that is really optimized for AR. What do you see emerging in mobile AR as we move beyond compass, GPS, camera, accelerometer based AR into markless image-based AR. What will the new use cases and where will we see mainstream users getting in AR. Will AR games be the first mainstream AR experiences?

Thomas Alt: My partner is actually, first of all, one of my best friends, second of all, very emotional, third of all, very intelligent, and he said the other day something I think very valuable in this area. He said, basically think about Mobile Augmented Reality, Thomas. There’s really a very limited number of use cases which you can do if you look at these classical Point and Find applications, ok? But there are almost unlimited number of use cases when Augmented Reality becomes a day to day companion, ok? So what he meant is, ok, I’m looking at my normal day… I’m looking at the city, I’m walking throughout this, I’m coming home, I’m having dinner basically. I can deploy Augmented Reality in a pure POI search fashion perhaps not even once. Ok when I’m travelling it’s a different story, but in an ordinary day I might use a POI search not even once.

But where this ultimately leads, you know, is even in the 15 minutes I’m having breakfast, I’m using AR – looking at the cereal box with my cell phone, I’m taking part in a sweepstakes or whatever. So from that we draw the conclusion that as a general strategy for Junaio, we should basically throw as much technology as possible into Junaio, make it halfway self-explanatory, and just give people the possibility to come up with ideas on how to deploy augmented reality continuously.

We have actually got a creative team from an Art School working on that, and just, you know, with very little programming skills coming up with things you can do with Augmented Reality on a day-to-day level. And it could be a scavenger hunt game, in the city, with monsters flying around, it could be the normal POI routine, it could be marketing purposes, and so on and so forth. And I think that’s really the roadmap, and this is a little bit similar on a more technical level, to what Qualcomm is doing, ’cause they’re floating out possibilities or capabilities I want to call them, and Metaio is doing that, but on a higher level [re the tools] meaning on a Junaio level.

Junaio is a capability platform. It is also a way for Metaio to demonstrate the capabilities of our technology. We will offer all the possibilities for AR and more that we have already demonstrated on PC augmented reality.

Tish Shute: What is the business model for Junaio? Are you encouraging developers to develop business on your platform ?

Thomas Alt: Junaio is our end consumer platform and our business model is similar to the way Google structures its business model. We work with OEMs, content partners, brands, and developers to offer free content to our end users. Where we do charge is on the advertising side, more specifically contextual and location based advertising. At the current stage, we are focused on building the content base, fostering our developer community, but in the near future, we will be introducing advertising channels.

First of all you have to have very good use cases in the platform basically. And then to put a business model on top of that from a technology stand point is not hard – its a pay channel. Its all prepared for this.

Tish Shute: You have quite a broad base as a company don’t you – you do everything from industrial AR, marketing to technology licensing and more?

Thomas Alt: Basically, there’s a lot of things people don’t see us. There is also the Unifeye PC SDK and we have a client base and partners who are sourcing software from us, and we are doing great pieces. I mean the hype of augmented reality is really coming to a peak. There are lots of pieces that are not even talked about any more. China’s GQ magazine launched with AR from Metaio, the biggest AR campaign anywhere – there are a lot of potential readers in China. And um, so that’s our business model… we have our IP, our patents and so on. And on this we can move on onto the mobile platform whenever it’s advisable or feasible.

Tish Shute: Right. Yeah. I mean uh, you’re very fortunate to have this base built on uh, years of developing IP. What are the most important areas of AR that Metaio holds IP and patents in, in your view?

Thomas Alt: there’s sleepless nights in that too…

So far we’re extremely excited about what’s going on with Junaio, it’s one of our big, big success stories. But we are sensible and trying to experiment because, you know, analogies from the past won’t really work in my view for Augmented Reality in a sense, that, you know, you better bring for a new system to fly, for a new technology to fly, you better bring a very concrete use case to the table, ok? And a well-defined use case. And we are, right now, with Junaio, in a state where we are checking out what could be such a very defined use case.

Tish Shute: So how many users does Junaio have now?

Thomas Alt: Let’s put it that way, we are, especially in the last 2 months, we are very satisfied. But we are not disclosing that, because users, and we’re seeing that from the competitive landscape, always needs 1 page of description what exactly a user is.

You understand what I’m saying? So this is why, ’cause we don’t want to up or downplay things, we are very careful saying, with users. Because I mean we have people who are actually also commercially very interested in Junaio…  We go through with them and discuss what exactly a user is. Cause there’s more then…a download is not a user. An app or something on your phone is not a user, basically, in my opinion.

Tish Shute: I am still waiting to see someone do something with AR and the Four Square API, or now the Facebook Places API. Do you see an interesting potential in the marriage of the rapidly emerging location based social networking space and augmented reality?

Thomas Alt: Definitely. Augmented reality offers a way for users to find information around them easily. Adding in the social networking component such as geo-tagging, rating, commenting can enhance the user experience and create engagement beyond just viewing the information. For example, within junaio, an average user can create their own personal channel and geo-tag photos or leave text messages at different locations. They can create a virtual tour of San Francisco and share it with their friends. By connecting the social side with good content, the augmented reality experience becomes more fun and interactive.

Tish Shute: And, of course, there’s the Junaio API. Are you beginning to see developers use that?

Thomas Alt: Yeah exactly, I mean if you go to Junaio.com, you can get a login, you have an API description. And the way it works is, that you bascially set up a call, which contains the information you would like to have in your individual channel. You submit it to us, it will get checked for profanity and other things, basically. And then we admit it to Junaio basically. The API is huge.   You can also use Junaio indoors.

This is very relevant. And there’s a tool chain for that, and so on and so forth. You can do mission-based search with Junaio. It’s in there, it’s called Junaio Glue. And there will be another very interesting feature coming up in a couple of weeks. And you can just do it, you can do a scavenger hunt, a game, normal POI search, and so on and so forth. And it’s all active. And that’s, what’s sometimes difficult for us to communicate, is it’s really a capabilities platform, but on the other hand it’s obviously very good to developers. And I mean on the developers side there’s huge interest.

Recently, Qualcomm announced an SDK for vision based augmented reality – currently in private beta and open to the public this fall. The Qualcomm augmented reality (AR) bonanza will launch with a $200,000 developer challenge and a SDK that will put vision based augmented reality into the hands of developers without licensing fees.

This is a big step forward for augmented reality and a very important move made by an industry giant to support the rapidly evolving AR industry. Innovation at all levels of the AR stack, particularly at the hardware level (CPU/GPU optimization) is vital for the full vision of augmented reality – media tightly registered to physical space, to take center stage. Vision based AR takes mobile AR beyond compass/GPS based AR post-its, which are only loosely connected to the world (but the staple of most current AR apps), towards the holy grail of AR – markerless tracking with the whole world as the platform.

Click on the image above or see here for a video demo of an AR version of Rock’em Sock’em Robots game.  Mattel, one of the first companies working with the SDK demoed AR Rock’em Sock’em, at the Uplinq 2010 conference (see Chris Cameron’s ReadWriteWeb write-up on Uplinq 2010).

The Qualcomm AR stack, which reaches from the metal to developer APIs, will give Android developers an important edge in AR development. And, when vision based AR starts getting integrated with visual search capabilities, and combined with cool tools like Unity, we will start to see the augmented world get really interesting.

Visual search is already an area of AR getting a lot of attention, with Google Goggles, Point and Find, Japan’s NTT DoCoMo set to launch “chokkan nabi,” or “intuitive navigation,” in September, and the recent partnership between Layar and Kooaba. Metaio’s mobile augmented reality platform Junaio is already integrated with Kooaba’s computer vision capabilities.

And, of course, I am particularly excited about including open distributed real time communications for AR in this stack, which is why I asked a group of developers who have been inputting into the ARWave project if they had questions for Jay Wright, Qualcomm.  Thank you Yohan Baillot, Gene Becker, Anselm Hook, Patrick O’Shaughnessey, Thomas Wrobel, Markus Strickler, and Davide Carnovale for your input.  [Note: see my upcoming post, about the future of ARWave and real time distributed communications for AR following this Google announcement.]

Jay Wright, “is responsible for developing and driving Qualcomm’s augmented reality commercialization strategy.” He “handles partnerships with leading innovators in industry and academia and leads Qualcomm’s efforts in enabling augmented reality within the mobile ecosystem.” In the interview below, Jay very generously answers our questions in detail.

A key contributor of questions for this interview is Yohan Baillot. Yohan is working on a full vision of AR – integrating computer vision, visual search, open distributed real time communications and AR eyewear. Yohan Baillot is founder of Simulation3D, a consulting and system integration company specializing in interactive visualization systems and eyewear-based AR systems. (I hope to bring you an interview with Yohan soon!).

Qualcomm was the title sponsor for are2010, Augmented Reality Event, and played a vital role in making this event an historic gathering of the talent and creative minds at the heart of the emerging AR industry. Watch out for the videos of the are2010 sessions to be posted at the end of August. My are2010 co-chair, Ori Inbar, is preparing them to go online while kicking his newly funded start up, Ogmento, into high gear! Ogmento is also one of the start ups pioneering vision based AR.

Metaio, (with Total Immersion, they are one of the first augmented reality companies), has played a key role in bringing a vision component to smart phone augmented reality apps with their Unifeye mobile SDK.  Junaio, Metaio’s own mobile augmented reality platform has gone beyond location based AR with “junaio glue” – “the camera’s eye is now able to identify objects and “glue” object specific real-time, dynamic, social and 3D information onto the object itself,” (see my upcoming interview with Metaio founder, Thomas Alt).  Also, recently, Layar – who continue to innovate at a breathtaking pace, announced a partnership with the computer vision company Kooaba.

Both Maarten Lens-FitzGerald, Layar, and Thomas Alt, Metaio, when I spoke to them recently, saw the Qualcomm SDK as a very positive development for AR, and they look forward to exploring its capabilities and integrating it where appropriate with their AR tools.  See more about Layar’s upcoming visit, to the US here – August 10th NYC, and August 12th SF.  Also save the date, Sept 27th, Munich, for InsideAR, Metaio’s upcoming conference.

It is clear that vision based AR will be driving the next wave of AR apps. And, as Maarten and Thomas both pointed out, it will be interesting to see which use cases capture the imagination of users the most. Having more tools freely available to AR developers will certainly be a boost to creativity. And, Qualcomm’s SDK is going to give Android developers, in particular, a big opportunity to take the lead.

Interview with Jay Wright, Director, Business Development, Qualcomm

Tish Shute: Before I start with questions on the new Qualcomm vision based augmented reality SDK, I want to briefly look ahead to what many people feel is vital for the full realization of augmented reality – head mounted displays, or more specifically, comfortable, sexy AR eye wear. Is Qualcomm going to be involved in the development of augmented eye wear and wearable displays?

Jay Wright: I think there’s some core technology that needs to come together so we can have what we think needs to be a see-through head mounted display with a decent field of view. And that looks like something that is quite possibly further than a three to five year horizon.

Tish Shute: Gene Becker asked some interesting general questions about the Qualcomm AR initiatives. He said, “I’m unclear exactly what Qualcomm’s goal is.” It would be interesting to hear from you the Qualcomm view, from the top down.

Jay Wright: Our largest revenue stream comes from sales of chipsets. And we see augmented reality as a technology that drives demand for increasing amounts of processing power. So we want to create demand for chips, higher-end chips, and augmented reality does that. Specifically vision based augmented reality because it is so computationally intensive.

Tish Shute: Yes. And I think that is why people are very excited by the Qualcomm SDK. It is not only the first free toolkit for developers to build vision apps from, isn’t it? There’s been nothing freely available before this, has there? But also Qualcomm is paying attention to the complete AR stack to support vision based AR development, from the chips to game/app development tools like Unity.

Jay Wright: That’s really the goal. We’re not here to be in the augmented reality applications business. Qualcomm’s role in the ecosystem has been to serve as an enabler. And that’s what we want to do with augmented reality: provide the enabling technology that allows the entire ecosystem to flourish.

“Augmented Reality has a number of attributes that make it a great fit for Qualcomm’s core competencies”

Augmented Reality has a number of attributes that make it a great fit for Qualcomm’s core competencies. It’s very computationally intensive, algorithmically complex, requires tight integration of hardware and software, and benefits from tight integration of multiple hardware components. And that’s the kind of problem we like here, where we can apply our core competence of really optimizing complex systems for performance, while at the same time minimizing power consumption.

And as you know Tish, mobile AR is really extremely power sensitive. We sometimes talk about it as a battery’s worst nightmare. It’s roughly equivalent to playing a 3D game and recording a video all at the same time.

Whenever there is something that takes a lot of power, that’s a definite opportunity for us to optimize it.

Tish Shute: Right. One of the core business is chips right, but for Qualcomm there’s basically a lot of profit in licensing. When I talked to the developer community about the Qualcomm SDK developers first question was, “What’s the licensing? What’s this going to cost us in the long run to develop on this SDK re licensing?” And they had all different takes on this. So everyone had different ideas about what your approach to licensing might or might not be. Could you clarify the approach to licensing, as I think this is a core concern for developers.

Jay Wright: Anytime you see something for free, you kind of say, “Hey, what’s the hook?” So yes, it’s definitely a logical question. Our intent is not to generate licensing revenue from application developers using the SDK. So the SDK will be made available free of charge for development, and it will also be free of charge for developers to deploy applications.

Tish Shute: Now, this is another question. You also include not just image recognition capabilities but Unity in the package you are offering developers. Unity products usually involve a license. They do have some free products too, I think. But how does this work? And how do you separate your part from their part, or don’t you?

Jay Wright: That’s a good question. What we’re trying to do with the platform is incorporate it into tools that people already know how to use. So we’re actually going to have the SDK support two different tool chains. One of them is the Android SDK and NDK. And then the other one, is Unity.

We’re working with Unity to create an extension to the Unity environment that will be available as part of the Unity installer when you install Unity from the Unity website. Developers will still be paying whatever license fees are associated with Unity’s products on their existing pricing schedule.

Tish Shute: One of Thomas Wrobel’s question is whether developers can just use the image recognition without Unity? Your answer is yes, you can work with the computer vision component of the SDK separate from Unity?

Jay Wright: Yes, you can.

Tish Shute: Good because we would like to build a completely open Android client for ARWave, and not tie it to Unity unless people choose to. He’s using the open Android JPCT 3D engine, which he’s adapting for AR. So he could actually use the part of the SDK that does image recognition and association with that, right?

Jay Wright: That’s correct. You are not required to use Unity. Unity is just one option for building the application.

Tish Shute: Great! That’s very good. But I’m sure many developers are going to jump on the chance to use Unity. But I mean it’s nice to be flexible because it’s so early for AR that people have different ideas and new use cases coming up all the time. I think it’s excellent you’ve divided that.

Another of Thomas’s questions was, “Can developers use their own positioning data sharing solution?” He’s really talking about AR blips.

Jay Wright: With data sharing solutions, I am assuming that by data he means referring to augmentation data or graphics?

Tish Shute: Yes, and I’ll ask him to elaborate. But, at the moment, everyone is using different ideas for POI, aren’t they?

“The goal with our platform is to make it just as easy for a developer to create 3D content for the real world as it is for a game world or a virtual world.”

Jay Wright: Yes. So let me answer it this way, Tish. The goal with our platform is to make it just as easy for a developer to create 3D content for the real world as it is for a game world or a virtual world. So all we’re really trying to do is provide the computer vision piece that makes the real world look like a bunch of geometric surfaces and potentially some meta data that is associated with this so you know what you are looking at.

So that means from a developer’s perspective, you are still doing all of the 3D content, all of the animations, all of the game logic, all of the rendering. You are still doing that all yourself. So if you think about doing an AR game, you are doing everything you used to do, except you are not creating a virtual terrain. You are just going to map it in the real world.

So if you want to do a browser that is doing POI’s, your POI data, or augmentation, or meta data, or whatever it is, that can be in your application, it can be in the cloud, it can be wherever you want to put it. We’re not putting any constraints on what that content is or where it’s stored.

Tish Shute: Right, and that’s what I hoped for. And I think that does answer the question. People are interested to know how far Qualcomm is going with this. For instance, Gene Becker asked: “do they see a business at a certain level in the AR stack?” As you said AR development basically feeds into the core business of chip development, right? But does Qualcomm also see some new business models developing?

Jay Wright: I think it’s foreseeable that Qualcomm could identify other business opportunities down the line. But we’re certainly not there today. Today, our motivation for the investment in AR is to create technology that is going to advance the chipset business.

Tish Shute: When the news came out about Qualcomm’s support of a game development studio at Georgia Tech at the same time as the SDK I think I wondered what was the scope of Qualcomm’s interest [for more on using Unity for AR development see Vision-Based Augmented Reality Technical Super Session video from Uplinq 2010].  For example, I am interested to know how the Qualcomm initiative in developing an AR stack connects to the effort to introduce an AR browser based on web standards, i.e., the Kharma/Kamra KML/HTML Augmented Reality Mobile Architecture from Blair MacIntyre and the Georgia Tech team (image below)? Are you supporting the open standards based browser development too?

Jay Wright: Blair is going to continue to work on the browser effort. And it’s our expectation that he will use our SDK and technologies for vision pieces of the browser effort where appropriate. So they are certainly not mutually exclusive. I would just think about our technology as one element of what may be used in that browser, as I expect it would be an element of what any other app developer would put in their application, whether it be browser, or game, or whatever.

Tish Shute: Yes Now, this is an interesting question, which is sort of connected…I’m trying to keep some form of narrative for this! It follows from the question about Blair’s web-based standards browser. A few people have asked me why we haven’t heard more from Qualcomm in all these various standard discussions that are starting to come up. I mean is it just too early, or are you too busy, or what?

Jay Wright: No, let me explain. The type of standards that have come up so far have been around how HTML should be extended for geo-browser type applications. And while that’s interesting, I think the standards efforts that Qualcomm would be more likely to be associated with in the near term are those related to API’s that are hardware accelerated.

So one of the things that we are in the process of doing right now, Tish – because as you know, Qualcomm is a company that adheres to standards and strives to produce a leading implementation of those standards on our hardware and software – is we are in the process of determining what API set within the existing SDK should be standardized.

Tish Shute: Right.

Now, my next question is, “Who are the other players at this level of the AR stack in the standards conversation? Who else is working at that level?” Obviously, the AR Lab in Graz was, but now they are Qualcomm, right?

Jay Wright: They are still independent. Qualcomm is the exclusive industrial partner of the Christian Doppler Handheld AR LAB in Graz.

Tish Shute: Does this compete with, say, the work that other AR start ups are doing?

Jay Wright: Our intent is not to compete with companies that have done augmented reality technology. Our intent is to enable the entire ecosystem. So we would like to work with both Metaio and Total Immersion to find ways that they can benefit from our technology. That would be the hope – that our technology can kind of lift and float all boats in the ecosystem.

Tish Shute: There are not many implementations of vision based AR right now? I mean obviously Microsoft is doing stuff because they have Georg Klein now, right, and there is Google Goggles, Total Immersion, Metaio, and it will be interesting to see where Layar’s partnership with Kooaba will lead?

Jay Wright: Yes. I think there are relatively few commercial implementations of vision based AR stacks.

Tish Shute: One of Patrick O’Shaughnessey’s question is he wants to understand what features are going to be in the vision component, very specifically. Patrick O’Shaughnessy, Patched Reality, working with Circ.us, Edelman, and Metaio used the Unifeye SDK to do a vision based AR app for Ben and Jerry’s that’s been getting all the attention lately. He was a speaker at are2010.

He very specifically wants to know what features will be included in the computer vision component. He says, “I’m most interested in understanding what features are going to be in the vision component. Is it marker based?” Well I know it’s more than marker based. I saw some of it in Chris Cameron’s ReadWriteWeb write-up on Uplinq 2010. Is it “NFT? PTAM? other? Also, are you are integrating any backend services.” That is an interesting question!

Jay Wright: So let’s get to the features on the client side, the vision based features. There’s support for, what AR aficionados would know as natural feature targets, or image based targets. And we use those to represent, obviously, 2D planar surfaces.

The other thing that we are trying to do to set expectations, Tish, about where these can be used is to let people know that they work best in what we’re calling near-field environments. So the idea isn’t that you use the system to create a large scale AR system that can recognize buildings indoors and outdoors. It’s the idea where I can recreate 3D experiences that take place on surfaces that are in my immediate field of view, whether that be on the table in front of me, or on the floor, or on the wall, or on the shelf.

Also, when you talk about near field experiences, there are some other constraints that are implied. Like, if it’s in front of me and my immediate field of view is probably going to be pretty well lit. And lighting, of course, is an important requirement.

So we’ll support these natural feature targets, or image targets. And we also have support for sort of a hybrid marker image type. It’s something called a frame marker, which has kind of a black border with some dots on it.

Click on the image above or here to view Vision-Based Augmented Reality Technical Super Session video from Uplinq 2010

Jay Wright: So there’s this additional type. And the reason for this additional hybrid marker type is it has a lower computational requirement than a natural feature target. So the idea is these things can be used as game pieces or elements of play where I want to have a large number of them detected and tracked simultaneously.

So you can have, for example, one big natural feature target that serves as a game board or game surface, and you can use these other things as smaller game pieces. And when you put them out, different types of content can appear on them and do different things.

Tish Shute: Yes, that’s nice! And the other thing I noticed was the virtual buttons. How well developed is that?

Jay Wright: The idea behind virtual buttons is, in addition to supporting augmentation, we want to support interaction. And we think there are going to be different types of user interaction with augmented reality content. It may be hand tracking and finger tracking, but another compelling form we’ve identified so far is the ability for me to touch particular surfaces and have an event fire within the application..

So virtual buttons are rectangular areas on image targets that a developer can define, and they serve as buttons. So you can create a target that is a game board, for example, and define certain regions. And when the user covers that region with his hand, like pushing a button, your application can detect that event and take some action.

Tish Shute: Nice! And what is the documentation on these capabilities that is offered by Qualcomm…For example Yohan Baillot, who is interested in integrating eyewear-based AR systems with smartphones asked. How deep does this go? Will there be full documentation on Snapdragon, people who want to work at that level? Is there a chip SDK?

Jay Wright: . Qualcomm’s model is to work with providers of the operating systems and deliver functionality of the chip through the operating system. So many operating systems APIs will take advantage of functionality that’s in the chip. But there is no separate chip SDK per se.

Tish Shute: I suppose that does come up a little bit with one of Anselm Hook’s questions, because there is some overlap with Google Goggles here, isn’t there, in terms of what you’re doing, right? Are you going to work closely with Google Goggles ?

Jay Wright: Google Goggles is performing what we’ve described ‘visual search’. So the idea is you take a picture, send it to the cloud and identify it and the results come back. I think if we see Google Goggles go in a direction where there’s an AR experience that would be a good area for us to collaborate with Google.

Tish Shute: Anselm Hook is very interested in having some kind of open standard around this physical tagging of the world, right, – the physical world as a platform. But I suppose that’s down the road but is there a plan to start talking about open standards here – visual search with image recognition? That’s a very powerful combination. (see my interview with Anselm Hook here).

Jay Wright: I think it is. And we’re very interested to hear from developers and others that have ideas about how they would want to integrate with the functionality that we have to best enable those kinds of combined experiences.

Tish Shute: Well, I know Anselm has a lot of very important ideas on that.

Jay Wright: I’d be very interested in hearing those because we want to do everything we can to enable the maximum number of applications and best user experience for anything that people want to do.

Tish Shute: Let’s go back to some specific questions about the platform, right? For example Yohan Baillot asked, “Is it arbitrary image/tag recognition supported? Is the tag / image specifiable by user? Is face recognition supported?” Not yet, face recognition, right?

Jay Wright: Not yet.

Tish Shute: What are the plans with that?

Jay Wright: I think we’ve identified it as an interesting area and something that there’s some interest in, but have not made a decision on a particular technology direction.

Tish Shute: You’ve answered some of these but 3D model based vision tracking. Yohan’s question was, “Is 3D model based vision tracking supported (that is recover the pose of the camera using a known 3D model and a 2D camera view of this model)?”

Jay Wright: That’s something we’re looking at very closely, but again, don’t have a plan, or don’t have a future date for.

Tish Shute: And you said with the natural landmark tracking that’s not supported, right?

Jay Wright: I don’t know if I know what that means, Tish. But we don’t have any APIs that provide compass or GPS functionality other than already exists in the operating system. So if you want to take advantage of the compass or other sensors, you can absolutely do that, but the SDK does not currently provide anything different or anything more than already exists in the OS.

Tish Shute: This is an interesting question, “Is Snapdragon offloading some processing to the GPU, if any?”

Jay Wright: Certainly rendering functionality that utilizes OpenGL is being offloaded to the GPU. We’re currently in the process of determining multiple methods for offloading functionality between both symmetric and heterogeneous cores on Snapdragon. Which would include the GPU, the apps processor, and DSPs.

Tish Shute: No one has truly solved optimizing the GPU/CPU for mobile AR yet have they?

Jay Wright: That really gets to the heart of the optimization here. Which pieces ought to be operating on which cores and when, and why? And that’s something that we’re looking at very closely.

Tish Shute: Right. The only AR – that is truly 3D media tightly registered to the physical world has been done for military and medical (and that has often been with a locked of camera!). But to take mobile AR to the next level I think many developers would like access to the CPU/GPU, for example a developer interested in the future of eyewear like Yohan?

Jay Wright: We’re very interested in hearing what kinds of tools developers would like to see.

Tish Shute: What is the best forum for discussing feature specifics?

Jay Wright: To provide feature requests to us?

Tish Shute: Yes. And discuss them.

Jay Wright: if people go to qdev.net/AR there’s an application up there for the private beta program. So if people do have ideas about features or other things they would like to see, they’re welcome to submit [their requests and ideas] there.

Tish Shute: I also have some questions about the specifics of the competition. Some people are a little confused about some things. Yohan asked, “What is the expected form of the project? Lab demonstration? Specific capability? Complete end to end system?”

Jay Wright: The only requirement is that they submit an Android application that we can then get running on a device. So if it has a backend component or backend server that it works against, great. If it does, it does. If it doesn’t, it doesn’t. But that’s really it. There’s no limit to the application category. It can be a game, it can be a museum tour, it can be a children’s learning game or learning experience. It can really be anything. The idea is we want to find experiences for which AR delivers some unique value. We’ll be announcing more specifics about the competition in the near-future.

Tish Shute: Right, because some people weren’t sure about the Unity being separated whether it was biased towards games. And it’s not really, is it?

Jay Wright: Unity is a bias toward just rapid development for 3D, I think. It’s most commonly associated with games, but there are also a lot of Unity customers that use it for medical simulations and other types of applications that aren’t really games at all.

Tish Shute: Yes. It’s very flexible, I know. You did bring up the backend services again. Are you thinking of offering any of that?

Jay Wright: There is a backend tool that we offer. And the backend tool is what you use to generate your targets. So if you want to create or use a particular image for a target in your application, you upload it to our target management application, and then it will evaluate that target and tell you how well it will work. So as you know, certain images are more likely to be recognizable than others. And so there’s metrics in that application that will give you some feedback.

And then you can download your target resource from the website that you can then incorporate into your application project.

Tish Shute: So this is available at the moment to people who are in the private beta and not to…you know, all of this information and documentation, right?

Jay Wright: That’s correct.

Tish Shute: So that’s an incentive. Now, just to encourage people to submit to the private beta is the other thing that people seem confused about. In one part you say 25 developers. And some people have thought that meant it was limited to 25 individuals. And some people have like maybe four people on their team, so they were going, “Well, are we going to be accepted because we have four developers, or do we count as one because we are all working at the same project?”

Jay Wright: it’s just 25 companies.

Tish Shute: OK. I think we’ve gone through the questions. Just to clarify and maybe give some incentive for people to apply to the private beta…the big advantage of getting in the private beta, aside from getting a month’s start on the competition, is that you get a chance to input, right?

Jay Wright: Yes. A chance to provide feedback, get early access to the technology. And then we are also providing a free HTC phone.

Tish Shute: Oh, yes. I forgot the phone. Yes, right. In the requirements, though, you basically seem to be asking for sort of a full app…some people get reticent about delivering their full application plan, right?

Jay Wright: Yes. I understand that. People should just reveal what they are comfortable talking about. Just so you understand the constraint on this end, this is early technology and we’re trying to understand exactly what the support requirement is going to be. And we have limited supported resources at this time, so we want to make sure that we can focus the resources that we have on folks that are really going to use the technology and have a sound plan to actually build something. So that’s really the motivation behind limiting the size of the private beta.

Tish Shute: OK. Yes, it’s good to reiterate that. We’re down to the last question that I have, and then I’ll ask you if there is anything that I missed. You say you are partnering with Mattel. Who are the developers? Because I mean Mattel isn’t an augmented reality development team.

Jay Wright: Mattel used a subcontractor, Aura Interactive.

Tish Shute: Nice. But that’s your only partner that I saw, right? Why Mattel?

Jay Wright: Well, to launch a new technology, companies will often find showcase partners to demonstrate compelling uses of it. And we thought Mattel and the Rock’em Sock’em™ toy was a great example of combining augmented reality with an existing toy.

Tish Shute: And I think people agree with you on Rock’em Sock’em (see Chris Cameron’s RWW post).

Jay Wright: And there’s other showcase partners and applications that we will continue to work on to kind of spur the ecosystem and show what is possible.

Tish Shute: OK. Now, is there anything I’ve left out that you think? What’s the core of this narrative that we need to get across, and if I’ve left anything out that is a key piece?

Jay Wright: I think you’ve done an excellent job of covering all the bases, Tish.

Tish Shute: [laughs]

Jay Wright: I think the important overriding message to get across is that we really see ourselves in an enablement role here, and that we are trying to provide….we’d like to provide fundamental technology that helps all developers build content for the real world.

Shortly after Augmented Reality Event – are2010, I talked with Bruce Sterling on skype and in gdocs about his experience there.  I am posting the conversation in two parts to make it a more blog friendly length!

The picture above is the Auggie Award for the best AR demo (above) designed by Sigal Arad Inbar.  It was won by YDreams! See, Ivan Franco recounts the team’s ARE 2010 experience, and winning the event’s first-ever Auggie Award, for more.   The video below was shot at the YDreams booth by Bruce Sterling.

“The Hotness” – YDreams rocking it at ARE2010 from brucesflickr

Rudy Rucker, who was hanging out with Bruce Sterling, captured the are2010 buzz and some great images in his post, Augmented Reality, Painting, Twitter. As Rudy put it:

“AR is hoping to be a next big thing, a cozier and more commerce-driven cousin of the old VR, or virtual reality.”

Bruce Sterling’s opening key note is up, ARE 2010 Keynote by Bruce Sterling: Bake a Big Pie!, and also the ARE 2010 Keynote by Will Wright: Brilliant Inspiration for the Augmented Reality Community with more videos from are2010 on the way.  One must read post on are2010 is Chris Cameron’s post, Augmented Reality’s Next Steps: Sitting Down with the Titans of AR.

Talking with Bruce Sterling, Part 1

The Auggie panel, Bruce Sterling, Jesse Schell, and Mark Billinghurst inspect the award.

Tish Shute: In your keynote at the 9am of the augmented reality industry you asked some questions of the are2010 audience: “What’s the mission statement?  You’re the world’s first pure play experience designers, except that user experience it’s mostly futuristic hot air.  But run with that, right?  What are your tactical steps?  You should get dressed, have a coffee, have a to-do list.”

How much of that did you see going on in the next two days?

Bruce Sterling: Well, I wasn’t privy to any of the business discussions.  I didn’t think it was an accident that this standard AR enabled tag thing came up from Bruno Uzzan, Total Immersion.  That seemed to me to be a useful thing.  I was always interested in the Augmented Reality Consortium.  It struck me as remarkable that there was this group of people who clearly all knew one another and it had some kind of game plan.  I applaud them for that, because these are not the 1980’s.  [laughs]  You know, it’s just a different world for young startup companies.

Tish Shute: I think you’re right. There seem to be some VC conversations going on, we don’t know what went on in the meetings, but it was noticeable in the atmosphere of excitement, and remarked on by a few people. So I think that kind of was definitely going on.

And, of course, I was so busy I never even got to see the expo properly! You said you wanted to be surprised.

Did anyone surprise you in any of the talks, in any of the expo?

AR used as interfaces for devices

Italian augmented robot from SEAC02 from brucesflickr

Bruce Sterling: I have to say I was a little bit surprised to see Andrea Carignano demoing a robot. I happen to know him because he’s here in Torino. He’s the guy that came out of Fiat and went into AR. I am not a particularly huge robot fan, but I think it’s of great interest that AR is used as interfaces for devices, as opposed to the Jesse Schell idea that AR is all about a “man with the X-ray eyes.”

My suspicion is that a lot of surprises will come out of mashups of AR.

Tish Shute: I didn’t get to see Andrea’s robot.  So what did it do?

Bruce Sterling:  It’s basically a sister device to that little helicopter that those Parrot AR Drone guys were doing.  It’s a little autonomous robot and it runs around with a webcam on it.  You can place video into the acquisition stream coming off the robot.  You can play a game, and blow away imaginary monsters or whatever.

Tish Shute: It’s interesting, because did you notice Will Wright and Patrick O’Shaughnessey, Patched Reality, spend some time hacking the Parrot AR drone in the hallway?  Did you come across them?

Bruce Sterling: Rudy was there with them.  You know, I didn’t want to watch Will Wright hack a robot.

[laughter]

Tish Shute: They seemed to be having fun even though as it turned out the power supply was dead.

Bruce Sterling:  I’m sure Will enjoyed that.  As a game designer, you want to go out and get your hands dirty with a plastic gizmo.

[laughter]

My Swiss Army knife can’t get through airport security, so I really don’t want to strip anything down.  But yeah, what else did I see that was of particular interest?  I was pretty happy about the Korean guys because they are a difficult group to get close to.

AR companies are like mini-global micro-startups.  They’re “glocal”.

“Korean elegance at the Zenitum booth” – from brucesflickr

Tish Shute: Zenitum.  What did you like from Zenitum.  They were one of our sponsors, along with Qualcomm.

Bruce Sterling:  I know that Seoul is like the number one center for augmented reality discussion.  But it’s  difficult to get behind the scenes as a journalist there and  track what’s going on in Korea.  I’m fine with Italian “realtà aumentata.”   And I feel like I’ve got a handle on French “réalité augmentée.”   The Germans were not hard to find, and the Dutch all speak English!  But the Koreans, and whoever the hell it is in Kuala Lumpur…  I have no idea what’s going in Kuala Lumpur, and only the vaguest idea of what’s transpiring in Singapore!  But I know that people there are paying a coherent interest.

So the Koreans show up, and they had some relatively predictable anime style 3D avatar conversion stuff.  But they had a really nice display space.

“Anime figures become three-d smartphone animated avatars,” from brucesflickr

Tish Shute: Ah, So Zenitum created a hot spot at the exhibition?

Bruce Sterling:  Yeah.  The Koreans had  IKEA furniture and some nifty little woven baskets.  They’d really classed up their presentation.  Most Koreans in tech tend to be kind of muscular.  The Koreans are not known for their refined presentations.  On the contrary, they tend to undersell everybody else.  But I don’t know, maybe they’ve been hanging out with Samsung and upgrading their design chops. [laughs]

Tish Shute:  Did you take some photos you could send me?

Bruce Sterling:  I took a few, but  I don’t consider myself a photographer.  They’re all up on my Flickr set. It was interesting to see so many people from so many different nations in such a collegial atmosphere.

Tish Shute: Yes – there were many different countries represented at are2010

Bruce Sterling:  It’s the beginning… and so global at such a young stage.

Tish Shute: Yes. As you said, it was 9 AM, so everyone was actually super excited to be gathered together from across the globe to start a new day together.  As you mentioned, there was a very warm affirmative vibe – everyone sharing a passion.

Bruce Sterling:   They have an online commonality. They seem to be aware of one another’s work through the Internet.

Clearly they had all heard about one another.  That’s a departure from earlier models of tech startup, where you usually have like three hippies in a local garage.  Now you’ve got German-American-Korean outfits like Metaio, and Total Immersion has a Russian affiliate.  They’re inherently multinational, both inside the company and out.

Tish Shute: It was the multinational garage, wasn’t it?

Bruce Sterling:  Yeah.  AR companies are like mini-global micro-startups.  They’re “glocal.” There’s something quite new to me about that.  I don’t find it’s shocking, because in Europe today it’s common to find startup teams who are multinational.  But to see such intense globalism at such an early stage of an industry is really different.

Tish Shute: Yes it made for a fun atmosphere?  It was wonderful running into Iguchi Takahito, Tonchidot.  You have a great rapport with each other despite the language barrier?

Bruce Sterling:  Yeah.  That guy from Tonchidot, he’s very charismatic.  He’s punchy.  That’s reflected in the very strong graphic design from his company.

Tish Shute: Using minimal English to make the case for Sekai No Camera at the Auggies, Iguchi Takahito still got through to the audience.

Bruce Sterling:  Well, his visuals were good.

What AR means for artistic practice…

Picture of Eric Gradman’s Cloud Mirror, from James Alliban post ARE2010 – Augmented Reality utopia in Silicon Valley – see for more on the are2010 ARt Gala

Tish Shute: So before I move on to wider themes, I’m going to wrap up on some of the different aspects of the conference.  I was chairing the technology track but you were more free roaming, was there anything that went on in the sort of hallway discussions and the presentation rooms that struck you?

Bruce Sterling:  Well, I did get collared by artists.   They really wanted to talk to me.  We got into some serious discussions on  what AR means for artistic practice.  How you can do this and reach that, how can one sharpen up one’s presentation?  I mean, they really wanted some art criticism.

Tish Shute: That’s very interesting.  Did you come up with anything that you hadn’t been thinking about already through the conversations?

Bruce Sterling: I’ve seen augmented reality installations before, and I certainly know many electronic artists.  But I don’t know.  People in the AR art space, they are looking for guidance and trying to find fellow spirits.  In their own way, they have the same pioneer spirit as the business people.

Helen Papagiannis shows Iguchi Takahito, Tonchidot, her AR Wonder Turner, an exquisite corpse inspired installation

Tish Shute: Yeah, it’s interesting, because we wanted the art gala to be even bigger, but it turns out, because of the logistics of putting up art in a conference space is fabulously expensive, because it has to be all installed and hung…

Bruce Sterling:  I’m keenly aware of that.  At Share Festival in Turin we bring in six installations, and it’s very heavy work.  It really takes a lot of logistics.  It was like a Battle of the Bands.  It’s like doing a rock concert.

Tish Shute: One of the installations I was really sad to not have there was Uber geeks’  “Steve” H.E.AI.D installation that Brady Forrest & Co. took to Burning Man.

So I was very happy that we actually did get the number of artists we did.

Bruce Sterling:  Well, there aren’t a million AR artists in the world, so it’s hard to judge.   I didn’t see many business people rushing up to have me critique their business plans.

Tish Shute: [laughs]  They were all in the meeting rooms.

Bruce Sterling:  Maybe it’s for the best.

VC and AR Startup Action

The Zenitum Booth, are2010, photo from Chris Cameron’s Flickr stream

Tish Shute: Do you know that why your talk started a few moments late is because we had 50 people who arrived from the Silicon Valley neighborhood I guess!

Bruce Sterling:  Did they not preregister?

Tish Shute: No. They all stood in the line for the same day registration!

Bruce Sterling: It ‘ll be interesting to see what transpires there, if there is a little wave of startup action.  God knows they need some place to put their money, because the VC scene in the US is pretty much moribund.

Tish Shute: Ogmento is the first US AR Games startup to get VC, I think.  I think there was some VC action at are2010 for sure.  And Qualcomm obviously seems to have commercialization plans for their AR technology, and to be scouting talent  and ways to deliver new AR experiences.

Jay Wright, Qualcomm presents Joe Dunn, e23 Games, winner of the are2010 StartUp Launch Pad with a check

Bruce Sterling:  Some  people don’t need venture capital.  I mean, Google Goggles isn’t going to be hurting for VC money, obviously [ see Chris Cameron’s RWW post, Google Goggles Coming Soon to iPhone] .  AR may come up through other methods, like people allying themselves with Hollywood, or peeling off of advertising companies.   There’s a lot of outfits who might conceivably want in-house AR skills.  Then when people set up a specialty AR shop,  they  peel off the list of clients.  I don’t know.  Those old days  of Silicon Valley venture capital seem like a lost world.

Tish Shute: Yes.  I, again, didn’t see anything really of the business tracks and production tracks.  Did you get back and forth between the tracks?

Bruce Sterling:  I went to the Hollywood tracks.  I mean, to the extent that I could.

Is Hollywood stirring? Who’s going to have the first breakout AR property?

Tish Shute: So what did you see from…Is Hollywood stirring?  Is it waking up?  I mean I know Kent Demaine, Oooii,  and Brad Foxhoven, Ogmento, spoke about the Hollywood AR scene.

Bruce Sterling:  There were guys there from LA who were sort of saying, look…they are aware of us, but they just want AR to promote their properties to some particular niche.  They realize that AR is potentially a mass medium and that you could do some real AR entertainment.  So they were batting around some ideas as to where that might happen.  Like, could it come out of a console gaming scene?  Who’s going to have the first breakout AR property?  A popular hit AR property, as opposed to like a neat way to sell shoes, or whatever.   Really, anybody’s guess is as good as theirs or mine.  But at least they were actively guessing.

Tish Shute: I know the breaking the fourth wall discussion has been going on for a while and now the question is, whether AR is going to take down the fourth wall and bring interactive storytelling into the mainstream.  Did you hear any of that?

Bruce Sterling:  Well, I always shy away from discussions of that kind because I don’t think there’s any “final thing.”  Practically everything that AR is involved in right now is  a transitional technology. Also, because I am a storyteller, I get alarmed whenever people in technology start saying, “Oh well, it’s all about telling stories.”  Because obviously it isn’t.

People can tell stories perfectly well orally, and absolutely nobody does that.  AR is not at all about telling stories.  It’s about a great many other things, such as user bases, niche audiences,  media saturation, urban informatics,  convergence culture, and the language of digital media.   I could list these factors until the world looks level. It’s really becoming pretty chaotic.  As I was saying in my speech, AR companies are media startups who almost never use the old-fashioned word “media.”

Tish Shute: Oh, that’s interesting.  Yes.  So why do you think that has happened that way?

Bruce Sterling:  Well, it’s because they are trying to do a different thing than media does.  I mean, they are trying to “augment reality.”  They don’t want you to know that you are using a medium.  They don’t want you to realize that you’re watching computer animation overlaid on some video acquisition stream.  That would defeat the whole point of AR.  It’s entirely different from an analog medium like television, where you turn on the television and there’s a constant stream of station identification alerts.   That’s like: “Don’t touch that dial!  You’re on channel 13!  Stay with us!”  Then it’s like, “And now a few words from our friendly sponsors!”  That medium was engineered to keep your eyeballs locked to a single stream that they’re feeding you.

In AR, it’s much more participative, more geolocative.  I’m not particularly interested in station-identification branding from my AR provider. What I really want to see is the interactivity of the augments they’re bringing to me.  It’s like  FlickR, the photo sharing site. You don’t have any TV-style splash page for FlickR.  “Hi! We’re FlickR! FlickR, bringing your photos to you!” No, FlickR is all about “you, you, you,” your photos, your tags, your friends, your activity around you.   It’s immediately trying to be very participative.

Tish Shute: Will Wright got to that point, didn’t he. He was trying to move us into an idea of blended reality. That the game is about the world, not about the dragons or the overlays per se.

Bruce Sterling: Right. I think that’s true. But see, the world isn’t a medium. A medium is something like this interview,  where I’m connecting to you and there’s a video Skype channel between us.  Whereas AR is more about spatial 3-D,  about 3-dimensional impositions.  Pieces of media: sound, vision, information visualization, tags, floating tags, air tags, icons, arrows, warning signs, warning sounds, tactility, whatever, being brought into the environment around us.

That’s why it’s properly called “augmented reality” instead of just augmented media.   If you call your work “augmented media,” you’re really in trouble. Because if it’s all about augmenting somebody else’s media, why doesn’t that medium just buy you, and augment their own selves?    If you think that way, instead of augmenting the world, you’ll just be a modest little plug-in for old-school media.

The World as the Platform

Blaise Aguera y Arcas, Microsoft, Santa Clara, are2010, photo from Chris Cameron’s Flickr stream

Tish Shute: Yes, which is why Blaise so generously gave the technical underpinning  for augmenting reality in his tech talk – about the trellis and the grapes,  he really explained how the world can become a platform for augmented reality.

Bruce Sterling: I wish I could have seen that. I did not see Blaise’s speech.

Tish Shute: We’re going to put the videos up in better quality.  People in the front row have put it up on the web already.  He really went into some of the challenges of mapping for augmented reality.

Bruce Sterling: His visual-mapping technique is important.  Registration is super important for AR.

Tish Shute: I think it was a really generous talk actually because he went step by step on how we will do this.

Bruce Sterling: I rather imagine that Microsoft has patented those steps.

Tish Shute: Oh, yes, I guess so!

Bruce Sterling: I could be wrong. Maybe they’ll open-source it. You never know.

Tish Shute: You never know. Because the world as a platform isn’t something one company can own, or go it on their own to exploit.

Bruce Sterling: I expect there to be a thorny path, but sometimes I’m surprised. Sometimes people really do try to fertilize the tech field in the hope of getting a good corn crop before they start fighting.

Tish Shute: We’ll I keep hearing that we may even see the unlikely marriage of Apple and Microsoft  – maybe wishful thinking, but there are motivations beyond AR for this unlikely match, and certainly between them these titans have what it takes to realize the grand visions of AR ? [laughs] But who knows…

Bruce Sterling: Well, yeah, it depends on where the thing catches fire.

Tish Shute: Yes. You mean whether AR catches fire in the form of  AR and mapping..

Bruce Sterling: It’s hard to say, but I’m convinced now that there’s more going on than I once thought. I thought that Bruno Uzzan made a very good speech for his company when he talked about how he worked on AR for eleven years.  Eleven years is no flash in the pan.   He has his long list of clients and successful applications. I thought he was right in his impatience with the press for not catching on. It’s gone on for quite awhile. The mere fact that you’re not aware of it, doesn’t mean it doesn’t exist.

The Illusive AR eyewear

My are2010co-chair, Ori Inbar, CEO and co-founder of the hottest new AR game development start-up, Ogmento, donning his goggles to open are2010 – picture from Chris Cameron’s Flickr stream

Tish Shute: Yes. So, the other theme you brought up in your opening keynote and I would be interested to know if anything you saw at are2010 changed your view is the illusive AR eyewear, and  if we actually got AR Goggles that worked they would bring AR’s gothic sister, VR, back from the grave right? [laughs]

Bruce Sterling: Right.

Tish Shute: It took quite a lot of work, but we pulled together a six-company HMD panel, right?

Bruce Sterling: Yeah. I was impressed to see so many of them there.  And I was chagrined to see how prototype-like all their gadgets were. But that doesn’t surprise me, because if any of those head-mounts were remotely working, they would be hyped out the wazoo. Everybody’s been waiting for them and hoping for the best. They’re obviously not ready for prime time. [laughs] Maybe in certain limited applications. Like maybe a diving mask. [laughs]

Tish Shute: No, I think what was nice though they got inspired and they all got together on the last day. I saw them having a meeting about standards. They got inspired to actually work together.

Bruce Sterling: Yeah, well, unless they’re going to invent mechanical eyeballs that those machines can fit onto, it’s going to be tough. OK, I’m a skeptic, but I’m prepared to be surprised. I’m also a skeptic in Artificial Intelligence, but as soon as they bring me an AI that can write a decent novel, I’m going to get it and review that book. [laughs]

Tish Shute: It’s interesting. Re AI, I’m totally in agreement with you. In terms of the way computers turned out, it wasn’t AI per se that they turned out to be good for, not in the way everyone had dreamed of it, rather it was the harvesting of human intelligence that turned out to be the big thing. But what is interesting is that despite all of that, AI or machine learning, as it is now called, permeates our whole society now from the stock market to how many businesses make many of their decisions.

Bruce Sterling: Well, there’s a lot of so-called collective intelligence.  But Marvin Minsky-style hard AI, no way. Alan Turing-style AI, forget about that.

Tish Shute: Yeah. So, that’s an interesting comparison with the HMDs.

Bruce Sterling: People stretch the definitions.  It’s like, well, my car engine is Artificial Intelligence. Yeah, so is your wall transistor. No, I don’t really think so.

And AR is a similarly big tent. I mean, Uzzan had to admit that he had denied that AR was AR, unless it was using his favorite technology. And he felt embarrassed to be rubbing shoulders with people who put AR into cell phones. And I can understand his feeling there, because, gee whiz, that’s certainly not what AR pioneers had in mind. But he had to admit he’d become more ecumenical about it. Obviously, they’re  there and doing business like gangbusters. You can’t very well ignore success, right?

I had a similar feeling about the goggles. Obviously, the goggles would be great, should they work. But if they did work, I rather think virtual reality would come very strongly to the fore.  You’d see people doing all kinds of elaborate immersive-style stuff.   A truly immersive technology doesn’t need to “augment” much of anything.

Tish Shute: Yeah, you’re right.

Social Augmented Experiences

Bruce Sterling: I think many of the most interesting AI aspects are not personal in the way goggles are.  They’re not about guys walking around with personal tech. They’re about big, communal, social-media experiences, like stage shows, and urban informatics, things where large numbers of people can interact with the same augmented reality. The projection mapping, which I go on and on about. Augmented public spectacles.

Tish Shute: Yeah, projection’s our best example of a social augmented experience right now because we are yet to have an easy way to do networked social augmented experiences easily – but that is of course the thrust of my interest in ARWave [see the slides for my presentation, AR Wave:  Federation, Game Dynamics, Semantic Search, Mobile Social Communications here].

Bruce Sterling: I think of Edison’s early days, when he wanted to sell movies to people for a nickel a clip.  You had to bend over and put your eyes on this visor and turn this crank. That coin-op device was easy for Edison to monetize, as opposed to getting a bunch of people to sit in theater seats. But people laugh at movies when they’re together in the seats.   Cinema is a more social, involving experience in a crowd situation.

Tish Shute: But it started with them, didn’t it, Hollywood – the movie biz? Basically Nickelodeons, right?

Bruce Sterling: That’s right. They were Nickelodeons. They were a lot like the goggles because they isolated the user.

Tish Shute: Yeah, that’s a really important point that the goggles are not Nirvana because of this question of whether they actually detract from the social augmented experience and blended realities, by drawing us into VR experiences?

Bruce Sterling: I’m tempted to claim that they’re more a VR technology than an AR technology.

Tish Shute: That’s a very interesting point because…

[thunder]

Tish Shute: Wow! What was that?

Bruce Sterling: Thunder storm.

Tish Shute: Oh, my God, how very Gothic! [laughs]

Bruce Sterling: It can get pretty loud up here in the mountains.

Tish Shute: Oh, you live in the mountains, better still!

Bruce Sterling:  Torino is in the foothills. This is Piemonte. So the Apennines are over there. The Alps are over here. We do get some rather spectacularly unstable weather.

Tish Shute: It sounded like a bomb to my NYC ears. [laughs]

Bruce Sterling: Yeah, it didn’t hit the building, but it was maybe half a kilometer away. I saw the flash.

Tish Shute: Oh, you did?   Well, I hope you don’t lose your power midstream here.    I was really happy to hear of that connection between Rudy Rucker and Layar  [Rudy was touched when Maarten Lens-FizgGerald from Layar said that he met the Layar co-founder at a Rudy Rucker lecture].

Bruce Sterling: That was very fun, yes.

Tish Shute: Wasn’t that wonderful? What was that experience like going around the conference with Rudy?

Bruce Sterling: Well, you know, Rudy’s very into graphics. He’s a mathematician, so he understands the underpinnings of this stuff. But he’s a skeptic. He thinks they’re kid toys. He’s not a gamer. He’s a good old-fashioned computer-science hacker. So he wanted to tell me all about his new eighth-order, fifth-dimensional fractals. He showed me a great many of them. They’re psychedelic. Rudy’s fractals are considerably trippier than most apps that help you find a barber or a train station. [laughs] Rudy really is a visionary. He’s into some very weird stuff.

Gamer Guys at are2010

Brad Foxhoven, Chief Marketing Officer, Co-Founder, Ogmento at are2010

Tish Shute: At are2010 there was a lot of discussion about how game dynamics and AR are going to intersect, right? Anything that you saw of interest there?

Bruce Sterling: Well, obviously, there are gamer guys there. Ori’s a gamer. The gamer guys are getting some money. The big buzz right now in gaming is, of course, social gaming.  Farmville has kicked everybody’s ass because it’s not even a game and yet it has more users than the entire gaming industry.

Tish Shute: I know, right! [laughs]

Bruce Sterling: Obviously that’s kind of humiliating. For a long time, I’ve seen people trying to do giant multiuser games on cell phones. It’s difficult to do because the interface on cell phones is crap, right? People aren’t going to run around responding to SMSs.

I can imagine people running around with little Wii-style bats that have audio and visuals on them. It makes a very large native AR game seem more plausible.

Tish Shute: Yes. that would be cool!

Bruce Sterling: Again, it’s not very gamelike to use those little fiduciary markers.

Tish Shute: No.

Bruce Sterling: Moving little cardboard chips, around like with card games…. It would be pretty easy to set up a little AR chess game.  Star Trek style hologram chess pieces,  and so forth. But it’s just cumbersome.

Tish Shute: And also, from what we’ve seen from things like Foursquare, the proximity based social gaming doesn’t have to offer very much [a crown badge, a mayorship] to get some mind share.. the social is the primary game dynamic…

Bruce Sterling:  I’ve seen a lot of different philosophies of gaming over the years. Who’s to say that Second Life doesn’t have the best idea? They built a little scene and then slammed their gate shut behind them.  But at least they’ve got a really nicely-paying little cult stuck in there. It’s different. And it’s manageable and it’s really theirs, theirs, theirs.  They don’t have to call in outside experts to try and run the monster.   They haven’t blown it up to the scale of Yahoo! where they’ve lost control of the enterprise, and gone into a tailspin of management overhead. Second Life has a very intense, almost a cultish atmosphere among the player-slash-developers.

Tish Shute: One thing that helped them was the thing they were always criticized, that the barrier of entry was so high. But once they got people they never left, right?

Bruce Sterling:  That’s not a bug, that’s a feature.

Tish Shute: One of the best features!

Bruce Sterling: Yeah, it’s like being in Mensa. Why don’t you lower your barriers to entry and get in some interesting stupid people?

Tish Shute: [laughs]

Bruce Sterling: In Mensa, we’d rather sit here making puns about neutrinos and fourth-order quadratic equations. [laughs] OK, that’s a business model, if that’s what you want.

The Man With the X-Ray Eyes!

Jesse Schell’s during his keynote, “Seeing,” at are2010, picture from Chris Cameron’s Flickr stream

Tish Shute: Ok!  Now to unpack the man with the x-ray eyes idea, Jesse Schell’s keynote theme.  This is a root metaphor for AR – making the invisible visible, seeing through walls. To me. I think you kind of wrote the book on this because all my ideas on what radical transparency might be come from you – your idea of Amazon.org is key to how I understand this..

Bruce Sterling: Oh, really? That’s funny.   I was touched that Jesse brought up that famous Corman film, because I was a judge in a fantasy film conference in Trieste earlier this year.  And Roger Corman was there.  He was the guest of honor.   “X: the Man with the X-ray Eyes” was one of the films shown during the conference, and I saw it.  I even had dinner with Roger Corman.  I had never met him before, so that was quite amusing.  The difficulty with a film of that kind is that what we science fiction writers call a “House of Cards Ending.”  In that story structure,  you ramp the thing up until the protagonist sees God, and then he has to be destroyed by the falling pillars of the temple.  That’s a classic science fiction structure:  like Frankenstein.  For the sake of the drama, Corman evades the issue of what’s really going on. For instance, let’s just suppose “the Man with the X-ray eyes” is not in fact a psychopath.  Let’s say he gets a grant from the Department of Health and Human Services, and he acts like a real scientist, not a stock B-movie “mad scientist.” So he has, like, backup guys, and some placebos, and a large group of people to test it on, trusted colleagues, and so forth.  You wouldn’t get any of that movie’s wild activity out of that.  What you would get is like a 5% improvement to people’s vision.

Then, in a year, there would be a 10% improvement in people’s vision.  There would be a  classic industrial story.  A rising star, you know, a cash cow.   Real tech isn’t done by a single guy as a divine curse.  It’s created by classic  tech startup culture.  So a runaway technology really behaves in the way that personal computers do.

Tish Shute: The things that get me all Utopian and happy about this are the ideas like those you first outlined with the notion of Amazon.org.

Bruce Sterling:  It would be easy to do an entirely different kind of film than “Man with the X-ray Eyes.”  Something much less B movie,  much less pat.  I mean, at the end of the film,  he destroys his own hardware and blinds himself.  Why?  For what rational reason would he do that?  Why doesn’t anybody else know the big secret of what he’s doing?  Why aren’t there Koreans doing it?  Why aren’t there Austrians doing it?  Why aren’t there Italians doing it?  Why?  AR doesn’t behave like that.  It’s not one lone guy with magic eye drops.  It’s entire teams of people that have been working on stuff for 17 years.  They all approach it in different ways.

Now, they are going to get scandals in AR.  I can guarantee you that.  They are going to get into  hot water eventually.  At least some people will surely come out and accuse them of being Roger Corman B movie monsters.  But unless they accidentally discover atomic fission or destroy the Gulf of Mexico with an oil spill [laughs], I don’t think they’re going to be particularly badly off!   The trouble I imagine  for AR people is very typical new media trouble.  It’s like movies being accused of corrupting our morals, or comic books being accused of leading to violence, or Google being accused of making us stupid and warping our brains.

I’m not an alarmist in that sense, but at least I’m concerned about real threats.  Roger Corman is a B-movie director who’s trying to sew up his lost plot ends by destroying his hero and his hardware. That’s not very plausible. It’s a nice science fiction movie device, but technology isn’t a movie.

Tish Shute: Yes. Well, the other thing that you always remind us of with AR is not to be saying it’s going to be this glorious moment when it’s no longer gimmickey, no longer pop culture. You always emphasize that’s actually part of what’s good about it.

Bruce Sterling: It’s not an accident that practically everybody in that audience knew about Roger Corman.  Nobody looked surprised; not the Austrians, not the Koreans. They were all like: “Oh, yes! Roger Corman!  Love him!”

Tish Shute: There were so many Rudy Rucker fans. Were you watching Twitter? People like Eric Gradman were succumbing to fanboyz moments..

Bruce Sterling: “Yeah. Rudy Rucker, he’s the best.”

“Rudy Rucker gripping an Augmented Reality shoe” from brucesflickr

Tish Shute: [laughs]  I noticed you inspired him to join Twitter..

Bruce Sterling: Well, I’ve got 8,000 followers and, obviously, a lot of them are Rudy’s fans.  Of course he’s going to be gang-rushed on Twitter. That’s not really any more surprising than two motorcycle stunt guys at the same attraction. And I’m a big fan of his Rudy’s blog.   He’s always got interesting things to say.

Tish Shute: Yes. AR does seem to bring out some of the coolest smartest people!  This morning I had breakfast with Joshua Kauffman in Central Park.  He is an advisor and entrepreneur working on design in the public sphere.  I was feeling rather brain dead and jet lagged.  I told Joshua I was wondering how to get the cottonwool out of my brains for this interview and he suggested,  the All Souls College one-word question interview!  Have you ever heard of that? – although apparently they recently scrapped it.

Bruce Sterling: Well, I’ve heard of All Souls College there in Oxford. What was their interview question?

Tish Shute: They used to use only one word, so they would only give you one word. It’s not a question. Basically, they throw out the word and then you had to spin off from there.

Bruce Sterling: You’re supposed to free-associate on a single word?

Tish Shute: I guess so. I hadn’t heard about it, but Joshua suggested it.

Bruce Sterling:  Well, it’s possible..

Tish Shute: Joshua came up with some good words..

Bruce Sterling: Right.

Tish Shute: We were talking about these proximity-based social work networks like Foursquare and Gowalla and how they may influence the emergence of social augmented experiences.

So Joshua’s suggestion for the first word was “territorialization” e.g. how do these new mobile social experiences like Foursquare,  and the observation that actually rather than breaking down territorialization, which would be a good thing, tend to support territorialization…but perhaps new forms of territorialization?

Bruce Sterling: Yeah, they’re re-intensifying it in a very odd, electronic fashion.

Tish Shute: Yes.

Bruce Sterling: I have noticed that.  It’s not true of stuff like projection mapping or the webcam fiduciary display stuff. But with the handheld stuff, and especially the urban informatic stuff, it really can’t help but take on a local flavor. Layar is like “Augmented Dutch Reality.”

And TonchiDot really is “Augmented Japanese Reality.” It’s hard to imagine a Layar interface going gangbusters at Tokyo.  Whereas the TonchiDot interface, which is very clearly influenced by Anime and cartoon graphics…. Maybe it could find some niche of hipsters in Amsterdam hash bars…

…to be continued in Part 2

The Augmented Reality Event: Bruce Sterling’s keynote from Ori Inbar on Vimeo.

Bruce Sterling’s keynote (aka the prophet of the augmented reality industry) set the bar high on the opening day with a keynote address reminding us all of how awesome it is to be the “world’s first pure play experience designers,”  but also not to forget “whose reality needs to be augmented most,” and “to cut ourselves a space of our own.”

“It’s 9 am in the augmented reality industry…without vision people perish…it’s your chance to build a big pie before you start slicing it up…it’s time for you to get dressed…good luck to you, I’ll be watching you”

Wow! Augmented Reality Event was hotness.  I know, as one of the chairs, it is not like I am a neutral observer.  But, if you were there, and you didn’t pick up on the “warm, affirmative feeling,” as Bruce put it, and the optimism, fun, and cool uber geekery, of are2010, I guess I will have to hear from you in the comments ‘cos the feedback has been incredibly upbeat, so far.

Anyway,  I am still basking in that rare glow of a geekgasm at 9am – the augmented reality industry embracing lovers whose dreams are in still in full flight – no burst bubbles, betrayals, train wrecks, or tragedies in sight yet?  Although Roger Cormanesque horrors were threaded through  Jesse ‘the man with the x-ray specs” Schell’s  closing keynote, “Seeing.”  So keeping watching Ori’s Vimeo stream for Jesse’s keynote  -  it will be up soon and it’s quite a ride!

I leave for England tomorrow to celebrate my mother’s eightieth birthday, so this will be a brief post for now.  But I am looking forward to posting soon a long interview with Bruce Sterling on his experience of are2010.

Bruce Sterling hung out with with Rudy Rucker (see Rudy’s post and pics on are2010 here), attracting fans wherever they went, visiting the expo, going to sessions,  talking generously with all – their table always crowded.  Bruce said he hadn’t had so much fun in a while.  And, apparently, Rudy was touched when Maarten Lens-FizgGerald from Layar said that he met the Layar co-founder at a Rudy Rucker lecture. Awesome!

As Ori says here:

“are 2010 is over. It was a blast.  Many thanks to 400 AR enthusiasts who joined us for 2 days of AR goodness.

Special thanks to our 90 speakers from 40 augmented reality companies, our exhibitors, sponsors, and above all – for Qualcomm.”

People’s slides will be up on the are2010 web site soon. My presentation, AR Wave: Federation, Game Dynamics, and Mobile Social Communications, is already up on slideshare.

You can catch up on some of the highlights of are2010 in Rouli’s linkfest, and check out these pointers he gave to following the event for more.

• Chris Grayson’s twitter account
• Patched Reality’s Patrick O’Shaughnessey and the augmented citizen Dan Romescu are also doing their share on twitter.
• Sophia Parafina ( @spara ) live blogs the event using Google Wave.

I will have a mega post up soon.  But for now here are a few photos that will give you a taste of some of the #are2010 magic.

Bruce gets a t-shirt from tonchidot CEO, Iguchi Takahito ( @iguchi ), after checking out Helen Papagiannis’ AR art

If  an event with Bruce Sterling, Rudy Rucker, Jesse Schell and Blaise Aguera y Arcas there wasn’t already enough awesomeness, check out how Sophia Parafina, Locatively, and Patrick O’Shaughnessy of Patched reality got to hang out with Will Wright – hacking the Parrot AR Drone.  And then, when a dead power supply stopped that adventure in its tracks, Will played some of Patrick’s AR games and gave him feedback.  OMG!  Will’s only the most important game designer the whole world.  There is already some videos shot from the front row up of Jesse Schell’s, Blaise Aguera y Arcas’ and Will Wright’s keynotes up, but the hi res versions are coming soon!

I have been so busy working on Augmented Reality Event, Jun 2nd, 3rd, Santa Clara, CA, in recent weeks that I have barely had time to post!  But it’s getting hard to contain my excitement about ARE2010.  If you haven’t already seen the sneak preview of the schedule  – see here.

Augmented Reality Event will be an unique opportunity to see how the complete vision of AR is emerging, one that will include visual recognition of real life objects, sensors to enable interaction with physical objects, and the long anticipated, comfortable, sexy, AR eyewear.  Six companies with be presenting and exhibiting AR glasses and HMDs at Augmented Reality Event.

Keynote luminaries will include: Bruce Sterling, “AR Prophet,” Will Wright (Spore, The Sims & now AR and The Stupid Fun Club), Jesse Schell (who will give the AR follow up to his now viral Dice talk), and Blaise Aguera y Arcas (Microsoft, Bing), whose talk on augmented reality maps was one of the most popular talks at TED this year.

Over 40 AR companies from all over the world are represented, and there will be speakers from Qualcomm, Google, Microsoft, Nokia, Intel, and Nvidia.  Over 80 augmented reality experts and entrepreneurs are presenting, including a AR titans, Total Immersion and Metaio.   Mobilizy, Layar, Ogmento, TagWhat, Tonchidot, and Zenitum (just to name a few of the stellar augmented reality start ups coming – for more, and a list of speakers see here) will demonstrate AR is not just a vision, but actually drives significant businesses today,

Five Augmented Reality industry start ups will be on the hot seat in a Launchpad competition with $10,000 prize for the winner, thanks to the generosity of  Qualcomm, a leading developer and innovator of advanced wireless technologies, and the featured sponsor of Augmented Reality Event.  Zenitum, a leading augmented reality company, with a technology portfolio that includes both location and advanced vision-based sensory input is the Gold sponsor for ARE2010, and Adobe are sponsoring the ARt Gala & Reception.  Many thanks to the sponsors for making ARE2010 possible.

Readers of this post can use my discount code TISH245 to register here for a $245 price for the whole 2 day event.

But just to add some balance to my total enthusiasm for the rocket like trajectory of augmented reality in recent months, this post will also look at an area that has been, to date, disappointing for AR developers.   While Nokia will have a couple of speakers at ARE2010, and certainly they have a very impressive history in augmented reality research, they have not delivered the magic brew to attract the augmented reality developer community.   Will this change with the release of the N8 (see video opening this post)?  I wanted to find out if AR developers see a light at the end of the tunnel with N8 as a platform for AR.  So I asked some AR developers and entrepreneurs, “How attractive is the N8 for the AR community?”

Nokia’s apparent nonchalance about augmented reality is in sharp contrast to the bear hug shown in this Verizon Droid Augmented Reality Ad featuring Layar!  As Bruce Sterling notes,

*Hollywood-up for Layar by cooking up some o’ those big-time sci-fi production values.
All magic, all the time, Mr. Roboto.

And it is clear that everyone has high hopes that iPhone SDK 4 delivers fully for AR developers this time! For more on this listen to @chrisgrayson’s 3rd video on Mobile AR here.

But before I get back to my question about the AR community’s thoughts on N8 as a platform for augmented reality, let me crow just a little more about the upcoming awesomeness of Augmented Reality Event.  As the chair of the technology track, ARE2010 is on my mind night and day at the moment!

The tools for the kind of augmented reality we have always dreamed of  – the zero click interface to a heads up, hyperlocal view, bringing you computing anywhere, anytime, with anything, are evolving fast.  Powerful algorithms for search, and machine learning, combined with cooperating cloud data services, will soon be bringing apps that learn by context accumulation to your AR view (see Siri).  Google Goggles, who will have a speaker at ARE2010, have already announced their intent to open APIs, putting a vital tool of visual search in developer’s hands.

A bunch of technologies are maturing these days and helping bring AR to consumers.  Mobile devices that pack all the ingredients for AR, location-based services, see thru video goggles, visual search, innovations in 3D mapping, Simple Geo’s innovative a approach to aggregating geo data, and the next generation of “Street View” such as Earthmine, and, very importantly, open standards and software for AR that will bring many new opportunities for creativity, and monetization.   These are just some of the aspects of this energetic technology convergence that will be represented at  Augmented Reality Event.

In the area of open standards, I am especially excited about the debut of Polaris at Augmented Reality Event.  Polaris is an AR browser, from Blair MacIntyre and the Georgia Tech team, based as much as possible on existing standards and tech.  More on this and ARWave in another post!  Also, I am excited to be demoing ARWave at the Com.Geo 2010 conference, June 21 – 23, in Washington.  Again more on this later.

But for now back to my question opening this post:

“Are Nokia back in the AR?”  and “How attractive is the N8 to the AR community?”

First I asked, my co-chair on Augmented Reality Event, Ori Inbar of Ogmento – one of the augmented reality start ups who’s star is on the rise, and Ori (also known for his writing on Games Alfresco) is one of the key thought leaders in the emerging augmented reality industry.

Ori Inbar:

“Well, the hardware spec is impressive – but that’s not what makes it a good AR device. What’s really important is reach, distribution, and ease of development – areas in which the N8 still has a lot to prove.

- Will the N8 become as prevailing as the iPhone or lead Android phones or will the Nokia landscape remain fragemented with too many variants of phones to support?

- Will the Ovi store pick up steam and become a lucrative chanel for selling apps?

While we wait for these questions to be answered – we’ll keep developing games for Android and the iPhone “

So there you go.

Ori is right that augmented reality developers like the hardware specs.  Here are some comments from Patrick O’Shaughnessey of Patched Reality, Peter Meier, Metaio, and whurley of Chaotic Moon and Robert Rice, Mirascape.

Patrick O’Shaughnessey:

“Looks like it has everything one would want to make both looking glass and magic mirror AR, and built-in face recognition software to boot!  I wonder if they allow you to use both cameras at once making AR video conferencing possilbe?”

“Dedicated graphics processor with OpenGL 2.0 enables 3D graphics” nice!

“Looks like it also does Flash (Flash Lite 4.0, anyway). Not sure if Flash Light provides access to the camera. Could be a nice way to do quick and dirty POC’s on the device”

Peter Meier:

“Metaio thinks the N8 is a very promising device for AR. Especially in Europe we expect a lot of reach.  Symbian is hard to develop for, but on the other hand is much more efficient in terms of real time performance than Android and less restrictive on APIs than iPhone.”

whurley:

“It looks really great and perfect for ar in many regards.”

“clarity of a 12 MP camera with Carl Zeiss optics will give the Nokia N8 some clear advantages in the AR world.  The clarity of this camera will allow for more advanced applications of augmented reality such as image/object recognition”.

“At first glance this is definitely very impressive from a hardware specification. I am surprised though, at the lack of buzz about the device. I have heard virtually nothing about it in the usual sources I monitor, and I wonder about the marketing.

Given the technical superiority and the difficulty Nokia has had penetrating at least the North American market, maybe something catchier like the GR-8 (Nokia Great) or M8 (Nokia Mate) might be a bit more user friendly and easier to market. I particularly like “Nokia M8″…take it with you wherever you go, can’t live without it, etc.

Anyway, as awesome as the specifications are, I would like to echo some of the other comments here…tools, community, ease of development, etc. are critical parts of the ecosystem that would make this ideal for AR development, above and beyond the pure hardware specs. I think Nokia has to really re-think some of its strategy to engage consumers, attract the developers, and establish a new beachhead in North America.”

Talking with Michael Halbherr at Where 2.0

I talked to Michael Halbherr briefly at Where 2.0 before the release of details on the N8, so, of course, he was unable to be as explicit as he might have been about the N8.  Below is a lightly edited transcript.  At Where 2.0, Michael Halbherr  gave a clear explanation of  Nokia’s emphasis on a global strategy.   And, as this blow by blow account on the Google Earth Blog noted, “They’re doing some cool stuff with their handheld maps. In particular, their Ovi Maps (turn-by-turn navigation) are now completely free and becoming very popular.”

But the question remains:  Where is AR is on the Nokia roadmap?

Michael Halbherr (Nokia) on stage at the O’Reilly Where 2.0 Conference 2010 in San Jose, California. Photo by James Duncan Davidson. See Where 2.0 Flickr set here.

Tish Shute: Can I ask you a few questions about Augmented Reality?

Michael Halbherr: So for me augmented reality and mixed reality is essentially a way you search.  So when you look at where we come from, as the company that acquired Gate 5, we believe the phone is essentially the gate that basically combines or connects your virtual and your real world in both ways.  So we can have virtual overlays on top of reality, and you can also record and push it back.

But when you look at the way we do it, I think it is actually an extension of mapping, right?  So you will have the map, which is an abstraction of reality, and then basically with a flick you can go to real reality.  And I think to really understand what you see, you need to actually really know what you see, and an image doesn’t do that.  But if you have compass, and GPS, and 3D models, which is what Navteq is capturing, we can actual do line of sight calculations and you can really say, “OK.  He is looking at that building.” And you can click on the building because you have that understanding.  And you can combine that.  And that leads to a completely new way for how people interact with their environment.

So the here and now, and me here and now, and what next is a key innovation in our focus hereafter.

Tish Shute: I suppose for AR enthusiasts we were all disappointed when we saw the N900 with Maemo, which was very exciting as an operating system, didn’t have a compass.

Michael Halbherr: Yeah, but you know, that is just a matter of timing.  At the end of the day, Meego is our, I think, high-end push.  You will see this Mego device have what you need it to have.  So I wouldn’t worry too much about that.  We are building all the  phones, and everything has everything.

I think it is not just Meego.  It is the whole thing around it.  It is the QT, it is the web run time, it is the whole Meego system.

Tish Shute: Yes we have used QT for some development on the ARWave project, it is very good.

Michael Halbherr: Yeah, and the web run times, it’s open…So Nokia is about open and emotional.  Others are maybe open and more utilitarian or closed, and we are more open and more emotional.  And I think that is what we want to basically do with this platform.

And we are working very hard.  And so whatever I say has been built almost a year ago.  So there is a lot that we are building that we don’t talk about before we announce.  And we try to do this as a global offer, so we can’t launch a little service in the US, which maybe is a problem for us.  But we are basically building complete solutions for global markets.

Tish Shute: Yes.  And when I heard you talk just now on Nokia’s global strategy just now, it all fits in.  But from the point of view from someone who lives here in the US, it seems Nokia is disappointing us.

So is Nokia enabling augmented reality in any ways that noone else is?   I mean if you are interested in augmented reality, the backend and data is vital, e.g. the Navteq part.  And Apple has excited us with the amazing performance of their hardware, and they have taken steps like controlling the manufacture of their chips, powerful cpu and gpu is very important for augmented reality experiences too? Is there a need for a chip set geared to augmented reality specifically?

Michael Halbherr: I’m not sure you have to go that far.  I think at the end of the day it is the data, right?  The map data and the 3D data that is actually extremely important.

Tish Shute: Yeah, I agree with you that the data is vitally important.

Michael Halbherr: Because at the end of the day, when you have a real abstract view, it is really important, because the camera or the view is only a viewer, right?  And people don’t really know what they see.  It is just they plaster on it believing that the person sees what he sees.  So I think real augmented reality or mixed reality is really basically having a real picture, but having, actually, the understanding of the 3D logic of the picture.  And for that, you need to capture the data which Navteq is doing.

Tish Shute: The original dream of augmented reality is 3D media and graphics tightly registered to the physical world.  But actually, it is the data…being able to integrate the streams of data and sensor fusion techniques that will make this interesting and possible right?

Michael Halbherr: Again, it is overlaying virtual elements on real elements.  It is like you are taking your glasses and now you can see information attached.  And of course, you need to make sure it is not overloaded, and that everybody works with it, and it is precise.  So that is basically it.  So I think that is why we are investing a lot in the precision of the maps and the 3D elements of the maps.

At this point we had to leave our seats has the set was being broken down for the next event at Where 2.0.  Check out all the videos for Where 2.0 here,  it was a watershed event for all things hyperlocal, geo, and augmented!

The video above, The Imawik commercial, is a collaboration between In The Can Productions and Paige Saez for Makerlab

“The Imawik (ImageWiki) is a visual search tool for mobile devices. It allows for the ability to turn images into physical hyperlinks, conflating visual culture with a community-editable universal namespace for images.”

Paige Saez is an artist, designer and researcher.  In 2007 she founded Makerlab with Anselm Hook, an arts and technology incubator focused on civic and environmental projects.

Paige and Anselm (see my interview with Anselm Hook here, Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook) have been asking a very important question:

“Who Will Own Our Augmented Future?”

But most importantly, they have been actually developing applications (again see my interview with Anselm for more background on this), to allow people to play with, hack and explore and create with the physical world platform, and to imagine new possibilities for physical hyperlinking and augmented realities.  This is pretty important stuff, and kudos to Paige and Anselm for beginning this work before the big players – Google Goggles, Point and Find, and SnapTell came hurtling into the field of visual search and physical hyperlinking  – see this demonstration of translation and optical character recognition in Google Goggle’s.  Also check out Jamey Graham’s (Ricoh Research) Ignite presentation at Tools of Change, 2010 – Visual Search: Connecting Newspapers, Magazines and Books to Digital Information without Barcodes, for more see ricohinnovations.com/betalabs/visualsearch.

We are only just beginning to get a glimpse of how contested the social commons of the physical world platform is going to be – see the Yelp controversy.

As Paige points out:

“The lens that you are actually looking through was as important as what you were looking at. And democratizing that lens became the most important thing that we could possibly do.”

I am in total agreement.  One reason I have so much enthusiasm for ARWave (note: if you are interested in following the developer conversations there are several public Waves) is I see this open framework playing an important role in the democratization of our augmented views, by creating an open, distributed, and universally accessible platform for augmented reality that will allow the creation of augmented reality content and games to be as simple as making an html page, or contributing to a wiki.

Federation, real time collaboration, linked data – ARBlips that contain metadata that is usable for semantic searches, and modified wave servers that can listen to and respond to  SPARQL HTTP requests properly (see Jason Kolb’s many interesting posts on XMPP and Wave).  These are just some of the reasons why ARWave could revolutionize augmented reality searches and more! (see my presentation at MoMo13 – video here)

For more on real time social augmented experiences see our panel, The Next Wave of AR: Exploring Social Augmented Experiences at Where2.0 2010, and don’t miss the Where2.0 conference which has been the crucible for the emergence of location technologies.

Augmented realities, proximity- based social networks, mapping & location aware technologies, sensors everywhere, linked data, and human psychology are on a collision course in what Jesse Schell calls the “Gamepocalypse”   See Jesse Schell’s Dice 2010 talk here, and check out his Gamepocalypse Now blog.  As Bruce Sterling’s notes in his post here:

*Another precious half hour out of your life.  However: if you’re into interaction design, ubiquity, social networking, and trendspotting, in the gaming biz or out of it, you’re gonna wanna do yourself a favor and listen to this.

And don’t forget to register now for Augmented Reality Event (ARE2010 in 2-3 June, 2010 – Santa Clara, CA).

Bruce Sterling, Will Wright, and Jesse Schell will be keynoting, and there is a totally awesome line up of AR innovators and industry leaders, including Paige and Anselm!

And:

You are in luck!

Here is a discount code for the first 100 folks to register to the event (before the end of March). Go to the registration page, type in code AR245 and “you’ll be asked to pay only $245 for 2 full days of AR goodness.”

“Watching AR prophet Bruce Sterling, and gaming legend Will Wright, visionary game designer Jesse Schell deliver keynotes for this price – is a magnificent steal.  And on top, participating in more than 30 talks by AR industry leaders will turn these $254 into your best investment of the year,” as Ori  put is so well on Games Alfresco!

If you want a preview of just how exciting it is to be involved in augmented reality right now check out Ori Inbar’s great round up on our latest monthly Augmented Reality Meetup NY (or as, Ori notes, we fondly like to call it ARNY.)  There is lots of video up now (much thanks to Chris Grayson, who  live streamed it).  Augmented Reality Magician, Marco Tempest, is an absolutely must see.  (developers note this is an awesome use of open Frameworks and OpenCV).   The video of the show includes a rare explanation of how it all works  – see here.

Talking with Paige Saez – “Software is candy now!”

Tish Shute: What interests me about ImageWiki is that you have thought about physical hyperlinking beyond the obvious of where to get your next good hamburger and beer, right?

Paige Saez: Right. It was interesting for me in just thinking about the two things. How do you design a tool to work in a way that people are getting value from it? And also, how do you make it work in a way where people can explore and hack it? I think the most interesting technologies, and this is probably something somebody else said sometime, are the ones that disappear, that we don’t see, instead we see through. They become just the intermediaries.  They don’t interfere with what we are trying to do.

It’s a struggle whenever you are developing a new way for people to get information or make something happen, because you are playing with magic a little bit. And you have to make it vanish the way a good magic trick makes an experience a magical one. But at the same time you also need to reveal just enough that you let people in and they can see how to change it and make it their own. That is the interesting tension for this space right now, the idea of augmented reality begins to lead the idea of a social commons for physical things. The Imagewiki project was a locus of just this tension. Tish you and I have previously discussed how difficult it was to even get people to understand the two concepts independently.

Tish Shute: Right, until recently most people hadn’t even heard the term augmented reality and I am not sure that a particularly high percentage of people would recognize it now despite the recent interest in smart phone apps.

Paige Saez: It’s very difficult to get people to understand the two concepts, and now you are adding in the third level of participation as well. So I don’t think it is impossible, but I do think it requires narrative. It is interesting that you were talking about the stories you heard this morning from the creatives at the event [Tish mentioned David Curcurito, Creative Director, Esquire gave an excellent presentation at Sobel Media event NYC] because it’s narrative and the attention to telling a story that help you walk through all of the ways you can understand how completely expansive this area is right now.

So I think we have to play with it, play with the space and the tools. I think we need to have an idea of what we want people to use the tool for, and we need to not only introduce them to the tool and the technology, but also introduce them to the concepts as well. So I see it as a three part process.

I’m really excited to be there with people, helping them do that. I think we need to do this face to face. I don’t think this can be only through a social network. The ImageWiki website is like one quarter of the entire picture, you know? The website is the resource center and the place where you can see people adding images, but what value is it to you to see an added image? It is more valuable for you to be interacting with the image or interacting with the object in the real world.

Designing for the experience of using the ImageWiki got very complicated very fast. I was trying to figure out the main thrust of the design for the UI for the ImageWiki and at a certain point I had to take a step back and say “Okay, this has to be good enough for now because we can lay it out and prototype as long as we want on the Web or mobile UI. What we need to be doing is going outside and actually aggregating and putting images into the database in order to see what exactly happens when we are adding.”  It’s not just like you are taking a picture of something and adding it to Flickr. Using the tool is very context specific and the information is context specific, and you can’t necessarily make that all happen at the exact same time. I think these are really fascinating spaces to be struggling in and I’m so glad to be working in this space.

Images by Chris Blow of unthinkingly.com

Tish Shute: Could you explain why we need ImageWiki? I mean I think I have ideas on this, but perhaps you can explain to me from you point of view why we need an ImageWiki, as opposed, to say, extending the image space of Wikimedia or something added on to Flickr.  I mean maybe something leveraging the geotagged photos sets and APIs we already have?

Paige Saez: Yes, definitely. It’s a really good question, I mean it really is. Like, do you need an entirely new place to be holding images outside of the places that we are already holding images? That’s a huge question; enormous. Especially when you take a look at the problems around that. Its’ exhausting for an end user. Who the heck wants to go and reload everything into yet another place, right?

Tish Shute: Right.

Paige Saez: Moreover, who is going to really bother? Another problem would be what happens to the existing datasets that people have already committed to? And then of course there is the problem of authority and explanations why….Gaining interest and authority in a space when nobody even understands why that space should exist in the first place. And those are just three, you know, off the top of my head problems with that idea.

And yet at the same time, I don’t actually know how else to go about thinking about the ImageWiki unless I think about it as it’s own thing. Then you start thinking about models of large independant image databases that exist already, examples of this from a product standpoint- references to consider. The Getty Foundation comes to mind. There are many other historical centers that have huge resources and images that are licensed out and used. So here we have a working example of people already doing this. But succesfully? I don’t know. We do have a ton of intellectual property rights and copyright issues and ownership and use issues with images currently. As a working artist these issues for me were a major red flag to consider. Working on the social commons for augmented reality starts paralleling issues found in digital rights management and intellectual property.

Tish Shute: But one good thing about Wikimedia, why I focused on Wikimedia, is Flickr and Wikimedia already use a creative commons licensing, right?

Paige Saez: Creative commons, you know they have their own resource center, too. But you know they haven’t been successful as great databases for images so far.

Tish Shute: What would you like to see that they don’t have? Like say maybe start with Wikimedia, right?

Paige Saez: There’s just still a lot of issues with how to encourage people to want to contribute. It’s hard to show the value to someone who doesn’t already understand the value for some reason. At least for me personally this is something I have run into frequently. I don’t know if it is necessarily what Wikimedia doesn’t have, I think it is a lack of understanding of what creative commons really means. And there is still a very strong sense of ownership and concern about creative property rights. Being paid to be creative is a tremendously difficult thing to do. People fear losing their livelihoods. They think this is possible. Is it? I dunno.

For example : Look at me, I take a photograph of something, I can sell that. And there’s a question about whether or not, as an artist, I want to have my photographs in a pool of images that is open and accessible when I could be making money on it instead. Now that is just an example. Me personally, I can see the value. But that is a common concern. The gist of the question being, ‘what value does it bring to give something away versus holding on to it?’ A hugely popular discussion right now.

This is the same crux of the problem we are dealing with when we talk about thinking about images in the social commons for the real world. It’s a conversation about ownership. It’s about, who does this belong to really? If I take a photograph of a Levi’s billboard, does that photograph belong to me or does it belong to Levi’s? We know the boundaries of that. But when the image becomes a living image, an image capable of transmutation; an image that provokes an action or hyperlinks to a product, experience, information….where are the boundaries in that?

Tish Shute: But how is ImageWiki handling that differently from Wikimedia, I suppose is my question.

Paige Saez: We haven’t solved the problem.

Tish Shute: Yes, I suppose it is not like we have fully solve the problem of a creative commons for images on the internet let alone the issues of a social commons for the real world! So neither one has solved the problem, right?

Paige Saez: Exactly. To be honest, it made my head spin. I realized we were building a web application and a mobile tool doing augmented reality, real time feedback on the world and suddenly we weren’t. Suddenly we were dealing with DNS and talking about physical hyperlinks and ownership and property. And basically at that point you just have to sit and really start looking at catching up on IP issues and figuring out how to deal with that space in a much more wholistic way. It became so important that we had to take a step back and go

“Oh my god I think we have really uncovered a real problem here.”

At the point when we were building out the tools we realized something was really going on with our project. Here we were thinking that this was just a beautiful experience of learning about the world around us. We really…Anselm and I both just really wanted this tool to exist. It was something that we both just really wanted to happen in the world, something that we felt really just thrilled to make. And we looked at and used it and realized that instead of it just being a beautiful experience, it was a fundamental shift in how we understood everything. That it impacted our world in the same way the Internet impacted our world. It was a fundamental shift in understanding. A sea-change.

So I put down the prototype and went back to researching, read a ton of books on IP and went and presented to friends, family, schoolmates and co-workers trying to explain the project and then the larger conceptual framework that had emerged from the project. I began using the metaphor of thinking about Magritte’s “Ceci n’est pas une pipe.” Thinking about a pipe that isn’t actually a pipe.

Tish Shute: Oh, yes!

Paige Saez: ..to try to help explain to people that the image that you see is actually not, you know, it’s not an image of a thing. It’s an image. And that image has a tone and that image has a voice, and that image was chosen. And there were decisions that were made through the interface of the camera, specific decisions that defined the view of what you were looking at. And that that wasn’t being acknowledged and that that was a fundamental part of what the ImageWiki was aiming to do. The lens that you are actually looking through was as important as what you were looking at. And democratizing that lens became the most important thing that we could possibly do.

Tish Shute: So the emphasis for you on ImageWiki was in fact the lens, even though you found obstacles to creating the interface, right?

Paige Saez: Yes. Definitely. That’s what I fell in love with first. I really wanted to be able to use my phone to learn about what kind of tree this was or to buy tickets for the band on the poster I just saw, or see a hidden secret. For me it was very much a story, a narrative experience that I just thought was magical. And that is how I fell in love with it, which is not where I ended up. Where I ended up was realizing it was a fundamental shift in not only my own understanding of how to use the world around me, but in our understanding of looking at the world.

Tish Shute: It would be pretty scary if an image DNS was basically in the hands of either one or very few people, right? I mean even ImageWiki would be stuck with this problem, that if you set up a bunch of servers, you are going to be holding a very, very large image database. I mean, whatever your motivation, right? I think at the minute that is why I am very into seeing everything through the lens of federation, I see that unless we have federation, these giant central, databases are inevitable aren’t they?

Paige Saez: Essentially, yes. I mean I wasn’t able to walk through it as quickly as that. It kind of just overwhelmed me. Looking back on it, it seems perfectly obvious. I was just like “Oh my god, what have we done? Like what is going on?” Particularly for me because so much of my life has been spent in art, it was really easy to immediately understand the connection between the view, the viewer, and what’s being viewed as all just different layers of ownership and understanding that it is a gaze. Right? We know that we are never able to look at something without passing judgment on it, but to see that become a part of the interface in a real-time fashion just blew my mind.

Tish Shute: Yes.

Paige Saez: I think you are right. Getty Images, Flickr images, no matter what you are always holding on to something and you have to be responsible for it. Right? So how do you deal with the responsibility but don’t take on too much ownership? Where is the boundary with that?

Tish Shute: And for me, the simple answer to that is loosely connected small parts, distributed systems and federation. Because there is only one way to be able to utilize these things is to have them distributed so that no one holds all the cards. Right?

Paige Saez: Definitely and I personally agree with you wholeheartedly. However, the idea of distributed power is a concept that most people just don’t know how to deal with.

Tish Shute: And it’s easier said than done because actually the root problems that you are talking about aren’t got rid through federation, because if someone really holds the, sort of, all the good image databases just because they have the potential to be federated, they may not choose to open them up on many levels.

Paige Saez: And even then you have to think about, sort of, like the next level of it, which is we want it to be all open and accessible, but everything is owned by somebody. Like, what really is public anymore, in general?

Tish Shute: And what is interesting though, regardless of what we speculate conceptually on this, we already set off down the road. I mean we have already several large…they are all in beta I suppose, Google Goggles, Point and Find, right? But we have applications that are beginning to implement this. They are beginning to implement search on it, and it is geo-located even if it’s not in an augmented view, right? So it is proximity based.

Paige Saez: Right, right. I mean maybe the solution is that if we follow that line of thinking then Flickr will be partnering with Google Goggles. And then my images would stay under my ownership through the authority of Flickr. And I would use Flickr as my place to add images and they would just be responsive via my devices via AR.

Tish Shute: That’s very interesting.

Paige Saez: Definitely I think so. It is also the shortest distance between things.

Tish Shute: Yes, and as Anselm kept pointing out, basically it is going to happen in the simplest way possible, really, regardless of the implications of that. But OK, getting back to ImageWiki. As you say neither Wikimedia nor Flickr were really designed to take this role, right?

Paige Saez: Right.

Tish Shute: With ImageWiki, you’ve had these ideas and a concern with the social implications of physical hyperlinking in your mind since it’s inception. Are there any design ideas you’ve come up with that you know, as opposed to sort of, as you say, connecting Flickr to Point and Find, or who knows, Google Goggles. How is ImageWiki going to be different, do you think? Is that a hard question at this point?

Paige Saez: It is, and it’s a great question, and it’s a question I really love to think about. I think we have to introduce the politics with the tools. It has to be acknowledged that it’s not just a place to hold information, that’s what I feel in my heart.

At the same time, is that too much for people to really grasp at one time? In my experience it really has been, so the design of the experience needs to allow for an understanding of the power of the tool and the level of authority that the tool offers, while not getting in the way of it; just using it. Because ultimately, at the end of the day, nobody will use anything if it isn’t valuable to them. And so I could talk for miles and miles and miles about how important it is that corporations don’t own all of the rights to all of the visual things in my life, right? For the rest of my life I could talk about that. The idea that advertising is dominating all of our views of anything in the world around us is horrifying. It doesn’t matter unless I can show somebody why it matters to them or how it affects them. It’s just that that is a tremendously difficult thing to explain through a user interface.

And I actually think that it’s great that tools like Google Goggles and Nokia Point and Find are here to do a lot of the hard work of showing people how it works. Recently somebody explained to me their experience of using Google Goggles. They went through this process of saying how the Google Goggles took a picture and then did this really complicated visual scanning thing over the image and it took a full minute.

And I said, “Well of course they did it that way.” And they said, “Well what do you mean?” I said, “Well, what they are really doing there when they are doing all these fancy graphics, is they are showing you how it works.” And even if it isn’t actually related at all to how it functionally works, algorithmically, that’s not the point. The point is that this gesture of the time taken to make it look like it’s scanning an image and going back and forth with pretty colors is giving people the time to process that as an experience. That’s a metaphor for what’s really happening. And these kinds of metaphors are crucial with user experience design. We have lots and lots of examples of them and how they work, and many of them aren’t necessary. Like you know, for example, the bar that shows you the time it’s taking for something to process.There is no relationship between that and reality. But it is really important.

Tish Shute: Yes those bars often have no relationship between the actual time..

Paige Saez: And that’s the thing. Like the idea of time versus our perceived understanding of time. Right? The length of time it takes for your Firefox browser to open and load your last 30 tabs, versus the reality of what’s actually happening. When you are doing that sort of research you are actually accessing millions and millions of places and points of interest all over the world, so we need more of that. We need more of the process shown. Anselm and I worked with a film maker named Karl Lind from In the Can Productions here in Portland to try and make a video about the ImageWiki. We made this little video and I can try to show it to you or send it to you if you want.

Tish Shute: One of the issues with this kind of visual search is that it is inherently dependent on large databases, regardless of where they are federated, are going to be very large. Right? I mean someone is going to have something big, and aggregated there. I suppose someone will figure out the challenges of federated search eventually but that is quite a big challenge!

So I suppose I am still trying to understand what ImageWiki can offer that we can’t get with any other existing service? How will their be a social commons and even a social contract for the world as a platform for computing and physical hyperlinks?

Eben Moglen brought up something when I talked to him about virtual worlds, he said we need code angels to let us know what was going on in the virtual space – who was gathering data and how, for example.

Paige Saez: Tell me more about that, I want to hear more about that.

Tish Shute: Eben suggested this metaphor for when I was asking him about privacy in virtual worlds. The fact that people just didn’t know that when they were pushing avatars around virtual worlds what metrics were being gathered on their behavior. And he basically said that what we need is code angels when we enter these spaces because having the rules of the game buried in a TOC was ridiculous.

Paige Saez: That is a really interesting idea.

Tish Shute: Maybe ImageWiki needs to be our code angel to navigate the augmented world. I mean that’s what I want to see it as. And when I hear you talk, what I hear is you talking in broad categories about what a code angel might be in the space of images and image links to the physical world. I mean that is what I hear from you.

Paige Saez: Yeah. No, I definitely agree with that. It is interesting. In that sense, it is kind of a protection layer. Is that what you are thinking?

Tish Shute: Yes, I suppose because we can’t be navigating a lot of complicated opt-ins and opt-outs just to get around our neighborhood safely (in terms of privacy (also see Eben Moglen’s definition of privacy here…) We will need a code angel that is sort of keeping up with you in real time!

Paige Saez: Right, right. I wonder how that would work in regards to images, though. That is a really interesting thing to try and put on an image. I guess why I am having such a hard time being specific about it, is I am just trying to work it in my head, thinking of a specific use case, like what would be an example of that?

Tish Shute: Well I suppose the example, and this is a crude one, is when you point your Google Goggles to the book jacket, the code angel, this is very crude, would say “You are right now drawing images from the Amazon database – they are collecting data such and such data from your search.

And then of course the ability to have crowd sourced tagging and corrections..

There was a wonderful book that came out last year on how we can have commercial intelligence -Dan Goleman’s new book: “Ecological Intelligence: How Knowing the Hidden Impacts of What We Buy Can Change Everything”…

how corporations various different stakeholders, including their customers will drive corporations to do the morally right thing because they will lose the commercial support of customers who won’t support them unless they are more green, fairer, do the things we would like them to do whatever that happens to be – physical hyperlinking and tagging I guess would be a big part of this.

Paige Saez: Sort of a transparency issue. And that almost becomes a page rank algorithm in and of itself. I mean now we are really talking about search more than anything, and what tool becomes the dominant search tool. Anselm and I talked a lot about one platform… I mean eventually we will have a unified platform. It will…No matter what, for the Internet and for physical objects and visual objects in the real world. It will just be a matter of, literally, who can find the best and most valuable, most relevant information on a thing. Currently we just have it very proprietary.

Tish Shute: Yes.

Paige Saez: That definitely won’t last. It just can’t, because of the exact problem that you are raising. And we already know too much about resources and information as they pertain to products for us to ever go back to a time where we are not considering other ways of getting information about it anyway. Right?

Like I have the same concerns nowadays when I look at fruit. I look at a piece of fruit in the store. I would never just assume that the person who put the sticker on that fruit, anymore, is the ultimate authority necessarily. I would always assume at this point I could go online and go find out more information about a company. Issues about like eco-footprint or how much toxicity, or pesticides or whatnot are now totally accessible already.

So I am thinking when you look at that piece of fruit and that sticker for Google, say what you are describing, do we just go immediately to the company’s website, or is it even more specific? Do we know that the sticker on that piece of fruit is going to tell us specific information about that? Or are we just getting back the nutritional resources, or are we getting a listing of all of the different options out of a page rank algorithm that shows us, “Well this is the website for the fruit. Here is the nutritional information. Here are the last 15 comments on it.” It’s basically just a basic search.

Have you heard of Good Search?

Tish Shute: you mean http://en.wikipedia.org/wiki/GoodSearch

Paige Saez: Right.

Tish Shute: A code angel interface would have to give you options, wouldn’t it on possible views available?

Paige Saez: Yes. You are then talking about filtering your view. Then it really gets really interesting, of course. I don’t even know if we have a choice in that. I think we are really kind of hitting a wall with who owns the space and the platform. Is it just a basic search because we are already familiar with search? If you had an option to choose, say, “I want to look at this apple sticker and I only want to get…programmatically only looking at my friend’s opinions of this company.”

Or I have a safety valve on it that only shows me certain information based on what the code angel knows about me, my preferences, my age, things like that. Then that gets really, really interesting, because we are trying to do all that work right now just with social media and the Internet. We are already overwhelmed with too much information. It is already past the point of comprehension. So to think that we would actually drill down even more specifics is very interesting.

Tish Shute: That was a point Anselm made about the fact that once you are into this mobile, just in time, one view kind of situation, it is quite different than the Internet where you can bring up all these different screens and go to another website.

Paige Saez: Well yes, mobile is a different level of engagement. Very contextual. Much less information. Much more about timeliness. I don’t want to look an apple and get back a Google search. Oh my God no. That’s the last thing I want. I would love to be able to look at an apple and my phone already knows exactly what I want, information-wise, to get back from that apple. But I don’t know. It’s all contextual and personal. So I think the code angle concept you are talking about is really interesting because you still need to think about who is the person that is adding or creating those level filters- is it you, a filtered friend network, an algorithm? How much work is too much work? Where do we draw the line? How much of this are we willing to let the machine do for us?

Tish Shute: Right.

Paige Saez: And then of course once you have those filters in place, you need control over them. You will need to dial them up and dial them down, be able to choose and add new ones, so on and so forth. It becomes very modal at that point. For example, I want to change my view: To walk into a grocery store and instead of finding out information, I’d want to see where the hidden Easter egg puzzles were that my friends left last week because we’re playing a game.

I’m still really attracted to the creative opportunities with the ImageWiki. I’m really attracted to changing this experience from being a one-to-one relationship (from Corporation to Consumer) to an open-ended relationship (From Person to Person). If I look at a book jacket, sure I can find out where to buy the book, but that’s boring. Who cares? I’d like to find out a link to a story or an adventure or a movie or something unthought-of before.

How do we build that in? How do we encourage serendipity? Mystery? I think the ImageWiki is the space for building that in, actually. Not how, that would be the one place, right? That’s my really big fear is that this relationship just stays one-to-one. Click an image of consumable object, get back objects retail value. How completely dull. We have to do better than this.

Additionally, what if I want to take a photograph of a book, an apple, or something and I don’t want to pull back data. Instead, I want to pull back music, or I want to pull back a video, or I want to pull back a song, or lyrics, or a story, or another image. It’s just a hyperlink at the end of the day, you know? That’s all we’re really doing. Hyperlinks can pull back so many different things.

Tish Shute: And that’s one of the reasons I’m into mobile social interaction utility building, because without that, if we don’t have that way to do that in mobile technology…that’s very available on the Internet, as we’ve seen, with Twitter. These applications are very easy to do on the Internet. They’re not easy to do natively in a mobile application..

hey, I’m just promoting AR Wave again. I should shut up.

Paige Saez: Oh, no. I think it’s a fascinating concept, I really do. I totally agree. As we’ve talked about it before, it’s amazing that marketing and advertising are helping push forward AR, and it’s great. It’s fantastic.

But it’s also the worst possible thing that could ever happen because it is such a singular way of looking at an overall ubiquitous computing experience. There are other ways.

The best experience I ever had was trying to explain to people about physical hyperlinks. I had to walk them through it. Good interactive isn’t something you present or show, it’s something you do. Nothing beats just walking around and showing people with a device or a tool or something else.

I mean, God forbid it always stays in our computers and our phones. I really hope we don’t have to be stuck living our entire lives with these horrible interfaces. But for the time being, we will. Having an AR app show you a puzzle, or a mystery, or a game, or an adventure is a magnificent experience, totally overwhelming, and people get it right away. There’s no question; they totally understand.

Tish Shute: Yes, I agree.

Paige Saez: You walk them through the experience with a physical hyperlink and then you say, “Here, I could use this device and I could show you where to buy this thing, or I could use this device and we could start playing a game.” Then everybody gets it.

Tish Shute: So then I have a question, because one of the things Anselm said to me when he wanted me to refer back to you is that he feels that the direction for ImageWiki should be perhaps to focus less on the technology and more on just the actual, I suppose, gathering of the images, how they’re going to be annotated, the metadata, right? But my question to him was, the problem if you do that, without the platform, there’s no experience or motivation for people to do that. Right? Is there?

Paige Saez: Yeah, I agree with you on that one. I’m curious what his…I think the reason why he wants to do that is he wants to be able to show people examples via the resources. Like to be able to show someone a library, essentially, which I think makes sense with some people. I definitely think that some audiences would really relate to that. For me, it doesn’t make sense because I’m just very experiential. I need to do it and I need to show other people how to do it and I need to grow that way. I think that at the end of the day, those are great ways to go about doing it. It’s just it’s a huge thing to do in either direction.

What Anselm’s really thinking on, I believe, is more about exemplifying how we read and understand images culturally. Then you’re really getting into Visual Studies and Critical Theory which is what I did for my Masters at PNCA. I worked on the ImageWiki while I was in grad school, it was something I was doing for fun. Independently of my studies, the project lead to issues on democracy and objects and property and I ended up right smack in the middle of what I was studying; the nature and cultural analysis of images Questions like, ‘what exactly do we get out of images?’ and how all these different things are happening in an image, and people get tons of totally different things out of an image depending on many factors.

The questions I began to ask myself got very philosophical. Questions like “Is this apple red? Is this apple red-orange? Is this a small apple? What’s my understanding of small versus your understanding of small?”

Because you supposed that you needed a text backup to the search, how would I be able to search for an apple? Because what if my understanding of apple is red and your understanding of apple is green. And so if I’m looking for a green apple, am I looking for the same green apple as you? It’s all semantics, sure. But at the same time, it gets bigger and bigger, and it’s fascinating.

Tish Shute: Google Goggles seem to work best on book jackets, basically.

Paige Saez: But book jackets are actually perfect for this. Book jackets are perfect for this problem, because book jackets are specifically designed art. So at the end of the day, we are still talking about creative works, artistic works, that have been designed as a communication tool. But that is not something that people can own. Creative works that are designed are a communication tool, with varying levels of skill to be sure, but still something anybody can do. What we need to do is we need to be using that language. We don’t need to be trying to reach as far as facial recognition. We need to develop our own logos, our own brand, our own…I mean not brand. Brand is a bad way of saying it. Another way of saying it would be like, just use it. Develop a visual language that we can use that is as effective and as well utilized as book jackets or the movie posters or something.

Tish Shute: What are some of the use cases for ImageWiki you would like to develop first?

Paige Saez: My dream…I have like four or five use cases that I want to see happen. One of them is I walk down the street and there is a new poster for my favorite band. And I can just go up to the poster and I use my device, whatever it looks like, and I download the latest album. It’s transactional. I am able to just plug in my headset and walk down the street and the transaction is done. I saw something I wanted. It was beautiful. I was able to get it and I was able to move on in my life. And that is totally possible.

Another one would be I walk down the street and there is a piece of graffiti. And I am able to use my device to find out who the artist was that made it and to give them props, and to point my other friends to the fact that the piece is there and it will most likely be there only for a short period of time- information retrieval and socialization.

Or, use my device to find an Easter egg, to find a narrative puzzle that ends up going on for weeks, and everybody is involved, and we are all playing this game together. Adventure-based, non-linear experiences. I want playfulness, not just purchases.

Tish Shute: Did you think of piggybacking on the Flickr API for geo-tagged photos as a way to work with those databases or not?

Paige Saez: Yeah, we definitely thought about that.

Tish Shute: And why did you decide not to, for any reason or…?

Paige Saez: Ultimately, we just…we were such a small group, we just had to tackle certain things at a certain time.

Tish Shute: Right. And you were so prescient, you were working slightly before we had the mediating devices, weren’t you? You were just before the mobile devices really got adequate for this.

Paige Saez: Yeah. We started on it…I believe it was January…No. December 2007. Basically, the iPhone had just launched like maybe six months prior or something like that.

Tish Shute: But not 3G and not 3GS, right?

Paige Saez: Not 3GS. It was the first generation iPhone. We built the ImageWiki before the App Store existed.

We knew that the App Store was coming out. And we knew that the App Store was going to be the biggest thing in the whole world. I remember getting into multiple fights with friends about how revolutionary the iPhone and the App Store were going to be and people thinking I was totally crazy; people just thinking I was absolutely nuts for being so excited about it.

It sucks that it is a closed proprietary system, but the App Store has done something for software that nothing has ever done in the whole world. Software is candy now. It’s candy. It is like when you are waiting at the grocery store at the checkout line and you are stuck behind somebody, and you have got all these little tchotchka’s, candy bars, magazines, nail-clippers and things. That is the equivalent of software now. It’s become an impulse buy, which is amazing. Nobody would ever have thought…that is actually revolutionary. That’s huge.

Tish Shute: Steven Feiner, who is one of the founding fathers of augmented reality said to me during a conversations at the ARNY meetup that one reason that augmented reality, despite the hype, is manifesting very differently from how virtual reality burst onto the tech scene is that it is about affordable apps on affordable readily available hardware.

ISMAR 2009 -  was an extraordinary mix of  high geek, academic eminence, gungho Dutch Cowboy entrepreneurial spirit, German engineering and industry, brilliant artistry, and invention, all fueled by a sense, and a very active presence in the case of Diamond Sponsor – Qualcomm, that the big technology players are waking up to augmented reality.

In the picture sequence above (click on photos to enlarge),  Noora Guldemond, Metaio, demonstrates Junaio (coming to an iphone near you Nov 2nd) to Dirk Groten, CTO of Layar (top left photo).  One of the nice social features of Junaio is that users can share the 3D augmented scenes they have created.  Noora is demoing this capability to Dirk, and as you can see he cracks up when he sees the  scene Noora has stored on her phone.  Dirk and I both recognize that this cute little dinosaur augmentation (close up above on bottom left) must have been created by Peter Meier, CTO of Metaio, during the Interoperability and Standards workshop earlier that day.  Metaio it seems were discussing standards while enjoying some 3D augmented back chat.

Both Dirk and I were active participants in the workshop too.  But little did we know that Peter Meier had introduced his little 3D dinosaur into our discussion while we diligently, and sometimes heatedly, debated the merits of XMPP, Wave Federation Protocol,  KML, ARML, VRML, X3D, and  more!  The photo I took is on the bottom right of the four pics above. It was probably taken very shortly after Peter’s augmented Junaio scene.  Of course there is no little dinosaur in my pic of  Dirk Groten with Joe Ludwig and Markus Tripp of Mobilizy who were discussing AR standards oblivious to Peter’s virtual pet in our midst.

I must say I had noticed an impish look on Peter Meier’s face (see photo above on the left – Peter is wearing glasses and holding a phone).  And Markus Tripp, of Mobilizy  revealed a little bit of gaming of his own, when he let out that, in part, ARML is a provocation.  But Peter was clearly unfazed and enjoying himself.  Dirk, tasked to summarize our discussion, stalwartly maintained an optimistic but serious tone fitting for a standards discussion:  “There is a willingness to standardize….,” he began (pic above on left – click to enlarge and read text).

But it was a little 3D dinosaur that, perhaps appropriately, had the last laugh. Fitting, as I am not sure whether anything anyone says about AR standards at the moment will hold up.  But, as Ori commented in his great post – an epilogue for ISMAR 2009, the vibe was “Peace and Love” in AR Browser land (although Chetan Damani of Across Air was not in the standards discussion because he attended the UX/content? workshop instead).  But as they say, “all’s fair in love and war.”  And it is my feeling the games have barely begun!  There are many players (virtual pets included) waiting in the wings. I met some at ISMAR, and they are just itching to join the frey.

Ori Inbar, Ogmento and Robert Rice, Neogence Enterprises, both founders of the AR Consortium, made great efforts to set our young industry off on the right foot -  in the  spirit of coopetition (a neologism coined to describe cooperative competition). See Curious Raven for Robert’s conference observations, and Ori’s post on Games Alfresco for more about Mobile Augmented Reality at ISMAR 2009.  The Mobile Augmented Reality Workshops were driven by an indomitable spokesperson for the new AR industry, Christine Perey.  Christine not only helped motivate discussion on the issue of oxygen to the system, i.e. business value, but also she was a very generous connector at the conference.

What’s Next From Augmented Reality’s Top Chefs?

As Ori pointed out, Kent Demaine, oooii (pic above is from the oooii web site), Minority report VFX designer was hanging out at ISMAR 2009 and he came to the panel I was on: “Augmented Reality in Sports,  Entertainment and Advertising.”  We chatted afterwords about instrumented environments and how this is such a key to development interesting augmented experiences.  Also I mentioned how back in the day I was involved in some of the early development of motion control software.  And it was great to hear Kent say they were still finding motion control cool at oooii.  As Ori notes, he is the “guy with the most enviable AR credentials in the world (the guy who designed VFX for minority report),” and oooii is busy and hiring.

One of the highlights of the Arts, Media and Humanities track for me was meeting Jarrell  Pair. He really brought the best out in panelists with his well tuned questions.  The recording of ISMAR was comprehensive and videos should be up next week.  I will post the slides on Ugotrade of my presentation:  “The Next Wave of AR: Shared Augmented Realities and Remix Culture.”.

“Mixed and Augmented Reality: ‘Scary and Wondrous’” – Vernor Vinge

“Imagine an environment where most physical objects know where they are, what they are, and can, (in principle) network with any other object. With this infrastructure, reality becomes its own database.  Multiple consensual virtual environments are possible, each oriented to the needs of its constituency.  If we also have open standards, then bottom-up social networks and even bottom up advertising become possible. Now imagine that in addition to sensors, many of these itsy-bitsy processors are equipped with effectors.  Then the physical world becomes much more like a software construct.  The possibilities are both scary and wondrous.” (Vernor Vinge -  intro to ISMAR 2009)

Vernor Vinge’s short intro to ISMAR 2009 (which can be downloaded with the ISMAR 2009 schedule here) captures the essence of the “Scary and Wondrous” dawn of the age of ubiquitous computing and mixed and augmented reality.  It is definitely worth a moment to download.  The future of augmented and mixed realities, as Vernor Vinge points out, is tied up in a “tension between centralized and distributed computing” that “will continue long into the future.” One of  my fascinations with Wave is that it offers a tantalizing opportunity to explore augmented reality in an open distributed architecture.

At ISMAR, I talked with as many people as possible about the AR Wave project – see my post here for more about Wave enabled AR.  Many people were very enthusiastic to join the AR wave and the only thing I really lacked was about 100 invites to hand out!

“Everything, Everywhere – making visible the invisible”

Some of the areas that I would have liked to see given more attention on at ISMAR were sensor networks, data curation, and user experience.  Not that these areas were entirely neglected with Pattie Maes, MIT as a keynote speaker, and Mark Billinghurst presenting on some fascinating work on social augmented experiences and user experience.  I highly recommend catching up on these and other ISMAR presentations when the videos go up.

And, I was very happy to meet and talk to Sean White whose work at Columbia University is one of my inspirations (for more about Sean’s work see here or click image above):

“the confluence of powerful connected mobile devices, advances in computer vision and sensing, and techniques such as augmented reality (AR) enables exciting new opportunities for interacting with this hidden network of dynamic information and shifts the locus of interaction from the desktop computer to the world around us”

And I had several very interesting conversations  at ISMAR about developing social augmented experiences that connect us to a physical world that is becoming “much more like a software construct” (Vernor Vinge).  Dirk Groten, CTO of Layar mentioned a few interesting projects Layar has up their sleeves, including something  Layar may be cooking up with The RoomWare Project.

The picture above is of RoomWare’s Social RFID Installation for Media Plaza in Utrecht (read more here).

Demos Galore!

In the demo rooms, Noah Zerkin (pic below left) pretty much single handedly carried the AR flag for a growing community of augmented reality Makers and Hackers.  But his presence was much appreciated, and he tirelessly demoed The Zerkin Glove. See the first of what may be several posts from Noah on ISMAR here.

And I got to try out the Vuzix goggles (picture above on right).   This was my first experience playing an AR game that was smart about real world gravity. It’s  “an augmented reality marble game that uses gravity as a game controller” – see Ori Inbar’s write up here.  It was a very compelling experience and I have to say I didn’t really notice the shortcomings of the Vuzix goggles while I was absorbed in the game. And  I turned out to be quite good at the game too. It is intuitive unlike the kind of  rule based games I never have time to learn properly.  But what is so special about this project is the tools that it is built with are open, and available for all, and affordable (see this list on Games Alfresco).

It was a great pleasure to meet Prof. Steven Feiner (picture on below the left) who heads Columbia University’s brilliant AR research team at The Columbia University Graphics and User Interfaces Lab.

Ori Inbar (pic below on right) also spent a lot of time in the demo room showing off Ogmento’s lovely AR learning game that delighted attendees, “Put a Spell: Learn to Spell with Augmented Reality.”

For a round up of  what’s next for augmented reality head mounted displays check out, Games Alfresco here, and Thomas Carpenter’s excellent review of the head mounted displays.

Ori Inbar on Games Alfresco asks is “Microsoft – the new big player to watch?”   “Georg Klein, inventor of PTAM-on-an-iPhone (and the smartest Computer Vision guy on the block)” has joined Microsoft to make Mobile AR.

The picture on the left above shows Georg trying out ARhrrr with Blair MacIntyre.   And on the right Blair is demoing his marker card pack to Senior Vice President of Cypher Entertainment, David Elmekies.  Yes ISMAR was abuzz with demos. See this post from Gail Carmichael for more video demos.

Next Year ISMAR 2010 in Korea!

At the banquet, I managed to find a seat at a table with Sean White (at left in photo above with Christine Perey to his right) and the Columbia University team.  The banquet culminated with the “Past and Future of ISMAR” Panel chaired valiantly by Jay Wright of Qualcomm.  We were asked to offer our input for ISMAR 2010.  I offered up an idea that I have been nurturing for a while now -  to stage a “Green Tech AR Competition.”  Perhaps, I suggested, we could base the competition around a conference (ISMAR 2010 in Korea?) and set up a target rich, instrumented environment for the occassion.  I think the Arduino open hardware community and AR developers have a synergy that is just waiting to be explored!  And, if we add the innovators of data curation to the mix, e.g., Pachube, AMEE, and Path Intelligence…(Markus Tripp left ISMAR to speak on a Web 2.0 Summit panel, “Humans as Sensors,” which also included Path Intelligence, Deborah Estrin on “participatory sensing,” and the brilliant work of Di-Ann Eisnor, Platial, on “Transactional Cartography”).  Anyway a big Green tech AR competition could get people working together across the broad spread of AR terrain on some of the sticky problems of user experience.  And, with a high level of support from Smart Phone companies, HMDs manufacturers and the chip makers we just might come up with some extraordinary magic.

The devil of course will be in the details.  But a competition like this could not only motivate key players to come together in the spirit of coopetition but also be an opportunity to show the world the power of AR to make visible the invisible ecosystems that are so important to the health of our planet.

One of the notable presences at ISMAR 2009 was the Qualcomm team.   Jay Wright’s presentation (an exclusive for ISMAR) not only outlined AR for 2012, but Jay also talked about some “close to the metal” innovation that we will see from Qualcomm very, very soon!  I had some time in the press room with Jay and his team prompted by MoMo’s Yuri van Geest.  When I twittered about Qualcomm’s presentation at ISMAR, Yuri replied:

vangeest “@tishshute: good stuff, hopefully you will integrate the neat new solutions and ideas in your talk in November ;)”

I will be presenting at MoMo #13 on AR, open AR, future of AR and GeoWeb,  and hopefully will bring some good news from Qualcomm too.  Anyway Jay seemed to like the idea of a Green Tech AR Competition, even though I did stress that I thought it needed some serious sponsorship and BIG prizes.

Where’s the beef? Tracking and Mapping at ISMAR 2009

On the flight from NYC to Orlando and ISMAR’o9 I dozed (I had been up late preparing my presentation) and I watched the Dew Tour Pro Skateboard competition and Top Chef on the Food Channel.  In this particular episode of Top Chef, the aspiring chefs were all given a brown bag of ingredients by an already famous chef who then judged whether the contenders managed to make a delicious meal with their allotment which was notably lacking in key ingredients of haute cusine.

This metaphor of  trying to cook up a great meal while perhaps missing the staples is apt for the current early stage of commercial augmented reality.  And when I arrived in Orlando, not only were the Dew Tour pro skateboarders staying at the same hotel as ISMAR, but ISMAR itself felt remarkably like an Augmented Reality Top Chef Coopetition.

Much of ISMAR was dedicated to the task of  providing the meat and potatoes of Augmented Reality, solutions to mobile tracking, mapping and registration, particularly in the Science and Technology track.

Industrial and Military Augmented reality solutions I found out, typically, solve the tracking problems by using fixed mounts which clearly wouldn’t translate well into the AR everywhere with everything mobile consumer culture expects.

In the picture on the left Fabian Doil stands by the VW engine that provided some of the outdoor targets for the ISMAR tracking competition.  On the right is a picture from the VW’s presentation on their research and development of AR.

I followed the tracking contest, organized by Daniel Pustka and Fabian Doil of Volkswagen, quite closely. And I learned a lot in the process. While  it is clear there has been progress in AR mapping and tracking, we still have a ways to go.

But hanging around the Tracking Competition was a good way to find out the state of play of this crucial part of the AR dream.  For example,  a little tidbit I learned is that Michael Goesele who has been reconstructing “high-quality geometry models from images collected from the internet (so called community photo collections, CPC)” is soon to be at the Institut Graphische Datenverarbeitung where top contenders in the tracking contest – Harald Wuest  and Folker Weintipper (in the foreground of the photo at the left and right respectively) are also to be found. [update Harold and Folker were the winning team see here for details of scoring and results!] Otto Korkalo and Tuomas Kantonen of VTT, Finland, Augmented Reality team are in the background. They have been working on the joint IBM, Nokia and VTT project that brings, Avatars and People Together for Virtual Meetings in Physical Spaces.

The picture on the right is another team that were doing very well. If my notes serve me well (and please forgive me if they don’t. I came back with my card wallet overflowing!) the photo on the right showsChristian Waechter (on the left) and Peter Keitler (on the right) of the Technische Universitat Munchen.

Germany is certainly leading the way in industrial AR. And I learned how small businesses like Metaio get to work with top research institutions and big companies like VW, thanks to very strong German funding program for AR and VR. The current iteration of a series of funding programs is  called Avilus.  Avilus  is putting 42 million Euros into AR and VR this year alone (click on the slide below to see more about Avilus ).

I wish we had the equivalent of Avilus here in the US.  But there is no equivalent to Arvilus for AR here, and  no AR is  being developed by the US car industry either it seems.  But look at the slide above to get a taste of some of the cool stuff Metaio and other small AR and VR businesses do for VW through the Avilus project.

I also got to meet many people from one of the world’s most important AR hubs -  The Department of Informatics, Technische Universitat Munchen, including Prof. Gudren Klinker on the far right in pic above.  And from left to right, Fabian Doil (VW, co-organizer of contest), Sebastian Lieberknecht , Selim Ben Himane (Metaio), Tobias Eble (Metaio).  Prof. Klinker is the engine behind much of German innovation in AR.

Metaio was one of the few teams to rely mainly on markerless tracking which in this contest was very challenging because of the very different light conditions (see pics below) between the windowless interior and dazzling Florida sunshine outside (pic on the right shows targets under ideal lighting conditions).  Many people in the US may be  familiar with Metaio’s consumer applications, like Junaio,  but thanks to Germany’s efforts to nurture augmented and virtual reality they are also respected software developers in industrial AR.  And I suspect that Metaio will spearhead markeless tracking in consumer AR too.

This post as usual has already expanded to something much longer than I originally attended – pretty typical for me! There is much I have not been able to cover including some of the interesting contributions by augmented reality artists at ISMAR – again I recommend the upcoming videos.

But I cannot end without a hat tip to, Oriel, Nate et al. who won the best student paper award for AR Sketch – again please see Games Alfresco for more on this (pic below from Games Alfresco). AR Sketch, Ori notes, is featured “in our top post and popular video.” And

“Their work is revolutionizing the AR world by avoiding the need to print markers – or any images whatsoever.”

Above is an image above  from Total Immersion’s augmented reality experience developed for the “Networked City” exhibition in South Korea, – “a fun scenario created for a u-City’s infrastructure and city management service”

“To the naked eye, the exhibit looks like a bare bones model of a city. But when visitors put on the special AR goggles a whole new world unfolds – as graphics overlaid on the city model.” (Games Alfresco)

“The Networked City,” is a large scale augmented virtuality of a scenario for a networked city. But my guess, reading the     Korea IT Times, is the plan is to move from an augmented virtuality to an augmented reality as Incheon Free Economic Zone  (IFEZ) realizes its vision to become a leading u-City – where reality is turned “inside out” (see Inside Out: Interaction Design for Augmented Reality ).  If you are not familiar with South Korea’s u-Cities, check out this post for a short primer (and note Google Trends search on Augmented Reality shows  South Korea leaving everyone else in the dust).

Ubiquitous computing and augmented reality are like adenine and thymine – a DNA base pair.

A sky view of Incheon Free Economic Zone (from Korean IT Times). For more on the IFEZ vision to become a leading u-City see here.

Korea IT Times writes about the u-city concept:

“Korea began using the term u-City after accepting the concept of ubiquitous computing, a post-desktop model of human-computer interaction created by Mark Weiser, the chief technologist of the Xerox Palo Alto Research Center in California, in 1998. There have been a lot of research in this field since 2002. As a result, many local governments in Korea have applied this concept to various development projects since 2005 based on a practical approach to it.”

The back story to many of my recent posts, including this one, is an understanding of a relationship between ubiquitous computing and augmented reality that emerged, for me, in a February conversation with Adam Greenfield, Towards a Newer Urbanism: Talking Cities, Networks, and Publics with Adam Greenfield.  In cased you missed it, here is the link again because I think it holds up very well considering the rapid developments of recent months.  Also, importantly for this post, it includes a discussion of  moving on from Weiserian visions.

Adam Greenfield’s Speedbird is one of my key sources for understanding “networked urbanism,” and the list he makes of the elements of networked urbanism here (also see the comments) – is my mantra for thinking about the DNA base pair relationship of augmented reality and ubiquitous computing.

Adam Greenfield’s, “summary of what those of us who are thinking, writing and speaking about networked urbanism seem to be seeing” is:

1. From latent to explicit; 2. From browse to search; 3. From held to shared; 4. From expiring to persistent; 5. From deferred to real-time; 6. From passive to interactive; 7. From component to resource; 8. From constant to variable; 9. From wayfinding to wayshowing; 10. From object to service; 11. From vehicle to mobility; 12. From community to social network; 13. From ownership to use; 14. From consumer to constituent.

Augmented Reality – Making Visible the Invisible

The screenshot above is one of  the coolest “making visible the invisible” AR applications. It was developed at Columbia University Graphics and User Interface Lab where Steven Feiner is Director (see the deep list of projects from the lab here).  This app “shows carbon monoxide levels projected over New York City. The height of each ball reflects concentrations of the pollutant.” Credit: Sean White and Steven Feiner  (via Technology Review).

The recent emergence of “magic lens” augmented reality apps for our smart phones – Wikitude, Layar, Acrossair, Sekai Camera, and many others now, have given us a new window into our cities. But we are yet to realize the full potential of the AR/ubicomp base pair that can “make visible the invisible” and give us new opportunities to relate to the invisible data ecosystems of our cities, not merely as a spectator experience,  but as an interactive, in context, real time opportunity to reimagine social relations.

Mark Shepard says in his curatorial statement for, “Toward the Sentient City:” (Much more soon on this very significant exhibit which runs from Sept. 17th to Nov. 7th, 2009.)

“In place of natural weather systems, however, today we find the dataclouds of 21st century urban space increasingly shaping our experience of this city and the choices we make there.”

Augmented Reality, as Joe Lamantia points out, is becoming the great “ambassador of ubiqitous computing.” AR is. “…mak[ing] it possible to experience the new world of ubiquitous computing by reifying the digital layer that permeates our inside-out world,” and we are only just glimpsing the razor thin end of the wedge in this regard.

I am still working on my Gov 2.0 Summit write up  and, amongst other things, I will talk about how an emerging new social contract around open data, here in the US,  will put augmented reality  apps center stage  – “doing stuff that matters.” At Gov 2.0 Expo Showcase Tim O’Reilly tweeted:

Tim O’Reilly Really enjoyed @capttaco (Digital Arch Design) @ #gov20e: “Augmented Reality could be a new public infrastructure” http://bit.ly/18iCx

Also see Tim O’Reilly and Jennifer Pahlka on Forbes.com discuss the The “Web Squared” Era -  “the Web Squared era is an era of augmented reality arriving (like the sensor revolution) stealthily, in more pedestrian clothes than we expected.… …our world will have “information shadows.” Augmented reality amounts to information shadows made visible.”

Again there is back story to how I came to think about Information Shadows in relation to augmented reality.  So in case your missed it the first time, here is the link to a conversation that began in a hallway meeting between Tim O’Reilly, Mike Kuniavsky, ThingM, Usman Haque, Pachube, and Gavin Starks, AMEE, at ETech earlier this year,  “Dematerializing the World, Shadows, Subscriptions and Things as Services: Talking With Mike Kuniavsky at ETech 2009.”

Slide from Augmented City Lab @ Picnic ’09

So What’s Next for Mobile Augmented Reality?

These videos from Daniel Wagner’s team from Graz University of Technology showing Realtime Panorama Mapping and Tracking on Mobile Phones and Creating an Indoor Panorama in Realtime, as Rouli from Games Alfresco points out,  indicate that there is a lot in store for us at ISMAR09.

We may not be so impressed by directory style/”post it” AR anymore, as these applications have become common place so quickly!  But while these early mobile AR apps may be disappointing in relation to some futurist visions of AR – merely AR/ubicomp appetizers,  there are still good implementations of this model coming out (see new comers to the app store Bionic Eye and RobotVision). And Layar, always on the ball, has upped the ante for the new cohort of AR Browsers with Layar 3D.

But as Bruce Sterling notes here:

*In AR, everybody wants to be the platform and the browser, and nobody wants to be the boring old geolocative database. Look how Tim [creator of RobotVision] here, who is like one guy working on his weekends, can boldly fold-in the multi-billion dollar, multi-million user empires of Apple iPhone, Microsoft Bing, Flickr, and Twitter, all under his right thumb

(watch video here)

But if  you looking for something more from AR, you probably won’t have to wait too long.  The two pioneering companies in AR, Total Immersion – founded in 1999, and Metaio – founded in 2003 are both coming out with “mobile augmented reality platforms” in a matter of weeks (see press releases here and here).  And both companies, it seems, will deploy much more sophisticated AR rendering and tracking than we have seen to date.

I approached Bruno Uzzan, founder and CEO of Total Immersion, for an interview as part of my look at the new industry of augmented reality through the eyes of the founding members of the AR Consortium. These consortium members are some of the first commercial augmented reality companies.

The interview below with Bruno began early this summer and then we both went on vacation and it picks up after the announcement of the partnership between Total Immersion and Int13.

The significance of this announcement is that Total Immersion is now positioned to take the augmented reality experiences they have developed for a number of top brands onto multiple mobile platforms with, “Int13′s very clever embedded solution that allows our [Total Immersion's] solutions to work across many [mobile] platforms,” while Int13 gets to extend their reach.

Total Immersion has a 50 person R&D team and their two main focuses have been, firstly getting:

“Augmented Reality to work with as many platforms as possible – PC, Mac, Mobile, Game Consoles, all those are the platforms that we are targeting. We are currently doing lot of work in the R & D team in cross platform compatibility….”

and, secondly:

“Our R&D guys are working on the real world interacting more with the virtual world.  And I have started seeing some results which are pretty much crazy and this will be ready for next year.”

Pandora’s Box – Shared Augmented Realities

Spes or “Hope”; engraving by Sebald Beham, German c1540 (see Wikipedia article on Pandora’s Box)

There are many weaknesses to the mobile smart phone AR experiences we have now, and the lack of near field object recognition (to date), and difficulties with accurate positioning aren’t the only ones.  Note re solving positioning problems in mobile AR, we are yet to see AR  leverage public libraries for analyzing scenes like Flickr’s geo tagged photos, see Aaron Straup Copes’s work on “The Shape of Alpha.” And for more on this my post here.

But, as Joe Lamantia points out:

“One of the weakest aspects of the existing interaction patterns for augmented reality is their reliance on single-person, socially disconnected user experiences.”

In my view, The Pandora’s Box of Augmented Realities is an open, distributed, multiuser augmented reality framework, fully integrated with the internet and world wide web.

As Yochai Benkler has pointed out many times, and argues again in, Capital, Power, and the Next Step in Decentralization, it is “open, collaborative, distributed practices that have been at the core of what made the Internet.”  We have to try to make sure that open, collaborative, distributed practices are at the core of mobile augmented reality.

Can Google Wave be the basis for an Open, Distributed, Multiuser Augmented Reality Framework?

I have been exploring the idea of using Google Wave protocol as the basis for a distributed, multiuser open augmented reality framework with a small group of AR enthusiasts and developers. And I am happy to say the proposal is beginning to get fleshed out a little.  New collaborators are welcome both for “gear heady” input and use case suggestions (but re the latter, you can’t just say everything you see in Denno Coil..!).

This effort started with Thomas Wrobel’s  proposal for an Open AR Framework prototyped on IRC – see here, and click to enlarge the image above of, “Sky Writer: Basic Concept for an Open Multi-source AR Framework.”

But recently we began looking at the Wave Federation Protocol.  And, if you check out this post, and this post, you may get a glimpse of why Google Wave protocol might be a good basis for an open, distributed, AR Framework.  You will notice, if you study what Google Wave has done with the XMPP protocol, that many of  the elements of networked urbanism that Adam Greenfield describes resonate strongly with what is being attempted in Wave.

But enough said for now!  Regardless of the details of implementation,  Google Wave or an AR protocol built from scratch (phew! the latter does seem like a lot of work) -  an open, distributed, multiuser AR framework integrated with the internet and web would explode the potential of AR, creating new possibilities for data flows, mashups ,and shared augmented realities.

And we are excited by Google Wave because, as Thomas puts it:

“The really great thing wave does ….(aside from being an open standard backed by a major player…hopefully leading to thousands of worldwide servers )….is that it allows anyone to create any number of waves, set precisely who can view or edit them, and for them to be able to be updated quickly and continuously (and even simultaneously!) Better yet, changes will (if necessary) propagate to all the other servers sharing that wave. It does all this right now. From my eyes this does a lot of the work of an AR infrastructure already.

I cant see any other protocol actually doing anything like this at the moment, although correct me if I’m wrong, as alternatives are always welcome :)”

Also, Thomas notes, “even the playback system (that is, the ability to playback the changes made to a wave since its creation) …this could give us automatically some of the ideas Jeremy Hight has mentioned in his visionary work here,  and here on “the geo spatial web, interlinked locations and data, immersive augmentation and open source geo augmentation.”

One of the many reasons why an Open, distributed AR Framework would be so cool is it would open up all kinds of possibilities for GeoAR by providing the over-arching standard protocol for communication of updates necessary for the substandards that will facilitate GeoAR.

Also important to note is the  Wave Federation Protocol allows anyone:

“to run wave servers and become wave providers, for themselves, or as services for their users, and to “federate” waves, that is, to share waves with each other and with Google Wave. – “the federation gateway and a federation proxy and is based on open extension to XMPP core [RFC3920] protocol to allow near real-time communication between two wave servers.” See Reuven Cohen’s blog for more here and here, “HTTP is Dead, Long Live the Real Time Cloud.”

Still some people have expressed concern that an AR Framework using Google Wave protocol would give Google disproportionate influence.   Will Google-specific functionality be an issue?  How much stuff is Google specific just because no one else is using it (yet)? And how much is Google specific because it holds no value to anyone else but Google? These are some of the questions that have come up.

You are going to see a variety of suggestions for standards and specs for open AR coming out out in the next few months which as, Robert Rice of the AR Consortium points out is: “a good thing, we need that competition early on to settle down on best case.” Recently,  Mobilizy have offered up an ARML (“an augmented reality mark-up language specification based on the OpenGIS® KML Encoding Standard (OGC KML) with extensions”) for consideration see here.

So it is, perhaps, also important to note, that an Open AR Framework should be neutral/transparent to techniques of  “reality recognition,”  and methodologies of registration/tracking, allowing various ones to work on the system as new techniques evolve, and to support as many evolving standards as possible.

Augmented Reality developers, like Total Immersion and others with powerful rendering/tracking AR software, should be able use an Open AR Framework to exchange the data which their tracking will use. And the tracking/rendering problems they and other researchers have solved are much harder than figuring out data exchange on on a standard infrastructure or protocol!

So I pricked up my ears when I heard Bruno Uzzan, CEO of Total Immersion -  the first and currently the largest augmented reality company, with a 50 person R&D team in France and offices in LA, where Bruno himself is now based, say: “Total Immersion is  only months away from launching shared mobile augmented reality experiences using near field object recognition/tracking across multiple platforms” (for more details read my conversation with Bruno Uzzan below).

I was happy when I asked Bruno about the possibilities for developing an open, distributed, multiuser augmented reality framework fully integrated with the internet and world wide web (possibly using Google Wave protocols), and he replied:

“I think this is feasible. I think that’s doable, that’s just  in my opinion. I mean some people might have another kind of opinion but I think that that’s definitely doable.”

Total Immersion – working with the “symbiosis between augmented reality and brands”

Total Immersion has created many of the best known and most ambitious augmented reality experiences for major brands to date, including Mattel’s new AR toys to be released in conjunction with the James Cameron film Avatar, and AR baseball cards for Topps, video here (or click screenshot above), and the UK’s first augmented reality books.

Bruno founded Total Immersion 10 years ago when he was just 27. And the kind of conviction it took to survive as an augmented reality business in the decade before augmented reality captured the world’s attention is remarkable.

AR’s first steps out into the world after 17 years as predominantly a lab science maybe “wobbly” (what new technology isn’t), and sometimes gloriously kitsch – check out this riotus video of the 3D Interactive Live Show Total Immersion produced in Korea (also see the Total Immersion Augmented Reality Blog for more on the TI’s turn key  Interactive 3D Live Show Solution).

As Lamantia points out here, ” projecting mixed realities into public, common, or social spaces makes them social by default.”

However, the potential for shared location based augmented reality experiences is as yet untapped.  So I see the entry of the most experienced commercial augmented reality company into mobile as pretty interesting.   While  smart phone AR still has significant limitations, and it certainly does differ from some of the futurist dreams of AR (see Mok Oh’s post here on his disappointment in this regard), it is significant that Total Immersion is committing to becoming a leader in mobile AR.

Our smart phones, the powerful networked sensor devices that so many people carry in their pockets, have proved themselves a “good enough for now”  mediating device for early manifestations of the ubiquitous computing and augmented reality base pair.  And now AR and ubicomp is mixed in the  rich, messy soup of everyday life, commerce, business, marketing, art, entertainment, and government, we should get ready to see these technologies grow up fast, and unfold in some surprising ways that lab science didn’t necessarily predict.

And, perhaps, the new dialogue between scientists and entrepreneurs may spur both communities to outdo themselves.

Particularly, as Joe Ludwig notes: “It seems to me that the biggest disconnect between the academics and the entrepreneurs is that they disagree on how far we are from the finish line.”

See the comments’s on Ori Inbar’s post, Augmented Reality Entrepreneurship: Natural Evolution or Intelligent Design?, for  a courteous but spirited discussion on the potential benefits and frictions of the newly expanded AR community of  researchers and  entrepreneurs.

As Blair MacIntyre (see my long conversation with Blair here) notes:

“not all academics and researchers are only interested in the traditional models of impact. Case in point: I wouldn’t be building unpublishable games, nor investing so much time talking to the press, entrepreneurs and VCs if I did not believe strongly in the value of the impact I am having by doing that — and I know others with the same attitude.”

In this vein, check out the Marble Game (video here) developed by Steve Feiner and his team at Columbia U. It’s enabled by Goblin XNA, an open source AR framework built on top of Microsoft’s XNA, which powers XBox live games, Zune games, and some Windows games. For more about Goblin XNA and AR from Columbia U see here.  (Hat tip to Brian Jepson for this link)

While we are still waiting for the kind of sexy AR specs – nothing totally game changing in Gigantico’s AR eyewear rounup (maybe note this Apple patent), that might get wide adoption. But at least researchers are not afraid to explore the possibilities of AR Goggles.

But how far are we now, with or without sexy goggles,  from a fuller expression of the base pair DNA of ubiquitous computing and augmented reality?

We may have a LAN of things before we have an Internet of Things

The picture above is a workshop I attended at Conflux last weekend – Fish ‘n microChips, with Natalie Jeremijenko. We are at the site of the Amphibious Architecture project (a commissioned work for Toward the Sentient City) and “a collaborative project with xClinic, The Living and other intelligent creatures.”

We are probably as far off some grand futurist visions of ubiquitious computing as we are some of the futurist visions of augmented reality. But as it turns out that may not be a bad thing! Recently, @mikekuniavsky noted in a tweet:

“Another argument for the LAN of Things before the Internet of Things: http://tinyurl.com/lgp9uq”

Bert Moore, in the article Mike linked to points out, the grand vision of an “internet of things” with everything connected to everything  can “distract people from thinking about the benefits of RFID in smaller, more easily implemented and cost-justified applications.”  The same argument I think applies to sensor networks and augmented reality.

In New York City, a series of commissioned works for the Architectural League of New York’s exhibit, “Toward the Sentient City” are giving us the opportunity to dip our toes into the ocean of a “networked urbanism.”   For only a small budget, two of the five commissioned works, Amphibeous Architecture and Natural Fuse demonstrate how sensor networks can allow us to explore new kinds of communities – connecting people to environments in interesting ways to create new forms of social agency.

“Amphibeous Architecture” -  from The Living Architecture Lab at Columbia University Graduate School of Architecture, Planning and Preservation (Directors David Benjamin and Soo-in Yang) and Natalie Jeremijenko, Environmental Health Clinic at New York University, uses a skillfully built (electronics and water are notoriously hard to mix) array of partially submerged sensors to pierce the blinding, reflective surfaces of the rivers  surrounding Manhattan and to create a new two way relationship with the ecosystem below – the water, our neighbors the fish and even a beaver that lives in the water surrounding Manhattan.

Image from Toward the Sentient City

In a similar spirit, “Natural Fuse” – Usman Haque, creative director, Nitipak ‘Dot’ Samsen, designer, Ai Hasegawa, designer, Cesar Harada, designer, Barbara Jasinowicz, producer, creates a network of people and electronically assisted plants to explore what it takes to work together on energy consumption and to experience the consequences of “selfish” and “unselfish” behavior interactively before it is too late to modify our actions.

The “Greedy Switch“ from Natural Fuse on the left. On the right “The System” – click to enlarge.

Much more to come in another post on these works, and “Toward the Sentient City.”  Also an update on how Pachube – an important part of both these projects and a very important contribution to ubiquitous computing because it creates the opportunity to connect environments and create mashups from diverse sensor data feeds – has matured since my interview with Pachube founder, Usman Haque, “Pachube, Patching the Planet,” in January this year.

In the picture above Natalie Jeremijenko, and Jonathan Laventhol give the Amphibious Architecture sensor array a last look over, as it will soon be lowered into the East River. Jonathan is on a busman’s holiday to help out at the pre launch of Amphibious Architecture, nr Manhattan Bridge, NYC.

I was very happy to get  a chance to talk to Jonathan Laventhol - more on our conversation in another post. Jonathan Laventhol is CTO of Imagination – one of the world’s leading design, events, and branding agencies. We talked about the importance of  Pachube, which Jonathan called the “The Facebook of Data,”  and  how the symbiosis between brands and augmented reality, and healthcare applications, would  be key to augmented reality emerging into the mainstream.

Natalie Jeremijenko’s workshop at Conflux on the social negotiation of technology and how “everyware” can give us the chance to experience new forms of agency and connection was a totally inspiring.  And I will cover this too in another post.  I have so much awesome stuff  to write about at the moment!

None of the projects in, “Toward the Sentient City,” included a mobile augmented reality, or “magic lens” component, but they all pointed to why “enchanted windows into our newly inside-out reality” are going to be so important. And why the DNA base pair of ubicomp and augmented reality can really do stuff that matters.

Shangri- La – “Transfigured City”

Screenshot from Shangri-La episode Transfigured City

In my AR Consortium founder member interview series, I have found that, understandably, the visionary founders of these first augmented reality companies are a little reticent about sharing their full vision.  They are basically on stealth mode in this regard.  So as you will not, from my interview with Total Immersion founder and CEO, Bruno Uzzan, get a fully drawn scenario of his vision for a next generation of shared augmented reality experiences, here’s a really interesting anime episode from the anime Shangri La called, Transfigured City, to mull over instead.

As you can tell from this rather long and circuitous intro to my my conversation with Bruno Uzzan, I  have been investigating shared augmented realities pretty intensively recently. And Mike Kuniavsky pointed me to Shangri-La, and Transfigured City, in a conversation with Mark Shepard, after Mark’s presentation at Conflux, Sentient City Survival Kit.

Mike Kuniavsky with Tod E. Kurt is founder of ThingM, a ubiquitous computing device studio. Also Mike Kuniavsky researches, designs and writes about people’s experiences at the intersection of technology and everyday life – see Mikes blog Orange Cone.  And I interviewed Mike at Etech- see here.

In Transfigured City, the “Metal Age” group has to figure out how to share and communicate in a city transfigured by augmented realities/virtualities, where no-one sees the same place in the same way.  Only one character can figure out from her previous experience of the city the relationship between the transfigured city and how it used to be.

The conversation I had with Mike Kuniavsky on The Transfigured City continued at a picnic in Washington Square Park the next day with Elizabeth Goodman, who I met at Etech when she gave a brilliant presentation, Designing for Urban Green Space.  We covered so many areas at the picnic related to ubiquitous computing and augmented realities that this conversation probably deserves a post of its own (my writing to do list is growing longer!).

The Plot Synopsis for Shangri La:

“In the mid-21st century, the international committee decided to forcefully reduce CO2 emission levels to mitigate the global warming crisis. As a result, the economic market was transferred mainly into the trade of carbon. A great earthquake destroys much of Japan, yet the carbon tax placed on the country is not lifted, so Tokyo is turned into the world’s largest “jungle-polis” that absorbs carbon dioxide. Project Atlas is commenced to plan the rebuilding of Tokyo and oversee the government organization, which the Metal Age group opposes due to its oppressive nature. However, Atlas is only built with enough room for 3,500,000 people and most people are not allowed to migrate into the city. The disparity between the elite within Atlas and the refugees living in the jungles outside of its walls set up the background of the story.”

Talking With Bruno Uzzan

Tish Shute: We won’t have fully opened the Pandora’s Box of Augmented Realities until we have ubiquitous, shared augmented realities, will we?

Bruno Uzzan: Yes. The most important for augmented reality is the experience we want to share. Now we are working on the cell phone, we can potentially do some marketing components that we already have developed now on cell phone. Done. It’s working.

But the most interesting part of it is how these new components [cell phone AR] will be used for marketing campaigns by brands. And we are also pretty much well positioned to transform some of the AR that we currently have working on Mac and PC and to transform these to applications working on mobile devices.

Tish Shute: We haven’t really experienced yet what it means to actually share mobile AR experiences?

Bruno Uzzan: It’s hard — we did a Facebook app. It’s a first try, it has a way to go.  But to go more and more into social, is the way forward for us – to share and expand AR experiences. But yes, I mean what you’re seeing is how two people on two different applications can share that same expanse.  For sure we are going in that direction. We are currently working on those kind of solutions. How people can share and experience together at the same time. That’s how we start creating excitement in augmented reality, and it’s coming up.

It’s a new market and there’s so much more in store for augmented reality. You know, some people are telling me, don’t you believe that augmented reality is a gimmick? It will be a trend for a few weeks or a few months and then gone? I say, you’re kidding me. This is only the beginning. I mean I can assure you that the applications that are on the market today are one percent of what we will have five years from now.

Tish Shute: I agree.

Bruno Uzzan: And I’m sure that augmented reality will be a part of a lot of components that we are currently using today – GPS, web browser, glasses, I mean there are so many applications that will come up shortly. This is only the beginning. I’m completely convinced that augmented reality will be in three years from now what virtual reality is today, which is a billion dollar market.  I know that it’s not just a gimmick of a few weeks or a few months, because so many brands are jumping into it, spending money, exploring solutions.  I know that it’s not just short term -what they are willing to do and we are willing to do, but also middle and long term. And that’s what makes this adventure pretty much unique and what makes creating a cutting edge technology, very, very much exciting for us.

Tish Shute: First could you explain more to me about your partnership with Int13. I am not sure I understand what is in the arrangement from Total Immersion’s POV. I mean what happens re your own mobile software development? Haven’t you only been licensed the Int13 SDK for a limited period of time and have limited access to all it’s power? Stephane from Int13 said to Ori on Games Alfresco, here, “we have licensed the SDK4 for two years,” and then Ori asks, “but you have basically kept the power to yourselves, right?” So if they are the only ones that can enhance it and develop the software, where will  TI be in two years in mobile if you haven’t really had the chance to develop your own software .

Bruno Uzzan: Actually it’s a real win-win situation. Int13 is a very small company and they have so many requests they can’t possibly fulfill them all. So  this is a way for both of us to be, as quickly as possible, the first mobile provider for all the requests we have. Also they give us exclusivity so nobody else can use INT13 SDK for such applications.  I think that it is a good partnership,

And concerning our own mobile application… First of all we have currently some mobile applications working. But with Int13 we have a mobile solution that can work on many different devices. That’s a fact and that’s working. And, believe me you will hear from us a lot more about this soon. We are fully independent on our mobile development. The reason we closed the partnership with Int 13 is  to be able to deploy mobile in a broad way.

I mean you know that the difficulty with AR mobile is that each separate device needs some customization. Working on the iPhone is different from working on the Nokia, different from working on the Palm; it’s different from working on the Samsung. Each of them have their own operating system inside and so we were interested in Int13′s very clever embedded solution that allows our solutions to work across many platforms.

The reason we are working with Int13 is that we are able to work on so many mobile devices, thanks to Int13. And in the mobile AR race that we are currently in, the next two years will be extremely important to us…

Tish Shute: OK, that definitely clarifies it a lot. So Int13 has done an embedded solution to allow TI developed AR solutions to work easily across many devices?

Bruno Uzzan: Yes  they have kind of an embedded solution, a way to address extremely quickly new cell phone… But, currently on our side, we are in discussions with a mobile company… and that only refers to some very specific mobile devices.  And what they have is also a way to embed deeper our technology into mobile, so that we can have quicker… applications that work on a large number of cell phones.

Tish Shute: So, basically it means you don’t have to go through some complicated negotiations with each of the cell phone companies, is what you are saying?

Bruno Uzzan: Not only negotiations, but also hard development. You know? Working on the Windows mobile is completely different from working on the Palm OS. You know, that’s different! Its a big work, to have a mobile application working on many other devices. So, INt13,  provides us a way for us to save some time and some development cost too.

Tish Shute: And Int13 doesn’t have powerful AR development tools like D’fusion right?

Bruno Uzzan: Right! That’s right. That’s why we say it’s a true win-win solution. They can benefit from our work too. And we can benefit from their work, in order to deploy quicker and faster mobile solutions.

Tish Shute: Now, the second thing is… there is a lot of debate and disagreement about how far mobile augmented reality is from delivering something more that the “post it” approach that has been much publicized in recent months, via all the AR browser apps.

But from my understanding from the conversation we had earlier this summer (see below), Total Immersion is targeting a much higher level of mobile augmented reality than we’ve seen to date?

Bruno: Yes the browser apps we have seen are a kind of augmented reality, but not exactly the way we see it. Let me explain you why. With this kind of application it’s true that you can overlay 3D-information and video. That’s a fact. So, in a sense, that’s augmented reality. But the way that they are working on the position of the 3D on that video is that they are using compass and GPS-information.. so it means that this AR solution will work only on some building and some physical objects that are FIXED. In a fixed and known position.

So you want to go to a theater?

The theater is here, for sure it will not move, so you know the position of the theater, and that’s a fact that you can superimpose an object on the theater. That’s what can be done currently. What we are achieving and what we are doing on mobile is more than that. We want to be able to port our solution with trading cards, with brands, into a smart phone.

I’m assuming that you want a can, a drink can, to be able to trigger an experience. The only way you can do it is to be able to understand what the can, it is. And the current solutions that are out there can’t do that, it’s impossible.

Tish Shute: Right, yes. There’s no near-field object at all in these early browser apps.

Bruno Uzzan: And the solution we have is that we can recognize a can and then — in a very, very precise way and that activates geo-location, so we can superimpose 3D. I mean in that case, it opens up all the applications that we currently have, so they could work on mobile.

Tish Shute: So for example, if you’re working with a soft drink company, people can trigger that experience wherever they see that can?

Bruno Uzzan: Correct.

Tish Shute: Yes. Yes, I assumed that was what you’re doing

Bruno Uzzan: We believe — and maybe that’s not the case, but we believe that our marker-less tracking technology is pretty much unique on the mobile devices.

I haven’t seen yet, from anyone, a full augmented reality mobile solution working.

I really see AR being part of the Web 3.0 next generation. I mean the vision I have is that, you know — today, when you want to have information, you go on a website and then you find your information. AR — and the future is that I think it will be part of the opposite. You want to have information about a product, you just show it to your computer and the information will automatically pop up. I see here a new way to market some key messages, a new way to get information is that some physical product by themselves could be a way to get information, and you don’t have to search anymore for them, it’s coming out to you.

AR is definitely for me, one of these components. Another thing that AR is a solution, another thing that AR itself will create these kind of results in how information is being displayed. But I’m seeing  here a way that could be part of a new way to have access to information. And that’s part of the vision I have. Whatever, if it is through mobile phone or web or PC, Mac, whatever, I really believe that now this kind of new generation of receiving information will come shortly and could be a kind of a new — could be part of the new 3.0 generation of the web.

Tish Shute: My friend Gene Becker did an interesting post recently on some of the current limitations of mobile AR where he pointed out the problem of:

“Simplistic, non-standard data formats – POIs, the geo-annotated data that many of these apps display, are mostly very simple one-dimensional points of lat/long coordinates, plus a few bytes of metadata. Despite their simplicity there has been no real standardization of POI formats; so far, data providers and AR app developers are only giving lip service to open interoperability. Furthermore, they are not looking ahead to future capabilities that will require more sophisticated data representations. At the same time, there is a large community of GIS, mapping and Geoweb experts who have defined open formats such as GeoRSS, GeoJSON and KML that may be suitable for mobile AR use and standardization.”

Bruno Uzzan: That’s interesting. I mean — I know exactly what his is referring to. He is mainly referring to a localization and how you can have a quick, accurate localization.  If you look at current solutions, and you look at this 3-D superimposing on the video, the 3-D is shaking a lot. I don’t know if you see that in some of these early efforts.

It’s hard to use because the 3-D, you know, is  part of the magic of augmented reality, that is when the 3-D is being inserted in a very easy way and smooth way in your solution. Here, when you see this overlay, 2-D or 3-D overlaid on the video, it’s shaking a lot. One reason for this is that the GPS compass is not accurate enough to coordinate the perfect location of the user. And here, what Gene says is interesting. I think we are addressing this localization issue in a pretty smart way.

But to be frank with you, I don’t believe mobile augmented reality in the extremely short term — I’m talking about three weeks, one, two months is mature enough for good AR applications.  It will be shortly.  But for now it is more proof of concept than a true and easy application to use.

But we are starting to see a lot of new application coming out, but I really believe that marketing and entertainment are the two key markets for AR right now.

I’ve been working ten years in augmented reality. And, eight years ago, when I was talking about augmented reality, I was E.T., you know? Nobody understood what I said, and I thought it was crazy. And now, today, yes it’s completely different.

Tish Shute: The Pandora’s Box of Augmented Realities, in my view, is an open, universal and standard, distributed, multiuser, augmented reality framework fully integrated with the internet and world wide web. I have been looking into Google Wave protocols as a basis for this would you be interested in this? Do you think it is feasable?

Bruno Uzzan: I think this is feasible. I think that’s doable, that’s just  in my opinion. I mean some people might have another kind of opinion but I think that that’s definitely doable.

Tish Shute: Yes I suppose an open AR Framework involves cooperation and collaboration, it is more about business and politics than technological problems.

Bruno Uzzan: Yes!  Actually the Web is politics. Business is politics.

Tish Shute: I would be interested if anyone in your R&D team would be interested in looking at some of the ideas that are emerging in our little discussion of Google Wave and an Open AR Framework  to offer feedback. it is an interesting time now to input on the Wave Federation Protocol docs because nothing is set it stone right now.

Bruno Uzzan: Just shoot me an email, I’ll try to put you in touch with the right person and, and a team member that can input on this.

Tish Shute: For mobile augmented reality the best thing we’ve got now is the phone, right?

Bruno Uzzan: Right.

Tish Shute: And the only way we can use the phone is by holding it up, right?  Isn’t this a bit of an an obstacle as you introduce better object recognition and tracking?  People are going to have to stop moving to use their phone. What do you feel about that experience? Isn’t AR eyewear and essential part of a tightly registered AR experience?

Bruno Uzzan: We don’t do hardware and we don’t have the current solution for eyewear that would do all we need for a good mobile AR experience, so I guess we don’t have the current answer for that.  But we are beginning to see the next generation of this — of these glasses.

Tish Shute: But you’re happy enough with the mobile experience of augmented reality on smart phones that you’re investing in this next generation of software for this.

Bruno Uzzan: Yes, I know. We know that some application will not work on the iPhone. And yes, whatever you do, you still need to hold the iPhone, so it means that you can’t play with your hands anymore. So we know that partially, some AR solutions  we have on other platforms will lose the magical effectivities on just the iPhone.

But I’m starting to see on the market some glasses that could perhaps be not too expensive — that’s a challenge!  And easy to use — that’s another big challenge. And, that could fit on anybody’s faces and head — there’s another big challenge. So yes, I’m starting to see that, but so far AR glasses are only applicable for some very, very specific application, like design or theme park or, you know, some specific location where it makes sense to move forward with glasses.

I don’t believe that kids will use glasses for — in our toys and for games in the next months or maybe othe next one or two years. But maybe something will come out shortly and that could be a big breakthrough, and enable us to think another way. But  from what we have seen so far and from what we know in this hardware market, I don’t believe that currently there is a workable solution.

Note: The following section of the interview took place earlier in the Summer.

Tish Shute: You are the first commercial AR company  – you started in 1999 right?

Bruno Uzzan: Yes you are right. We started the extremely early in this augmented reality market. We were the first company worldwide to start doing augmented reality and to start promoting augmented reality. So it’s true, we are pretty old players although the market has been getting bigger and bigger for the last year and a half. So for a long time we were only in the market, and the market was not really there.

But for the past 8 months, the company has been growing really fast.

Tish Shute: Yes I’m sure. Congratulations for hanging in there long enough to get the pay off!

Bruno Uzzan: You know, my background is Financial. So I have been driving the company for many years in a very cash efficient way. So we have been waiting for the markets to reach maturity before starting make some investments. So that’s the reason we are still here, and that’s the reason I think we managed pretty smartly the cash that we raised for the company.

Tish Shute: Yes there is a saying that when a market takes off you can tell a pioneers because they are the ones with the arrows in their backs. But I am glad you are dodging the arrows!

Bruno Uzzan: You know, I’ve always driven the company with revenue. And because revenue was not there at the beginning I was extremely cautious about the cash. So now that the company is getting some revenue, for sure we are making more and more investments, and taking advantage of our situation as a worldwide leader of augmented reality.

This situation is not easy as it appears today but it’s now getting better, as you can see, AR, Augmented Reality, has very good momentum and we are benefiting a lot from all this momentum for augmented reality right now.

Tish Shute: You’ve been very involved in researching developing augmented reality tools. Are you still as active in the research area, or are you too busy keeping up with work for hire now, to be working on research and building new technology for Augmented Reality?

Bruno Uzzan: Both. First of all, we are part of lot of projects either directly with clients like Mattel or with some partners that are using our technology to promote and develop other AR projects. From what we he have seen, many, many, many, projects augmented projects have been done currently with our solutions.

To continue with your previous question. So we are being perceived as this leader in that space, and we  have some pretty heavy demand for our services. But we are coming up with new technology, of course, still connected to Augmented Reality.  But, our R & D is working in two different directions, which of course also bind together.

The first one is platform developments. So we want Augmented Reality to work with as many platforms as possible – PC, Mac, Mobile, Game Consoles, all those are the platforms that we are targeting. We are currently doing lot of work in the R & D team in cross platform compatibility.

Tish Shute: Robert Rice said recently, “markers and webcams equal Photoshop page curls…”

Bruno Uzzan: Yes. There are so many concerns with markers. The quality is extremely bad. As soon as you hide a part of the marker, a slight part of the marker, you’re dead. You can’t track any more of the object. So compared to our solution where I want to say play with cards or where you are going to play with a Mattel toy, even if you hide a part of the toy, it’s still working.

Tish Shute: But you haven’t offered the public an SDK to your engine right? Basically the way people get access to your tools is working in a partnership with Total Immersion right?

Bruno Uzzan: Correct.

Tish Shute: Do you think in the future you might open your SDK? Are you considering that?

Bruno Uzzan: Yes, it would be interesting.

Tish Shute: So that is something we can see coming soon?

Bruno Uzzan: Maybe, because it’s true that Total Immersion is starting to be mature enough for these kind of tools. The only thing is that we have to respect good timing for that.  It’s a big decision. You know what I mean?  It is a big, big decision. We would then compete with others using our technology.

Tish Shute: Oh I know, it is a big decision when you have so much skin in the game! But it would be nice to have your SDK being THE platform for AR, wouldn’t it?

Bruno Uzzan: It is a really big decision that we can’t just take like that, you know.  There are a lot of friends who told me you have to be extremely careful about timing. This timing is pretty much connected to the maturity of the market. For sure, we see the market being more and more mature. But, there are a lot of low hanging fruits we still want to address

To get the best value possible for all the publicity we have and all the clients we have now.

Tish Shute: Yes, I know. You’ve been in this game so long. Now, there is an interesting question here though about tools and platforms because you know, A.R., augmented reality has already expanded  beyond its kind of original purist definition. And when I talk to people  about augmented reality, there are actually lot of different ideas and priorities of where the tools should go right now. You know, obviously we have these kind of browser-like applications, but these browser like applications are not dealing with recognizing near field objects yet.  What are your priorities for tool development and what are your priorities for AR development in the future? What areas are you going to focus on? Oh dear that is a rambling question!

Bruno Uzzan: [laughter]  So, one of our first priorities is we need to create our software with one development, one installer, one software that can be spread on different platforms. The same application, the same software can be used either on a PC, Mac, phone or console. So that’s a lot of work, because that means that our platform has to address many many different devices and that’s a big priority for us because we received this request from our clients. We want to be able to use one application on many different platforms and devices. So, that’s the first one.

And the second one is to add more and more interactivity between the real and the virtual world. So, we are working on some improvements to add some real components that will interact with virtual, and that also part of our big strategy and direction and these two worlds can more and more be bridged together, linked together so they can interact  one with the other.

Our R&D guys are working on the real world interacting more with the virtual world.  And I have started seeing some results which are pretty much crazy and this will be ready for next year.

There are so many different directions for interaction between the real world and virtual world to develop.  I’m sure ten years from now you’re going to have AR applications everywhere.  Its not just temporary fashion stuff or a gimmick for few months. I mean we are getting there, its getting stronger and stronger and we are getting a good adoption rate from our consumers. They like it, they test it, they play with it and brands wants more, people want more and its getting bigger and bigger.

Tish Shute: Yea and I totally agree, its not a gimmick because the interaction between “virtual” and “real” enhances the magic of both. Another question about you RandD operation. Is your R&D still in France or have you moved totally out to LA.

Bruno Uzzan: We are 50 people in France and I started this LA office two years ago and I moved permanently two years to LA. So I’m now permanently located in the US to take care of the US office, knowing that revenues are really getting bigger and bigger in the US. So it means that we are getting a lot of traction, working with large company and now I’m currently located in the US.

Tish Shute: My sister lives in Paris. Could I visit your R&D lab at some point? I’d love to visit!

Bruno Uzzan: Yeah sure sure sure. I mean if you want to go. You won’t have access to all the research. But if you want to go out and meet all the team please do.

Tish Shute: I’d love to.

Bruno Uzzan: No problem. Shoot me an Email you and I will introduce you to Eric Gehl, COO, he is the COO of the French team. And he can definitely take care of that.

Tish Shute: That would be fun. Thank you!

Recently, AR browser applications have really caught the imagination of the web community, eg., Layar and Wikitude?  Where do you think the most important market for AR is at the moment, entertainment,  green tech, business, education?

Bruno Uzzan: I think that all that you mention will be important. The first one that did grab my attention is entertainment particularly dual marketing, because they always searching for new ways to interact with players or the consumers.  But it’s just the tip of the iceberg, you know, I mean medical applications could be huge using augmented reality. Education, and edutainment is definitely using more and more augmented reality components.  And I will just be submitting with big companies – that are considering using augmentation for education. Museums are very important too. Also augmentation as a kind of free sales tool, you know there are so many applications, design, architecture – so many directions that it’s hard to say today which one will take the lead.

But I do believe that on the short term the ones that are really really moving fast are the entertainment business and the digital marketing business.

Tish Shute: What do you think are the biggest shortcomings with current augmented reality and what are the obstacles that no one has solved yet?

Bruno Uzzan: I think the cell phone is not fully ready for augmented reality – a lot of people are working on that but there are still a lot of constraints to get the augmented reality working on a cell phone and I think that from what I heard a lot of manufacturers and a lot of companies are working from direction that are going to help us a lot to develop some great cell phone applications.

And I think that’s one of the biggest part of the game. All the applications that you see on cell phones so far are just gimmicks – the next big key is how to transform some gimmick cell phone application to a real, industrial, robust application that’s going to work on a cell phone. So I think that’s a big challenge for this year.

Most of what we see now is just matching and overlaying some 2d components in a video. This is not what I call AR.  You’re far away – with this kind of application, you are far away from doing the registration that we need to do – you can’t do it. So here’s the challenge: “how can you get a Topps is an application working on cell phone. That’s the big challenge  How we can make that work!” You can’t today get a real AR Topps application working on cell phone because there’s no cell phone  that’s actually ready. But we are working on it and the first one that can make that work, it’s going to be huge.

When you are working with good AR components you need a lot of CPU and GPU programs. So today new cell phone have started to be more and more ready for augmented reality but you need a really good cell phone to make it work. You can’t choose an old cell phone to make it work because you have some recognition, you have some tracking, you have some rendering, so you can’t choose a Nokia cell phone two years old to make that work. For sure the newest iPhone is the one that can make it work, but that’s it for now. There is a lot of research – from large cell phone companies – to get more CPU and GPU into their cell phone.  But so far we are also waiting for these devices to be released to consumers.

Tish Shute: And the current economic climate has put a damper on MIDs hasn’t it. But who can tell? It depends what price points some new MID came out at right?

Bruno Uzzan: Correct.

Tish Shute: Yes,I agree. But basically what’s interesting, the interesting thing is, the iPhone can deliver so much of what is necessary and even if Apple hasn’t given access to the full power of the iphone to AR developers yet, there is really no going back now – the mobile augmented reality cat is out of the bag!

Bruno Uzzan: You’re right, you’re fully right.