Picture on the left Mirrorshades, picture on the right a Metroid Hud.

“Augmented Reality is like a Philip K Dick novel torn off its paperback rack and blasted out of iPhones,” Bruce Sterling in Beyond the Beyond

“a techno visionary dream come true – those are rare, really rare, you have to be patient,  it’s super cyberpunk”… Bruce Sterling, “At the Dawn of the Augmented Reality Industry.”

The Dawn of the Augmented Reality Industry continues to brighten, and now we have two augmented reality companies, Total Immersion and Ogmento, firmly established in Hollywood – the dream mother of so many of our augmented realities.

Ogmento is the most recent of these two pioneering augmented reality companies to set up shop in LA.  Total Immersion’s CEO Bruno Uzzan moved to LA from France two years ago, although he still has a fifty person RandD team in France.   Total Immersion began 10 years ago in the quiet, lonely, hours before the dawn of an AR industry.  But Total Immersion’s AR toys for Mattel, and augmented reality for Topps baseball cards, fired CNet writer Daniel Terdiman up enough to say, “I have seen the future of toys, and it is augmented reality” (see full post here on CNet).

Recently, I talked with  Ori Inbar, one of the founders of Ogmento and  the premier augmented reality blog Games Alfresco about his new venture in Hollywood. Bruce Sterling, @bruces, had some fun with my invention of brand new augmented reality trade jargon here!  Ori pointed out Ogmento brings two important new facets to the rapidly growing augmented reality field: firstly they are bringing leadership from veterans of the entertainment industry into augmented reality development. Brad Foxhoven and Brian Selzer from Blockade have partnered with Ori on Ogmento.  And, in an another important step forward for a young industry, Ogmento announced they will be acting as publishers for a fast growing cohort of augmented reality application developers and helping AR development teams out there bring their concepts to the market.

So I was very happy also to have the opportunity to talk with Brian Selzer.  Bruce Sterling pointed out in his seminal sermon from the flatlands at the Layar Developer Conference, AR is kind of a “Hollywood scene.” We have seen the web early adopter/developer/blogger community  embrace augmented reality browser experiences in recent weeks in an awesome wave of enthusiasm. Are Hollywood creatives equally smitten? For the answers see the full interview with Brian Selzer below.

Brian Selzer (www.brianselzer.com and twitter – brianse7en ) has an extensive involvement with emerging platforms:

“from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC [user generated content] projects (mashade.com, instafilms.com).”

I began my own career in Hollywood doing motion control photography and creating software that bridged the language of robotics and servo motors with the visions of  film directors. Eventually our little company, NPlus1, moved on to 3D vision systems and image recognition stuff.  So yes, I have been really, really patient waiting for this particular techno visionary dream.  And, while I have been waiting for augmented reality to manifest, I have grown to love the internet.  But now, how awesome, It is OMG finally for mobile AR!

Augmented reality is busting out all over – through our laptops, our phones, on the streets, toys, baseball cards, art installations, sticky light calligraphy and more.

Many of my questions to Brian were directed at at how and when we will see augmented realities with near field object recognition, image recognition and tracking and, of course, the illusive eyewear.  As Bruce Sterling points out we are just at the very, very beginning – the dawn of an industry.  I created the photomontage below on the right to compliment Tonchidot’s illustration suggesting the evolutionary inevitability of holding our phones up (below on the left).  The Evolutionary Reality of AR will not end there.  It is just a step into eyewear, hummingbirds or Nano Air Vehicles, and more…….

The Evolutionary Reality of AR

Cartoon on the left by Tonchidot on the right a collage of a stock photo and the Pentagon’s Robot Humming Birds – “Nano Air Vehicles.”

While we finally we have, an affordable mediating device with the horse power, mindshare and business model to bring AR mainstream with the iphone.  The much anticipated Apple 3.1 Beta SDK to be released in September will not, I am sure, open up the Video API at the levels that augmented realities with near field object recognition and tracking require (I would love to be proved wrong though). But the magic wand to deliver even tightly registered AR graphics/media (that require a lot of CPU and GPU) to a wide audience is in our hands, so full access to may not be far off. And others, of course, can/will/might knock the iphone off its current pedestal.  AR made it’s mobile phone debut on the Android after all.

Like everyone else who loves AR, I wish that Apple would open up faster (and I wish Android would manifest on some rocking hardware). But we will see enough of the iphone Video API open for the next generation of mobile augmented reality games and applications to emerge in the coming months.

One of these will be Ogmento’s. Although Ogmento is in stealth mode, they have released a teaser for their first game, “Put A Spell,” developed by ARBalloon – screenshot below.  Ori did reveal to me in this interview that they are doing image recognition and using the Imagination AR engine.

As Brian notes, Hollywood has had the AR bug for a long time. AR has been everywhere in Science Fiction Movies and video games. Nintendo’s SPD3 head Kensuke Tanabe, “effectively the man in charge of overseeing all the Metroid franchise underneath original co-creator Yoshio Sakamoto,” explains the story of Metroid to Brandon Boyer of Offworld here (an image of a Metroid Hud on the right opening this post) :

“the idea of the different visors you use in the Prime games to interact with the world: the scan visor, for instance, set the game apart from other first person shooters in that the player was using it to proactively collect information from the world, rather than having the story come to them passively, in the form of cut-scenes or narration. “Prime could have adventure elements with the introduction of this visor,” says Tanabe, “That’s how we came up with the genre — first person adventure, instead of shooter.”

But as Brian points out:

“the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear).”

Talking with Brian Selzer

Tish Shute: Bruce Sterling’s sermon at the Layar Developer conference, “At the Dawn of the Augmented Reality Industry,” was absolutely awesome. He spread the future feast/orgy of augmented reality before us  – and described many of the dishes we will tasting both delectable and diabolical.  One of the many things he points out is, AR is kind of a “Hollywood scene.” And, as Ogmento is one of only two augmented reality companies in Hollywood, I am interested to hear how it looks from your neck of the woods. We have seen the web early adopter/developer/blogger community  embrace augmented reality browser in recent weeks in an awesome wave of enthusiam – are Hollywood creatives catching the buzz?

Brian Selzer:  It was a thrill to hear Bruce Sterling mention Ogmento. I devoured all of his Cyberpunk books back in the 80′s, along with writers like Gibson, Rucker, Shirley… To me, sci-fi writers are the visionaries who define and influence our technological paths into the future. They make science and tech sexy enough to want to manifest those experiences in the real world. Clearly Bruce sees the AR industry as being sexy. I love that he called it “a techno-visionary dream come true… and super-cyberpunk.”  And yes, kind of a Hollywood scene.

Hollywood creatives caught the AR bug before they knew what AR was.  Look at science fiction movies and video games to see AR everywhere. Terminator, The Matrix, Minority Report, Iron Man.. the list goes on.  Look at any video game with an integrated heads-up display.  It’s clear Hollywood loves AR.  It’s only been in the past few months though that the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear). So yes, the buzz is here and it’s strong.  With that, we all have to be prepared for the good, the bad and the ugly as AR goes mainstream.

It certainly goes to show how young this industry is when Ogmento and Total Immersion are currently the only AR companies based in Los Angeles. It’s very exciting to be the only company right now demonstrating a natural feature tracking (markerless) iPhone experience in Hollywood. We are in talks to bring some very big brand and properties to the mobile AR space. The goal is to deliver experiences that create added engagement and value to the consumer.

Tish Shute: Also in his landmark sermon Bruce Sterling noted that augmented reality has been around for 17 yrs and now at last we are seeing the dawning of  an augmented reality industry. What inspired you to take up the challenge of launching an augmented reality company in Hollywood?  Oh congrats that Bruce Sterling name checked Ogmento in his list of companies that prove that this really is the dawn of an industry!

Brian Selzer: I’ve always been involved in emerging platforms… from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC projects (mashade.com, instafilms.com). Working with all these great CG game assets, I continued to think about what’s next, and that’s when I started to follow AR very closely and started engaging with those who were pioneering in the space.

I remember swapping instant messages with Robert Rice (@robertrice) right after the 2008 Super Bowl.  We were not chatting about the football game, but rather about some of the commercials that aired during the event as a sign that AR was making its way into the mainstream.  A lot of people became aware of AR for the first time when the GE SmartGrid commercial aired.  There were all these YouTube videos popping up of people blowing on holographic wind turbines.

The commercial that really got me excited though was the Coke Avatar commercial.  In that commercial people in the city were sporadically being portrayed as their digital persona’s, avatars, gaming characters, etc..  For me that spot did a great job showing how many of us already have these ‘alter egos” that live in cyberspace, and how the line between these worlds can sometimes be blurred. I remember watching that commercial and thinking that is exactly the type of experience I’d like to create with mobile AR.  I want to overlap the virtual world into our every-day reality. Why cant I bring my World of Warcraft or Second Life persona with me into the real world?

I am big on the notion of “Games and Goals.” I believe that games have the power to motivate people in a very powerful way. By challenging ourselves while playing a game we can climb mountains.  Augmented Reality is the perfect platform to bring gaming into the real world.  By mixing the virtual world with the physical world, this added layer of perception provides a very powerful experience for something like a role-playing game.

One of my earlier social-gaming projects was a website called Superdudes.  This was a “Be Your Own Superhero” concept that celebrated and motivated kids to create superhero avatar/persona’s online, and we gave members all sorts of games, challenges, and rewards, some of which carried into the real world. The site recognized members for teamwork, creativity, volunteer work and things like that. So the Superdudes were often involved in charity events and benefits to help children.  Everybody called each other by their Superhero names, and the line between fantasy and reality were being blurred.  This project really got me thinking about what happens when you take positive role-playing like this and mix it into the real world.  I started to work on a plan for location-based activist missions for points and rewards, but never got to complete that. So I have some unfinished business here.

I think it would be fantastic to be able to show up to some type of fun event with friends, and everybody could see each others alter ego personas standing before them. When you can turn the world into a playground, and use the power of gaming to make a positive impact on the planet… well, I don’t think there is anything better than that.  These are the types of projects that drive me, and I think AR is the best platform to support these types of social gaming experiences.

Tish: Does Ogmento have any RPGs under development?  I noticed in the Google Wave on RPG someone has been working on doing something with the Dungeons&Dragons API.  I am interested in exploring the web of protocols underlying Wave as a transport mechanism for multi-person, mobile, AR experiences (not requiring downloads), on an open global outdoor AR network. If not Wave, what do you see as the potential infrastrucure and protocols we could harness for an open augmented reality network?

Brian:  Ogmento has a deep background in video games and we interact regularly with most of the major game publishers. As a company we are not so much developing our own RPGs right now, but rather exploring what mobile AR extensions make sense for existing brands.  There are many limitations to location-based gaming, but a global AR network is exactly along the lines we are thinking.  Lots of discussions are taking place on protocols, platforms, API’s, and there are numerous ways to approach this.  We need to be able to use what’s available now and continue to refine and customize for AR’s specific needs and issues as we progress.

In general though, Ogmento is focused on what types of experiences can be had today and over the next couple of years. I still think we are several years out from a truly open augmented reality network.  We are certainly looking at launching our own “Ogmented Network” which would support some fun treasure hunt type experiences, or add an entertainment layer on top of traditional outdoor marketing campaigns.

Tish: I don’t know whether you have read Thomas Wrobel’s ideas for an open augmented reality network that I just published here on Ugotrade.  The principals he talks about are very important for augmented reality to become a major part of our lives – .  Considering the difficulty open networks can pose for emerging business models how can we fund the development of an open framework for augmented reality?

“a future AR Network, I mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.
Where people can just point their phone, webcam, or pair of AR glasses anywhere were a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.”

Brian: I think funding for these types of projects will definitely come from Venture Capital groups in the near future.  It’s early in AR, but the VC’s are watching and deciding which horses to bet on.  Until that time, it’s about service work, and developing AR experiences for others with what is possible today. That work will help fund internal development of original AR products, and platform development.

Tish: How did you get started with Ogmento?

Brian: My first conversation with Ori was actually about my interest in Location Based RPG concepts.   We had a long conversation about the possibilities with AR, and it was clear that we shared similar interests, but were coming from different complimentary backgrounds. The idea of collaboration was exciting, so we just kept talking until the timing felt right. Now, with Ogmento we bring a unique blend of AR development experience with a deep backgrounds in AR technology, animation, video games, entertainment, social media, etc.   I think this is a powerful mix that will allow us to do some great things.

It’s still so early, and things are just getting started in AR. There are only so many webcam magic tricks you can enjoy before you are ready for something else.  The location-based apps have the most potential in my opinion, which is why we are really focused on mobile AR.   We have some board-game type projects, which do not instantly scream location-based gaming, but if you look at something like the ARhrrr board game, you can see how much more compelling it can be when the game invites the player to be actively moving around during the experience.

Tish: I am interested in your perspective on how we can create the kind AR experiences that really embody what has always been so exciting about AR – the tight alignment of graphics and media with real world objects and ultimately a rich immersive 3D experience, so I am going to hit you with a bunch of those, “Is this really eyewear or vaporware?” questions.  The real deal eyewear changes everything!

While eyeware is a big challenge technically and aesthetically,  I am pretty sure that there are several outfits out there that can pull off the optics and projection. 
Will the entertainment industry get excited enough to put a major push into delivering the eyewear in short order instead of the 5 to 10 year project that some people still think it is?    The business development challenge is bigger perhaps than the technical obstacles perhaps? What is your view on this?

And, perhaps, the eyewear is a clear example of a need for partnerships. For example, we have seen efforts from companies like Vuzix and Lumus, and recently a Japanese Company, Masunaga.

I have no reports from people who have tried the Maunaga eyewear yet.  But,  limited by small field of view, and tethered, currently eyewear offerings, available at a reasonable price point, are not workable solutions for augmented reality experiences. But the problems are not insurmountable. What will facilitate the real deal?  


It seems that it is critical to start creating hardware relationships now. The industry is costly and slow moving and as Robert Rice put it to me in a recent conversation, “once the software cat is out of the bag, its going to go wild and if the hardware isnt there, its going to stutter.”

As Ori notes some of the hardware companies like Intel and others don’t seem to be paying enough attention to AR.  Ori points out they don’t see the demand yet.  But in order to create an awesome AR experience and demand from a mass audience, don’t we need to work in conjunction with hardware designers?

Brian: It’s fun to think about who will eventually deliver a great hardware solution for AR glasses. It will happen. It would be cool to see somebody like an Oakley or Nike partnered up with a company like Vuzix to deliver something people actually might wear in public.  Perhaps a hardware manufacturer like Apple or Nokia will bring us something like the iSight or the NGaze down the line.  I’d love to see a set of glasses designed by Ideo.  Microsoft or Sony are already playing with technologies like Project Natale and the EyeToy, so I think its only a matter of time before they deliver an eyewear solution. I would even look to the toy companies to eventually make an investment here.

Gamers will be the early adopters, and in a few years we may start to see people running around in the park wearing glasses with headsets, but it will be acceptable because it’s clear they are using them for a game.  It’s going to take a very sexy and stylish piece of hardware for everyday people to be willing to wear AR glasses in public while going about their everyday business.   It’s like the recent cover of Wired magazine where Brad Pitt is wearing a mobile headset in his ear, and the editors point out that even he can’t pull that look off, so why do you think you can.  When AR glasses come in designer frames, and you can’t tell them from non-AR glasses, to me that’s when things get really interesting from a mass-adoption perspective.   Compare how many people were carrying around a mobile phone in the 80s to now.  I think it will be the same thing with glasses.

I was in an AR pitch meeting the other week at a very significant media company, and brought up the point that today’s handheld Smartphones will eventually evolve into tomorrows Smartglasses. My comment was quickly shrugged off as sort of a sci-fi notion that was irrelevant to the business at hand.  Probably true, but I think it is important to understand where digital media and entertainment is going, so you can adapt quickly, and evolve into those spaces more naturally.  The more we see people walking around with their Smartphones in front of their face (like a camera), the sooner it will be that we make the jump to eyeglasses as a key hardware device for AR experiences.

At Ogmento, we definitely are working on AR experiences with the hardware and software available today.  We will get some product out this year, and 2010 will be a banner year for markerless mobile AR in general.  I think the entire AR community is looking forward to bringing this technology to the mainstream in the form of games, marketing campaigns, virtual docent apps, and much more.  It might not be the full experience we are all dreaming about for some time, but we can see the path and the true potential, and it’s pretty spectacular.

You mention the tight alignment of graphics and media with real world objects.  That is really our focus. A lot of well-deserved attention is going to the browser overlay “post-it” approach right now, which uses compass and GPS.  We are focused on markerless natural feature tracking, so once you identify something that is AR enhanced in your environment, you can interact with that integrated experience.  On an iPhone that can be as simple as using your touch screen to interact.  When you are wearing glasses, it becomes more about visual tracking. There are lots of smart people thinking through these issues. Many of which you have interviewed. It is my hope that there are exciting collaborative efforts to be had in the coming months to get us all there together and faster.

Tish: Bruce touched on some of the hard problems that have to be solved for augmented reality – and he noted for instance security needs to be tackled in the early stages. Robert made a nice list, “privacy, media persistence, spam, creating UI conventions, security, tagging and annotation standards, contextual search, intelligent agents, seamless integration and access of external sensors or data sources, telecom fragmentation, privilege and trust systems, and a variety of others.” Will Ogmento be leading the way in solving some of these hard problems?

And, won’t trying to solve these hard problems for networked AR in walled garden scenarios one company at a time lead to a lot of reinventing the wheel wasted energy?

Brian: These are all important issues, and again there are a lot of smart people thinking about solutions to these problems on a daily basis.  Ogmento is interested in partnering with developers and supporting their efforts as a publisher of mobile AR experiences.  While we intend to roll up our sleeves in these areas, we are currently more focused on taking AR mainstream with the hardware and software available today.  As the industry evolves, so will Ogmento. As the opportunities evolve, our ability to make a greater impact tackling these issues will be realized.

Tish: Another area of development that could really kick AR into high gear might be creating augmented reality hotspots  where we use can deliver the kind of location accuracy/instrumentation necessary to create interesting AR experiences (partnership with Starbucks, perhaps ?!).  Augmented reality hots spots, could deliver the kind of high quality AR experience that isn’t possible ubiquitously at the moment, and may be a real way to get people really exploring the potential of AR now, rather than later?

Brian:  Agreed. I see a great opportunity here with this approach.

Tish: Although there are many obstacles to Green AR – the energy hogging servers at the backend for starters! Last week I had a conversation with Gavin Starks, AMEE, and Robert Rice and Jim Purbrick about how to work with AMEE and the technology available and encourage Green Tech AR development (see an early exploration of green tech AR from Pachube here).

We came up with the idea of holding a competition perhaps centered around a targeted instrumented space. But I would really love to hear your thoughts on the topic of Green Tech AR (the energy hogging servers at the back end being the first cloud on the horizon!.)  Cool GreenTech AR imaginings, social gaming ideas, RPGs, not even necessarily even tied to the immediately practical, would be like rain in a drought!

Brian:  I go back to “Games and Goals”… If you make environmental and other activist efforts fun and rewarding, more are likely to be motivated and participate.  Can you imagine having a personal “carbon footprint stat” floating over your self at all times? Or over your home or factory?  How would that change your behavior?  We all love stats. Look at how the Nike+ campaign has used technology and gaming to motivate people to run.  I think there is a lot that can be done to make being green fun. It starts with the individual, and spreads from there.  Keep me posted on that one!

Tish: I would also like to explore further the RRW suggestion that ambient intelligence is both the Holy Grail of AR and possibly snake oil:

“The holy grail of the mobile AR industry is to find a way to deliver the right information to a user before the user needs it, and without the user having to search for it. This holy grail is likely in a ditch somewhere beside a well-traveled road in the district of the semantic Web, ambient intelligence and the Internet of things. Be wary of any hyped-up invitation to invest in a company that claims to have gotten the opportunity right. What we’ve seen in the commercial industry to date is a rather complex version of a keyboard, mouse, and monitor.”

So Holy Grail, Snake Oil, or a ditch somewhere….?

Brian:  I instantly think of Minority Report, where Tom Cruise’s character is being bombarded with holographic ads personalized with his name and to his current situation.  In the future, Spam is a nightmare, especially when it knows who you are.  I think the key thing here is delivering “the right information”, and we still dont have that down. I do see a day where we can truly customize what comes to us, how we want it, when we want it.  My future vision of ambient intelligence is the ability to “turn everything off” if I want to… block out the stimuli and replace it with images of nature, or natural surroundings, etc.  Where I live in Los Angeles, we have those digital billboards everywhere, so it’s like advertising overload wherever you look (hints of Blade Runner).  I personally don’t mind them, but I know there is great debate on there being simply too many billboards everywhere. So AR would only add to the noise of life by adding yet another digital overlay of information, right?

Perhaps the holy grail is to use technology to filter things out. AR might become a solution to leading a simpler life, or a perfectly customized life if you want that. Ultimately the control needs to be with the individual.  I guess I am talking about something like TiVo taken to the extreme.

Tish: And then that other biggy – augmented reality search! I am asking this next question of  Wikitude and Sekai camera too and now I must also ask Acrossair and several others I guess! Obviously a huge area of opportunity in this broader landscape that uses location-awareness, barcode scanners, image recognition and augmented reality is to harness the collective intelligence – a whole new field of search. There is the beginning of a discussion on this in the comments here.

What will it take, in your view, to become a leader in augmented reality search?

Brian:  I’m more of a content guy, so I tend to focus on things like UI, quality of creative, etc..  From that perspective, I am looking forward to evolving beyond the “post-it” text overlay user-experience we see now in AR search. I was impressed with the TAT Augmented ID concept and hope we start seeing more smart design solutions like that emerging in the space.  There are some great new design approaches coming out of the location-aware space that should be applied to AR search. I’ve been studying the heads-up display designs being used in video games, and re-watching movies like Iron Man for ideas. This is another example where Hollywood has painted a polished picture of what AR can and should look like, and the masses have already accepted these design approaches.  So from that perspective, from my view the leaders in search will be delivering sexy, smart and simple solutions. It’s all about the S’s.

Blair MacIntyre is one of the original pioneers of  augmented reality and an extraordinary amount of creative work is coming out of his Augmented Environments Laboratory at Georgia Tech – see YouTube videos here.  The screenshot below is from, ARhrrrr, a very impressive augmented reality shooter game created at Georgia Tech Augmented Environments Lab and Savannah College of Art and Design, (SCAD- Atlanta), and produced on the NVidia Tegra devkits – watch the demo here.

Blair has spent much of his career working on immersive augmented reality and more recently the integration of augmented reality with mirror worlds. Blair explains:

“I am interested in the intersection of mobile devices – whether they are head mounts or handhelds – and parallel mirror worlds…I think that parallel mirror worlds are a direct manifestation of the intersection of the virtual world we now live in (the web) and geotagging.  As more and more information is tied to place, and as more of our searching become place-based, we will want to do those searches about places we are not at.  A 3D mirror world may provide one interface to that data.  Want to plan your trip to London;  go their virtually and look around, see what is there (both physically and virtually), teleport between areas you want to learn about, and so on.  More interestingly, talk to people who are there now, and retrieve your location-based notes when you are on your trip.”

But, at a time when many augmented reality developers are focusing on AR apps for smart phones, including Blair (the picture on left opening this post is Blair’s augmented reality iphone app ARf), I was interested in finding out from Blair what the state of play was for the real deal Rainbow’s End style AR, as well as the potential he sees in smart phones to mediate meaningful AR experiences.

There is enormous amount of  innovation in mapping our world, see my post, “Location Becomes Oxygen at Where 2.0 and WhereCamp,” and  Ori Inbar’s  Where 2.0. conference roundup. But as Ori notes, to move augmented reality forward:

My point is not a shocker: all we need is to tap into this information and bring it, in context, into people’s field of view.

And this is what Blair MacIntyre’s work is all about.

Talking With Blair MacIntyre

Tish Shute: There do seem to be broader implications to augmented reality today than when this term was first coined. I am interested to have your perspective on how augmented reality may go beyond some of our early definitions?

Blair MacIntyre: I still think the original definition of the term is useful:  media (typically graphics) tightly registered (aligned) with the physical world, in real time.  Many people talk about many things that relate virtual worlds to places, spaces, objects and people.  There is room for many of them, and they don’t all have to “be” augmented reality.  I like using Milgram’s definition of Mixed Reality as everything from the physical world (at one end) to the virtual world at the other;  it’s a spectrum, and augmented reality just sits at one point.

The reason I like the old definition is I believe there is something special about graphics that are tightly, rigidly aligned with the physical world.  When things appear to stick to the world, and an obviously identifiable location, people can start leveraging their natural perceptual, physical and social abilities and interact with the mixed world as they do the physical world.  We’ve found this with the two studies we’ve done of tabletop AR games (Art of Defense and Bragfish);  one key to those games is that the graphics were tightly aligned with identifiable landmarks in the physical world (gameboard).

Art of Defense (pic on left) Bragfish (pic on right)

Tish: I know that you are involved with ISMAR 2009 which is the key US augmented reality conference.  What do you think will be the hot themes, applications, innovations at this year’s conference? Do you think this will be the year that AR really breaks out of eye candy into truly useful and sustained experiences?

Blair: Unfortunately, I won’t be involved this year.  I was supposed to be helping run the technical program, as well as the art/media program, but sickness in my family prevented me from having the time, so I am not helping this year.

First, I would not agree with the implication of the last question — I don’t think AR has just been eye candy up to now.  I do agree that the “high profile” uses of it have largely been that, which is mostly because of the limits of the technology.  I don’t think we’ll see huge changes in that regard by ISMAR this year.  However, we will hopefully see a mixing of communities that hasn’t happened at ISMAR before, and I do believe that this year (independent of ISMAR) we will see more and more AR apps.  Whether they go beyond eye candy is still a question.  I’m hoping that some folks (including myself and other ISMAR folks!) will help push AR in new directions.  But I also expect many folks new to ISMAR and AR to play a big role, because it is this new blood, especially those folks with real problems to solve, new art and game ideas, and a fresh perspective, that will open new doors.

Tish: You have been working on integrating augmented reality with virtual worlds. You mentioned that the way you use Sun’s Wonderland is really about pulling the virtual world into the real world, i.e., Wonderland, “is just a place to put data.”  How is your use of the persistent virtual space different from what we have become accustomed to call virtual worlds?

Blair: The approach we are taking in our project at Georgia Tech is to use the virtual world as the central hub of the information space, and allow the virtual world to be the element that enables distributed workers to collaborate more smoothly.  This is work we are doing with Sun and Steelcase (and the NSF), and is an outgrowth of a project (the InSpace project) that’s been going on for a few years.

What we are trying to do is use mixed reality and ubicomp techniques to pull as much of the physical activity into the virtual world, and then reflect that activity back out to the different participants as best suits their situation.  So, folks in highly instrumented team rooms will collaborate in one way, and their activity will be reflected in the virtual world;  remote participants (e.g., those at home, or in a cafe or hotel) may control their virtual presence in different ways, but the presence of all participants will be reflected back out to the other sides in analogous ways.  We may see ghosts of participants at the interactive displays, or hear their voices in 3D space around us;  everyone will hopefully be able to manipulate content on all displays and tell who is making those changes.

A secondary benefit, I hope, is that by putting the data in the virtual world and making that the place that gives you more powerful and flexible access to the data (e.g., by leveraging space and giving access to history), distributed teams will begin to have the virtual space become a place they go to work, bump into each other and have those casual contacts co-located workers take for granted.

Creating the Information Landscape of the Future

Tish: At the end of my interview with Ori Inbar he said, in order to have a ubiquitous experience “you’ll need to 3d map the world. Google earth like apps are going to help but it is not going to be sufficient. So let’s leverage people. Google became successful in part by making people work with them.  Each time you create a link from your blog to my blog their search engines learn from it.  So let’s find ways to make people create information that can be used for AR.” What ways do you think people can create information that can be used for AR?

Blair: I think the big part of that is the creation of models and environments, the necessary “baseline” for specifying experiences.  Google and Microsoft are clearly working toward this;  recent videos from Microsoft show them starting to move the photosynth work toward Virtual Earth.  Similarly, I came across a page where people are finally starting to mine geotagged Flickr [see my post, “Location Becomes Oxygen,” and here for more on the “The Shape of Alpha” project from Flickr]  images to create models.  It’s that kind of thing that will be useful first;  using the data we all create to enable modeling and (eventually) vision-based tracking in the real world.

After that, it’s a matter of time till more of what we “create” (e.g., Tweets and blog posts and so on) are all geo-referenced;  these will become the information landscape of the future, the kinds of things people think about when they read “Rainbow’s End”.   The big problem will be filtering, searching and sorting.  And, of course, safety and security.

Tish: You are working with Unity3D to research the integration of mobile location based AR with persistent mirror world like spaces.  What has attracted you to Unity? What is the difference between this and your Wonderland project? I know you mentioned. you will be using head-mounted displays are part of this Unity project. What are your goals for this project?

Blair: We started to use Unity3D because it gave us what we wanted in a game engine.  Most importantly, it’s very open and let us trivially expose AR technologies into the editor.  Similarly, it can target the iPhone, so we can begin to work with it on that platform, too.  The biggest problem with creating compelling experiences is content;  and a show stopper for creating content is not getting it into your engine.  Unity has a nice content workflow.

Unity3D is a front end engine, for creating the game;  Wonderland is both a front end, and a backend.  We are actually looking into using the Wonderland backend with Unity as well.  Wonderland also has growing support for doing “real work” in a virtual world, which is key to our other projects.

Eventually, we’ll be using HMD’s.  The goal for the Unity3D project, initially, was to explore what you can do with an AR/VR mirror-world; this is a project are working on with Alcatel-Lucent, and demo’d at CTIA this year.  It’s continuing to grow, though, and now includes a number of our projects, including some work on mobile social AR and soon, some performance and experience design projects in the area of AR ARG’s.  It’s really quite interesting to imagine what you can do when you have an “MMO of the real world” (which we now have for part of campus) that supports both VR-style desktop access simultaneously with mobile AR access.

Tish: Have you taken another look at OpenSim as a possible backend for augmented reality?  Recently I talked to David Levine, IBM and he is thinking about some possibilities to optimize OpenSim to dynamically load a large amount of objects at once (i.e how fast OpenSim can bulk load into an existing sim) and make it better suited to augmented reality/mirror world type projects.

Blair: I haven’t looked at OpenSim recently.  We will probably look at it this summer.

Tish: Why did you select Unity as a good client for augmented reality?

Blair: Unity is a 3D game authoring environment so at some level it is no different from using Ogre, if all the associated stuff was just as well done. It has integrated physics, scripting, debugging, etc. – you can write code in javascript or C# or whatever.   It has a good content pipeline, as well, and supports a range of platforms.

It has simple networking built in, so multiple unity engines can talk to each other but it is not a virtual world platform out of the box – there is no back end …

Tish: Someone described Unity to me as a great client waiting for a great backend? So what are you going to use as a back end?

Blair: There is no real processing except in the client right now.  We will eventually have to create a back end.  We are thinking of using Dark Star because someone on the Sun Wonderland community forums has already built a set of scripts connecting Unity to Darkstar.

But for us, we are not proposing right now to build a real product.  This is research to demonstrate what you could do if you actually had the back end.

Tish: What are the most important aspects of the backend from your POV?

Blair: We want to simulate a variety of the interesting aspects of the back end.  So I very much care about notions of privacy and security and how these sorts of AR/VR Mirror Worlds would work in practice.  But I care about how those things as they impact user experience, not really about how we would really implement them.

Tish: So looking at some of the big problems from the perspective of user experience? Are we are going to go through the same growing pains that the web and VWs have seen, for example, will we have to type in passwords to get into everyone’s little worlds….

Blair: Well you know the SciFi background to this, you’ve mentioned it in other posts on your blog.  Because when you look at the Rainbow’s End model where you have security certificates flying around, that is in effect what cookies and so on are now.  You can authenticate yourself once and then have those certificates hang around. So you can easily imagine how it could be done.  But the big question is how does that change user experience.  There are all kinds of things that start coming into play – like what happens if nearby people see different things – it goes on and on!

Tish: Sounds Like this is very valuable research.  It seems to me that there will be a lot of investment soon in putting the pieces together to do location based markerless AR and it would be nice if we knew more about it from the user experience POV.

Isn’t it vital for a productive intersection between mobile AR and persistent mirror world spaces for us to have markerless AR?  Aren’t we right at the beginning of people really saying yeah markerless AR is doable now? But it seems to me not many people researching or working on fully immersive AR and its integration with mirror worlds?

Blair: I think some of the AR community is thinking about this. There’s probably people who are doing stuff in some other non technical communities. It wouldn’t surprise me to find out that people in the digital performance or ARS electronica world who are thinking a little bit about these sorts of things. Although not necessarily at the level of actually trying to build it, because they probably can’t right now.  But experimenting with the precursors.  My colleagues in digital media like to point out that this is often the purpose of digital art, to point out new directions and push the boundaries.

Obviously Science Fiction has explored the possibilities because that is what Rainbow’s End and the Matrix were all about.

Tish: and Denno Coil…

Blair: There has been some research – people like my adviser Steve Feiner up at Columbia, Mark Billinghurst in New Zealand, myself and people at Graz University in Austria .  But partly it has been so hard to do mobile AR up to now – so many people mock head worn displays and can’t get past current technology – you have had  to be willing to ignore the bulky back packs and cables and batteries and so on.  That is changing which is good.

My current response to the anti-head-mounted display people is if 5 years ago you told me you told me that fabulously dressed people who care about their looks and wear stylish clothes would have had big things hanging from their ears that blink bright blue light, so they could talk on the phone, many of us would have said you were crazy, because it would be ugly and so on.  But because there is an intersection of demonstrable need and benefit…Bluetooth headsets are really useful and the sort of early gestalt feeling that grew up around them – that people who use them are so important that they always have to be in touch, they wear these things – so people accept them.

It will likely be a similar thing with head mounted displays. And I don’t know if it will be that people wearing them so that they can read their mail while driving, god forbid. But it will be something.  And when we get the 2nd generation of the wrap glasses that look more like sun glasses and are not bulky and so on, we will have the potential for them catching on because you will look at them and you will think that the person is wearing because they are doing x…

X might be surfing a virtual world or reading their email or keeping in touch, or being aware. It will happen. But they have to get unbulky enough and there has to be more  than one important application, not just watching TV.

Picture above shows  an outside view of the KARMA AR system;  the knowledge based maintenance system Blair built in his first year of grad school (“first AR system Steve Feiner, Doree Seligmann, and I worked on”).  Blair noted, “The Communications of the ACM paper on it (from 1993) is a pretty widely cited AR paper.”

Tish: I think the need for  full on transparent, immersive, wraparound, Gucci stylish eyewear with a decent field of view are the elephant in the room in terms of realizing the full potential of augmented reality.  There are a few new players in the field Digilens,  Vuzix, others?  What is the progress in this area and what do you hope for in terms of near term solutions?

Blair: I agree with that sentiment.  I think that, in the near term, there is a lot we can do with handhelds, as we’ve been doing in the lab.  However, because it’s awkward and tiring to hold up a device, even a small one, for any length of time, handhelds will only be good for “focused” uses of AR.  Such as the table-top games we’ve been doing, or the constellation viewing app that I heard came our recently for the Android G1.  I don’t even see something like Wikitude as that compelling (beyond the “gee wiz” factor) for a handheld form factor.   Many proposed AR apps only really become compelling when users have constant awareness of them, and that requires a see-through head-worn display.

I’ve seen the mockups of the Vuzix ones;  they seem pretty interesting, and are getting to were early adopters could use them (they will be cheap enough, and will hopefully be good enough).  Microvision’s virtual retinal display is also promising;  the contact lens displays will be the most interesting, if anyone can ever make them work.   I don’t know of anything else out there.

“its not really a killer app you care about, it is the killer existence that all of the technology and small applications taken together facilitate”

Tish: While location based services are accepted now and people are understanding that it is something that opens up a new relationship to everything, we still haven’t found the experience that will get everyone holding up their mobile devices?

Blair: Well that is actually the killer problem.  Gregory Abowd is one of my colleagues who does ubiquitous computing research here at Tech.   Way back when we started the Aware Home project (Aware Home Research Institute at Georgia Tech) when I first got here about ten years ago, there was always this question of what is the killer app.  So Gregory comment in a meeting once that its not really a killer app you care about, it is the killer existence that all of the technology and small applications taken together facilitate. It is not that any one of these AR demos we see right, whether it is seeing your photos in the world or whatever, is important. Its that when taken together, there is enough of a benefit that you would use the whole environment.

In the original context we were talking about an instrumented home, but it is the same thing here with AR.

The problem with the mobile phone as a AR device is that problem of awareness. If I have a head mount on and I walk down the street and there is bunch of probably-not-useful-but-potentially-useful information floating by me, that’s a good thing, because I may see something that is useful or makes me think of something else.  But if I have to hold up my phone to see if something might be interesting nearby, I will never hold up my phone because at the time there is a high probability that there won’t be anything particularly important there.  You might imagine you can get around this by using alerts or something like that, but then you overload whatever alert channel you use.  For example, I forward maybe 5 or 6  people’s updates from Facebook to my phone – started with my wife, a few friends, my brother, and the net result of that is I never get SMSs’ anymore because when my phone buzzes, usually I ignore it because it is probably just somebody’s random Facebook update. So if we start overloading channels like that with “oh there might be something useful here in the real world, if you pick up the phone and look through it you will see it … and I will buzz you.” People  just start ignoring the buzzes.

So it is a very hard problem if you think about the kinds of applications that people always imagine with global AR — names over peoples heads and other random information floating in the world — until you have a head mount and all that information is around you all the time. That is when those sort of applications will actually happen.

Tish: Robert Rice notes: “AR is inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.” (see my interview with Robert, “Is it ‘OMG Finally’ for Augmented Reality?). And I think the iphone experience has laid the foundation for the increasing desire to experience the network wherever we are – and not be stuck behind a pc.  We cannot perhaps do all we want to do yet. But even in the range of things we can do know, we are not even sure exactly what it is we want to do where yet is it?

“imagine your iphone Facebook client supports AR and that all data on Facebook might be georeferenced – pictures, status updates etc…….”

Blair: Yes that is a huge problem. I have been lucky to be able to teach two fun classes this year that let the students and I start to explore some of the potential that handheld AR might bring.  Last fall I taught a handheld AR game design class — coordinated with a class at the Savanna College of Art and Design’s Atlanta campus — and we had the students build a sequence of prototype handheld AR games, which was a lot of fun.   This spring I taught a mixed reality/augmented reality design class with Jay Bolter (a professor in the School of Literature, Communication, and Culture here at GT).  Jay and I have been teaching this class off and on for about 9 years; this semester we decided to say to the students “imagine your iphone Facebook client supports AR and that all data on Facebook might be georeferenced – pictures, status updates etc…….” and have them do projects aimed at such an environment.

Tish: Not many of our favorite social media today have much sense of location do they? But Flickr  are  utilizing the geo-referenced pictures to create vernacular maps…..The Shape of Alpha

Blair:Yes that is because lots of cameras put geo location data into the exif data so they can extract it…

Some mobile Twitter clients like the one I use in my iphone will let you add your location.  But in general Facebook and other sites don’t have any notion of location. But if you look at all the things people do in Facebook, such as sending gifts and other games, its easy to imagine what these might look like with geo-reference data.  So, the high level project for the class is the groups have to design experiences people might have using mobile AR Facebook.  We told them to assume Facebook as it stands now, but add geolocation and AR to the client.  The class boiled down to “What would you imagine people doing?” So it has been kind of fun.

And we are using Unity for the class too – the same infrastructure I am working on in my research linking mobile AR to persistent immersive mirror world type spaces – and we having the students mock up what a mobile AR Facebook experience would be like.

Tish: Can you describe some of the ideas you class came up with that you think have potential? I know Ori mentioned that from the games class he liked Candy Wars.

Candy Wars

Blair: In the end, they had a nice range of projects in the Spring class.  One created tag clouds out of status messages over spaces, others looked at analogies to virtual pets and gift giving out in the world, one looked at leveraging geolocation to help with crowd-sourced cultural translation, and three groups did straight-up social games.

[See all of the projects from the handheld AR games class on YouTube here]

iphone, Android, or NVidia Tegra devkits or the Texas Instrument’s OMAP3 devkits?

Tish: Is anyone in the class working on Android?

Blair: Nobody is using Android because no-one in the class has the phones. We have ATT microcell infrastructure on campus.  Some ATT people joke that we are better off than them because we have a head office on campus so we can build in the network applications which people even at ATT research can’t do.  But because  we have this infrastructure on campus, and a great relationship with ATT and the other sponsors, we have the ability to provision our own phones without having to pay for long-term contracts, which is vital for research and teaching.

Tish: So does this lock you into the iphone?

Blair: Well the G1 is of course not AT&T but it is GSM so we could probably buy them unlocked and put them on our AT&T network. But the students I work with are much more interested in the iphone right now.

Tish: Is that because the iphone has the market?

Blair: For me the reason I am not interested in the G1 is because you can’t do AR on it – there is Wikitude and a few other apps, but it is all hideously slow.  Worse, because the Java code isn’t compiled like it would be on the desktop, you can’t do computer vision with it, so you can’t do anything particularly interesting on the current commercial G1s.  We could probably take the NVidia Tegra devkits or the Texas Instrument’s OMAP3 devkits (both are chipsets for next gen phones — high end graphics, fast processing),  and install Android on those and we may actually do that yet.  But, it seems like a lot of work right now, for not much benefit.

Augmented Reality shooter game ARrrrr from Georgia Tech and SCAD Atlanta on the NVidia Tegra devkits – watch the demo on YouTube here.

Tish: Everyone seems very excited about the iphone OS 3.0 and the addition of compass. Compass is pretty essential for AR right?

Blair: It is necessary if you can’t do other forms of outdoor tracking, but the problem is that the compass on the G1 isn’t very good, relatively speaking and the iPhone one probably won’t be much better. It does not have very high accuracy, nor is it very fast (compared to, say, the high end 3D orientation sensors we use, from Intersense and MotionNode). As far as I can tell, it doesn’t even give full 3D orientation. I don’t have a G1 (although I have pre-ordered an iPhone 3Gs), but people have told me it only has absolute 2D orientation, so you can only line things up if you are careful.  Your can’t look around arbitrarily…

Tish: You can’t sweep your phone?

Blair: You can look left and right, but if it doesn’t have full 3D orientation, you can’t go up and down. You can’t tilt it in weird directions. It is not fast in the form that you would want to look around quickly.  So it is nice demo.  And it is good for what the Android people use it for which is to let you do your Google street view by looking around, which is actually really useful.

I think there are lots of really useful things you can do with such a compass.

And, it is clear that compass is a necessary feature if we want to do AR.  It’s just not sufficient.

Outdoor Tracking and Markerless AR

Tish: Isn’t it essential for markerless AR? I guess not I just saw this post about SREngine on Augmented Times!

This wasn’t up when we spoke so perhaps you have some comments about what it brings to the table?

Blair: Maybe. The folks at Nokia are working on outdoor tracking, they demoed some stuff at ISMAR last year on the N95 handsets that is all image based.  We are trying to do some work with them, one of my students is working on it.  And probably Microsoft is going to do more on this as well, they had a video up showing that they are also working on vision based techniques.  If you give the phone the equivalent of those panoramic Google Street View images (assuming they are up-to-date) and you are standing at the right place, you don’t really need a compass, you can figure out which way you are looking by looking at the camera video. Ulrich Neumann (USC) did some work on tracking from panorama’s years ago, I don’t know what ever became of it.

Regarding SREngine, that project appears to be a pretty simple first step, but is probably just a demo at this point, and limitations like “only works on static scenes” and “doesn’t work for simple scenes” means it’s probably extracting some simple features out of the image and then matching those to some database.  The trick would be getting this to work on a large scale, where the world changes a lot.   It’s not obvious how to get there.

Tish: So forget RFID for AR…

Blair: RFID is not really useful.

Tish: not at all?

Blair: RFID is useful for telling you what things are near you.  The problem is it doesn’t give you any directional information – it just tells you you’re in range of the tag. So can use it to tell you when you are near a certain product for example.  So it is useful in terms of telling you what thing you are near, and then you can load up a vision system or something else that will recognize that thing.

In that way, it could be useful as a good starting point.

Similarly for computer vision, the compass and the gps are very useful for giving you an initial guess at what you may be looking at that can then speed up the rest of the process.  But, computer vision by itself will not be a complete solution because if I have my panoramic Google Street view (or whatever image database I use for tracking) and you are standing between me and the building -  I am not going to see what I expect to see, I am going to see you.

So I think it is all going to be part of one big package – you are going to see accelerometers, digital compasses, and gps and then combine that with computer vision and other sensors, and then maybe we are going to start getting the things that we have always dreamed about.  I like to show this video from the U. of Cambridge (work done by Gerhard Reitmayr and Tom Drummond) of an outdoor tracking demo because it gives a sense of what will be possible.  Techniques like this will be an ingredient in the future of things.  It becomes especially interesting when you have these highly detailed mirror worlds.  It is sort of one of those chicken and egg problems where if I have an highly detailed model of the world then techniques like they have can be used to track.  But that mirror world needs to be accurate or you can’t use it for tracking, and why would you create the mirror world if you couldn’t track?

Tish: I noticed in your comment to “my interview with Robert Rice” that you said you thought that is was important not to collapse AR into ubicomp – “forgetting what originally inspired us about AR” is, I think if I remember correctly, the suggestion you made. But aren’t ubiquitous computing and AR basically coextensive?

The vision of ubicomp Mike Kuniavsky describes – “sharing data through open APIs and the promise of embedded information processing and networking distributed through the environment” demonstrates how much can be done with very little processing power.” In its most immersive form augmented reality requires a lot of processing power. I think we have all become very conscious about trying minimize levels of consumption.  Can you explain why you think people shouldn’t see AR as the Hummer (energy squandering indulgence) of Ubiquitous Computing?

Blair:  I think there will be a hierarchy of interfaces. You are going to have the rich Rainbow’s End like experience – you are totally submerged in a mixed environment, if you have a head mount on (its not going to be Rainbow’s End for while) but if you don’t have the headmount on that information might be available to you other ways, whether it is a 3D overlay using your handheld or just a 2D mashup with Google maps.  But there will be some circumstances and people who will want to get the compelling experience you can only get with the headmount.

Tish:  Are you doing any research on how all these hierarchies of experiences will fit together – what aspects of this are you looking at?

Blair: The thing that really needs to happen is you need to have this backend architecture that allows you to collect your data from different sources and aggregate it much like the web. Right now Google Earth and Microsoft’s Virtual Earth are much like the old pre-web hyper-text systems that were all centralized. And what we really need is to have the web equivalent where Georgia tech can publish their building models and I.B.M. can publish their building models and their campus models, and your client can aggregate them, as opposed to Microsoft or I.B.M. puts their building models into Google Earth and then somehow you get them out with Google’s google earth browser. That’s just not going to fly.

Tish: so what does it take then to get us to this backend architecture, because I’m in total agreement?

Blair: The nice thing about augmented reality versus virtual reality is that you don’t need everything modeled. You can do interesting AR apps like Wikitude with absolutely no world model.

Tish: So that means we can start with what we have – utilize cloud services without a full blown backend architecture?

Blair: It may very well be that Google Earth and MS Virtual Earth act as a portal because people go and build models and link them with KML, and they can see them in google earth but they can also download the KML’s through some some other channel. So it may be that those things end up being something that feeds some of this along. Then people start seeing a benefit to having these highly accurate models so then you start integrating the Microsoft photosynth stuff and leveraging photographs to generate models.

It’s just keeping up with it and building it in real time is the challenge. A lot of folks think it will be tourist applications where there’s models of times square and models of central park and models of Notre Dame and the big square around that area in paris and along the river and so on, or the models of Italian and Greek history sites – the virtual Rome. As those things start happening and people start building onto the edges, and when Microsoft Photosynth and similar technologies become more pervasive you can start building the models of the world in a semi-automated way from photographs and more structured, intentional drive-by’s and so on. So I think it’ll just sort of happen. And as long there’s a way to have the equivalent of Mosaic for AR, the original open source web browser, that allows you to aggregate all these things. It’s not going to be a Wikitude. It’s not going to be this thing that lets you get a certain kind of data from a specific source, rather it’s the browser that allows you to link through into these data sources.

So it’s that end that interests me. It’s questions like “what is the user experience”, how do we create an interface that allows us to layer all these different kinds of information together such that I can use it for all my things. I imagine that I open up my future iphone and I look through it. The background of the iphone, my screen, is just the camera and it’s always AR.

I want the camera on my phone to always be on, so it’s not just that when I hold it a certain way it switches to camera mode, but literally it’s always in video mode so whenever there’s an AR thing it’s just there in the background.

When we can do that I can have little alerts so when I have my phone open I can look around and see it independent of the buttons and things that I’m tapping and pushing to use the phone. That’ll be a really a different kind of experience.

Of course it is not known yet if the next gen iphone will have an open video API.  And of course, the current camera is pretty low quality, so why would they give it an open API until they put in a better camera?  I am not expecting anything one way or the other until the 3Gs comes out and people start using it.

But there are many things about the iphone 3.0 OS that are hugely important, like the discovery API that allows people to play games with other people nearby, that don’t have much to do with AR.

Tish: You have an iphone AR virtual pet application ARf.

Macrumors wrote it up and suggested that the neg gen iphone will have compass and open video API.  What are your plans for ARf?

Blair: ARf is just a demo right now.  I know what we’d like to do with it, but it would require tons of work;  imagine what it would take to do a multiplayer, social version of Nintendogs?  It’s not clear what we’d really learn by doing that, but there are lots of other game ideas we have that we want to explore.

Tish: I think it was on Twitter where Tim O’Reilly said, “saying everything must have a RFID tag is like saying we can’t recognize each other unless we wear name tags. Look at what’s happening with speech recognition, image recognition et.al. and tell me you really think we need embedded metadata.” What would you say to that?

Blair: I think that whatever extra data is there will be used. So if we put machine readable labels on some objects then they’ll be used if they make the identification and tracking problem easier. But it’s pretty clear that people are already working on tracking and so on.

A lot of these mobile AR apps are clearly putting ideas in people’s minds things that won’t really be doable in the near future. Like being able to look down the aisle of the store and it recognize all of the products. Given the distances and complexity of the scene, the number of pixels devoted to each of those objects, and so on – you just can’t recognize things in that context. But if I’m standing in front of a small set of objects, or looking at one thing, or I’m standing in front of a building, or if I’m in the store and because of the location API — imagine an enhanced location API that can tell me within a few feet where I am, and then combine that with some use of the discovery API that allows the store to tell your device you’re in the toothpaste section. Now you only have to look for different brands of toothpaste. So now you can recognize the big letters “Crest” or whatever. It’s all about constraining the problem.

That’s why I like that particular piece of Drummond’s work, the tracking web site I mentioned above. The general tracking problem of looking around and recognizing objects and tracking is still impossible. But if I know roughly what direction I’m looking in and I have a good estimate of my position, and I have models of what I should be seeing when I look in that direction, then it becomes a tractable problem. And so it’s not that a compass and a GPS are 100% necessary. But if you have them it certainly makes things possible that you wouldn’t otherwise be able to do.

Imagine for example  if there’s a new version of GPS, I just noticed that some of the new satellites going up have this new L5 channel. There’s the L1 & L2 signals  that the military and civilian ones use and they added this civilian L5 signal, which should make GPS more accurate. I haven’t found anything online that says how much more accurate.

But someday, hopefully, all GPS will get to be the quality of survey-grade GPS. Right now, if you get an RTK GPS from one of these companies that make the survey grade GPS systems, they give you position estimates in the range of two centimeters, and update 10 to 20 times a second. When you have that kind of positional accuracy combined with the kind of orientational accuracy you get from the orientation sensors we use in the lab from Intersense and MotionNode, everything is easier because you’ve pretty much got absolute position. You put that into a phone and now when I look up, it’s still not perfectly aligned because there will still be errors (especially in orientation, since the compasses are affected by metal and other magnetic noise). But it does mean if you and I are standing 5 feet apart from each other and look at each other, I can pretty much put a little smiley face above your head. Whereas now, with GPS, if I look at you and we’re 5 feet apart our GPS’s might think we’re on the opposite side of each other because they’re only accurate to two to five meters.

And that depending on the time of day and weather!

Putting RFID tags everywhere is easy; the problem is the readers – they currently require lots of power and they have a limited range.  Sprinkling RFID tags everywhere is fine. But you have to be able to activate those tags and read back the signal.  In certain contexts it works.

Tish: And one final question!  What do you think can be done re beginning to think about standards for AR.  Is there a meaningful discussion going on yet? Thomas Wrobel left this comment on my blog rcently and I was wondering what your position was on some of the ideas he raises?

Wrobel wrote, “The AR has to come to the users, they cant keep needing to download unique bits of software for every bit of content! We need an AR Browsing standard that lets users log into an out of channels (like IRC) and toggle them as layers on their visual view (like Photoshop).Channels need to be public or private, hosted online (making them shared spaces) or offline (private spaces). They need to be able to be both open (chat channel) or closed (city map channel) as needed. Created by anyone anywhere. Really IRC itself provides a great starting point. Most data doesn’t need to be persistent, after all. I look forward too seeing the world though new eyes.I only hope I will be toggling layers rather then alt+tabbing and only seeing one “reality addition” at a time.”

Blair: I agree with him, in principle.  But, I’m not sure there’s a point yet.  It can’t hurt to try, of course, from a research perspective, and I’m interested in the experience such an infrastructure would enable (as we’ve talked about already).