Interview with Vernor Vinge

Tish Shute: Many of the pioneers of the emerging AR industry who will be speaking at, and attending Augmented Reality Event, consider “Rainbows End” one of their key inspirations. [Note: If you want to attend ARE2011 readers of this post can use my discount code TISH295 ($295 for two days, or for one day only TISH1DAY11 for $149]

What is the best and worst, in your view, about the way Augmented Reality is emerging from science fiction into science fact?

Vernor Vinge: Progress that sets the stage:
The worldwide market penetration of cellphones in the era 2000-2010 was of a size and speed that would have counted as foolish implausibility even in science-fiction of earlier times. More than half the human race suddenly had access to knowledge and comms. Being in the middle of this firestorm of progress, we can’t really judge ultimate effects, but I expect that smart phones and the empowering aspects of social networks and AR are still massively underhyped. (This is not to say that individual innovation enterprises can’t fail; the treasure is there for those who dare, and ultimately the whole human race can benefit.)

But I can still whine:
Some — mostly political/legal — issues are disappointing. These affect AR but also the broad range of our progress with technology:
o Software patents and some styles of cloud computing are blunting the ability of average people to innovate. In the 2010-2020 era, average people should have the building blocks to empower them to create (and throw away at the end of the workday) tools that in olden times would have been the whole purpose of a business startup.
Unfortunately, some companies restrict and compartmentalize their releases like we’re still living in the twentieth century.
There are also some mostly tech issues that I’m impatient with (speaking as a never-satisfied consumer and fan:)
o The low pixel counts in contemporary head up displays.
o The poor position coordination in current HUDs.
o The lack of mass market acceptance of HUDs.
o The lack of progress in distributed store-and-forward between
mobile devices (sub-femtocell, ad hoc and transitory forwarding).
o The lack of progress in uniform solutions to centimeter-scale
localization.

Tish Shute: What do you feel will be the most impactful application of AR in people’s everyday lives?

Vernor Vinge: There are nebulous and fairly high likelihood answers: AR apps that let each person/team see those aspects of physical reality that are important for their current activity. Pointing technologies that coordinate with that AR vision. The combination is a revolution of interfaces, and the probable physical disappearance of more and more of the gadgets that twentieth century people associated with high tech.

There are also more specific, spectacular, and necessarily uncertain impacts (that depend on social acceptance and the development of network infrastructure for consensual sharing of local imagery).
o Economic disruption of the trend toward huge, expensive display devices.
o Bottom up social networking, arising from GPL’d tools. I see this as very disruptive, in good, bad and arguable ways, as illustrated by descriptive terms such as “consumer protection clubs”, “belief circles” and “lifestyle cults”. Some of these could be as public as our topdown social networks. Some might be quiet and widespread, perhaps growing out of pre-existing groups that already have a lot of intermember trust. (See:http://www-rohan.sdsu.edu/faculty/vinge/C5/index.htm)
o More farfetched, but in the tradition of the last 50 years: the digitization of external visual design: building architecture could give less priority to physical appearance and more to cheap physical strength, network access support, and physical modifiability.

Tish Shute: I interviewed Bruce Sterling earlier this week – http://www.ugotrade.com/2011/05/06/augmented-reality-transitioning-out-of-the-old-fashioned-legacy-internet-interview-with-bruce-sterling/.  And, I’m really looking forward to your “fireside chat” with Bruce at the end of Augmented Reality Event to sum up the event [see the full schedule for ARE2011 here].  But was there anything that particularly rung a bell for you in my conversation with Bruce?

Vernor Vinge: Bruce says:  “… it’s pretty clear that the people who would weep for joy to have Augmented Reality are people whose reality is already damaged. People who need reality augmented as a prosthetic …” This really rings a bell with me. And social networks with AR may have a special impact at small sizes, even just _two_ players. At such a scale, they might be better called “joint entities” than “social networks”. For example, two differently disabled persons, where one is mobile. There’s a lot more that could be said about this, including applications that could be done (maybe are being done) already.

Picture via IEEE Spectrum: Augmented Reality in a Contact Lens

Tish Shute: As Jesse Schell pointed out last year at ARE2010, “The whole point of AR is to see things from a different point of view … How can there be a more powerful art form than one that actually changes what you see?”

The magic lens of the smart phone, screens – large and small, projection, audio and sensory devices are mediating our AR experiences today. Bruce pointed out last year in his opening keynote, that these less immersive forms of AR have their own merits.

But eyewear has always been integral to the big vision of AR. Do you see some interesting futures for AR without eyewear? And, How long before AR eyewear is part of our everyday lives?
Vernor Vinge: This importance of vision is a visionist claim :-), but for the majority of us who have sight, binocular vision is by far the highest bitrate input we have, and we have enormously sophisticated wetware for analyzing what we see. Current display tech is far short of fully exploiting this input channel.

Along the way to this goal, I expect we’ll pass through mini-eras of exploiting the best-available tech. Right now, that is the tablet and the smartphone. Sometimes I almost wish for slower progress: in the nineteenth century, you could profitably spend your tech lifetime mastering one mechanism (for instance, black-and-white silver halide photography). The whole world would benefit from your career. Now, we rattle through the mini-eras so fast that we never fully exploit what’s zooming past before we’re on to the next stage.

How fast (or if) HUDs like in Rainbows End show up will probably depend on network and localizer tech as much as the HUDs themselves, with clear generational differences within such eyeware. In fact, it’s fun to imagine the mini-eras you could get with different combinations of HUDs tech, localization, and networking.

(Aside, a quibble: I think AR should not be restricted to visual only. There are tactile and kinesthetic possibilities, at least.)

(Aside, a whine: If only we had an output channel with the bitrate and flexibility of vision! Wearables plus voice and gesture could do some of that. Going further might involve scary human re-engineering. In Fast Times at Fairmont High, I speculated that a small re-engineering (eidetic memory) could give a form of highrate output,
simply by allowing selection from very large menus.)

Tish Shute: Augmented Reality and Ubiquitous Computing are intimately connected. Is a distinction between AR and Ubicomp still useful? (This recent PARC blog post: http://blogs.parc.com/blog/2010/03/defining-ubiquitous-computing-vs-augmented-reality/ takes a look at the definitions.)

Vernor Vinge: In a literal sense there is a distinction, and there is enough technical challenge in AR to justify specialists spending all their time with AR. But Augmented Reality’s importance to humanity is in its role as a portal to the power of ubicomp and human cooperation.

Tish Shute: Augmented Reality, as we understand it now, is a human centered experience. But even now some of the most important aspects of our lives are governed by machine to machine intelligences that operate for the most part beyond the reach of human perception, e.g., the trading bots of Wall Street. What role can augmented reality play in better mediating between human intelligence and machine to machine intelligence? Does AR hasten the arrival of the technological singularity?

Vernor Vinge: I see four or five concurrently active paths to the Singularity:
a) Artificial Intelligence: We create superhuman artificial intelligence in computers.
b) Digital Gaia: The worldwide network of embedded microprocessors, sensors, effectors, and localizers becomes a superhumanly intelligent entity.
c) Internet Scenario: Humanity with its networks, computers, and databases becomes a superhuman being. (Bruce’s story “Maneki Neko” is a beautiful and subtle illustration of this possibility.)
d) Intelligence Amplification: We enhance individual human intelligence through human-to-computer interfaces.
e) Biomedical: We directly increase our intelligence by improving the neurological function of our brains. (I regard this last item to be the weakest of the possibilities.)

AR is central to progress with possibilities (c) and (d).
If we humans want to keep our hand in the game, AR is an important thing to pursue.

Tish Shute: Powerful computer vision apps are emerging for smart phones and face recognition technologies are beginning to appear in consumer apps. Do you think we need a major shift in the way we handle data ownership? And, is “there is a real risk of our augmented reality world being owned by interests which are not our own?” (see my conversation with Anselm Hook last year. http://www.ugotrade.com/2010/01/17/visual-search-augmented-reality-and-a-social-commons-for-the-physical-world-platform-interview-with-anselm-hook

Vernor Vinge: Yes, there is such a risk. (See also my political/legal comments in response to your question (1).)
More broadly, I see DRM and the Law being used to reify our intellectual heritage as permanent private property. If this could work, it would be the biggest grab in history — and a major roadblock on human progress.

But even setting aside all the open/closed/free ideological questions, there is another important issue here: anytime laws are passed making popular and easily accomplished behavior illegal, things get very ugly. It may seem frivolous to compare this to the first stages of the War on Drugs, but that’s where serious enforcement would lead.

Tish Shute: We have seen gestural interfaces go mainstream in the last year. What are the most interesting innovations with gestural interfaces that you have seen in recent months? What sessions will you go to at ARE this year?

Vernor Vinge: I’m way behind the curve as to what is happening right now. Collecting data points on real hardware and applications is a high priority for me in attending ARE 2011.

Tish Shute: Are you reading/writing any new fictional literature about AR? And/or, What design fictions for AR are most interesting to you in the moment?

Vernor Vinge: As to writing: My novel The Children of the Sky should come out this October from Tor Books. It’s set in the far future and is the sequel to A Fire Upon the Deep. Alas, the story has only indirect connections to our present technological interests.

As to reading: I got a big kick out of Daniel Suarez’s duology Daemon and Freedom(TM).

Planetary from Bloom Studio, Inc. on Vimeo.

It is just over a week until Augmented Reality Event, and I know there are a lot of people, including me (full disclosure I am co-chair and co-founder) who are totally psyched to see what unfolds there this year.   Bruce Sterling, Vernor Vinge, Blaise Aguera Y Arcas,  Jaron Lanier, Will Wright, Marco Tempest and Frank Cooper will join 107 speakers from 76 augmented reality companies on a single stage (see my previous post) to tell a momentous story of a technology of our time (also see my previous post here).

As Bruce Sterling points out, Augmented Reality is “truly a child of the twenty-teens, a genuine digital native,” and one visible indication that:

..the Internet really could look like a “legacy.” The Legacy Internet as an old-fashioned, dusty, desk-based place best left to archivists and librarians, while the action is out on the streets (see the full interview below).


(photo by Jasmina Tesanovic)

Opening this post is a video of Ben Cerveny’s Planetary app, which “turns your music into a universe,” and enchants all who try it.  Planetary shot into #3 on the Top Ten Free ipad app list soon after its release.

Ben Cerveny’s talk at Augmented Reality Event will be one of the must attend talks (see the full schedule for Augmented Reality Event here, and note my discount code for Augmented Reality Event, TISH295, is still good, if you want to register).

Planetary, while it is not an AR experience,  points the way for AR to take us out of the old-fashioned, “Legacy Internet.”

“Planetary is just the sort of science fiction experience you expect when using an object from the future like iPad,” developer Bloom Studio writes on the app’s iTunes page. ( from Mark Brown’s Wired post).

In his interview on cnet Daniel Terdiman, Ben describes how popular computing will evolve beyond those, “dusty, desk-based place best left to archivists and librarians,” (Bruce Sterling).

Ben points out:

“The tablet is a total disruption of how we understand popular computing. The next era of experiences will be driven by visceral gesture-based input, and rich fluid responsiveness in native graphics contexts. I see the potential for Bloom to help define a “killer pattern” for application design. Because apps have been deconstructed into discrete tasks that flow across devices….”

Bruce Sterling had some interesting comments on the Bloom app:

I’m a big fan of Ben and his good works in infoviz — and urban informatics, too. I admit I’m not sure the I entirely need the metaphor of a solar system in order to play a few Texas blues tracks. But I could be persuaded. Ben Cerveny is a significant thinker and a very well-spoken guy.

I’m not saying that happens tomorrow; I’m just saying, why not? Why not aspire to that?

I too am a huge fan of  The Bloom team, Ben Cerveny, Tom Carden, and Jesper Sparre Andersen (also see my post here about Fizz, the Bloom team’s app used by The Locker Project for their Strata demo).  And, if you haven’t already heard about The Locker Project and Telehash – get on it!  This is one of the most important projects of our time – an infrastructure for a better future!

Interview with Bruce Sterling by Tish Shute and Ori Inbar

Tish Shute: As you so memorably put it, “AR is a technovisionary dream come true – something really rare, and you have to be really patient for those….”

What is best and worst, in your view, about the way Augmented Reality technovisionary dream is coming true and emerging to flourish in the wild?

Bruce Sterling: The best part is that AR is truly happening and is a lot of fun, and the worst part is that it’s happening in a Depression. If AR had broken loose in the dotcom days when cash flew around like soap bubbles, man, that would have been psychedelic.

AR that is even more of-our-time than “social media.” AR has arisen directly from modern technical factors that just didn’t use to exist. It’s made from shiny new parts, and is truly a child of the twenty-teens, a genuine digital native. It’s a little kid and it has to walk before it can run, but it’s great to see it walking.

Tish Shute: As Jesse Schell pointed out last year at ARE2010, “The whole point of AR is to see things from a different point of view…How can there be a more powerful art form than one that actually changes what you see?” What do you feel will be the most impactful application of AR in people’s everyday lives?

Bruce Sterling: I’m all for impact, but it’s pretty clear that the people who would weep for joy to have Augmented Reality are people whose reality is already damaged. People who need reality augmented as a prosthetic, in other words, so that they can achieve an “everyday life.” This is like the impactful but underappreciated role of the Internet in the lives of people who’ve been shut-in. If you’re laid-up in a hospital bed, a laptop is a revolution in convalescence.

But that kind of “impact” doesn’t sound too exciting or too profitable. My guess would be that the biggest arena for “impactful AR” would be augmenting cityscapes for foreign people who can’t speak the local language, can’t read the signs, and lack time to learn the local reality. Imagine, say, the Brazilian overlay for Moscow. You could show up, read your native Brazilian overlay of that city, do your business, eat, sleep, buy, leave, and scarcely “be in Moscow” at all. Constructed right, the AR Brazilian Moscow might even be a better Moscow — a Moscow that Russians themselves would pay to visit.

Tish Shute: You pointed out last year, in your opening keynote for ARE2010, that less immersive forms of AR have their own merits. We are still not seeing much “head mounted display weirdness” yet, but many other forms of AR are emerging – mobile, webcam, projected video, sonic augmented reality, even sticky light. You noted, practically everything that AR is involved in is a transitional technology. But since you spoke last year at ARE2010, which of these transitional technologies have shown the most promise for AR?

Bruce Sterling: It’s got to be handsets. Smartphones. The stats there are just amazing. The smartphone biz makes the personal computer business look like a Victorian railroad. When I read a guy like Tomi Ahonen, who talks about transitioning out of the old-fashioned “Legacy Internet,” that idea is startling. But AR is one visible indication that the Internet really could look like a “legacy.” The Legacy Internet as an old-fashioned, dusty, desk-based place best left to archivists and librarians, while the action is out on the streets.

Tish Shute: This year we have seen gestural interfaces go mainstream. What are the most interesting directions for gestural interfaces that you have seen emerge in recent months?

Bruce Sterling: To me, the most “interesting” part is seeing people do gestural stuff in public. William Gibson, my fellow author, observes that cellphones have stolen the gestural language of cigarettes. There’s lots of fidgeting, box tapping, ash-swiping, slipping boxes in and out of pockets… People quickly learn to do that without thinking twice, and they forget how weird it looks. It’s “design dissolving in behavior,” as Adam Greenfield puts it.

The gestural hack scene for the Kinect has been amazing. It’s like watching 1950s Beatnik dancing go mainstream.

Tish Shute: You have observed that Augmented Reality is Glocal which not only gives us different flavors of augmented experience but is “a departure from earlier models of tech startups, where you usually have like three hippies in a local garage. Now you’ve got German-American-Korean outfits like Metaio, and Total Immersion has a Russian affiliate. They’re inherently multinational, both inside the company and out.” What flavors of glocalness do you hope/expect to see at Augmented Reality Event this year.

Bruce Sterling: I’d be pretty happy to see some AR input from Brazil, India, and South Africa. I seem to be picking up a lot of followers in my Twitter stream from those locales. If I saw some Augmented Bollywood Reality, that would pretty much make my day.

Ori Inbar: What sessions will you go to at ARE this year? Who do you want to meet at ARE 2011?

Bruce Sterling: I make it my business to hang out with artists, but I’m hoping to drill down more on the technical aspects. For instance, where exactly are the bottlenecks in building animated augments? It looks like we’re about a sneeze away from jamming some crude Hanna-Barbera cartoons into real spaces. But the devil is in the details there.

Ori Inbar: Your commentary about the evolution of the AR industry over the years had significant focus on style. Is the AR industry dressed to kill yet? Any glimpses of promise in that direction?

Bruce Sterling: I’m not “pro-style” in every possible aspect of life, but as an Augmented Reality critic, it’s clear to me that if you claim to “augment” reality, then you should work hard to augment it — struggle to make it better. Otherwise you might as well call yourself “Defaced Reality,” or even “3D Spam.” When I see that kind of crudity and carelessness in AR, I’m gonna call people out on it. I know there will be the AR equivalent of cheesy billboards and gang graffiti, but I never much cared for those, either.

The industry’s videos have improved radically in the past year and a half. It used to be all about “look at my grainy, shaky handheld video of my cool new AR hack,” but nowadays the biz has really pulled its socks up.

If AR is about “experience design,” as I think it basically is, then eventually, as a matter of intellectual consistency and professional pride, everything you create will be considered part of “the experience.” That’s the industry’s way forward — that’s what it would do if it was grown-up.

AR people already look better than most similar geeks in the gaming business, and some day, I really do believe that augmentation people will become glamorous. They won’t be supermodels, but they’ll be about as chic as, say, professional set designers. Because AR is set design, in a way; it’s real-time interactive set-design for three-D spaces.

Ori Inbar: In the Layar Launch in 2009 you said “it’s the dawn of AR…”, at ARE 2010, you followed up on the theme saying “it’s 9am in the AR industry.” What time is it now?

Bruce Sterling: I’d be guessing it’s around 9:30 AM, but come on, that’s just a metaphor! ARE we all gonna blow off at 4:30 PM and have a beer, or is AR one of those cruel tech startups where nobody ever gets a personal life?

Ori Inbar: Are you reading any new fictional literature about AR that inspires you? And/or What interesting design fictions for AR have you come across recently?

Bruce Sterling: Well, I’m always interested in creative people who just plain make stuff up. Because that’s why I commonly do myself. The stuff that “inspires” me is usually stuff that I just didn’t expect to see. But when I don’t expect it, that usually means I wasn’t paying enough attention. I plan to pay a lot of attention to AR this year.

I’m not sure it makes a lot of sense to write fiction nowadays “about AR,” because it’s no longer a fictional topic. It’s become like writing fiction “about cinema.” You can write good fiction about someone who works in cinema, but not fiction about cinema itself. AR is not sci-fi “Augmented Reality” any more, it’s become a real-world phenomenon, a new industry of real augmentation.

With that said, I must remark that I sit up straight whenever I see Marco Tempest do stuff. Magicians are all about mystery and wonder. You wouldn’t see a magician, say, using AR to work an assembly line, or re-order library books, or find a pizza joint in Barcelona. And that’s great. Marco is always gonna do something freaky and out-there, and even though he’s a tech magician, it’s never about the tech first. It’s always about his ingenuity in finding new ways to employ new tools in creating a magical experience for his audience.

Marco’s not an entrepreneur, he’s not gonna revolutionize people’s daily lives or invent Web 4.0, but even if AR becomes “old hat” some day, it’s never going to be old hat when he’s doing it. The guy is a pro, and I’m quite the fan.

Magic Projection Live @ TEDxTokyo 2010 from Marco Tempest on Vimeo.

The prophets of Augmented Reality, Bruce Sterling (best know for his science fiction works, his non-fiction includes “The Hacker Crackdown,” “Tomorrow Now,” and “Shaping Things”), and  Vernor Vinge (author of “Rainbows End” and “The Coming Technological Singularity”) are joining  Will Wright (Legendary game designer of SimCity, The Sims, and Spore), and Jaron Lanier (a computer scientist, composer, visual artist, legend of virtual reality research, and the author of You Are Not A Gadget: A Manifesto) to judge the Auggies at the Second Annual Augmented Reality Event, 2011, Santa Clara, Ca., May 17th, May 18th!  Pictures top row, Bruce Sterling, Vernor Vinge, Will Wright, 2nd row, Jaron Lanier, Blaise Agüera y Arcas, and Frank Cooper III.

Augmented Reality has been prototyped in science fiction writing and films for many years now.  But Augmented Reality Event, the first global event dedicated to the emerging industry of AR,  is your chance to be part of a momentous transformation of science fiction into science fact.   Bruce gave a seminal keynote last year at ARE2010, “At the 9am of the Augmented Reality Industry…” But we are edging closer to high noon this year – see John Markoff’s recent New York Times post, “Augmented Reality Comes Closer to Reality,” and Bruce Sterling’s opening keynote at ARE2011 will raise the bar for us all again.

Bruce Sterling  and Vernor Vinge will wrap up the show with fireside chat.  Jaron Lanier will be keynoting at are2011 for the first time and Blaise Agüera y Arcas (a leader in Microsoft’s mobile strategy), will return to top his Ted talk demonstrating innovations in Bing Maps and Augmented Reality.  Frank Cooper III, who serves as Senior Vice President and Chief Consumer Engagement Officer of PepsiCo, will give, an up to now, missing perspective of  the big brands’ on Augmented Reality (AR).

“The main stage at ARE 2011 will be blessed with science fiction visionaries, ground breaking scientists, mind blowing technologists, and legendary game designers and more!”

Come prepared to Augmented Reality Event.  Read as much Bruce Sterling and Vernor Vinge as you can, and follow @bruces blog on Wired, Beyond the Beyond.  With only a few weeks to go until Augmented Reality Event, myself and co-chairs Ori Inbar, and Chris Grayson, have been working hard on a preliminary schedule ( by the time you click on this link it should be updated again).  You can follow us on twitter for updates as they come in – @tishshute, @comogard, @chrisgrayson,  and please follow @arealityevent and join our ARE facebook posse.  And, of course, don’t forget to register soon while we are still offering discount codes.

Readers of this post can use the code TISH295 for $100 of the already sweet price if $395.

If you joined us for Augmented Reality Event last year, you will know that the”Auggies” – an award for the best AR demo presented live with commentary American idol style,  is a chance to join the best and brightest in AR as they write the next chapter for AR in the sharp and often amusing reparte between judges and contestants.  Submit your demo proposal here under the “Auggies” track.

The picture below is the Auggie Award for the best AR demo designed by Sigal Arad Inbar.  (See, Ivan Franco recounts the team’s ARE 2010 experience, and winning the event’s first-ever Auggie Award, and the video shot at the YDreams booth by Bruce Sterling. “The Hotness” – YDreams rocking it at ARE2010 from brucesflickr)

ARE2011 will include more than 90(!) speakers in 30 sessions organized into 3 tracks: business, technology & programming, and production & design.

It will feature special activities such as a Startup launch pad (submit your proposal here under track: “Startup Launch Pad”), and an ARt Gala with live performances.  There will be displays by top AR artists of AR art projects and live performance art by the world’s best AR artists featuring: Helen Papagiannis, Amir Baradaran, Sander Veenhof, Ina Centaur and more to be announced.

The exhibition area will be open throughout the 2 days of the event and will include the latest product demos by the leading AR companies, as well as a career fair to help grow the fledging industry.  A small number of booths is still available in the exhibition hall ($995 for 10′x10′).Grab them while they last!

A press conference will kick start the event, as a vehicle for AR companies to launch new products and services. Submit your proposal here and add: “Press Conference” in the title. The event wrap up will feature Marco Tempest with a live AR Magic show

I have been so busy working on are2011 (and another project yet to be named -  hint I am focused on social augmentation and the world as a platform for a geo-situated social interest graph), that I have had no time to blog since I was on The Potential for Augmented Reality in Education Panel at SXSW, 2011. But I have posted my slide deck here for my talk, “Enchanted Objects and People: Data Driven AR.”

We are entering a new era of collective engagement possibilities for augmented reality, consumer to consumer brokerages which could unleash the the visions of the collaborative consumption movement and tap into our collective energies in totally new ways.  As Seth Praebatsch, Scavngr, suggested in his SXSW keynote, The Game Layer on Top of the World, the power of communal game play may even help us address even the most intractable problems like failure in education and global warming!!

One of my favorite new AR ventures exploring the potential of new forms of social engagement through AR is Suwappu.  We are working on how to get them across the pond and a few time zones to are2011.

“Dentsu London are developing an original product called Suwappu. Suwappu are woodland creatures that swap pants, toys that come to life in augmented reality. BERG have been brought in as consultant inventors, and we’ve made this film. Have a look!”

Click here or on the image above to watch the video.

“Suwappu is a range of toys, animal characters that live in little digital worlds. The physical toys are canvasses upon which we can paint worlds, through a phone (or tablet) lens we can see into the narratives, games and media in which they live.”

Many new players in Augmented Reality will join the pioneering AR companies, Layar, Metaio, Occipital, Ogmento, Total Immersion, YDreams, and renowned innovators, Mark Billinghurst, Steven Feiner, Blair MacIntyre and the Georgia Tech team and many others will return to surprise and amaze us this year – see the still growing list of speakers here.   Qualcomm, who announced an SDK for vision based augmented reality, and $200,000 developer challenge last year, will also be back with several of their team at are2011!

Visual Search and OCR augmented reality apps have flourished in the last year and they will have a strong presence at are2011.  Long anticipated and oft fretted over face recognition apps are coming to a phone near you soon.  Viewdle, working locally on the phone, is taking a new approach to face recognition.  Google has so far said that they will not use or release the face recognition technology that they apparently already have.  But Shailesh Nalawadi, Google Goggles, will return to are2011 to discuss building out a computer vision search database and 3D Classifiers to solve AR challenges and show us what Google Goggles has been working on recently.  Aurasma, out next month, is perhaps the next big thing in this space, so we are all eager to try it out and see what they bring to ARE2011.

My interview with Anselm Hook, Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, is still a great reference point for important questions in this space.

Wordlens, wowed us all with OCR done locally on the phone, and they will be  showing off new features at ARE2011. Pulling out WordLens to translate at subway poster for the first time is a magic moment, and guaranteed to impress the person sitting next to you too!    Tactical Information Systems, who Fred Wilson said had the potential to be a Shazam for faces at Web 2.0 Expo, NYC, will show us what they have been up to since then. 

Data driven AR is still in the early stages, but it is data that will unlock augmented experiences of the world around us in many powerful ways.   Also, we are seeing exciting new efforts to put people at the center of their data.  Jeremie Miller who ushered in the Twitter era with his invention of XMPP has brought us a new real time protocol, TeleHash and the Locker Project, that will unleash the full potential of communication at the edge of the network.   And don’t miss Ben Cerveny and Bloom,  at ARE2011, with “pop-cultural instruments for data expression and exploration” and “where data visualization meets game design,”  I highly recommend Ben’s talk!

Also I am very excited that Mike Kuniavsky will be speaking at  ARE2011 for the first time.  He  pointed out in 2007 that Magic is a powerful core interaction metphor for Ubiquitous computing [and AR].  Offices. libraries, and the desktop provided us with common metaphors that unleashed the power of the PC through files, desktops and trash cans, browsing, etc.,  but metaphors of magic – “enchanted objects and people,” are keys to augmented reality experiences.   And, if Magic is a core interaction metaphor for AR, kinect hacks have shown us that gesture will be the soul of the AR experience.  Jaron Lanier’s keynote will be a must see and make sure you catch Chris Arkenberg’s presentation on the Future of Hands Free AR.

This post is just a glimpse at some of what will be going on at Augmented Reality Event.  I cannot do justice to everything in a single post, so I hope to see you there!  And, if by any chance you have a few moments to daydream between now and then, feel free to ponder your own script for a Green Lantern movie:  “In brightest day and darkest night, no evil will escape my sight.”  I will too.  As @brendanscully noted, “Green lantern’s power is Augmented Reality!”

“The internet eats everything it touches,” write Brady Forrest and Nathan Torkington, O’Reilly Media, Inc., in their must read 2006 companion essay The State of Where 2.0 (PDF).  Now in 2010 that statement is more true than ever.

Last week,  I talked to Brady about what we can look forward to at Where 2.0, 2010,  and what he thinks will be the “internet eating” trends emerging this year.  Brady is uniquely positioned to get a glimpse of things to come.  His job for O’Reilly Media is tracking changes in technology and organizing large scale events, including Where 2.0 which he chairs, and Web 2.0 Expo in San Francisco and NYC which he co-chairs.  Brady also runs Ignite, and previously worked at Microsoft on Live Search.  And, when not doing his day job, he participates in such Uber Geek activities as Steve the Robot H.E.AI.D – A Human Energized Artificial Intelligence Device…with lasers and generative sound, (click on pic above or see video here).  Look out for Steve the Robot H.E.AI.D, at Augmented Reality Event, June 2nd and 3rd, Santa Clara, CA,  and a presentation from Brady.

As Vernor Vinge pointed out in his intro to ISMAR 2009 – the “possibilities are both scary and wondrous” as “the physical world becomes much more like a software construct.”  Brady Forrest has taken a lead role, since 2004 – when “‘local search’ was interesting but not yet real,” in shaping this transformation.

Where 2.0, together with WhereCamp (this year at Google) constitutes WhereWeek – a crucible for emerging trends in web mapping platforms, and location based technologies.  This year augmented reality, proximity-based social networking, local search, and the rapidly maturing field of Crisis Management are in the  mix along with the huge and long established GIS industry which has moved rapidly into the Where 2.0 space.

But what business models will oxygenate the system is still a key question – one Brady discusses in the interview below.  Certainly, the usefulness of location based analysis, mapping, new interfaces, and bringing this data to every application is clear.

Crisis management is center stage this year Jeffrey Johnson (Open Solutions Group), John Crowley (Star-Tides), Schuyler Erle (Entropy Free LLC) who will present on, Haiti: CrisisMapping the Earthquake.  And Chris Vein & Tim O’Reilly will “discuss how cities and application developers will benefit from open data and what these programs will look like in the future”  in the plenary City Data.

Mobile social, proximity- based social networking, which may soon emerge as a challenger to web based social networks, and augmented reality are the sexy rockstars of  the Where 2.0′s 2010 showcase of potentially disruptive technologies.  Augmented Reality has had a breakthrough year, and this is reflected in its strong showing on the Where 2.0 schedule.  But, as Brady notes, AR awaits the killer app, that will drive it to the next level  Of course, we hope to unveil that at are2010!

At Where 2.0, I am presenting on The Next Wave of AR: Exploring Social Augmented Experiences panel.  We will look at how social augmented experiences will be key to the next wave of mobile augmented reality.  Mike Liebhold, in a complementary presentation, looks at Truly Open AR. If you have been reading Ugotrade, you already know I am an advocate for an open, distributed, real time communications framework for AR – see ARWave.  Wave Federation Protocol is an open fast, compact, federated, communications protocol that is a dream come true for AR.  And, I would hazard a guess that in 2010, real time communications plus location will become oxygen.

But also key to the next wave of AR, as I discussed with Anselm Hook in this post on Visual Search, Augmented Reality and a Social Commons for the Physical World Platform, will be a view constructed through complex “hybrid tracking and sensor fusion techniques” (Jarell Pair), cooperating cloud data services, powerful search and computer vision algorithms, and apps that learn by context accumulation.”

And as Brady notes in the interview below,  a key step forward would be “to take advantage of your location, but it doesn’t need to have been mapped before.”

For some interesting news on the mapping front (and a discount code for Where 2.0 for Radar readers) see Brady’s post, Flickr Photos in Google Street View. These kind of human built maps have the potential to develop into “photo-based positioning systems” that could create new opportunities for augmented reality.  Brady asks:

“how often the Flickr photos get updated, where else these Flickr photos are going to show up in Google’s services (Google Goggles perhaps?) and will they show up in new search partner Bing? I am doubly curious if Facebook will ever let its photos be used in a similar way.”

Lior Ron of Google Goggles will be at Where 2.0 to tell us all about, Looking into Google Goggles.  And if you want to learn more about how our view of the physical world will be ” rooted in powerful computing, pervasive connectivity, and the cloud” don’t miss this one.  I will be there.  And I very much hope there is a Q and A with this session.

During our conversation (see the full conversation below) Brady gave me his short list for breakthroughs that he sees as having big significance in 2010:

“Well, I think Google Goggles is one of the most exciting things to me.  Having access to a visual search…having someone actually release a visual search engine in that way, to consumers, I think is huge.  You know, you see stuff like that in the labs. But I don’t see it… it’s rare to see it out.

I think Android is huge.  And the way Google is pushing hardware to show off the platform; so the Nexus One being another example and the fact that it’s breaking free from the carriers.  Because I think when we get away from the carriers we are able to see more innovation, it’s what’s going to allow people or developers and companies to really innovate.

And I think Twitter adding geo-location to their APIs and buying MixerLabs is a huge move. I think Twitter may end up becoming the end-all be-all of location services. They are going to be updated constantly by people; they are going to have a really good grasp, real-time, of what is happening in any one place, at least based on the people.

And then with the addition of the MixerLabs data, they’re going to have more datasets at their ready, as well as any data that they start to collect from the clients themselves, like from TweetDeck.

So there are global clients that are updating Twitter.  I think those are some of the most exciting things.  And again, just to come back to Yelp, I think Yelp’s Monocle is also pretty significant, just because it’s an AR [augmented reality] app that’s being pushed into consumers’ hands.

And we’ll see how useful they find it.”

Talking With Brady Forrest

Pic above from WhereCamp 2009, Brady Forrest, facing camera, checks out Mark Powell’s Food Genome Project.  Check it out here – it just woke up!

Tish Shute: So last year when you were interviewed by Michael Calore for WebMonkey before Where 2.0 you said, “Location is no longer a differentiator it’s going to become oxygen.” And after attending Where Week 2009, I agreed with you and wrote about it here.  But, in what ways did this prediction exceed expectations, and what ways were you disappointed now as we get close to Where 2.0, 2010?

Brady Forrest: Well, it exceeded expectations in that there are now five different mobile OS’s where you can load on third party applications that active users’ locations that can then be shared out.

And so, what it is making is the possibility of real-time social location aware applications.  And this is something that hasn’t truly been possible in years past. Looking back three years ago when the iPhone launched, it was the first major phone, especially in the US, to be location aware.  And a year later, the Apps Store launched, giving developers full access to location, which previously had been held onto very, very, incredibly tightly by the carriers.

And now, a year and a half later, you have Android, you have Palm Pre, you have Blackberry working on their SDK to make it better, but it still is there.  You have Windows Mobile working on their SDK.  And, you know, who knows?  Maybe even BREW will get into the mix.

And AT&T is opening up their own interactive store.  And so, AT&T and Verizon and all their smart phones may now be looking at BREW.

Tish Shute: Right. It was very exciting last year at Where 2.0, where we had all these new toolsets announced and then the iphone being location aware. What were the best implementations of these new capabilities that became available in 2009, do you think?  What, in your view, was the most creative, surprising and disruptive?

Brady Forrest: Well, I am a huge fan of Yelp Monocle. I think, you know, that is just a real life example of using Augmented Reality.  You are on a street.  You have got a bunch of restaurants.  You have got a bunch of businesses.  And just to be able to swing through and look for people…I mean and look for ratings and reviews.

They have just started to institute check in, so you will be able to know where your friends are and where your friends have gone.  And that type of real-time, incredibly useful data is what will make augmented reality a standard part of the landscape.

I think it is that type of data, more so than, say, reference data, that will make people want to have all the possible sensors.  So, what do you need for that?  You need a camera.  You need a compass for orientation.  You need a GPS or, at least, a decent location service.  And then you need a screen where you can actually see the data, and then you need an Internet connection.

So it is not like any phone can handle this.  And so, you are going to need those killer apps to actually drive people to the type of phones that can support this.  I don’t think AR is quite there yet.

Tish Shute: I agree, for true AR you need more that compass, camera, and GPS.  There are some missing pieces for the real deal experience – and not just a pair of sexy AR spec.  As you mention, hybrid tracking and sensor fusion techniques that can combine computer vision technology with  compass and GPS are vital.  We need the compass.  We need the GPS.  We definitely need the camera!  But we need this combined with computer vision techniques to get the tracking, mapping and registration for true AR, or even to deliver a stable experience with the post-it/geonote AR that we see emerging with Layar, Wikitude, and others. At the moment we need to put together the tools for a true AR hyper-local experience.

And, of course, another aspect of this is the kind of physical hyper-links that applications like Google Goggles are building.

Do you have a speaker from Google Goggles at Where 2.0.  I would be absolutely fascinated to hear more about their road map?

Brady Forrest: I was loading Google Goggles onto the program yesterday.

Tish Shute: Oh, you did?  Oh, fantastic. And you have Lior Ron speaking!

Brady Forrest: It is actually possible it is not up on the website, but I talked to them and got them to agree to do a talk on it.

Tish Shute: I very much want to hear more about their road map.  Google Goggle’s is a very, very significant step towards the physical internet and this integration of computer vision with sensor fusion techniques necessary for true AR.

Brady Forrest: I mean that combination with Computer Vision is going to be incredibly valuable, because,  and then the other issue you have there is like is it on the client,  or is it on the server?  And right now, Google Goggles is definitely on the server, and that is not fast enough in real-time AR.  So that is like more of a 10 blue links IO interface.

Tish Shute: And also, they haven’t got an open API, have they?

Brady Forrest: No, not yet.

Tish Shute: Maybe they will announce that.  Can you nudge them?  For true AR,  we need to move forward in several areas – of course, there is the mediating device issues, like access to the video buffers in the iphone, and the development of cool AR eye wear would be nirvana!

But my recent obsession has been working on a real-time communications infrastructure for AR, because that is quite doable now, yet we don’t really have that real-time infrastructure, i.e. a real-time mobile social utility that is really up to the real time requirements of AR [see more about this here and on ARWave wiki].

But we certainly don’t have the integration of computer vision and sensor techniques, and the access to the big image databases we need, let alone the clients we need to put it all together either!

Brady Forrest: Google has done work to help out the community with their support of Open CV.

It is based out of Willow Garage, but I believe that Google has done quite a bit of work on it.

Tish Shute: Could you talk a bit more about Open CV?

Brady Forrest: O’Reilly has  a 500 page book on it.  It came out of the Darpa Project, or the Darpa Contest, where unmanned vehicles are raced.  And that has since become, at least in my mind, the primary computer vision library that people work with.

I actually used it…or, one of the teammates did, on our project we did this summer.  We implemented an Open CV pretty quickly that detected where people were, and then we would play music based on that.

Uber Geek Meeting from ShellyShelly’s photostream
Tish Shute: Is that your Burning Man project? Do you have a link for that, and some pictures, video?

Brady Forrest: Yeah.  Heaid.com.  Human Enhanced Artificial Intelligence Dancing.

Tish Shute: Thank you! This year the augmented reality story has been fairly basic – relying on basic sensors, compass, gps, accelerometers.  But it has also been an exciting year because  we hadn’t even had  smart phones with the camera, and GPS, and compass before this.

But now, the big adventure is to hook this all these sensor fusion techniques up with computer vision so that we can actually do reverse positioning for example from photos from what we are looking at, right?

Brady Forrest: Yeah, and start to use it in a more ad-hoc manner so that as you are traveling around, yes, it will take advantage of your location, but it doesn’t need to have been mapped before.

Tish Shute: Right – moving from mapping to context awareness.  Could you give like a quick explanation of what you did in your Burning Man project and how that relates to this kind of,  ad-hoc, on the fly, beginning to know what you are looking at without it having been mapped before, that is fascinating.

Brady Forrest: Sure.  So we mounted a camera about 30 feet off the ground.  And as people would move underneath or dance, they would move from block to block.  And we had kind of created kind of bitmap of the area underneath and set up different sound zones.  So as people moved from zone to zone, it would play different music.

And we used Maxim FP to handle the computer vision, although it has Open CV library to handle the computer vision part and to handle determining which of the audio to fire off.  And then, also, we had a laser that would play at the same time.

And then we used Ableton Live, which is a very popular DJ software to actually handle the music.  So as someone moved from, say, square A to square B, it would fire off various MIDI signals and Ableton would interpret that.  And each person who went in, up to…well, theoretically, up to 4- 8 people.  But because of how small the stage was and how the sounds are played, realistically, more like 4-6 people.

Each person had there own set of sound.

Pic from extramatic‘s Flickr stream here

Tish Shute: Wow! Awesome.

Brady Forrest: We would be able to detect different people, assign them a sound, or a set of sounds, so, like bass, drums, vocals.  And then we would have clips that played well together that were 3-5 seconds in length.

Tish Shute: At what distance could you detect people?

Brady Forrest: We had a 22 foot area underneath the camera.  That was mostly based on what the lens could capture.

Tish Shute: OMG I love this!  This is really the next step for augmented realities – not just attaching reference data to the world but exploring new shared “cosensual realities” (see Anselm Hook’s interview part 2 upcoming).

I am very interested in how in something you talk about a lot in your “State of Where 2.0″ essay, about lifestyle coming first for a potentially disruptive technology, not commercial considerations.  I still have to post the second half to my interview with  Anselm Hook but Anselm has some brilliant ideas in this area.  He is working on a project called Angel, where part of the vision is for people to actually find what they need without explicitly having to ask for it having to ask for it.

And this brings me to something that is very, to me, noticeable about Where 2.0 this year, and very exciting.  This is that location aware technology and crisis management basically has matured, hasn’t it?  We are beginning to see really useful stuff in this area now.

What is different this year that has brought crisis management and location aware technology together, a world in crisis?

Brady Forrest: Well, I think the primary thing that has brought all these technologies together is Haiti.  Without Haiti…A lot of times, future crises benefit from the current one, because people put in a lot of work.  And so, there is new infrastructure being laid with things such as Ushahidi, which is an open source platform for tracking…well, originally for tracking election violence in, but now is being used to track people and their locations and food requests in Haiti.

Also, Haiti did not have solid, accessible, good maps at the time of the of the earthquake.  And there have been two volunteer projects that have sprung up to help with that.  One being headed by the Open StreetMap Wood Foundation and many volunteers.  And then the other, Google Map Maker. And in both cases the activity around Haiti on these programs went up exponentially…or, I don’t know about exponentially, but a lot.  In the case of Map Maker, it was up 100 times and was the most worked on country for that week.  And one of the most downloaded for that week.

Tish Shute: Yes the work being done in CrisisCamps around the country is very encouraging.

Brady Forrest: And then also, you know, not just Ushahidi or Open Street Map, but also the People Finder which had open API so that different organizations could share their data, thus learning from Katrina.  There are all these different pieces of technology will be used in the future and hopefully be able to save more lives.  I didn’t see…there are iPhones apps that were released.  But I’m not aware of any Android apps.  I’m not aware of any AR apps.

Tish Shute: We don’t have smart phones devices distributed widely enough for them to be appropriate, do we, in a lot of areas where crisis strikes.

Brady Forrest: Yeah and there was criticism that they shouldn’t have been on iPhone.  You know, that iPhones were a waste of time. Because they aren’t…a lot of on the ground agencies aren’t going to have iPhones.  However, a lot of people who are going from the States will, and if the apps are there, then people will start to have them.

But relatively speaking, an iPhone is not that expensive.

Tish Shute: One thing I noticed and actually I discussed this in the second half of the interview I did with Anselm which I am getting ready to post.  But one of the aspects of the crisis filter was having people working as curators looking at messages coming out of Haiti, and while integrating the streams that would be useful is still probably a challenge, many curators will be on iPhones because they are based in the US.

We need to work across all platforms probably.

Brady Forrest: Yes.  Patrick Meier of Ushahidi, who runs Crisis Mappers, he ran a 24/7 emergency room. It was out of the Fletcher School in Boston.

They had volunteers all over the States and Canada.  They had volunteers in Vancouver that were translating Creole messages in under ten minutes.

Tish Shute: Yes and another point that is interesting in terms of the reconstruction and rebuilding of  Haiti is  the whole idea of leap frogging, and the idea that you can really… there’s always, as we’ve seen in other parts of the world, opportunity, when you miss pieces of basic infrastructure, to skip a whole stage and go onto the next one, like how virtual banking took off in Africa because of the absence of brick and mortar infrastructure.

Brady Forrest: To skip to a topic that been in my head, I’m just so bummed that the iPad does not have a camera.

Tish Shute: I was bummed is barely the word I would use.  Particularly as we had just been planning our ground breaking AR/next generation ebook in the days leading up to the announcement!

I suppose there is the hope they’re going to put it in the next one.  But I suppose the play for conventional content delivery is so big that everything else is trivial in comparison – especially in seems jump starting the emerging augmented reality industry!

So we might get thrown a camera and compass in the next round but will we get access to the video buffers?  AR enthusiasts may have to live on table scraps from Apple a bit longer it seems.

But what blows my mind is why hasn’t the iTouch got a camera, been AR enabled?  AR gaming would get an enormous boost from that alone. My son loves even the simple minded AR games available now on the iphone, and he loves iphone games – he has 110 games downloaded!

Brady Forrest: Ridiculous.  Yeah.  I don’t know what they don’t like about cameras.  And I plan on getting an iPad, but because of the limitations I plan on using it for base content and will probably get the bottom line model. I can’t imagine…I don’t know.

Tish Shute: It is very interesting, who actually puts together the big enabling mediating device for AR is still an open question, isn’t it?  I mean, that’s the truth; we have sort of mediating devices but we don’t have the magic brew yet do we?

Brady Forrest: No. Not yet.

Tish Shute: Good enough in some ways, and certainly a start but not quite the real deal.  For me, Where 2.0 this year covers the groundwork for true AR, mobile social proximity-based social networking, visual search, computer vision and sensor fusion techniques….   And because all these things have a chicken and egg relationship laying the groundwork is basically as important as having the mediating device otherwise you can’t do interesting things when we get the mediating device, right?

Is this the year we get the magic brew for AR, i.e., the business model, the killer app, and the mediating device?

Brady Forrest: This is not the year.

Tish Shute: Then I should ask you. Are you in the Goggles camp? That is do you think AR needs eyewear to go mainstream?

Brady Forrest: I think this may be where we get…we start to see what is going to be the killer app that gets people to buy the hardware that will support AR.  You see what I mean?  And then from there the apps will come out and the hardware will advance in that direction.

I don’t think AR has made that leap yet.  It hasn’t, to use almost a cliché, it hasn’t crossed the chasm yet and it hasn’t proven that it will.  Because I don’t know if…I think it’s difficult to tell right now.  Is it going to be games?  Is it going to be data layers? What is going to drive people to an AR device, especially one fully dedicated to it?

Tish Shute: I think in terms of AR games taking off a bit of help from the mediating device e.g. access to the iphone video buffers would probably be enough to stoke up AR games into being a hot commodity.  But in terms of AR data layers  going mainstream, we need some of the other players in the location space to put together the magic brew on the business model, don’t we?

Brady Forrest: That’s why I’m so curious though…that’s why I gave Yelp their own talk.  They are…Those guys are gang busters, they’re a consumer company, very consumer facing website.  They’ve got amazing data stores.  They do a lot of interesting stuff with their data.  And I don’t think people always give them the geek credit they deserve.

Tish Shute: You began Where 2.0 back in 2004, when as you point out, “‘local search’ was interesting but not yet real” and you have always stressed something that’s proven to be absolutely true which is lifestyle before commerce, right?  And that if location based services were going to be big it was because they meant something in terms of our lifestyle, not just because they told us where to get another good burger.  Right?

I think there’s been a lot of breakthrough in that area this year in terms of what location based services and proximity based social networks are to us now, how they’re changing our lifestyle.  What do you see as the breakthroughs for in 2009 and what are you hoping for in 2010?

Brady Forrest: Well, I think Google Goggles is one of the most exciting things to me.  Having access to a visual search…having someone actually release a visual search engine in that way, to consumers, I think is huge.  You know, you see stuff like that in the labs. But I don’t see it… it’s rare to see it out.

I think Android is huge.  And the way Google is pushing hardware to show off the platform; so the Nexus One being another example and the fact that it’s breaking free from the carriers. Because I think when get away from the carriers we are able to see more innovation, it’s what’s going to allow people or developers and companies to really innovate.

And I think Twitter adding geo-location to their APIs and buying MixerLabs is a huge move. I think Twitter may end up becoming the end-all be-all of location services. They are going to be updated constantly by people; they are going to have a really good grasp, real-time, of what is happening in any one place, at least based on the people.

And then with the addition of the MixerLabs data, they’re going to have more datasets at their ready. As well as any data that they start to collect from the clients themselves, like from TweetDeck.

So there are global clients that are updating Twitter. I think those are some of the most exciting things. And again, just to come back to Yelp, I think Yelp’s Monocle is also pretty significant, just because it’s an AR app that’s being pushed into consumers’ hands.

And we’ll see how useful they find it.

Tish Shute: Gary Gale, Yahoo! Inc., is going to talk on overcoming the business, social, and technological hurdles so we can reach the long promised [Laughs] Hyperlocal Nirvana. I think you’ve outlined some of these obstacles in relation to  AR, where there are obstacles are in terms of mediating device, and bringing all the pieces together including computer vision techniques in order to have an AR view. That’s the AR side of it. But the layer below that, which is the layer where actual location based apps that are beginning to go mainstream now,  are these presenting successful business models for location-based services.

So in short, in your view, what are the big hurdles to Hyperlocal Nirvana before we get to AR, even just for these location-based services?

Brady Forrest: Well, how do you make money?

Tish Shute: Yeah, to put it bluntly. I like John Battelle’s way of putting it [laughs] how do we oxygenate the system!

Brady Forrest: So are location-based services something that you can make money in the long-term? Nokia bought NavTec for $8 billion. And then two years later, they’re giving it away free as part of Ovi Maps.

Tish Shute: Right.

Brady Forrest: I’m assuming that that’s actually part of the plan.  And that although their hand may have been forced by Google with their release of Turn-By-Turn that…but it’s still got to be a hard nut to swallow that this huge investment in location ends up becoming a loss leader to sell more phones.

So, can you make money through subscriptions, through selling apps? And I think that is still being proven. The other one is, can you use advertising? And it’s kind of scary to see that Apple is restricting the use of advertisers to use location.

It came out yesterday or two days ago that advertisers cannot use location, or app developers cannot use location for ads. They can only use location to show something interesting or useful to their customers.

And there’s a lot of speculation that it’s because Apple wants to control the location-based ads that go on the iPhone.

Tish Shute: Yes. I heard a strange rumor.  Actually its an un-strange rumor, a likely rumor in fact,  that Apple and MS are getting together to replace some of the Google aspects of the iPhone like search and maps?

Brady Forrest: Yes, …. Microsoft employees get 10% off at the Apple store. There’s a longstanding relationship between those two companies.

And Android is definitely more of a competitive threat than Windows Mobile is.  And it’s well-known what the relationship between PCs and Macs are. So I don’t think…I don’t find that to be that surprising of a rumor.  I do wonder if it would hurt the iPhone, but it doesn’t surprise me that they would consider it.

Tish Shute: I do know, certainly from the AR point of view, Microsoft has recently hired some of the key researchers, including Georg Klein. And they are looking for more people in the image recognition area so it seems currently MS is going to be making a bigger push not just with PhotoSynth, but with image ID.

So it could be a pretty powerful combo between the iPhone, and Microsoft – they have some of the key computer vision research that would be needed for full AR.

Brady Forrest: Oh, yeah. Microsoft has amazing research depth. They’ve got an amazing team.

Tish Shute: But it is a bit of a mystery to me why Microsoft haven’t done more with Photosynth.  As I noted in my previous post, Nokia’s ImageSpace is beginning to do what many thought Microsoft would do with photosynth two years ago.  And “photo-based positioning systems” -  3d models of the environment to cover every possible angle, and then software that can work out in reverse based on a picture precisely where you are and where your facing could be hugely important to AR.  But that brings me to another mystery why haven’t we seen more from Nokia in this space yet – the N900 doesn’t have a compass?

Brady Forrest: Yeah, I don’t know why Nokia hasn’t made more of a space for themselves in these things. They did a lot of early work in these areas. I think they are trying to…my guess is that they’re trying to restructure themselves. They made some pretty big changes on the web-Ovi made its own division. And they’ve been doing a lot of location-based acquisitions: Places, Gate 5 several years ago, Gossler, just the past six months. And so I think that’s really been their focus…and the research team.

And a large company, since they haven’t found a business model, which is what we’ve been discussing here, they are hesitant to launch it, or to…they don’t really know if this is a business that they need to launch, or if this is an app that they should have there out for fun.

Tish Shute: Yeah. And that’s back to the oxygenation of the system and location.  We really still have some work to do to with the business models

Final question!  At the core of many of today’s business model is the idea of hoarding data – that’s an underpinning.

But ultimately, for open AR, we want a situation where we can really share data so that we don’t really have the data all locked inside one particular browser or app. The current crop of AR browsers aren’t really browsers in the sense that we understand a browser on the web today, because the data’s locked inside each service, Wikitude, Layar, Acrossair etc.

I have become very interested with Federation as a model for solving this, so that we can begin to have an opportunity to build consensual relations around data, sometimes sharing, sometimes not. Federation is my big dream at the moment.  And now we even have something to work with in the Wave Federation Protocol. But how do we get from here to there, where we really have a federated world of data for AR and location-based services? But you think people need to solve the question of business models first?

Brady Forrest: I think people need…I think one potential is ads; so serving up content.  And by ads, I also mean coupons, meals, the Foursquare…. what it looks like Foursquare’s going to do, featured content, which is Layar’s.

So we need to see, is that the way we’re going to sell these? The other is to have the best viewer, which in some ways is a race in selling that, but that’s potentially a race to the bottom, price-wise.

Tish Shute: Right. Do you think Google Wave Federation Protocol has a chance of taking off and changing the game for real-time communications, federation, real-time…

Brady Forrest: Quite possibly with the real-time. I think they need to work on the UI.

Tish Shute: Oh dear we can’t discuss the Wave UI right at the end of the interview – of course I believe it would do better in an AR view!   I know you have to go  now but I have to say Google Wave not standardizing the client/server interface – so we could seem some new UIs for Wave [we are working with PygoWave for ARWave because of this], and  iPad’s lack of camera were two huge disappointments in recent months.

Brady Forrest: Yeah. It’s [the Wave client] is very difficult to use.

Tish Shute: But the Wave Federation Protocol is an open fast, compact protocol that is a dream come true for AR.  Open, distributed, real time communications is a very big enabler for AR.  I would hazard a guess that in 2010 real time communications plus location becomes oxygen.

ISMAR 2009 -  was an extraordinary mix of  high geek, academic eminence, gungho Dutch Cowboy entrepreneurial spirit, German engineering and industry, brilliant artistry, and invention, all fueled by a sense, and a very active presence in the case of Diamond Sponsor – Qualcomm, that the big technology players are waking up to augmented reality.

In the picture sequence above (click on photos to enlarge),  Noora Guldemond, Metaio, demonstrates Junaio (coming to an iphone near you Nov 2nd) to Dirk Groten, CTO of Layar (top left photo).  One of the nice social features of Junaio is that users can share the 3D augmented scenes they have created.  Noora is demoing this capability to Dirk, and as you can see he cracks up when he sees the  scene Noora has stored on her phone.  Dirk and I both recognize that this cute little dinosaur augmentation (close up above on bottom left) must have been created by Peter Meier, CTO of Metaio, during the Interoperability and Standards workshop earlier that day.  Metaio it seems were discussing standards while enjoying some 3D augmented back chat.

Both Dirk and I were active participants in the workshop too.  But little did we know that Peter Meier had introduced his little 3D dinosaur into our discussion while we diligently, and sometimes heatedly, debated the merits of XMPP, Wave Federation Protocol,  KML, ARML, VRML, X3D, and  more!  The photo I took is on the bottom right of the four pics above. It was probably taken very shortly after Peter’s augmented Junaio scene.  Of course there is no little dinosaur in my pic of  Dirk Groten with Joe Ludwig and Markus Tripp of Mobilizy who were discussing AR standards oblivious to Peter’s virtual pet in our midst.

I must say I had noticed an impish look on Peter Meier’s face (see photo above on the left – Peter is wearing glasses and holding a phone).  And Markus Tripp, of Mobilizy  revealed a little bit of gaming of his own, when he let out that, in part, ARML is a provocation.  But Peter was clearly unfazed and enjoying himself.  Dirk, tasked to summarize our discussion, stalwartly maintained an optimistic but serious tone fitting for a standards discussion:  “There is a willingness to standardize….,” he began (pic above on left – click to enlarge and read text).

But it was a little 3D dinosaur that, perhaps appropriately, had the last laugh. Fitting, as I am not sure whether anything anyone says about AR standards at the moment will hold up.  But, as Ori commented in his great post – an epilogue for ISMAR 2009, the vibe was “Peace and Love” in AR Browser land (although Chetan Damani of Across Air was not in the standards discussion because he attended the UX/content? workshop instead).  But as they say, “all’s fair in love and war.”  And it is my feeling the games have barely begun!  There are many players (virtual pets included) waiting in the wings. I met some at ISMAR, and they are just itching to join the frey.

Ori Inbar, Ogmento and Robert Rice, Neogence Enterprises, both founders of the AR Consortium, made great efforts to set our young industry off on the right foot -  in the  spirit of coopetition (a neologism coined to describe cooperative competition). See Curious Raven for Robert’s conference observations, and Ori’s post on Games Alfresco for more about Mobile Augmented Reality at ISMAR 2009.  The Mobile Augmented Reality Workshops were driven by an indomitable spokesperson for the new AR industry, Christine Perey.  Christine not only helped motivate discussion on the issue of oxygen to the system, i.e. business value, but also she was a very generous connector at the conference.

What’s Next From Augmented Reality’s Top Chefs?

As Ori pointed out, Kent Demaine, oooii (pic above is from the oooii web site), Minority report VFX designer was hanging out at ISMAR 2009 and he came to the panel I was on: “Augmented Reality in Sports,  Entertainment and Advertising.”  We chatted afterwords about instrumented environments and how this is such a key to development interesting augmented experiences.  Also I mentioned how back in the day I was involved in some of the early development of motion control software.  And it was great to hear Kent say they were still finding motion control cool at oooii.  As Ori notes, he is the “guy with the most enviable AR credentials in the world (the guy who designed VFX for minority report),” and oooii is busy and hiring.

One of the highlights of the Arts, Media and Humanities track for me was meeting Jarrell  Pair. He really brought the best out in panelists with his well tuned questions.  The recording of ISMAR was comprehensive and videos should be up next week.  I will post the slides on Ugotrade of my presentation:  “The Next Wave of AR: Shared Augmented Realities and Remix Culture.”.

“Mixed and Augmented Reality: ‘Scary and Wondrous’” – Vernor Vinge

“Imagine an environment where most physical objects know where they are, what they are, and can, (in principle) network with any other object. With this infrastructure, reality becomes its own database.  Multiple consensual virtual environments are possible, each oriented to the needs of its constituency.  If we also have open standards, then bottom-up social networks and even bottom up advertising become possible. Now imagine that in addition to sensors, many of these itsy-bitsy processors are equipped with effectors.  Then the physical world becomes much more like a software construct.  The possibilities are both scary and wondrous.” (Vernor Vinge -  intro to ISMAR 2009)

Vernor Vinge’s short intro to ISMAR 2009 (which can be downloaded with the ISMAR 2009 schedule here) captures the essence of the “Scary and Wondrous” dawn of the age of ubiquitous computing and mixed and augmented reality.  It is definitely worth a moment to download.  The future of augmented and mixed realities, as Vernor Vinge points out, is tied up in a “tension between centralized and distributed computing” that “will continue long into the future.” One of  my fascinations with Wave is that it offers a tantalizing opportunity to explore augmented reality in an open distributed architecture.

At ISMAR, I talked with as many people as possible about the AR Wave project – see my post here for more about Wave enabled AR.  Many people were very enthusiastic to join the AR wave and the only thing I really lacked was about 100 invites to hand out!

“Everything, Everywhere – making visible the invisible”

Some of the areas that I would have liked to see given more attention on at ISMAR were sensor networks, data curation, and user experience.  Not that these areas were entirely neglected with Pattie Maes, MIT as a keynote speaker, and Mark Billinghurst presenting on some fascinating work on social augmented experiences and user experience.  I highly recommend catching up on these and other ISMAR presentations when the videos go up.

And, I was very happy to meet and talk to Sean White whose work at Columbia University is one of my inspirations (for more about Sean’s work see here or click image above):

“the confluence of powerful connected mobile devices, advances in computer vision and sensing, and techniques such as augmented reality (AR) enables exciting new opportunities for interacting with this hidden network of dynamic information and shifts the locus of interaction from the desktop computer to the world around us”

And I had several very interesting conversations  at ISMAR about developing social augmented experiences that connect us to a physical world that is becoming “much more like a software construct” (Vernor Vinge).  Dirk Groten, CTO of Layar mentioned a few interesting projects Layar has up their sleeves, including something  Layar may be cooking up with The RoomWare Project.

The picture above is of RoomWare’s Social RFID Installation for Media Plaza in Utrecht (read more here).

Demos Galore!

In the demo rooms, Noah Zerkin (pic below left) pretty much single handedly carried the AR flag for a growing community of augmented reality Makers and Hackers.  But his presence was much appreciated, and he tirelessly demoed The Zerkin Glove. See the first of what may be several posts from Noah on ISMAR here.

And I got to try out the Vuzix goggles (picture above on right).   This was my first experience playing an AR game that was smart about real world gravity. It’s  “an augmented reality marble game that uses gravity as a game controller” – see Ori Inbar’s write up here.  It was a very compelling experience and I have to say I didn’t really notice the shortcomings of the Vuzix goggles while I was absorbed in the game. And  I turned out to be quite good at the game too. It is intuitive unlike the kind of  rule based games I never have time to learn properly.  But what is so special about this project is the tools that it is built with are open, and available for all, and affordable (see this list on Games Alfresco).

It was a great pleasure to meet Prof. Steven Feiner (picture on below the left) who heads Columbia University’s brilliant AR research team at The Columbia University Graphics and User Interfaces Lab.

Ori Inbar (pic below on right) also spent a lot of time in the demo room showing off Ogmento’s lovely AR learning game that delighted attendees, “Put a Spell: Learn to Spell with Augmented Reality.”

For a round up of  what’s next for augmented reality head mounted displays check out, Games Alfresco here, and Thomas Carpenter’s excellent review of the head mounted displays.

Ori Inbar on Games Alfresco asks is “Microsoft – the new big player to watch?”   “Georg Klein, inventor of PTAM-on-an-iPhone (and the smartest Computer Vision guy on the block)” has joined Microsoft to make Mobile AR.

The picture on the left above shows Georg trying out ARhrrr with Blair MacIntyre.   And on the right Blair is demoing his marker card pack to Senior Vice President of Cypher Entertainment, David Elmekies.  Yes ISMAR was abuzz with demos. See this post from Gail Carmichael for more video demos.

Next Year ISMAR 2010 in Korea!

At the banquet, I managed to find a seat at a table with Sean White (at left in photo above with Christine Perey to his right) and the Columbia University team.  The banquet culminated with the “Past and Future of ISMAR” Panel chaired valiantly by Jay Wright of Qualcomm.  We were asked to offer our input for ISMAR 2010.  I offered up an idea that I have been nurturing for a while now -  to stage a “Green Tech AR Competition.”  Perhaps, I suggested, we could base the competition around a conference (ISMAR 2010 in Korea?) and set up a target rich, instrumented environment for the occassion.  I think the Arduino open hardware community and AR developers have a synergy that is just waiting to be explored!  And, if we add the innovators of data curation to the mix, e.g., Pachube, AMEE, and Path Intelligence…(Markus Tripp left ISMAR to speak on a Web 2.0 Summit panel, “Humans as Sensors,” which also included Path Intelligence, Deborah Estrin on “participatory sensing,” and the brilliant work of Di-Ann Eisnor, Platial, on “Transactional Cartography”).  Anyway a big Green tech AR competition could get people working together across the broad spread of AR terrain on some of the sticky problems of user experience.  And, with a high level of support from Smart Phone companies, HMDs manufacturers and the chip makers we just might come up with some extraordinary magic.

The devil of course will be in the details.  But a competition like this could not only motivate key players to come together in the spirit of coopetition but also be an opportunity to show the world the power of AR to make visible the invisible ecosystems that are so important to the health of our planet.

One of the notable presences at ISMAR 2009 was the Qualcomm team.   Jay Wright’s presentation (an exclusive for ISMAR) not only outlined AR for 2012, but Jay also talked about some “close to the metal” innovation that we will see from Qualcomm very, very soon!  I had some time in the press room with Jay and his team prompted by MoMo’s Yuri van Geest.  When I twittered about Qualcomm’s presentation at ISMAR, Yuri replied:

vangeest “@tishshute: good stuff, hopefully you will integrate the neat new solutions and ideas in your talk in November ;)”

I will be presenting at MoMo #13 on AR, open AR, future of AR and GeoWeb,  and hopefully will bring some good news from Qualcomm too.  Anyway Jay seemed to like the idea of a Green Tech AR Competition, even though I did stress that I thought it needed some serious sponsorship and BIG prizes.

Where’s the beef? Tracking and Mapping at ISMAR 2009

On the flight from NYC to Orlando and ISMAR’o9 I dozed (I had been up late preparing my presentation) and I watched the Dew Tour Pro Skateboard competition and Top Chef on the Food Channel.  In this particular episode of Top Chef, the aspiring chefs were all given a brown bag of ingredients by an already famous chef who then judged whether the contenders managed to make a delicious meal with their allotment which was notably lacking in key ingredients of haute cusine.

This metaphor of  trying to cook up a great meal while perhaps missing the staples is apt for the current early stage of commercial augmented reality.  And when I arrived in Orlando, not only were the Dew Tour pro skateboarders staying at the same hotel as ISMAR, but ISMAR itself felt remarkably like an Augmented Reality Top Chef Coopetition.

Much of ISMAR was dedicated to the task of  providing the meat and potatoes of Augmented Reality, solutions to mobile tracking, mapping and registration, particularly in the Science and Technology track.

Industrial and Military Augmented reality solutions I found out, typically, solve the tracking problems by using fixed mounts which clearly wouldn’t translate well into the AR everywhere with everything mobile consumer culture expects.

In the picture on the left Fabian Doil stands by the VW engine that provided some of the outdoor targets for the ISMAR tracking competition.  On the right is a picture from the VW’s presentation on their research and development of AR.

I followed the tracking contest, organized by Daniel Pustka and Fabian Doil of Volkswagen, quite closely. And I learned a lot in the process. While  it is clear there has been progress in AR mapping and tracking, we still have a ways to go.

But hanging around the Tracking Competition was a good way to find out the state of play of this crucial part of the AR dream.  For example,  a little tidbit I learned is that Michael Goesele who has been reconstructing “high-quality geometry models from images collected from the internet (so called community photo collections, CPC)” is soon to be at the Institut Graphische Datenverarbeitung where top contenders in the tracking contest – Harald Wuest  and Folker Weintipper (in the foreground of the photo at the left and right respectively) are also to be found. [update Harold and Folker were the winning team see here for details of scoring and results!] Otto Korkalo and Tuomas Kantonen of VTT, Finland, Augmented Reality team are in the background. They have been working on the joint IBM, Nokia and VTT project that brings, Avatars and People Together for Virtual Meetings in Physical Spaces.

The picture on the right is another team that were doing very well. If my notes serve me well (and please forgive me if they don’t. I came back with my card wallet overflowing!) the photo on the right showsChristian Waechter (on the left) and Peter Keitler (on the right) of the Technische Universitat Munchen.

Germany is certainly leading the way in industrial AR. And I learned how small businesses like Metaio get to work with top research institutions and big companies like VW, thanks to very strong German funding program for AR and VR. The current iteration of a series of funding programs is  called Avilus.  Avilus  is putting 42 million Euros into AR and VR this year alone (click on the slide below to see more about Avilus ).

I wish we had the equivalent of Avilus here in the US.  But there is no equivalent to Arvilus for AR here, and  no AR is  being developed by the US car industry either it seems.  But look at the slide above to get a taste of some of the cool stuff Metaio and other small AR and VR businesses do for VW through the Avilus project.

I also got to meet many people from one of the world’s most important AR hubs -  The Department of Informatics, Technische Universitat Munchen, including Prof. Gudren Klinker on the far right in pic above.  And from left to right, Fabian Doil (VW, co-organizer of contest), Sebastian Lieberknecht , Selim Ben Himane (Metaio), Tobias Eble (Metaio).  Prof. Klinker is the engine behind much of German innovation in AR.

Metaio was one of the few teams to rely mainly on markerless tracking which in this contest was very challenging because of the very different light conditions (see pics below) between the windowless interior and dazzling Florida sunshine outside (pic on the right shows targets under ideal lighting conditions).  Many people in the US may be  familiar with Metaio’s consumer applications, like Junaio,  but thanks to Germany’s efforts to nurture augmented and virtual reality they are also respected software developers in industrial AR.  And I suspect that Metaio will spearhead markeless tracking in consumer AR too.

This post as usual has already expanded to something much longer than I originally attended – pretty typical for me! There is much I have not been able to cover including some of the interesting contributions by augmented reality artists at ISMAR – again I recommend the upcoming videos.

But I cannot end without a hat tip to, Oriel, Nate et al. who won the best student paper award for AR Sketch – again please see Games Alfresco for more on this (pic below from Games Alfresco). AR Sketch, Ori notes, is featured “in our top post and popular video.” And

“Their work is revolutionizing the AR world by avoiding the need to print markers – or any images whatsoever.”