Augmented World Expo 2013 was really an amazing experience. I’m co-founder and co-organizer of the conference, along with Ori Inbar, so it has meant a lot to me to see our event grow over the last four years, and thrilling to make such a big splash this year.  There were 1,163 attendees, and the expo show cased an ecosystem of emerging technologies – augmented reality, gesture interaction, eyewear, wearables, and connected hardware of  many stripes, that mark the beginning of natural computing entering the mainstream. It was a unique opportunity to get up close and personal with what it feels like to be an augmented human in an augmented world!

Videos of AWE 2013′s 35 hours of educational sessions and inspirational keynotes are now available on our YouTube channel. I am sharing my own talk (my slides are also up on slideshare here), and a few of my favorites in this post, but there are far to many to post here, so please browse further on the Augmented World Expo youtube channel.

One notable high point of AWE2013, for me, was the showcase sponsored by Meta, a startup developing the first device allowing visualization and interaction with 3D virtual objects in the real world using your hands. It was made possible by the generous contribution from the private collections of Paul Travers, Dan Cui, Steven Feiner, Steve Mann, and Chris Grayson, and passionate volunteers who are helping advance the industry. Sean Hollister of The Verge did this excellent report on the eyewear showcase 35 years of wearable computing history at Augmented World Expo 2013
Also for more on Meta see this article by Dan Farber.

My colleagues at Syntertainment, Will Wright, Avi Bar-Zeev, Jason Shankel, and LaurenElliott all gave great talks. Ironically, we’re not building augmented reality apps or H/W. We all just happen to continue to be very interested in the field.  

Thank you to everyone for supporting the event!

The press coverage was truly extensive:

In the shadow of Google Glass, an augmented reality industry revs its engines
The Verge, Sean Hollister, June 9, 2013, 271 Tweets

The next big thing in tech: Augmented reality
CNET, Dan Farber, June 7, 2013
Pick up on Current News Daily
350 Tweets

AWE 2013 Conference Report: Augmented Reality and Marketing
The Persuaders Marketing Podcast on Dublin City FM, June 23, 2013

AR Dirt Podcast – Ori Inbar AWE2013 Extravaganza Recap
AR Dirt by Joseph Rampolla, June 18, 2013

35 years of wearable computing history at Augmented World Expo 2013
The Verge, Sean Hollister, June 9, 2013
7 Tweets

Augmented Reality: Bruce Sterling, keynote at Augmented World Expo 2013
Wired, Bruce Sterling, June 9, 2013
9 Tweets

On the road for VR: Augmented World Expo 2013
Doc-Ok, Staff, June 7, 2013
3 Tweets

My Interview from Augmented World Expo 2013 [VIDEO] Wassom.com, Brian Wassom, June 7, 2013

Augmented World Expo
ZenFri, Staff, June 7, 2013

AWE2013: Hardware for an augmented world
FBNSantos.com, Felipe Neves Dos Santos, June 6, 2013

Augmented Reality Will Be the New Reality
InvestorPlace, Brad Moon, June 6, 2013

Wearable computing pioneer Steve Mann: Who watches the watchmen?
TechHive, Armando Rodriguez, June 6, 2013

Expo puts augmented reality in the limelight
ABC 7 News, Jonathan Bloom, June 5, 2013

These OLED microdisplays are the future of augmented reality
DVICE, Evan Ackerman, June 5, 2013

Visualized: a history of augmented and virtual reality eyewear
Engadget, Michael Gorman, June 5, 2013

Wikitude announces Wikitude Studio and in-house developed IR & Tracking engine
PapiTV, KC Leung, June 5, 2013

Augmented reality expo aims for sci-fi future today
USA Today, Marco della Cava, June 5, 2013

Augmented Reality: High Dynamic Range (HDR) Video Image Processing For Digital Glass
Wired, Bruce Sterling, June 5, 2013

Will Wright at Augmented Reality Conference: Don’t Augment Reality, Decimate It
AllThingsD, Eric Johnson, June 4, 2013

Philip Rosedale’s Second Life with High Fidelity
CNET, Dan Farber, June 4, 2013

Google Glass competitors vie for attention as industry grows
PC World, Zack Miners for IDG News Service, June 4, 2013

4D Augmented Reality Leader Daqri Announces $15 Million Financing
Press Release, June 4, 2013

CrowdOptic Powers Lancome Virtual Gallery App, Crowd-powered Heat Map
TechZone 360, Peter Bernstein, June 3, 2013

Augmented humans, enhanced happiness?
Crave Culture, Angelica Weihs, June 2, 2013

Metaio & Vuzix to Showcase AR-Ready Smart Glasses at the 2013 Augmented World Expo
Press Release, May 30, 2013

Four ways augmented reality will invade your life in 2013
Quartz, Rachel Feltman, May 30, 2013

Augmented Reality: Augmented World Expoâ„¢ is next week
Wired, Bruce Sterling, May 28, 2013

Strike it Rich with Cachetown and AWE 2013 Playing the Gold Rush 49’er Challenge In Augmented Reality
Press Release, May 24, 2013

Local Community College Student Headed to Silicon Valley to Learn More about Augmented Reality
St. Louis Post-Dispatch, Staff, May 24, 2013

Explore an intricate labyrinth with smartphone AR
CNET Australia, Michelle Starr, May 21, 2013

Dartmouth firm lands super app
Herald Business, Remo Zaccagna, May 21, 2013

Augmented World Expo 2013–The Future of Augmented Reality
Silicon Angle, Saroj Kar, May 17, 2013

“The game is about the world not dragons,” Will Wright, Founder and Chief Executive  Stupid Fun Club, Creator of Spore and The Sims.

I had a brief chat with Will Wright after his talk at Engage!, and I was delighted to hear that augmented reality is high on his agenda at the moment:

“a lot of our stuff is kind of in the experimental format right now, but definitely one of our strong interests is AR.”

Will Wright will be coming to speak at Augmented Reality Event, Santa Clara, CA., June 2nd, 3rd,  2010.  But, for now, here are a few hints at some of the directions that are intriguing him, e.g., the game potential of 3D mapping like Blaise Aguera y Arcas’s  demo of augmented reality maps at TED -  see the full conversation below.

There has been a vital shift, Will Wright points out.  Before the Wii,  immersive was understood as  how much we were pulled into the world of the game.  Now immersive is how much the game pulls us deeper into our world, e.g., our relationship with the people we are playing with as in Rock Band, or engaging with other people’s crazy antics when playing Wii games.

“Computers are imagination amplifiers and toys are imagination constructors.”

The slide above is from Will Wright’s talk at Engage!

Will Wright’s talk was extraordinary, dense, layered, and deeply thought provoking.

I have picked out a few samples from Will Wright’s vast tome of slides here.  They are just a glimpse of the many insights he offered.  If you are still wondering what will transform augmented reality into a mainstream experience, I suggest studying this talk carefully (I think the audio will be posted on the Engage! web site).  Also watch Will Wright’s, Games For Learning Institute talk at NYU, February 17th, 2010, archived here.

Will Wright and Stupid Fun Club are getting ready to takes us to the next level of imagination amplification and construction.

“Smart” things can make us dumber by overriding our instincts

Just one of the many wonderful anecdotes Will Wright told  was the story of his experiences with a new “smart” car (he bought this car with the intent of exploring the pinnacle of the “smart” car experience). The slide above is from Will Wright’s talk at Engage!

Increasingly, artifacts are being designed to send us more and more data, and this car was endowed with an array of sensors supplying data aimed at assisting parallel parking – a notoriously challenging aspect of driving.  But the car  failed miserably in helping.   While parallel parking had been easy for him prior to being deluged with all this data, Will Wright pointed out, ironically, he had to learn to ignore this stuff to park the “smart” car.

Instinctively, we learn to filter the information necessary for parking to the relevant stuff.  This kind of pre-conscious filtering is a key challenge for augmented reality, and one that Will Wright, as a game designer, has given great deal of thought to.

As Will Wright pointed out, a  lot of our ideas about augmented reality, and sensor enabled artifacts, are rooted in trying to give us more data, to “take over our instincts.”   Not only do these artifacts attempt to give us more data, which as in the case of the HUDs for parallel parking can get in the way of our own highly effective intuitive instincts.  But, as Will Wright also noted, these artifacts also have more data which they can deploy independently to override our instincts, e.g., the car detecting your head has turned back to talk to a passenger and applying the brakes!

“Toys Encourage Agency”

Toys can be the antidote to instinct blocking “smart things.”  In contrast to “smart” data spitting cars that “take over” our instincts, toys encourage agency.  Will Wright gave the example of  high dynamic lighting photographs that make the world “toy like” and encourage us want to reach in and play with it (photo above from HDRCreme).

“What Computers are really good at is harvesting human intelligence”

Another key insight that Will Wright explored in depth in his talk was the significance of  crowd sourced intelligence (the slide above is from Will Wright’s talk at Engage!).  If the crowd is training the filter, he suggested to me, this might build the kind of context we need to build meaningful augmented reality experiences (for more on this see the conversation below).

Talking with Will Wright at Engage!, NYC, 2010

Tish Shute: I was very interested by the idea you put out that this deluge of information gathered by sensors is not necessarily a kind of nirvana for augmented reality, in fact it can be just the opposite.  In the embryonic world of augmented reality, we have two streams it seems at the moment – one is the idea of a kind of like hyper local nirvana imagined for AR, in which we get information relevant to us, when and where we need it.   But you talked about some of the problems in realizing this, didn’t you?  The other strand  is the emerging stream of play which you are exploring..

Will Wright: Right.  I think part of it is like what I was talking about-the way our senses are set up to know how to filter out 99% of what is coming into them.  That is why they work, and that is what is beneficial.  I think that is why AR needs to focus on…

You look at what I can find out on Google or whatever, the amount of information is just astronomical.  The hard part, the intelligent part, is how do you figure out that one tenth of 1% that I actually care about at this given second?

Tish Shute: Yes.  Have you seen any examples of AR beginning to do that?

Will Wright: No, not at all.  I think that you have to have a contextual understanding of where I am at, where my mindset is, what my situation is, what my goal state is in a moment by moment basis.  And then it is still a complex task.  But the very first thing we need is more context for building a filter.  See, that filter is changing every few minutes, you know, what I am filtering into my senses is changing, and my context is changing moment to moment.

Tish Shute: I really liked your emphasis on crowd sourced intelligence as the key power of a networked world, is this the seed..?

Will Wright: Well, you can imagine crowd sourcing that filter…it would affect a million people and get a sense of what mental context that they were in and what filter they turned on.  And so, in a sense, the crowd is training the filter.

Tish Shute: Yes.  The problem with projects like SIRI, that is driven by the big DARPA AI project, CALO, is it is centralized – although I am not sure what they intend to do in terms of crowd source corrections?  But if it was all open and we could crowd source as well that would be interesting.  But in the end we need a framework for AR that is as open as the internet, don’t we?

Will Wright: Right.  I think the technological infrastructure needs to be much lighter so that it can be grounded in more like a Twitter feed or something.

Tish Shute: Yeah.  I’m actually working on a project using the Wave Federation protocol as the basis for an open communications framework for augmented reality, AR Wave – not the Wave user interface, just the real-time federation protocol.  But, of course,  for it to become an open framework that could be a vehicle for crowd trained augmented reality it would need good take-up!

Will Wright: Right.  You really want a million people involved.

Tish Shute: Yes our dream is that the creation of augmented reality content will be as open, accessible and  simple as making an html page, or contributing to a wiki.

So in terms of AR games what is interesting on the horizon, presumably games also have to solve the problems of  delivering a hyper local experience.  The car that you described in your talk tried hard to use augmented reality to solve the problem of parallel parking and ended up making it harder.  So giving us the information we need, where we need it, when we need it, and specific to who we are is going to be a big challenge.  But I mean in terms of games, what kinds of hyper local experiences will be most fun and what have you seen that is interesting in terms of augmented reality games up to now?

Will Wright: I’ve not actually seen much at all.  I’ve seen people doing interesting stuff with like Google Maps.  They aren’t really entertainment oriented, but I think you can start thinking about…

I mean I think for a lot of people, Google Street View is entertainment.  But I haven’t really seen something that was really leaning into an entertainment application using existing technology and data that is already out there.

I mean I have seen some cool experiments-people playing Pac-Man in Washington Square and stuff like that, but nothing really serious.

Tish Shute: Yeah.  of course I think one of the missing links is that the barrier of entry is way to high for creating social augmented experiences for smart phones, and as you point out in your talk it is the social implications of the game is what makes it compelling.

Will Wright: Also, I think using them [smart phones] as data aggregation devices rather than just data consumption devices…so that people out there are using their phone, cameras, microphones, or whatever to gather data and get an experience where they are rewarded for gathering data.

Tish Shute: Like foursquare where you get the badges, and people can become the mayor of like a cafe or something.

Will Wright: Right.  Yeah, you can imagine people using their phones to actually kind of pull information…

Tish Shute: A Dutch developer/artist/game designer, Thomas Wrobel,  Lost Again, came up with the original concept for the AR framework we are building on the Wave Federation protocol.  Thomas and his partner Bertine van Hovell design alternate reality games, amongst other things they do…so they are deeply immersed in the potential of the world as game.

Will Wright: Yeah, one of my programmers actually works in Amsterdam….there is a whole sub-community…
Well, yeah.  The possibilities are tremendous.  And Wii is actually training us that way [to be as much engaged with the other players in the physical space as the virtual game], so it is going to happen.

Tish Shute: What are the most exciting things you see at the moment, and for the next 12 months for augmented reality?

Will Wright: Gosh.  I mean I just think there is cool stuff happening in mapping, in general.

Tish Shute: Like Blaise Aguera y Arcas’s  demo of augmented reality maps at TED?

Will Wright: Yeah, I thought the 3-D mapping with Microsoft…I think like the next level of that is going to be really compelling.

Tish Shute: You see game potentials in that?

Will Wright: Yeah.  You start overlaying really cool game potential on top of that.

Tish Shute: Might you get interested and do something?

Will Wright: Oh, yeah.  I mean in terms of games, that is one of my biggest interests, is AR.

Tish Shute: Are you allowed to talk about anything specific at all?

Will Wright: Not yet, no.  I mean a lot of our stuff is kind of in the experimental format right now, but definitely one of our strong interests is AR.

Tish Shute: Yeah, absolutely.  We are over being tied to our desks to use computers -we want to be doing it anywhere, anytime, with anything…

Will Wright: Now the game is about the world instead of about dragons.  I love that.

I first started talking to Robert Rice, CEO of Neogence Enterprises, Chairman of the AR Consortium, in 2008.  Robert was already actively working on creating the world’s first global augmented reality network.  But it took a few months before what Robert had said to me about impending explosion of  augmented reality into our lives really sunk in – “this is going to be much bigger than the Web!,” he extolled.

By January, 2009 I was convinced and I posted my first interview with Robert, “Is it OMG Finally for Augmented Reality?..” As I mentioned in the intro, I had recently tried out Wikitude and Nathan Freitas’s grafitti app on the streets of New York City and I was impressed.  Now, 7 months later, Augmented Reality has  not disappointed and there is an explosion of new applications, and the arrival of some of first commercial and practical toolsets, SDKs, and APIs for aspiring developers.

For more on this see my previous post, Augmented Reality’s Growth is Exponential: Ogmento – “Reality Reinvented,” talking with Ori Inbar, which is an introduction to my series of interviews with the key players in augmented reality and founding members of the ARConsortium – Int13, Metaio, Mobilizy, Neogence Enterprises, Ogmento, SPRXmobile, Tonchidot, and Total Immersion.

As I mentioned before, Maarten Lens-FitzGerald of SPRXmobile told me the other day that my first Interview with Robert Rice, in January of this year, was a key inspiration for SPRXmobile to get started on the development of Layar – a Mobile Augmented Reality Browser. Much more on Layar and Wikitude – world browser in my upcoming interviews with Maarten Lens-FitzGerald and Mark A. M. Kramer, respectively.

Recently, both Layar and Wikitude earned a mention in the white paper by Tim O’Reilly and John Battelle, Web Squared: Web 2.0 Five Years On. Web Squared is essential reading not only because it covers the underlying technological shifts of “Web Meets World,” which augmented reality is a vital part of;  but, crucially, Web Squared focuses on how there is a new opportunity for us all:

“The new direction for the Web, its collision course with the physical world, opens enormous new possibilities for business, and enormous new possibilities to make a difference on the world’s most pressing problems.”

I am currently working on a post on Green Tech AR which is one of the areas augmented reality can play an important role “in solving the world’s most pressing problems.” Augmented Reality has a lot to offer Green Tech development.  As Gavin Starks of AMEE said at Euro Foo in 2006, “climate change would be much easier to solve if you could see CO2.”

But really useful Green Tech AR requires still hard to do markerless object recognition (going beyond feature tracking and modified marker recognition), and a tight alignment of media/graphics with physical objects, in addition to a quite a high level of instrumentation of the physical world.  And for Green Tech AR to really shine, we are going to need innovators like Robert Rice who are working on, and solving, multiple really hard problems like:

“privacy, media persistence, spam, creating UI conventions, security, tagging and annotation standards, contextual search, intelligent agents, seamless integration and access of external sensors or data sources, telecom fragmentation, privilege and trust systems, and a variety of others.”

Recently Robert Rice presented at MoMo Amsterdam. Here is a drawing of him in action (picture below from wilgengebroed‘s Flickr Stream).

In his Twitter feed Robert Rice ( @RobertRice ) Robert reminds us: “By the way folks, what you see out there now as “augmented reality” is not what it is going to be in two years.”   Robert plans to show the first public demo of his “platform for platforms” at  ISMAR 2009.

Robert is writing up a series of White Papers currently.  I got a preview of the first, “The Future of Mobile – Ubiquitous Computing and Augmented Reality.”  Robert points out, “AR through the lens of the mobile industry and ubiquitous computing is almost overwhelming compared to AR as marker based marketing campaign.”

I asked Robert, “What are the key take-aways for investors interested in the augmented reality field at the moment:

“First, Mobile AR is going to be bigger than the web. Second, it is going to affect nearly every industry and aspect of life. Third, the emerging sector needs aggressive investment with long term returns. Get rich quick start ups in this space will blow through money and ultimately fail. We need smart VCs to jump in now and do it right. Fourth, AR has the potential to create a few hundred thousand jobs and entirely new professions. You want to kick start the economy or relive the golden days of 1990s innovation? Mobile AR is it.

Don’t be misguided by the gimmicky marketing applications now. Look ahead, and pay attention to what the visionaries are talking about right now. Find the right idea, help build the team, fund them, and then sit back and watch the world change. Also, AR has long term implications for smart cities, green tech, education, entertainment, and global industry. This is serious business, but it has to be done right. I’m more than happy to talk to any venture capitalist, angel investor, or company executive that wants to get a handle on what is out there, what is coming, and what the potential is. Understanding these is the first step to leveraging them for a competitive edge and building a new industry. Lastly, AR is not the same as last decade’s VR.”

Talking with Robert Rice

Picture of Robert Rice at MoMo from Guido van Nispen‘s Flickr Stream

Tish Shute: So perhaps we better start with an update on state of play with Neogence?

Robert Rice: Neogence is doing well actually. We don’t talk much about the fact that we are still a small startup and we face a lot of the usual obstacles related to that and being a small team. Fundraising has been extra difficult, mostly because people are just now beginning to see the potential in AR, but that is still colored by perceptions based on a lot of the gimmicky AR ad campaigns out there. Still, it is better than it was two years ago the idea of an AR startup was a bit of a joke to a lot of VCs we talked to. However, we do have an agreement from a new venture fund in Europe (which we can’t talk about yet) for our first round of funding, but we don’t expect to close that for several months.

If all goes well, we hope to debut our first public demo at ISMAR 2009 in Orlando to select individuals and a few press folks. We might release a few viral videos before then that are conceptual and about what we are building in the long run, but that depends on how things go over the next several weeks.

We are also very active in looking for and building strategic partnerships and relationships with other companies, and this is not restricted to the augmented reality or mobile sector. As I have said before, we are looking at this as a long term business venture and the industry as something that will be bigger than the web itself within ten years. We are doing typical contract work and custom AR solutions to keep the cash flow going and build up the corporate resume a bit. So, if you want something done, and better than the stuff you are seeing now with all of the generic “look at our brand in AR with markers and a webcam” you should definitely give us a call.

Tish Shute: Just to clarify because most of the recent press has been about browser type AR like Wikitude and Layar which are not in the purist sense AR ‘cos they do not have graphics tightly linked to physical world. Neogence, if I am correct, is focused on building a true AR platform in the sense I just described?

Robert Rice: Hrm, I have argued with a few others about the actual definition of AR. Some people prefer a narrow and limiting view (3D overlaid on video), but I think in terms of the market and the end-user, it is better to have a wider definition. In that sense, AR is purely the blend of real and virtual, with or without full 3D overlaid on video. If we go with that, then Wikitude, Layar, Sekai, NRU, and others all fit into the AR definition.

Anyway, you are correct. We are building a true platform for AR, and this is quite different from what others are marketing as AR browser “platforms.”

There are a few problems with the “AR Browsers” approach that no one seems to be noticing. One is that they are all trying to get people to build new applications for their browsers, when they should be trying to get people to create content that they can share and browse.

Second, someone using Layar is not going to see anything that is designed for Sekai or Wikitude.

Third the experiences are generally for one user. While I love all of these guys and think each of the teams has some real talent on it, the model is flawed until someone using Wikitude can see the same thing that someone using Layar or Sekai camera is seeing (provided they are in the same physical location).

While we are working on our own client side technologies that we hope will be useful and integrated with every mobile device and AR browser out there, our core focus is on connecting everything and everyone together, and facilitating the growth of the industry with the tools to create content, applications, and so forth. We want to solve the really difficult technical problems (some of which most people haven’t even considered yet, because of the perspective they are looking at the potential of AR with), and make it easy for everyone else to do the cool stuff. We want to be the facilitators.

If you really want an idea of where we are going or some of what has inspired us, you have GOT to read Dream Park, Rainbows End, and The Diamond Age. If you have heard me speak anywhere or read my blog, you know that I am continually suggesting these and others.

Anyway, short answer, yes, we are building a true platform for ubiquitous mobile augmented reality, and we are absolutely the first to be doing so. I hope to demo some of this in October at ISMAR, with a full commercial launch next year (10/10/10 at 1010am Hehe, seriously). We will probably launch a website soon for people to start signing up and building a community now (especially if you want in on the beta testing of the whole kibosh).

Tish: So just to clarify,  how will Neogence’s approach differ and fit into the  growing world of Augmented Reality tools that we have now, e.g.,  ARTookit, Imagination, Unifeye?

Robert: I guess you could say that we are trying to build the infrastructure for the global augmented reality network. This could be viewed as a service, or even a platform for platforms. If Neogence does its job right, anything you create using ARtoolkit, Unifeye, or Imagination would be applications you could ultimately link to, integrate with, or deploy on or through, what we are building, and not be tied to a specific set of hardware, browser, or walled garden.

Tish: You mention Neogence is going to provide a platform for platforms. Without knowing the details that sounds like a lot of centralization which prompts the inevitable question: “Who owns the data?” Do you think other AR applications or providers would resist a “Platform for Platforms?” I know the potential centralization power of Google Wave has already got people talking about these issues (one of the comments in my recent blog post was about how Google Wave protocol may be interesting for a least some parts of augmented reality communication).

Robert: It really depends on perception and how we end up building it. We aren’t talking about creating a closed system. As far as who owns the data, it depends on what data we are talking about. For the most part, I think that if the end-user creates something, they should own it and have control over it. They should also be able to do what they want with it, independent of everything else.

This is one thing that proponents of the smart cloud and the thin/dumb client don’t like to talk about. It sounds great on paper, but when you start thinking about it, all that does is strip away power from the end user. Case in point…Amazon recently wiped every copy of George Orwell’s 1984 from all Kindle devices. They claimed they didn’t have rights to distribute/publish it and it was available on accident. The scary thing though, is that they literally went into every kindle out there, found copies, and deleted them.

How would you like it if Microsoft suddenly decided to delete every copy of Microsoft Office? Or every file that had a .doc extension? That is a huge violation…we feel like we own what is on our computers. But with the whole cloud thing, your data is at the mercy of whoever is running the cloud servers. No privacy, no ownership, no control. And if the system breaks, all you will have is a pretty dumb device that can’t do much on its own. Now, that isn’t to say that the technical merits and benefits of a cloud model aren’t worth pursuing, they are.

But I think there needs to be some hybrid model. Don’t dumb down my computer or my smart phone, let’s keep pushing how much these devices can do. We should take full advantage of centralized and distributed systems, but in a hybrid mashup sense. That is what we are pursuing with our AR platform, while trying to protect ownership and intellectual property rights of the end user.

Tish: Earlier today I was telling you how impressed I was by Google Wave – it is quite mind blowing to experience massively multiplayer real time interaction on what will be an open internet wide platform – Wave is breaking new ground here and more than one person has mentioned its potential role in AR to me (see the comments to my recent post on Ogmento).

I know you are a strong advocate of this kind of real time shared experience being part of AR.  But we are only just beginning to see it emerge via Wave on the existing web – what will it take to have this kind of real time shared experience in AR!  We got briefly into the thick client, thin client, cloud versus P2P discussions – what is your approach to delivering a massively shared real time experience that is like Wave not confined to a walled garden?

Robert: I’m not a fan of any of those models as being stand alone or mutually exclusive. Again, the hybrid model with the best of both worlds is key. In the early stages of the emerging industry, you are likely to see some walled gardens (or perhaps a walled garden of walled gardens…).

No one knows how things are going to turn out in the next five to ten years and few people are thinking about it actively. For us though, I favor Alan Kay’s quote (pardon the paraphrasing): “To accurately predict the future, invent it”. That’s what we are doing. In the short term, there will be plenty of experimentation in the industry and a lot of model testing.

Tish: Do you think though Wave protocols might be useful as at least part of the picture for AR standards?  As you point out open standards and open protocols are going to be vital for shared experiences of AR.  Is it important to build off existing protocols to get the ball rolling and what do you see as being the important early protocols for AR?

Robert: I think for now, we will use a lot of existing protocols for communications and whatnot, as well as the usual standards for things like 3D models, animation, and so forth. This is only natural. However, as the industry and technology evolves, we will need entirely new ones. As far as I know there is no existing market standard for anything like the Holographic Doctor from Star Trek Voyager, and that type of thing is definitely in the pipeline for the future (sooner than you would think).

Tish: All the excitement at the arrival of the browser like mobile reality developments has been really great – I feel people are getting a taste for what it means to compute with anyone/anything, anywhere and and anytime.

Wikitude started the ball rolling. And with Wikitude.me it is the first to support user generated content. Now there is Layar, Sekai Camera also. But as you mentioned to me in an earlier chat, with Layar and Wikitude opening up “their are probably half dozen other apps coming out in short order with similar functionality (even the AR twitter thing has some similarities).”

What has been most exciting to you about these developments up to this point? What will these apps/platforms need to do to stand out in a crowd.  Up to now, these browser like AR experiences do nothing with close by objects. Do you see “world browsers” with near object recognition coming out in the near future. Could Wikitude do this with an integration of SRengine or Imagination?

Robert: Yes, Wikitude or Layar could do this (integrate with something else for “near” AR) and it would be a step in the right direction. Tagging things in the real world is the basic functionality that will grow from text tags to photos, videos, 3D objects, and all sorts of other types of data and meta data. This gets really fun when that data is generated by the object itself. First is just giving people the ability to tag something and share that tag with their friends, everything else grows from that. This sort of functionality is probably the most exciting in terms of near future advancement.

However, I think the idea of a stand-alone browser platform is a bit awkward…unless you also consider firefox a website browser platform. After all, you can create widgets (applications) for it. Anyway, the point is having access to the same data…if you put three people in a room, one for each browser, they should see and experience the same content, although the interface might be different (based on what browser and of course which hardware they are using). This means there needs to be some communication between whatever servers they are storing their data on (meaning, user tags) and some standard for how those tags are created.

Of course, if all they are doing is grabbing the GPS coordinates of the nearest subway station and telling you how far it is and in what direction, then they should all be able to see the same thing, regardless of the platform. But then, that isn’t really interesting is it? I could get the same info on a laptop with google maps.

This is part of the problem right now though…no one seems to be thinking about the bigger picture much. All of the effort is either on making the next cool ad campaign for a car or a movie, or creating a tool to tell you where the nearest thingamajig is, but in a really cool fashion on a mobile device.

No one is talking much about filtering data, privilege systems, standards, third party tools, interoperability, and so on. There is also little conversation about where hardware is going. Right now everyone is developing software based on what hardware is available. This needs to change where hardware is being developed to take advantage of new software coming out (this happened in the PC industry a while back and growth accelerated dramatically).

These are some of the reasons why I led the effort to start the AR Consortium. We brought CEOs from 8 different AR companies and startups together to start talking about these issues. We are still getting organized and have plans to expand the membership to other companies, but we want to do this right and we aren’t rushing things. The important thing is that we have started and there is at least a line of communication open now, where there wasn’t before.

I would expect to see the early movers expanding what they offer very soon, and they will probably lead the way in the short term. Definitely keep an eye on the companies involved in the AR Consortium. There are lots of very smart and motivated people there, and they are far ahead of all the experimental dabbling in AR we are beginning to see on youtube, twitter, and elsewhere.

Tish: When we had a discussion about what were the basics for an AR platform and an AR browser earlier, you talked about the difference between tools, a platform, and a AR browser – like Wikitude and Layar which should be about features/functionality e.g. to create treasure hunts AR geocaching, invisible AR yellow sticky notes you can leave at restaurants you don’t like, etc. Also you noted it should let you explore (browse) multiple formats, and open content content for AR – any data, information, or media that is linked to something in the real world and the visualization/interaction with the same.

Wikitude is a stepping stone to a true browser by your definition. But are we also seeing what you would define as an AR platform emerging – Unifeye, Wikitude (you can recap your definition if you like too)?

I think Wikitude hopes to provide the lego blocks for  augmented reality readers, browsers, applications, tools, and  platforms?

Robert: I expect some segmentation among the various AR companies that are out now, as they find their individual strengths and focus on them. Some will emphasize the client software (the browser), others will develop robust tools for creating content, SDKs/APIs will advance and facilitate rapid development of applications, etc. Neogence is ultimately working on the glue in the middle that ties everything together, makes it massively multiuser, persistent, and ubiquitous. Things like Unity3D have the potential to fill a need in the middleware space.

Tish: I know Blair McIntyre (see my interview with Blair here) and others are using Unity3D as an AR client, Could Unity3D become increasingly important?

Robert: It has the potential to become a favored middleware for providing the rendering layer. It already works nicely in regular browsers, and on several mobile platforms. Why code all the graphics rendering stuff from scratch when you can just license something and extend its features with AR functionality?

Tish: Now to ask your own question back to you! There seems to be a lot of reason to think that, eventually, there will be the kind of access to the iphone video API that augmented reality really requires and by that I mean more than we will get with OS 3.1 which is rumored to deliver only about half of what we really need for AR on the iphone – “not truly useful when you want to align video. with graphics.”  So:

“The iphone…future or failure? Seemingly anti-developer stance regarding augmented reality, and only a sliver of the global market share. Are we letting the short term glitz of Apple and the iPhone fad pull us in the wrong direction? Shouldnt we be focusing on symbian devices that have the lion’s share of the market? or should we be looking more at either other OSs (winmobile, android) or not at all and trying to create a new platform that is more MID and less smart phone with a hardware partner?”

Robert: Apple and the iphone are a bit problematic right now. There is no way I can go to a venture capitalist (at least in North America) and say hey we are building awesome AR applications for winmobile or symbian…they would either laugh or they simply wouldn’t get it. There is this false perception that the iphone is the ultimate mobile device, it is the sexiest, and the only thing that people want. Everyone wants a demo on the iphone, the media is mostly interested in iphone developments, and the apple fanatic market could give a fig about other devices. Other devices may have a larger market share or even better hardware, but we have to focus on the iphone right now at least in the demo stage to get any market attention and traction worth the time and effort.

In the future though, unless Apple changes its stance with their SDK and APIs, and starts adding hardware that is key for mobile AR (beyond what is there now), the market will move on without them. This is a really easy decision to make given Apple’s draconian policies and the fact that their percentage of the global market is miniscule. The smart companies are looking at the whole picture and not putting all of their eggs in the Apple basket.

Of course, once the wearable displays are commercially viable everything changes. Wearable computers with small screens or even no screens are going to be what everyone wants. The interface will go from handheld touch screens to virtual holographic interfaces that you interact with using your bare hands.

So for now, (the immediate short term), its all about the iphone. Taking mobile ubiquitous AR to the global market and building for the future will be based on something else. Hardware risks becoming a commodity or a closed platform. Do you really want to buy the Apple iGlasses and only see AR content that is compatible, where your best friend has a pair of WinGlasses and sees something entirely different? No. The hardware, and the client software (what people are calling the ar browser now) will become common and it won’t matter what brand you use, they will all be accessing the same content.

But at least for the forseeable future, we are building software for specific hardware, and the sexiest mobile on the block is the iphone. The second someone comes out with something much better and the paradigm shifts (software driving hardware instead of vice versa) everything changes.

Tish: How is the quest for sexy AR eyewear going.  I know we were checking out the Japanese eyewear with Adam Johnson from Genkii just now.  For the Neogence project – as you are going for a fully developed model of AR doesn’t this necessitate going beyond the iphone and getting the hardware companies moving on the eyewear?

Robert: The guys making wearable displays really need to get off the pot and stop paying lip service to mobile AR. If they don’t do something quick, I,  and others, are going to be scouring the planet looking for someone capable of building the lightweight stylish wearable displays with transparent lenses we are begging for. We aren’t going to be waiting around for hardware anymore. The AR Pandora’s box has been opened. I should note that many of us (AR Consortium members) have had less than pleasant experiences or communications with the half dozen companies or so that are making wearable displays. Either their visual design is terrible, the materials feel flimsy, the field of view is limited, or the companies are preoccupied with other business and government contracts. Any attention to the growing AR market is an afterthought and in a few cases condescending. AR is going to be a billion dollar industry in a very short time, and these guys are just leaving money on the table. If they were smart, they would be begging the CEOs from the AR Consortium to fly out to their offices and collaborate on building a pair of wicked sick glasses. The smart phone manufacturers should be doing the same thing, but I have to say that they at least seem to have some ambition and zeal to create better devices, so I can’t really complain too much there.

Anyway, to answer the rest of your question, we have to assume that the hardware guys, especially regarding the eyewear, is going to take a long time to develop and release the things we need for the ultimate AR experience. So, our goal is to start building things now for what is available. That means scaling things down and handicapping what AR can do, so it works on the “sexy” iphone. The important thing though is to start creating applications -now- so when the glasses are commercially available, there will be a wealth of content for people to access and use on day one.

As long as Apple isn’t playing nice, it is going to hurt everyone. Is it any surprise that they shut down Google Voice? There is a huge opportunity for someone to step up and leapfrog the rest of the industry. Give us the hardware and we will create amazing software for it. Don’t compete with the iphone, surpass it.

Tish: What is the state of play of current AR technology and toolkits?

Robert: The current crop of AR technology and toolkits is absolutely critical for this stage of the industry, and everyone should be leveraging it as much as possible. I talk down marker and image based tracking a lot, but I also like to point out that it is the necessary baseline that the industry is going to be built on. The problem is that there is only so much you can do with marker driven apps, and as creative people and marketing types start conceptualizing about all sorts of cool stuff for the future, they risk setting the expectations too high. It is one thing to show someone the future, it is another to say this is the future and its happening right now. This is why I cringe everytime I see a conceptual video presented as “our product DOES this” instead of “our product WILL DO this.” Something that simple can still cause the butterfly effect of raising expectations too high and contribute to overhyping.

Tish: One of the things that seems very exciting about the new Ogmento partnership is that experienced content producers  Brad Foxhoven and Brian Selzer from Blockade are now taking a leading role in AR.  What are the most exciting directions for content that you see emerging for AR in the next 12 months?

Robert: Virtual (well, augmented) pets, and multiuser mobile AR games (2-4 people) are probably going to lead in the next 12 months for content. Easy, accessible, engaging.

Tish: And are you at Neogence also involved in content partnerships?

Robert: Yes, we are in the process of finalizing some content partnerships with an eye for long term relationships. We are specifically looking for partners that want to find substantive ways to leverage AR technology, and not use it as a superficial gimmick or attraction that wears off after five minutes. I’m still cringing over the Proctor & Gamble Always campaign with AR.

Tish: So back to your observation about some of the tricky problems re creating a true global massively multiuser, ubiquitous, mobile AR platform – what are some of the main obstacles to this mission in our view? (aside from getting investment!)

Robert: Trying to explain it to people. The technical problems we can handle or have already solved. But trying to communicate what exactly we are doing is still tough. Not because it is overly complicated, but rather because it is so new and different. People are having a hard time grasping augmented reality beyond marker/webcam.

Tish: Which AR tools are most important right now?

Robert: Content is critical right now to show what the technology is capable of and to continue building the presence of augmented reality in the public mind the big benefit to integrated / unified platforms now is speed of development for content. I think that the flash artoolkit = papervision is rocking the planet right now. It is accessible, easy to learn, and lets people create something very quickly. More tools and middleware are coming out and this increases options for designers and developers.

Tish: What are your favorite papervision apps?

Robert: Hrm, I don’t have a favorite papervision app just yet, although I think the tech is solid. I expect to see a lot of stuff built on that platform in the near future. Especially as more ad agencies get on the bandwagon and start telling their IT guys to learn how to program flash so they can make something. Have you seen www.ronaldchevalier.com Not so much for the actual AR stuff, but because the whole thing is just brilliant. Its exactly like some cult figure spiritual guru would do with AR. I wish I had thought of it first actually. This is probably one of the best -seamless- implementations of AR in marketing where it fits…it isn’t just jammed in there for the sake of saying they used AR.

Tish: Do you think Apple is going open the iphone to the full potential of augmented reality anytime soon – a lot of expectations have been raised?

Robert: Apple is like that guy has a party at his house and owns this really awesome state of the art home theater in his basement, but makes everyone watch a movie in the living room on a regular TV with a VCR.

They need to get over themselves and quit being a wet blanket. Otherwise, we are taking the beer and pizza we brought, and going to someone else’s house. Sorry, the Apple thing is a bit of a sore point with me.

Tish: But will people leave all that candy and soda at the appstore?

Robert: I tell you what though, there is an opportunity for certain mobile phone manufacturers to give me a call and start talking to Neogence and the other members of the Consortium. We have some ideas and specs that could have a radical impact on the mobile market and stuff the IPhone in a box. Hint hint.

Tish: So what is your vision for the ARconsortium.  I know it kicked off with a letter to Apple about the video API.  What is the next step? There was a lot of hope that this year would be big for MIDs but this really hasn’t happened yet – do you think there is hope for a MID take off despite the lousy economy?)

Robert: MIDs? No, not yet. smart phones are too lucrative and too hot. It isn’t time yet for the MID to go mainstream. For that to happen, there needs to be a driving need (cough ubiquitous AR cough)

The AR consortium is mostly an informal affiliation. I expect that representatives from each member will probably meet at every significant conference to catch up over drinks. We are also going to be planning for our own members conference at least once a year. That will happen after we expand the membership though.

The main idea behind the consortium though was to open up a channel of communication between the CEOs so we could work together on standards, solving problems, collaborating, forming some partnerships, and using the collective to bang on the doors of companies like Apple and others. There is power in a group.

Tish: You mentioned there is a whole long conversation we can have about getting the eyewear.  As you point out true AR eyewear changes everything.  Can give a little road map of where this has to go?

Robert: There are essentially four or five main approaches, depending on whether or not you make the lenses special or if they are just plain. You would normally want them to be plain so people with prescription lenses wouldn’t have problems and would have the option to switch them out. Some types use a more prismatic approach for top down projection, or a corner piece mounts lasers and bounces them off the lens into the eye.  Another approach is embedding OLEDs or something else into the lenses themselves.

I really like the Lumus approach, but their product design isn’t quite there yet. If the wearables don’t look cool, people won’t use them. To be honest, if I had the money, I’d probably ask the Art Lebedev guys to design them based on someone else’s optical engineering. They designed the optimus maximus old keyboard…   brilliant industrial designers, loaded with engineers too. If these guys couldn’t build the glasses and make them look damn bad ass, I’d be shocked. Heck, I bet they could build the next gen MID while they were at it.

Tish: Getting the hardware innovation and software innovation feeding into each other would be really great.

Robert: Absolutely.

Tish: That would push the eyewear forward too wouldn’t it?

Robert: All it takes is one, and then the competitive landscape would fire right up.

Tish: What applications would the accurate gps enable?

Robert: Everything. for example, you know exactly where the phone is and where it is facing, that means you can put it on a table and hit a button, then move it somewhere else and do the same thing in a few minutes, you have a nearly accurate “mental” model of the whole place now you go back and start dropping virtual flower pots everywhere.

This is one area where I think the smart phone guys are missing the boat and taking the cheap route. It is possible to have very accurate GPS (down to a six inch area) with better chips and firmware, but it is cheaper to stick in old tech. Most apps today dont need that hyper accuracy, so they aren’t bothering. Mobile AR though, thats a different story.

With that level of accuracy, you would know exactly where the mobile device is, so all you would need to know is the direction it is facing (orientation), and you could solve one of the problems with registering exactly where 3D objects and augmented media is (it is more complicated than I am describing it, but we don’t need to get into that much detail here). You wouldn’t need markers anymore.

Tish: Isn’t Wikitude doing this with Wikitude.me their tagging app.?

Robert: Not really. That type of approach is on a very large scale using the accelerometers compass and GPS to determine where you are and what is in the distance. They (and others like Layar) don’t handle “near” AR. They effectively poll your GPS and then check a database to see what is nearby and what degree/distance it is and then they draw a representation on the screen. They don’t even need a mobile device’s camera at all.

Even if they did things up close, its still based on finding landmarks or on things that are broadcasting their location. For example, if they were standing near me, they might get “robert, 37 degrees, 15 meters away” but they wouldn’t be tracking me exactly as I walk around or have the ability to overlay graphics on ME.

Tish: I retweeted your #ar marketing using ARToolkit + flash (markers/webcams) = Photoshop pagecurl <six months. Bad design kills innovation. I know you like Dr Chevalier though!  What are some of the other AR marketing projects that you like. What would you like to see in terms of innovation in the next 6 months?

Robert: The marker/webcam approach is already becoming overused and cliche (tremendously fast). Older readers will remember the ubiquitous photoshop page curl that adorned nearly every website and graphic on the internet back in the day. It was horrible. Yes, the Dr. Chevalier stuff cracks me up.

I want to see some big companies or ad agencies really try to do something different with AR, preferably mobile. Take some risks, do something different. Don’t follow the crowd. Innovation? I want to see some wearable displays with transparent lenses, I want a mobile device specifically designed for ubiquitous AR, I want to see some experimenting with AR in the green tech sector, and I’d like to see someone get that GiFi wireless technology from that researcher in Australia and jam it into a smart mobile. I would also like my flying car and lunar vacation now, thank you. It is almost 2010 and no one has found that black obelisk yet.

Tish: So a few closing thoughts! What do you see as the next big thing? Hopes for the ar consortium?  Biggest bstacle for commercial AR?  And what is the coolest thing you have seen this year?!

Robert: The next big thing is what I’m working on hahaha. I hope the AR Consortium will grow and be the active catalyst in making AR mainstream, practical, and world changing.

The biggest obstacle is making sure that the right funding finds the right developers to develop the right technology and create kick ass applications.

The coolest thing I’ve seen this year would probably be the facade projection stuff (see below): Now, imagine that, but without the projector. Thats part of what I envision for AR in the future.

Blair MacIntyre is one of the original pioneers of  augmented reality and an extraordinary amount of creative work is coming out of his Augmented Environments Laboratory at Georgia Tech – see YouTube videos here.  The screenshot below is from, ARhrrrr, a very impressive augmented reality shooter game created at Georgia Tech Augmented Environments Lab and Savannah College of Art and Design, (SCAD- Atlanta), and produced on the NVidia Tegra devkits – watch the demo here.

Blair has spent much of his career working on immersive augmented reality and more recently the integration of augmented reality with mirror worlds. Blair explains:

“I am interested in the intersection of mobile devices – whether they are head mounts or handhelds – and parallel mirror worlds…I think that parallel mirror worlds are a direct manifestation of the intersection of the virtual world we now live in (the web) and geotagging.  As more and more information is tied to place, and as more of our searching become place-based, we will want to do those searches about places we are not at.  A 3D mirror world may provide one interface to that data.  Want to plan your trip to London;  go their virtually and look around, see what is there (both physically and virtually), teleport between areas you want to learn about, and so on.  More interestingly, talk to people who are there now, and retrieve your location-based notes when you are on your trip.”

But, at a time when many augmented reality developers are focusing on AR apps for smart phones, including Blair (the picture on left opening this post is Blair’s augmented reality iphone app ARf), I was interested in finding out from Blair what the state of play was for the real deal Rainbow’s End style AR, as well as the potential he sees in smart phones to mediate meaningful AR experiences.

There is enormous amount of  innovation in mapping our world, see my post, “Location Becomes Oxygen at Where 2.0 and WhereCamp,” and  Ori Inbar’s  Where 2.0. conference roundup. But as Ori notes, to move augmented reality forward:

My point is not a shocker: all we need is to tap into this information and bring it, in context, into people’s field of view.

And this is what Blair MacIntyre’s work is all about.

Talking With Blair MacIntyre

Tish Shute: There do seem to be broader implications to augmented reality today than when this term was first coined. I am interested to have your perspective on how augmented reality may go beyond some of our early definitions?

Blair MacIntyre: I still think the original definition of the term is useful:  media (typically graphics) tightly registered (aligned) with the physical world, in real time.  Many people talk about many things that relate virtual worlds to places, spaces, objects and people.  There is room for many of them, and they don’t all have to “be” augmented reality.  I like using Milgram’s definition of Mixed Reality as everything from the physical world (at one end) to the virtual world at the other;  it’s a spectrum, and augmented reality just sits at one point.

The reason I like the old definition is I believe there is something special about graphics that are tightly, rigidly aligned with the physical world.  When things appear to stick to the world, and an obviously identifiable location, people can start leveraging their natural perceptual, physical and social abilities and interact with the mixed world as they do the physical world.  We’ve found this with the two studies we’ve done of tabletop AR games (Art of Defense and Bragfish);  one key to those games is that the graphics were tightly aligned with identifiable landmarks in the physical world (gameboard).

Art of Defense (pic on left) Bragfish (pic on right)

Tish: I know that you are involved with ISMAR 2009 which is the key US augmented reality conference.  What do you think will be the hot themes, applications, innovations at this year’s conference? Do you think this will be the year that AR really breaks out of eye candy into truly useful and sustained experiences?

Blair: Unfortunately, I won’t be involved this year.  I was supposed to be helping run the technical program, as well as the art/media program, but sickness in my family prevented me from having the time, so I am not helping this year.

First, I would not agree with the implication of the last question — I don’t think AR has just been eye candy up to now.  I do agree that the “high profile” uses of it have largely been that, which is mostly because of the limits of the technology.  I don’t think we’ll see huge changes in that regard by ISMAR this year.  However, we will hopefully see a mixing of communities that hasn’t happened at ISMAR before, and I do believe that this year (independent of ISMAR) we will see more and more AR apps.  Whether they go beyond eye candy is still a question.  I’m hoping that some folks (including myself and other ISMAR folks!) will help push AR in new directions.  But I also expect many folks new to ISMAR and AR to play a big role, because it is this new blood, especially those folks with real problems to solve, new art and game ideas, and a fresh perspective, that will open new doors.

Tish: You have been working on integrating augmented reality with virtual worlds. You mentioned that the way you use Sun’s Wonderland is really about pulling the virtual world into the real world, i.e., Wonderland, “is just a place to put data.”  How is your use of the persistent virtual space different from what we have become accustomed to call virtual worlds?

Blair: The approach we are taking in our project at Georgia Tech is to use the virtual world as the central hub of the information space, and allow the virtual world to be the element that enables distributed workers to collaborate more smoothly.  This is work we are doing with Sun and Steelcase (and the NSF), and is an outgrowth of a project (the InSpace project) that’s been going on for a few years.

What we are trying to do is use mixed reality and ubicomp techniques to pull as much of the physical activity into the virtual world, and then reflect that activity back out to the different participants as best suits their situation.  So, folks in highly instrumented team rooms will collaborate in one way, and their activity will be reflected in the virtual world;  remote participants (e.g., those at home, or in a cafe or hotel) may control their virtual presence in different ways, but the presence of all participants will be reflected back out to the other sides in analogous ways.  We may see ghosts of participants at the interactive displays, or hear their voices in 3D space around us;  everyone will hopefully be able to manipulate content on all displays and tell who is making those changes.

A secondary benefit, I hope, is that by putting the data in the virtual world and making that the place that gives you more powerful and flexible access to the data (e.g., by leveraging space and giving access to history), distributed teams will begin to have the virtual space become a place they go to work, bump into each other and have those casual contacts co-located workers take for granted.

Creating the Information Landscape of the Future

Tish: At the end of my interview with Ori Inbar he said, in order to have a ubiquitous experience “you’ll need to 3d map the world. Google earth like apps are going to help but it is not going to be sufficient. So let’s leverage people. Google became successful in part by making people work with them.  Each time you create a link from your blog to my blog their search engines learn from it.  So let’s find ways to make people create information that can be used for AR.” What ways do you think people can create information that can be used for AR?

Blair: I think the big part of that is the creation of models and environments, the necessary “baseline” for specifying experiences.  Google and Microsoft are clearly working toward this;  recent videos from Microsoft show them starting to move the photosynth work toward Virtual Earth.  Similarly, I came across a page where people are finally starting to mine geotagged Flickr [see my post, “Location Becomes Oxygen,” and here for more on the “The Shape of Alpha” project from Flickr]  images to create models.  It’s that kind of thing that will be useful first;  using the data we all create to enable modeling and (eventually) vision-based tracking in the real world.

After that, it’s a matter of time till more of what we “create” (e.g., Tweets and blog posts and so on) are all geo-referenced;  these will become the information landscape of the future, the kinds of things people think about when they read “Rainbow’s End”.   The big problem will be filtering, searching and sorting.  And, of course, safety and security.

Tish: You are working with Unity3D to research the integration of mobile location based AR with persistent mirror world like spaces.  What has attracted you to Unity? What is the difference between this and your Wonderland project? I know you mentioned. you will be using head-mounted displays are part of this Unity project. What are your goals for this project?

Blair: We started to use Unity3D because it gave us what we wanted in a game engine.  Most importantly, it’s very open and let us trivially expose AR technologies into the editor.  Similarly, it can target the iPhone, so we can begin to work with it on that platform, too.  The biggest problem with creating compelling experiences is content;  and a show stopper for creating content is not getting it into your engine.  Unity has a nice content workflow.

Unity3D is a front end engine, for creating the game;  Wonderland is both a front end, and a backend.  We are actually looking into using the Wonderland backend with Unity as well.  Wonderland also has growing support for doing “real work” in a virtual world, which is key to our other projects.

Eventually, we’ll be using HMD’s.  The goal for the Unity3D project, initially, was to explore what you can do with an AR/VR mirror-world; this is a project are working on with Alcatel-Lucent, and demo’d at CTIA this year.  It’s continuing to grow, though, and now includes a number of our projects, including some work on mobile social AR and soon, some performance and experience design projects in the area of AR ARG’s.  It’s really quite interesting to imagine what you can do when you have an “MMO of the real world” (which we now have for part of campus) that supports both VR-style desktop access simultaneously with mobile AR access.

Tish: Have you taken another look at OpenSim as a possible backend for augmented reality?  Recently I talked to David Levine, IBM and he is thinking about some possibilities to optimize OpenSim to dynamically load a large amount of objects at once (i.e how fast OpenSim can bulk load into an existing sim) and make it better suited to augmented reality/mirror world type projects.

Blair: I haven’t looked at OpenSim recently.  We will probably look at it this summer.

Tish: Why did you select Unity as a good client for augmented reality?

Blair: Unity is a 3D game authoring environment so at some level it is no different from using Ogre, if all the associated stuff was just as well done. It has integrated physics, scripting, debugging, etc. – you can write code in javascript or C# or whatever.   It has a good content pipeline, as well, and supports a range of platforms.

It has simple networking built in, so multiple unity engines can talk to each other but it is not a virtual world platform out of the box – there is no back end …

Tish: Someone described Unity to me as a great client waiting for a great backend? So what are you going to use as a back end?

Blair: There is no real processing except in the client right now.  We will eventually have to create a back end.  We are thinking of using Dark Star because someone on the Sun Wonderland community forums has already built a set of scripts connecting Unity to Darkstar.

But for us, we are not proposing right now to build a real product.  This is research to demonstrate what you could do if you actually had the back end.

Tish: What are the most important aspects of the backend from your POV?

Blair: We want to simulate a variety of the interesting aspects of the back end.  So I very much care about notions of privacy and security and how these sorts of AR/VR Mirror Worlds would work in practice.  But I care about how those things as they impact user experience, not really about how we would really implement them.

Tish: So looking at some of the big problems from the perspective of user experience? Are we are going to go through the same growing pains that the web and VWs have seen, for example, will we have to type in passwords to get into everyone’s little worlds….

Blair: Well you know the SciFi background to this, you’ve mentioned it in other posts on your blog.  Because when you look at the Rainbow’s End model where you have security certificates flying around, that is in effect what cookies and so on are now.  You can authenticate yourself once and then have those certificates hang around. So you can easily imagine how it could be done.  But the big question is how does that change user experience.  There are all kinds of things that start coming into play – like what happens if nearby people see different things – it goes on and on!

Tish: Sounds Like this is very valuable research.  It seems to me that there will be a lot of investment soon in putting the pieces together to do location based markerless AR and it would be nice if we knew more about it from the user experience POV.

Isn’t it vital for a productive intersection between mobile AR and persistent mirror world spaces for us to have markerless AR?  Aren’t we right at the beginning of people really saying yeah markerless AR is doable now? But it seems to me not many people researching or working on fully immersive AR and its integration with mirror worlds?

Blair: I think some of the AR community is thinking about this. There’s probably people who are doing stuff in some other non technical communities. It wouldn’t surprise me to find out that people in the digital performance or ARS electronica world who are thinking a little bit about these sorts of things. Although not necessarily at the level of actually trying to build it, because they probably can’t right now.  But experimenting with the precursors.  My colleagues in digital media like to point out that this is often the purpose of digital art, to point out new directions and push the boundaries.

Obviously Science Fiction has explored the possibilities because that is what Rainbow’s End and the Matrix were all about.

Tish: and Denno Coil…

Blair: There has been some research – people like my adviser Steve Feiner up at Columbia, Mark Billinghurst in New Zealand, myself and people at Graz University in Austria .  But partly it has been so hard to do mobile AR up to now – so many people mock head worn displays and can’t get past current technology – you have had  to be willing to ignore the bulky back packs and cables and batteries and so on.  That is changing which is good.

My current response to the anti-head-mounted display people is if 5 years ago you told me you told me that fabulously dressed people who care about their looks and wear stylish clothes would have had big things hanging from their ears that blink bright blue light, so they could talk on the phone, many of us would have said you were crazy, because it would be ugly and so on.  But because there is an intersection of demonstrable need and benefit…Bluetooth headsets are really useful and the sort of early gestalt feeling that grew up around them – that people who use them are so important that they always have to be in touch, they wear these things – so people accept them.

It will likely be a similar thing with head mounted displays. And I don’t know if it will be that people wearing them so that they can read their mail while driving, god forbid. But it will be something.  And when we get the 2nd generation of the wrap glasses that look more like sun glasses and are not bulky and so on, we will have the potential for them catching on because you will look at them and you will think that the person is wearing because they are doing x…

X might be surfing a virtual world or reading their email or keeping in touch, or being aware. It will happen. But they have to get unbulky enough and there has to be more  than one important application, not just watching TV.

Picture above shows  an outside view of the KARMA AR system;  the knowledge based maintenance system Blair built in his first year of grad school (“first AR system Steve Feiner, Doree Seligmann, and I worked on”).  Blair noted, “The Communications of the ACM paper on it (from 1993) is a pretty widely cited AR paper.”

Tish: I think the need for  full on transparent, immersive, wraparound, Gucci stylish eyewear with a decent field of view are the elephant in the room in terms of realizing the full potential of augmented reality.  There are a few new players in the field Digilens,  Vuzix, others?  What is the progress in this area and what do you hope for in terms of near term solutions?

Blair: I agree with that sentiment.  I think that, in the near term, there is a lot we can do with handhelds, as we’ve been doing in the lab.  However, because it’s awkward and tiring to hold up a device, even a small one, for any length of time, handhelds will only be good for “focused” uses of AR.  Such as the table-top games we’ve been doing, or the constellation viewing app that I heard came our recently for the Android G1.  I don’t even see something like Wikitude as that compelling (beyond the “gee wiz” factor) for a handheld form factor.   Many proposed AR apps only really become compelling when users have constant awareness of them, and that requires a see-through head-worn display.

I’ve seen the mockups of the Vuzix ones;  they seem pretty interesting, and are getting to were early adopters could use them (they will be cheap enough, and will hopefully be good enough).  Microvision’s virtual retinal display is also promising;  the contact lens displays will be the most interesting, if anyone can ever make them work.   I don’t know of anything else out there.

“its not really a killer app you care about, it is the killer existence that all of the technology and small applications taken together facilitate”

Tish: While location based services are accepted now and people are understanding that it is something that opens up a new relationship to everything, we still haven’t found the experience that will get everyone holding up their mobile devices?

Blair: Well that is actually the killer problem.  Gregory Abowd is one of my colleagues who does ubiquitous computing research here at Tech.   Way back when we started the Aware Home project (Aware Home Research Institute at Georgia Tech) when I first got here about ten years ago, there was always this question of what is the killer app.  So Gregory comment in a meeting once that its not really a killer app you care about, it is the killer existence that all of the technology and small applications taken together facilitate. It is not that any one of these AR demos we see right, whether it is seeing your photos in the world or whatever, is important. Its that when taken together, there is enough of a benefit that you would use the whole environment.

In the original context we were talking about an instrumented home, but it is the same thing here with AR.

The problem with the mobile phone as a AR device is that problem of awareness. If I have a head mount on and I walk down the street and there is bunch of probably-not-useful-but-potentially-useful information floating by me, that’s a good thing, because I may see something that is useful or makes me think of something else.  But if I have to hold up my phone to see if something might be interesting nearby, I will never hold up my phone because at the time there is a high probability that there won’t be anything particularly important there.  You might imagine you can get around this by using alerts or something like that, but then you overload whatever alert channel you use.  For example, I forward maybe 5 or 6  people’s updates from Facebook to my phone – started with my wife, a few friends, my brother, and the net result of that is I never get SMSs’ anymore because when my phone buzzes, usually I ignore it because it is probably just somebody’s random Facebook update. So if we start overloading channels like that with “oh there might be something useful here in the real world, if you pick up the phone and look through it you will see it … and I will buzz you.” People  just start ignoring the buzzes.

So it is a very hard problem if you think about the kinds of applications that people always imagine with global AR — names over peoples heads and other random information floating in the world — until you have a head mount and all that information is around you all the time. That is when those sort of applications will actually happen.

Tish: Robert Rice notes: “AR is inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.” (see my interview with Robert, “Is it ‘OMG Finally’ for Augmented Reality?). And I think the iphone experience has laid the foundation for the increasing desire to experience the network wherever we are – and not be stuck behind a pc.  We cannot perhaps do all we want to do yet. But even in the range of things we can do know, we are not even sure exactly what it is we want to do where yet is it?

“imagine your iphone Facebook client supports AR and that all data on Facebook might be georeferenced – pictures, status updates etc…….”

Blair: Yes that is a huge problem. I have been lucky to be able to teach two fun classes this year that let the students and I start to explore some of the potential that handheld AR might bring.  Last fall I taught a handheld AR game design class — coordinated with a class at the Savanna College of Art and Design’s Atlanta campus — and we had the students build a sequence of prototype handheld AR games, which was a lot of fun.   This spring I taught a mixed reality/augmented reality design class with Jay Bolter (a professor in the School of Literature, Communication, and Culture here at GT).  Jay and I have been teaching this class off and on for about 9 years; this semester we decided to say to the students “imagine your iphone Facebook client supports AR and that all data on Facebook might be georeferenced – pictures, status updates etc…….” and have them do projects aimed at such an environment.

Tish: Not many of our favorite social media today have much sense of location do they? But Flickr  are  utilizing the geo-referenced pictures to create vernacular maps…..The Shape of Alpha

Blair:Yes that is because lots of cameras put geo location data into the exif data so they can extract it…

Some mobile Twitter clients like the one I use in my iphone will let you add your location.  But in general Facebook and other sites don’t have any notion of location. But if you look at all the things people do in Facebook, such as sending gifts and other games, its easy to imagine what these might look like with geo-reference data.  So, the high level project for the class is the groups have to design experiences people might have using mobile AR Facebook.  We told them to assume Facebook as it stands now, but add geolocation and AR to the client.  The class boiled down to “What would you imagine people doing?” So it has been kind of fun.

And we are using Unity for the class too – the same infrastructure I am working on in my research linking mobile AR to persistent immersive mirror world type spaces – and we having the students mock up what a mobile AR Facebook experience would be like.

Tish: Can you describe some of the ideas you class came up with that you think have potential? I know Ori mentioned that from the games class he liked Candy Wars.

Candy Wars

Blair: In the end, they had a nice range of projects in the Spring class.  One created tag clouds out of status messages over spaces, others looked at analogies to virtual pets and gift giving out in the world, one looked at leveraging geolocation to help with crowd-sourced cultural translation, and three groups did straight-up social games.

[See all of the projects from the handheld AR games class on YouTube here]

iphone, Android, or NVidia Tegra devkits or the Texas Instrument’s OMAP3 devkits?

Tish: Is anyone in the class working on Android?

Blair: Nobody is using Android because no-one in the class has the phones. We have ATT microcell infrastructure on campus.  Some ATT people joke that we are better off than them because we have a head office on campus so we can build in the network applications which people even at ATT research can’t do.  But because  we have this infrastructure on campus, and a great relationship with ATT and the other sponsors, we have the ability to provision our own phones without having to pay for long-term contracts, which is vital for research and teaching.

Tish: So does this lock you into the iphone?

Blair: Well the G1 is of course not AT&T but it is GSM so we could probably buy them unlocked and put them on our AT&T network. But the students I work with are much more interested in the iphone right now.

Tish: Is that because the iphone has the market?

Blair: For me the reason I am not interested in the G1 is because you can’t do AR on it – there is Wikitude and a few other apps, but it is all hideously slow.  Worse, because the Java code isn’t compiled like it would be on the desktop, you can’t do computer vision with it, so you can’t do anything particularly interesting on the current commercial G1s.  We could probably take the NVidia Tegra devkits or the Texas Instrument’s OMAP3 devkits (both are chipsets for next gen phones — high end graphics, fast processing),  and install Android on those and we may actually do that yet.  But, it seems like a lot of work right now, for not much benefit.

Augmented Reality shooter game ARrrrr from Georgia Tech and SCAD Atlanta on the NVidia Tegra devkits – watch the demo on YouTube here.

Tish: Everyone seems very excited about the iphone OS 3.0 and the addition of compass. Compass is pretty essential for AR right?

Blair: It is necessary if you can’t do other forms of outdoor tracking, but the problem is that the compass on the G1 isn’t very good, relatively speaking and the iPhone one probably won’t be much better. It does not have very high accuracy, nor is it very fast (compared to, say, the high end 3D orientation sensors we use, from Intersense and MotionNode). As far as I can tell, it doesn’t even give full 3D orientation. I don’t have a G1 (although I have pre-ordered an iPhone 3Gs), but people have told me it only has absolute 2D orientation, so you can only line things up if you are careful.  Your can’t look around arbitrarily…

Tish: You can’t sweep your phone?

Blair: You can look left and right, but if it doesn’t have full 3D orientation, you can’t go up and down. You can’t tilt it in weird directions. It is not fast in the form that you would want to look around quickly.  So it is nice demo.  And it is good for what the Android people use it for which is to let you do your Google street view by looking around, which is actually really useful.

I think there are lots of really useful things you can do with such a compass.

And, it is clear that compass is a necessary feature if we want to do AR.  It’s just not sufficient.

Outdoor Tracking and Markerless AR

Tish: Isn’t it essential for markerless AR? I guess not I just saw this post about SREngine on Augmented Times!

This wasn’t up when we spoke so perhaps you have some comments about what it brings to the table?

Blair: Maybe. The folks at Nokia are working on outdoor tracking, they demoed some stuff at ISMAR last year on the N95 handsets that is all image based.  We are trying to do some work with them, one of my students is working on it.  And probably Microsoft is going to do more on this as well, they had a video up showing that they are also working on vision based techniques.  If you give the phone the equivalent of those panoramic Google Street View images (assuming they are up-to-date) and you are standing at the right place, you don’t really need a compass, you can figure out which way you are looking by looking at the camera video. Ulrich Neumann (USC) did some work on tracking from panorama’s years ago, I don’t know what ever became of it.

Regarding SREngine, that project appears to be a pretty simple first step, but is probably just a demo at this point, and limitations like “only works on static scenes” and “doesn’t work for simple scenes” means it’s probably extracting some simple features out of the image and then matching those to some database.  The trick would be getting this to work on a large scale, where the world changes a lot.   It’s not obvious how to get there.

Tish: So forget RFID for AR…

Blair: RFID is not really useful.

Tish: not at all?

Blair: RFID is useful for telling you what things are near you.  The problem is it doesn’t give you any directional information – it just tells you you’re in range of the tag. So can use it to tell you when you are near a certain product for example.  So it is useful in terms of telling you what thing you are near, and then you can load up a vision system or something else that will recognize that thing.

In that way, it could be useful as a good starting point.

Similarly for computer vision, the compass and the gps are very useful for giving you an initial guess at what you may be looking at that can then speed up the rest of the process.  But, computer vision by itself will not be a complete solution because if I have my panoramic Google Street view (or whatever image database I use for tracking) and you are standing between me and the building -  I am not going to see what I expect to see, I am going to see you.

So I think it is all going to be part of one big package – you are going to see accelerometers, digital compasses, and gps and then combine that with computer vision and other sensors, and then maybe we are going to start getting the things that we have always dreamed about.  I like to show this video from the U. of Cambridge (work done by Gerhard Reitmayr and Tom Drummond) of an outdoor tracking demo because it gives a sense of what will be possible.  Techniques like this will be an ingredient in the future of things.  It becomes especially interesting when you have these highly detailed mirror worlds.  It is sort of one of those chicken and egg problems where if I have an highly detailed model of the world then techniques like they have can be used to track.  But that mirror world needs to be accurate or you can’t use it for tracking, and why would you create the mirror world if you couldn’t track?

Tish: I noticed in your comment to “my interview with Robert Rice” that you said you thought that is was important not to collapse AR into ubicomp – “forgetting what originally inspired us about AR” is, I think if I remember correctly, the suggestion you made. But aren’t ubiquitous computing and AR basically coextensive?

The vision of ubicomp Mike Kuniavsky describes – “sharing data through open APIs and the promise of embedded information processing and networking distributed through the environment” demonstrates how much can be done with very little processing power.” In its most immersive form augmented reality requires a lot of processing power. I think we have all become very conscious about trying minimize levels of consumption.  Can you explain why you think people shouldn’t see AR as the Hummer (energy squandering indulgence) of Ubiquitous Computing?

Blair:  I think there will be a hierarchy of interfaces. You are going to have the rich Rainbow’s End like experience – you are totally submerged in a mixed environment, if you have a head mount on (its not going to be Rainbow’s End for while) but if you don’t have the headmount on that information might be available to you other ways, whether it is a 3D overlay using your handheld or just a 2D mashup with Google maps.  But there will be some circumstances and people who will want to get the compelling experience you can only get with the headmount.

Tish:  Are you doing any research on how all these hierarchies of experiences will fit together – what aspects of this are you looking at?

Blair: The thing that really needs to happen is you need to have this backend architecture that allows you to collect your data from different sources and aggregate it much like the web. Right now Google Earth and Microsoft’s Virtual Earth are much like the old pre-web hyper-text systems that were all centralized. And what we really need is to have the web equivalent where Georgia tech can publish their building models and I.B.M. can publish their building models and their campus models, and your client can aggregate them, as opposed to Microsoft or I.B.M. puts their building models into Google Earth and then somehow you get them out with Google’s google earth browser. That’s just not going to fly.

Tish: so what does it take then to get us to this backend architecture, because I’m in total agreement?

Blair: The nice thing about augmented reality versus virtual reality is that you don’t need everything modeled. You can do interesting AR apps like Wikitude with absolutely no world model.

Tish: So that means we can start with what we have – utilize cloud services without a full blown backend architecture?

Blair: It may very well be that Google Earth and MS Virtual Earth act as a portal because people go and build models and link them with KML, and they can see them in google earth but they can also download the KML’s through some some other channel. So it may be that those things end up being something that feeds some of this along. Then people start seeing a benefit to having these highly accurate models so then you start integrating the Microsoft photosynth stuff and leveraging photographs to generate models.

It’s just keeping up with it and building it in real time is the challenge. A lot of folks think it will be tourist applications where there’s models of times square and models of central park and models of Notre Dame and the big square around that area in paris and along the river and so on, or the models of Italian and Greek history sites – the virtual Rome. As those things start happening and people start building onto the edges, and when Microsoft Photosynth and similar technologies become more pervasive you can start building the models of the world in a semi-automated way from photographs and more structured, intentional drive-by’s and so on. So I think it’ll just sort of happen. And as long there’s a way to have the equivalent of Mosaic for AR, the original open source web browser, that allows you to aggregate all these things. It’s not going to be a Wikitude. It’s not going to be this thing that lets you get a certain kind of data from a specific source, rather it’s the browser that allows you to link through into these data sources.

So it’s that end that interests me. It’s questions like “what is the user experience”, how do we create an interface that allows us to layer all these different kinds of information together such that I can use it for all my things. I imagine that I open up my future iphone and I look through it. The background of the iphone, my screen, is just the camera and it’s always AR.

I want the camera on my phone to always be on, so it’s not just that when I hold it a certain way it switches to camera mode, but literally it’s always in video mode so whenever there’s an AR thing it’s just there in the background.

When we can do that I can have little alerts so when I have my phone open I can look around and see it independent of the buttons and things that I’m tapping and pushing to use the phone. That’ll be a really a different kind of experience.

Of course it is not known yet if the next gen iphone will have an open video API.  And of course, the current camera is pretty low quality, so why would they give it an open API until they put in a better camera?  I am not expecting anything one way or the other until the 3Gs comes out and people start using it.

But there are many things about the iphone 3.0 OS that are hugely important, like the discovery API that allows people to play games with other people nearby, that don’t have much to do with AR.

Tish: You have an iphone AR virtual pet application ARf.

Macrumors wrote it up and suggested that the neg gen iphone will have compass and open video API.  What are your plans for ARf?

Blair: ARf is just a demo right now.  I know what we’d like to do with it, but it would require tons of work;  imagine what it would take to do a multiplayer, social version of Nintendogs?  It’s not clear what we’d really learn by doing that, but there are lots of other game ideas we have that we want to explore.

Tish: I think it was on Twitter where Tim O’Reilly said, “saying everything must have a RFID tag is like saying we can’t recognize each other unless we wear name tags. Look at what’s happening with speech recognition, image recognition et.al. and tell me you really think we need embedded metadata.” What would you say to that?

Blair: I think that whatever extra data is there will be used. So if we put machine readable labels on some objects then they’ll be used if they make the identification and tracking problem easier. But it’s pretty clear that people are already working on tracking and so on.

A lot of these mobile AR apps are clearly putting ideas in people’s minds things that won’t really be doable in the near future. Like being able to look down the aisle of the store and it recognize all of the products. Given the distances and complexity of the scene, the number of pixels devoted to each of those objects, and so on – you just can’t recognize things in that context. But if I’m standing in front of a small set of objects, or looking at one thing, or I’m standing in front of a building, or if I’m in the store and because of the location API — imagine an enhanced location API that can tell me within a few feet where I am, and then combine that with some use of the discovery API that allows the store to tell your device you’re in the toothpaste section. Now you only have to look for different brands of toothpaste. So now you can recognize the big letters “Crest” or whatever. It’s all about constraining the problem.

That’s why I like that particular piece of Drummond’s work, the tracking web site I mentioned above. The general tracking problem of looking around and recognizing objects and tracking is still impossible. But if I know roughly what direction I’m looking in and I have a good estimate of my position, and I have models of what I should be seeing when I look in that direction, then it becomes a tractable problem. And so it’s not that a compass and a GPS are 100% necessary. But if you have them it certainly makes things possible that you wouldn’t otherwise be able to do.

Imagine for example  if there’s a new version of GPS, I just noticed that some of the new satellites going up have this new L5 channel. There’s the L1 & L2 signals  that the military and civilian ones use and they added this civilian L5 signal, which should make GPS more accurate. I haven’t found anything online that says how much more accurate.

But someday, hopefully, all GPS will get to be the quality of survey-grade GPS. Right now, if you get an RTK GPS from one of these companies that make the survey grade GPS systems, they give you position estimates in the range of two centimeters, and update 10 to 20 times a second. When you have that kind of positional accuracy combined with the kind of orientational accuracy you get from the orientation sensors we use in the lab from Intersense and MotionNode, everything is easier because you’ve pretty much got absolute position. You put that into a phone and now when I look up, it’s still not perfectly aligned because there will still be errors (especially in orientation, since the compasses are affected by metal and other magnetic noise). But it does mean if you and I are standing 5 feet apart from each other and look at each other, I can pretty much put a little smiley face above your head. Whereas now, with GPS, if I look at you and we’re 5 feet apart our GPS’s might think we’re on the opposite side of each other because they’re only accurate to two to five meters.

And that depending on the time of day and weather!

Putting RFID tags everywhere is easy; the problem is the readers – they currently require lots of power and they have a limited range.  Sprinkling RFID tags everywhere is fine. But you have to be able to activate those tags and read back the signal.  In certain contexts it works.

Tish: And one final question!  What do you think can be done re beginning to think about standards for AR.  Is there a meaningful discussion going on yet? Thomas Wrobel left this comment on my blog rcently and I was wondering what your position was on some of the ideas he raises?

Wrobel wrote, “The AR has to come to the users, they cant keep needing to download unique bits of software for every bit of content! We need an AR Browsing standard that lets users log into an out of channels (like IRC) and toggle them as layers on their visual view (like Photoshop).Channels need to be public or private, hosted online (making them shared spaces) or offline (private spaces). They need to be able to be both open (chat channel) or closed (city map channel) as needed. Created by anyone anywhere. Really IRC itself provides a great starting point. Most data doesn’t need to be persistent, after all. I look forward too seeing the world though new eyes.I only hope I will be toggling layers rather then alt+tabbing and only seeing one “reality addition” at a time.”

Blair: I agree with him, in principle.  But, I’m not sure there’s a point yet.  It can’t hurt to try, of course, from a research perspective, and I’m interested in the experience such an infrastructure would enable (as we’ve talked about already).

I am excited about going to Where Week 2009 – Where 2.0 and WhereCamp, this week (for more see Brady Forrest’s post).  Where Week will be total immersion for five days in a think tank with creators of the information landscapes of the future.

As you know, if you have read my previous post – here, I think the “Mobile Reality” panel is a must.  And I have been looking forward to hearing more about The Shape of Alpha from Aaron Straup Cope, Flickr, since Etech 2009 when I was introduced to Aaron by Mike Kuniavsky (see my interview with Mike Kuniavsky at Etech here and more on Mike’s concept “information shadows” in his Etech talk).

Shape of Alpha is revealing some fascinating possibilities for mining geotagged Flickr images.

As Tim O’Reilly noted in a tweet, Aaron Straup Cope’s recent post, The Absence and the Anchor, describes, “some of the surprising things Flickr is learning about people from geotagged photos.” Aaron’s post also announces that the “donut hole shapes” are available for developers to use with their developer magic via the Flickr API.

“If the shapefiles themselves are uncharted territory, the donut holes are the fuzzy horizon even further off in the distance. We’re not really sure where this will take us but we’re pretty sure there’s something to it all so we’re eager to share it with people and see what they can make of it too.”

For more on shape files see Aaron’s blog post about “some experimental work that I’d been doing with the shapefile data we derive from geotagged photos.”

Creating the Information Landscapes of the Future

I have been thinking and writing a lot about augmented reality lately.  And key thought leaders in this space like Blair MacIntyre, Robert Rice (see my interview here), and Ori Inbar (see my interview here), have clued me in to how vital it is, for an ubiquitous experience, for us to find ways to allow people to fill in the stories that can be used for augmented reality.

As Ori noted in conclusion to our recent conversation:

“in order to have a ubiquitous experience like Robert Rice and others are striving for, you’ll need to 3d map the world. Google earth like apps are going to help but it is not going to be sufficient. So let’s leverage people. Google became successful in part by making people work with them.  Each time you create a link from your blog to my blog their search engines learn from it.  So let’s find ways to make people create information that can be used for AR.”

Jim Purbrick, another key thinker in this area (interview upcoming), also notes:

“you can imagine a crowd sourced set of hints for any location so, AR knows roughly where it is and can do photosynth style matching  to find out exactly what it’s looking at and get the extra data it needs about that thing (humans are really good image recognition systems, and are also pretty good at interfacing with networks) instead of marking up real objects with ids you take pictures of real objects, tag them and then search them based on images from your ar system.”

Ori Inbar suggested to me an idea that I really liked – the notion of bread crumbs where, “You don’t have a constant view of what is happening when you walk but you get images and text and all sorts of things from people who walked there before – like breadcrumbs.“  And as Design Under Sky points out about Shape of Alpha:

“The truly amazing part of this process is how the “community” has the authority to provide areas previously unmapped.  By uploading personal photos of areas not covered by mapping software, members have the power of further shrinking our world through greater visual access and understanding of locations one might not be willing or unable to visit.”

Aaron Straup Cope, Flickr, Todd E. Kurt, ThingM and Mike Kuniavsky, ThingM

The Locative Media Manifesto

@stamen’s tweet brought André Lemos’ brilliant, thought provoking, ” Locative Media Manifesto,” to my attention.  I am also looking forward to hearing about how old maps “can shed light on modern geography when placed in counterpoint to the state of art in modern maps from Google or Microsoft” from Michal Migurski, Stamen Design, who will present Flea Market Mapping at Where 2.0.

André Lemos writes:

“After uploading to Matrix up there – Internet 1.0 – now is the time to “download cyberspace,” information about things down here – Internet 2.0. We are not dealing with what is virtual up there, but of what to do with all this information about things and places down here! How can we relate to things and places? And now that these things and places are provided with digital information and Internet connections? Do we invoke Heidegger and Lefevbre?”

I will leave it to people smarter than I to invoke Heidegger and Lefevbre as Andre Lemos does so eloquently in Locative Media Manifesto. But by reminding us artists and activists created the term “locative media” to “question the mass use of LBS (location based services) and LBT (location based technologies,”  the manifesto delivers 30 principles to inspire creators of Locative Media and explorers of the, “current dimension of cyberculture, comprising the era of “cyberspace leaking into the real world” (Russel, 1999); an era of the “internet of things.”

I feel well primed for Where Week by my visit to the ITP Spring Show, 2009 last Sunday. It was an interaction riot, jam packed with brilliance and off beat explorations of locative media which I experienced through the senses of my 9 year old.  His pick for best of show is below. But he had many favorites and I have put some pictures up on my FLickr stream with links to the creator’s sites.  One of my favorite projects Alexander Reeder’s S Ring – “seducing people by talking with your hands,” is up on my Posterous blog.  You can see a list of the extensive media coverage the show got here.

Loose Interaction Topologies

The picture above is of a game of mud pong  in Tom Gerhardt’s Mud Tub.  The mud interface – “a smart tub with some mud” knows the topology of the mud and where your hand is. Mud Tub takes advantage of  a complex material – to explore loose interaction topologies, including as seen above a game of Mud Pong.  Loose interaction topologies are a way we can explore meaning in “the internet of things.”

Tom explained his own exploration of the internet of things to me very succinctly:

“I am not trying to make mud better. I am trying to make computers better with mud.”

He elaborates on the value of Mud Tub in this regard on his site, dirtycomputing:

“The Mud Tub occupies a space similar to other experimental human-computer interfaces, like, multi-touch surfaces, body controllers, augmented reality systems, etc, which push the boundaries of codified interaction models, and drive the development of innovative software applications. Beyond its role as a research topic, the Mud Tub also exists as an open-sourced hardware/software platform on which interactive artists and designers explore new methods for creating and displaying their work.”

Recently, I talked to Ori Inbar (above), formerly senior vice- president at SAP.  Ori is on a mission to make augmented reality commercially successful not in 5, 10, or 15 years, but now. Ori is the founder of Pookatak Games – a video game company, “with a vision to upgrade the way people experience the world.” Ori will be participating May 20th, in O’Reilly’s Where 2.0 panel, “Mobile Reality” -  an event not to be missed IMO.

The taste for computing anywhere anytime has entered human culture via the iphone and is spreading like chocolate cake and pizza at a preschool party (see why the iPhone changed everything).  And while the full flowering of the next step is yet to come – computing anywhere, anytime by anyone and anything (“the internet of things”), our love for these first devices capable of being mediating artifacts for ubiquitous computing (Adam Greenfield) is a vital first step to free us from our tethers to computer screens, and fulfill the promise of augmented reality.

If you need more convincing on the pivotal role augmented reality will play as the web moves into the world, check out Tim O’Reilly’s recent comments in this video clip posted on Augmented Times and here early last year.

From another perspective, the gloomy specter of economic and environmental catastrophe  is driving a movement to “infuse intelligence into the way the world work’s.” But the challenge for a smart planet is not just about making environments smart, it is about using smart environments to enable people to act smarter (see my interview with Adam Greenfield).

We need a rapid upgrade in both the way the world works, and the way we experience the world.

((Note:  It is time to read (if you haven’t already) Bruce Sterling’s Caryatids (Cory Doctorow’s book of the year for 2009) “as a software design manual” (see Julian Bleeker) because Caryatids reveals the Gordian knots of human folly, greed, compassion and desire entwined in near future designs for technologies to save the world.))

Ori Inbar, worked with Shai Agassi (Shai is now leading the world changing Better Place ) driving Netweaver from a mere concept to a “major, major business for SAP.” So Ori has already been through the cycle of working in a very small startup and growing it into a billion dollar business.  He has both the experience and the passion to realize his vision for augmented reality.

At Pookatak, he explains :

“We design “reality experiences” that make users’ immediate environments more significant to them. We wish to free young and old from getting lost in front of the screen. By delivering the world’s information to people’s field of view, and by weaving real world objects into interactive narratives, we help people rediscover the real world.”

Pookatak will release their first game this summer. Currently it is under wraps. But Ori gives us some glimpses of what is to come in the interview below.

In addition to founding Pookatak, Ori is involved in a broader effort to move augmented reality forward. On his blog, Games Alfresco – he recently welcomed a new partner, Rouli Nir, Ori has focused his eye of wisdom on every significant recent advance in Augmented Reality (check out this essence of Ori’s thinking in a fast paced video presentation for WARM ‘09).

Also Ori is one of the organizers of the interactive media track at ISMAR 2009.  At ISMAR this year, Ori explained, “we are trying to bring in people that develop interactive experiences for consumers, beyond the traditional attendees coming from a research perspective.”

In the interview below, Ori explains much of his thinking on how augmented reality will become commercially successful.  Enjoy it, think about it, and share it. And most importantly, if you can, get involved with ISMAR 2009.

Ori  has inspired me to participate in ISMAR this year.  Ori pointed out:

The call for papers is on, and this year it targets well beyond the typical research papers audience and into interactive media and art folks.

There are plenty of opportunities such as:

Art Gallery

Demonstrations

Tutorial

Workshops

It’s a huge opportunity to shape the emergence of augmented reality.

Interview With Ori Inbar

Making Augmented Reality Commercially Successful

Tish Shute: You are considered a key trail blazer in AR and you have the go to blog for augmented reality!  What are the most important lessons you have learned researching, writing, and developing AR in the last couple of years?

Ori Inbar: You need to have a vision. You need to know where this is going to go in ten or fifteen or twenty years. But you’ve got to start with something really simple that makes use of the technology you have on hand. And do something that is practical, that people will like, and something they would actually want to buy. Its as simple as that. I’m currently looking at what we could do with existing technology. First of all, you have to put it in front of people. Right now most people have never heard about the term augmented reality. Go into the street, and ask 100 people about it, maybe 2 would know about it. So you need to put it in front of people because most people think it’s still science fiction or a special effect you see in movies, not something you can experience in real life.

Tish: It seems to me to that for augmented reality applications to become popular with existing technology the key breakthrough would be getting people to hold up their phones. What are the obstacles to getting people to use their mobile devices like this?

Ori: There’s a really nice cartoon by Tonchidot (below) – the Japanese company behind the Sekai Camera. It’s an illustration showing the evolution of man, from ape to man (holding a cell phone looking down), to the developed man holding a device like a camera – in front of its eyes.

Which is exactly what you’re talking about. People ask, “are people going to walk with this like that all day long?” Probably not. I mean you have to build it in a way that doesn’t require them to hold it like that all the time. People are used to this gesture with the ubiquitous digital cameras. I tested one of my prototypes on a two and a half year old girl. She had no problem holding it just like she holds a camera.

Tish: Blair MacIntyre mentioned, “The problem with the mobile phone as a AR device is a problem of awareness,” i.e., you have to have a way of letting people know when there’s something interesting wherever they are. One of the issues regarding this is if you get too many alerts, then you tune them out.

Ori: First of all Blair is one of the people in academia that get it. Because he looks at it from an experience perspective. Not just as an interesting technical problem to solve. Let’s start with getting people to enjoy this new experience. The AR demos so far were mostly eye candies, and mostly for advertising – the GE AR ad created a lot of buzz; but you look at it for 10 seconds and you forget about it.  You need to build something that people would want to experience over time and would be willing to pay for. I think that’s the big test, right?

Now in terms of having a ubiquitous experience where you’re continously connected, it doesn’t have to be an overwhelming experience. Just like some of the social media tools we’re using today, we decide when to connect, and we filter out the trash. You could get alerts only for things that really matter to you, not for everything that happens in your immediate environment.

There will be many layers of information, and it’ll be up to you to pick the ones you want to experience. The real benefit is that you get the information in your own field of view and in context of where you are or what you do.

Tish: So what are you working on these days?

Ori: We are working on a little app that targets a very different audience than what you’d expect: pre schoolers. We think we can encourage them to get away from a PC or TV screen and learn something while playing – in the real world. You’ll hear more about it as soon as this summer. Nuff said.

But, it is a small application that will run on the iPhone. People ask how many pre-schoolers own iPhones? Well, their parents do.

Tish: Yes there are certainly many New York kids with iPhones – my kid now has my old iphone.  He has pretty much switched from playing games on his DS to the iPhone. I noticed in your WARM video you place a big emphasis on AR as something that will get kids away from screens and engaged with reality.  This is something parents will approve of!

Ori: Yes I saw something really interesting at my kids’ party one day; they were all sitting around the room – looking down at their own DS screens.  You could play the DS anywhere, but kids would usually play it on the sofa, looking at the screen, isolated from the world. With an iPhone and a camera, and the application we’re producing, reality becomes part of the game. Yes that makes it all of a sudden much more interesting for parents. Because kids are spending so much time in front of the screen, all of a sudden they’re something that will encourage them to interact with real objects, real things. Every parent I’ve talked to loves that idea.

Tish: Yes that is what is cool about the work of Kati London – I think I saw someone say this on Twitter, “Kati puts the computer in the game not the game in the computer.”

Ori: Yes, kids are spending more time in front of games and the computer because it’s more interesting. It captivates them with “game pleasures ” that tap into their brain’s dopamine circuitry – constantly seeking reward and satisfaction. So you’re not going to be able to tell them to go back to playing in reality without these pleasures. We have to study these mechanics from games and bring them into reality. It’s about programming real life; and augmented reality helps you achieve that.

Here’s an example: cause and effect; in a game when you do something you always get an immediate effect. You’re good, you get a reward. You’re not good, you get a cue to improve. In real life you do things and you could wait 2 or 3 years until you actually get feedback (if you’re lucky). Augmented Reality allows you to bring these mechanics into the real world. I think that’s going to help kids rediscover reality, in a new sense, which is what every parent is dreaming about.

Tish: I don’t know how much you can say about your app. But in regard to doing augmented reality on the iPhone.. there’s no compass. Is this a limitation?

Ori: True, no compass yet. But the camera gives you a lot of information that you can interact with. When you run the application, you see the world in front of you, and if the app can recognize real life objects – it can put virtual elements on top of it.

Tish: But not with any accuracy unless you’re using markers. Are you using markers?

Ori: We’re using natural feature recognition. It doesn’t have to be an ugly looking marker. It can be any image.

Tish: So you’re using image recognition. Are you working with one of these image recognition startup companies (list here )?

Ori: We’re working with one of those. What’s unique about it is it runs very nicely on any cell phone, and on the iPhone it works the best. For this first app, it doesn’t really matter where you are physically; the geolocation is not part of the experience.

Tish: For a truly engaging AR experience we will need more of a backend than is currently available?

Ori: I call the backend the cloud, where you have all this information and ways to access it from anywhere. Actually I think it’s become pretty mature today. If you look at the different elements required to enable an augmented reality experience to work, you have – first – the user whose always in the center. Then you have the lens. The lens can be an iPhone, or glasses, even a projector. The lens allows you to watch, sense and track information in the real world: people, places, things. Then in the backend you have the cloud where you store and retrieve information.

So if you look at the maturity of these different elements, I think the cloud is in pretty good shape. Because there’s so much information we’re collecting and storing. Anything from Google, Wikipedia, Facebook, all that kind of stuff, it’s a lot of useful information you can access from anywhere using APIs. And a lot of it is also starting to include geolocation information. Take Loopt or Google’s friends service that allows you to see where your friends are and what they’re doing. There’s tons of information out there and it’s pretty easy to access it. Now what do you do with it is the question?

Wikitude is such a simple and brilliant application and nobody thought about doing it until this guy from Salzburg did. It doesn’t have any sophisticated visual tracking. It knows your position and it’s simply looking at the angle you’re pointing to. Based on these parameters it brings information from Wikipedia that pertains to your field of view. So most of it was already there. It’s just a matter of connecting the pieces in an experience that is valuable for people.

Tish: It is the uptake of even a very simple technology that puts the magic in it.

Ori:  Yes, take Twitter. If you go to its homepage it looks like a very simple boring app but it is something that is both enjoyable and very useful to people.

Why you should participate in ISMAR 2009

Tish: I know that you are involved in organizing  ISMAR (picture above from Ori’s post on “ISMAR 2009: The World’s Best Augmented Reality Event…,“) and there is a call out for papers and for volunteers, can you tell me more about it?

Ori: Yes, we hope to have the first ISMAR where we practice what we have just discussed: let’s build on all the research invested so far and instead of thinking only about 5-10 years from now, let’s see what we can do today. So we are bringing people in from other disciplines – artists, interactive media developers and people from the entertainment industry.  The goal is to use the technology to make something interesting for people – again, something that people would buy, and making it commercially successful.  Many people either don’t know about ISMAR because in the past it was a pure engineering-orientated event and people  from a commercial perspective of AR weren’t attracted to it.  The Chair of the Event this year is based in Florida and he is going to bring in a lot of people from the entertainment industry such as Disney. I think this will transform this event into something more like SIGGRAPH – more of an industry event.  As one of the organizers of the interactive media track we are trying to bring in people that want to build applications for consumers.

Tish: In terms of AR applications what are the flagships today?

Ori: There are very few because it’s just the beginning. There’s one tiny studio in France called Int 13 . They’ve created maybe the first commercial game running on a mobile device using AR technology. It’s called Kweekies. It was one of the contenders for the Nokia Mobile innovation awards. They were one of the ten finalists, but they didn’t win it. It’s looks really cool. It’s somethng that runs on your desk, with a marker. Many AR folks say markers are the past, markers are ugly. But it’s still a cool experience. I think people will go for it.

Tish: Yes I think we will have to look to small companies that are free to think creatively to lead the way.  It seems many games companies are tied up pulling off huge big budget projects and enterprise is still catching up on how to use social media!

Ori: Yes, last year I was in the game development conference (GDC); there was no mention of augmented reality – not on the exhibition floor, none of the sessions, nobody talked about it. I was stunned. Then this year, there was a little a change. There were like three demos on the exhibition floor, Metaio, Vuzix and a Dutch company called Beyond Reality.  And then there was Blair’s talk, which was very very cool. The room was packed with people. And after the talk there were dozens of people lining up to talk with him about the topic. There was definitely interest, but still on the very edge. The video game industry is still a hit driven business and publishers spend upward of 20-30 million dollar to create the best AAA game possible. They just can’t take the risk. So it’s going to come from smaller companies, from outsiders coming in with a vision and understanding on how to put the AR pieces together to create a totally new experience.

Tish: But the basic tool set is there isn’t it?

Ori: I talked to some folks at the games developer conference, many folks with MMO background, and they have great ideas about AR. It’s great to see different people with different views on what’s needed first. “Joe the Programmer” had this idea of creating a small piece of hardware that you can put in every house and provide accurate geospatial information in your home. That could  open up many opportunities for AR experiences in homes.

Tish: Don’t you think we have enormous resources in terms of image databases that provide a great basis for augmented reality.  I was talking to Aaron Cope at ETech about The Shape of Alpha – Flickr’s vernacular mapping project using all the geotagged photos in Flickr. That is such cool project. Aaron will be speaking at Where 2.0 also.

Ori: Think of Google Earth. Google Earth leveraged communities to basically map all the major cities around the world into 3D models. And that is an essential step to be able to do augmented reality outdoors. Because if you had to model everything from scratch, it wouldn’t be realistic.

Augmented Reality and Becoming Greener.

Tish: I am really interested in how AR interfaces might be useful to some of the emerging energy identity/metering projects like AMEE and WATTZON because I think it is very important that people have very intuitive, immediate, and enjoyable ways to relate to energy data so they can make greener choices.

Ori: Back in the day I had an idea to build an Augmented Reality application to become greener. You look at things around your home with the camera and it  recognizes its green gas footprint and makes recommendations to reduce it.  I guess it was a bit too early to do that based on visual recognition alone…you’d need  additional sensors that would provide related information about what you are looking at.

Tish: Well as there is more interest in Green technology do you think we may see VC interest in some green AR projects now?

Ori: I talked to some of the investment folks, Angels as well as VC’s about AR and they had no clue what it is. There’s a need for a whole lot of education. And there are no proof points (as in successful investments in this domain), and counter to popular belief – they don’t like risk so much…

Tish: And consumer adoption must lead the way, right?

Ori: Just like with every emerging technology in history, people never bought the technology, they bought the content, the apps, the benefits that came on top of the technology. Whether it was VHS winning over Beta Max, or BluRay winning over HD. It’s always because of more/better content. Look at the video game console war: Xbox, and Nintendo did better than Sony just because they had more and better games. Even Windows was a success thanks to its applications. People bought it for the applications not the OS. The content is the first to drive demand.

Tish: One of the challenges to giving people new ways to relate to their energy consumption is that you can just have them looking at graphs of how bad they have been in the past you – that may make them feel bad but that doesn’t necessarily give them ways or motivation to change. There perhaps needs to be more immediate relationship to the data to facilitate change. I think the mantra for optimization of anything from energy usage to supply chains is timely, actionable data?

Ori: There are a lot of ideas about measuring information and displaying it to people. For example, the Prius hybrid car, one of its interesting features – which is kind of game like – is a constant display of your current fuel consumption. That alone changes how people drive because they try to beat the “Score” and as a result conserve more fuel. That model can be applied to our homes…

Tish: Yes that is something I am very interested in. I have been following several projects in this area – one of my favorites is the Arduino, Current Cost/Tweetawatt, Pachube integrations I saw at Homecamp.

You joined a start up with Shai Agassi which was bought out by SAP right? He has a brilliant approach with Better Place.

Ori:  I think what’s really unique about Better Place’s approach is that he doesn’t require people to change their behavior. People are still going to have their own cars. They’ll be able to drive as far as they want, and for the same (or lower cost). Its not necessarily about a new technology, electric cars have been around for a long time but there was no way people were going to be limited by the 50 or 70 mile range and Better Place is solving that problem. With its infrastructure of charging spots and battery switching stations, drivers are going to be able to drive anywhere. And it’ll be similar to having to stop once in a while to refuel your car. The price maybe even lower than what you pay today for your transportation needs – and you’ll stop generating green gas. It’s a clever way of taking technology to a whole new level without changing the behavior of people.

Tish: Better Place is a classic example of things as a service isn’t it?  It is basically a utility company.

Ori: It is similar to a phone carrier model.  You pay for a membership that gives you access to the car (equivalent to the phone) and electricity (equivalent to the phone line) for the same price of fuel cost today. And as bonus you get to save the world.

How the iphone changed the game for AR – and the iphone versus Android

Picture from Ori’s post, “GDC 2009: Why the iphone changed everything”

Ori: And back to AR, you have to take the same approach, because nobody’s wants to don those huge head mounted displays or backpacks. You have to take advantage of people’s current behavior: they already carry their iPhones or similar devices.

Tish: As we discussed, you just have to get people raising up their phones and looking through them when that is a useful thing to do. Both Wikitude and Nathan Freitas’s graffiti app were enough to get me interested in the evolutionary step of raising my phone! Nathan’s graffiti app is nice. You leave a marker for your graffiti so other people can find view/add their own – a nice primal experience like pissing on the lamp post to let your pack know where you’ve been.  Also the graffiti app taps into a long history of  NYC street culture around tagging and graffiti art (see my interview, “Is it OMG finally for Augmented Reality?”).

Ori: The app store has fundamentally changed the mobile gaming industry. Last year they were in shambles. There was no growth. Everybody was complaining, “we can’t handle it, there’s a million phones, and you have to test it on each phone. And carriers suck, they don’t care about sharing and promoting your content. Everything was bad. This year mobile gaming is the hottest thing. And it’s all because of the iPhone. It changed the game.

Tish: How do you think Android is going to get traction against the iphone?

Ori: Well the number one thing is the form factor – the iPhone is just much cooler than the G1. Its OK but it doesn’t have the same feel. People thought it was going to be easy to clone the iPhone but none of the attempts succeeded so far.

Tish: How much does it matter for AR not being able to runs things persistently in the background on the iphone?

Ori: Actually they have add a such a capability in OS 3.  You can now make use of a background service.

Tish: OS 3 will open up new possibilities for AR?

Ori: The access to the video API is still not public.  But there is a new Microsoft application – Microsoft Tag that makes use of that API which means it is probably OK to use it.

Tish: (I ask Ori for his card and he shows me how to read it with my iphone.) Oh nice you have an AR card, of course!

In Search of Pong for Augmented Reality

Tish: So how will AR begin to, as Blair’s friend put’s it, “facilitate a killer existence,” particularly as we are probably looking at some new and perhaps pricey hardware?

Ori: You could take the Better Place approach. We’re going to give you a great experience and we’ll include the devices as part of that experience for the same price. Let’s say you subscribe to an AR experience  which offers access to multiuser, support, and all the information you need wherever you go – exactly according to the vision. You pay for a subscription on a monthly basis and included in that cost we give you a better device that offers a  better AR experience. It’s following the phone carrier approach, but in a good way.

But first of all we do need our Pong! I was sitting with a couple of AR game enthusiasts at the GDC and we were asking ourselves, “how do we create the first pong for AR?”

Was Pong a multiplayer game? Not necessarily! Did it connect to the network? No! We have to create the first dot in a long line of dots that will bring us to our destination.

Tish: You haven’t seen a Pong yet have you?

Ori: Not yet. I mean there’s maybe a handful of games and apps out there, but I don’t think any of them is a Pong yet. Still, it’s getting closer.

Tish: Kati London is doing some very interesting work on bringing games into reality, isn’t she?

Ori: Yes, she works with Frank Lanz at Area/Code. He teaches at NYU and has designed games for the “Come Out and Play” festival here in Manhattan. And a lot of these games are actually low tech.

Tish: Yes I have a big alternate reality game blog brewing that I haven’t had time to write yet!

Ori: The city is the gameboard is their slogan. It’s going to be a great playground for AR games. The city becomes a theme park. The city could become an even bigger touristic attraction. People will come to the city to be part of these games. So you’re having thousands of people running around the city playing all sorts of games from laser-tag style to history adventures, to treasure hunts.

Composing Reality

Tish: So why haven’t you focused on one of these kinds of games with your company?

Ori: We have a couple of scenarios along these lines that we’re planning for 2010-11. But first focus on what’s possible today.

Tish: And what’s stopping you from doing those kind of games today?

Ori: Many things. The devices are not there yet, location services are not accurate enough, ubiquitous sensors are not  there yet.

Tish: You think alternate reality gaming needs more “ubiquity” than is currently available?

Ori: Not necessarily. People are doing alternate reality games with no “ubiquity” at all. But my interest is to add the visual aspect. I believe humans are mostly driven visually.

Jane McGonigal said in a talk at GDC, that AR would allow us to program reality, which is exactly how I look at it. Once you can recognize things, some of it with WiFi and RFID and all sorts of sensors. But visual sensors is always going to be the ultimate way to recognize things. And once you recognize things and know what they are, and can pull information about those things (or people and places) from the internet, you can program it (visually). You could program it to be fictional, like in a video game, or it could be programmed as non-fictional, like a documentary. And that allows you to do things that before were unimaginable.

Tish: But you can’t forget the visual, it is primary the connection to peoples’ primary sensory relationships.

Ori: Yes, it’s like you go to a grocery store and you pick your vegetables, a lot of it is by sight and by touch. And what if you could also see just by looking at it that it’s from a local store, and that it’s organic?

Tish: It goes beyond overlays really?

Ori: By the way, I don’t like the term ‘overlay’. I know that’s how it looks: you either overlay or superimpose, but I’m still searching for a better term. A term I prefer to use is “composing reality”. Just like painters, they use brushstrokes and colors and compose a painting. We need to take the real element and the virtual element and compose them into something new. It’s not just about slapping one on top of the other.

Tish: yes I think the idea of dashboards is not so appealing.

Pookatak Games

Tish: Do you want to explain the evolution of your company? You have an interesting history of success with high end enterprise applications.

Ori: Since I was a kid I wanted to invent and create things. When I discovered software, that was a really cool way of actually creating things from nothing. From thin air; and you can do it very quickly. That’s what brought me into software. But I was always looking for the intersection between technology and art. Looking for ways to bring these things together. In the early nineties virtual reality was doing it. It had the appeal of cutting edge technology that can be combined with art. But then, as we all know, it crashed. So I joined Shai Agassi’s startup (who is now doing Better Place) back in the early nineties. I was one of the first employees in his startup which was developing multimedia products. I was leading the development of one of its flagship product. At some point we realized the technology could be great for an enterprise environment.

It was a really great experience. First going through this cycle from a very small startup and growing into this multi billion dollar business. I was responsible for defining and marketing SAP’s platform, which was called Netweaver. It was just an idea when we joined SAP and by the time I left it was a major, major business for SAP. I learned about the challenges of building a platform. No matter what purpose you’re building it for, it typically has similar rules. It’s definitely not just about the technology; the content that comes with it is really key to making a platform successful.

The third part of this platform trifecta is the community. If you don’t build a community, you won’t get the critical mass required for adoption. It may be your own platform but it’s not necessarily the people’s platform. That experience is very key to what we’re doing today. Now, a new industry is being born on the basis of a remarkable technology. But to drive adoption, first we’ll need good content. The content will be created using today’s technology with internal tools developed to simplify the process. Next step would be to make the tools used internally – available to other developers. Help scale the industry, enable innovation on a larger scale. That way we have a chance to create a platform. So it isn’t really just about my company. I’m so passionate about augmented reality, I want to it to become a healthy and successful industry for the next 5, 10, 15 years.

Tish: Yes I am so ready to be liberated from the sitting behind a computing screen! And I know that all this hardware is murdering the environment.

Ori: There’s ‘s the book by Rolf Hainich which is called “The End Of Hardware. ” It’s about hardware for augmented-reality. Once you use goggles or other AR interfaces you eliminate the need for screens, laptops, etc. It’s going to be great for the environment. You have read Rainbow’s End, right? According to the book in few years there will barely be any (visible) hardware. At least it’ll have a much smaller footprint for the environment. And it’ll touch every aspect of life, everything you do. It’ll change the way you interact with the world.

The Illusive Eyewear for Immersive AR.

Friend of Ori’s in San Francisco wearing retro AR goggles (from Games Alfresco, Ori’s roundup of GDC 2009)

Tish:OK lets talk about goggles.

Ori: Goggles are going to happen, we want to be hands free.

It’s going to happen because it’s just a more intuitive way to use this technology. But above all it has to look cool. Because if it’s not, if it’s a big headset, then maybe a small percent of the population might use it, but most people won’t. It has to look like an accessory, like new cool eyeglasses that you just must wear.

I recently talked to a friend, who runs an industrial design firm, and has experience in designing such glasses for companies like Microvision and Lumux. He says that when you try to bring the images so close to our eyes – there are some really hard problems to solve. Otherwise it can become really annoying and cause dizzyness.

But I’m optimistic. I believe it’s going to happen 3 to 5 years from now. It’s already starting now: Vuzix announced goggles that will be available this year. Some AR apps that are going to take advantage of next year. Initially only a fraction of the population will use it. And that’s going to help advance it and make it better and better. But it’s going to take time until it reaches the mass market.

Tish: In virtual worlds we have seen, I think, a lot of mistakes in terms of reinventing the wheel and producing too many proprietary versions of the same thing and not enough concerted effort on standards and open platforms that could create a vibrant ecosystem.  How can augmented reality not make the same mistakes?

Ori: There are some early AR open source efforts ARTookit, ARtag but it is not a movement yet.  One of the things we’re trying to do at ISMAR this year is to put together  discussions around key industry issues, such as standards. Some people say it’s too early, you have to have a defacto standard to start from. But pretty soon it’s going to be too late. Just like with virtual worlds, all of a sudden you have all these islands that don’t talk to each other. Why get to that point if we can plan to avoid it? Let’s start thinking about it right now. On the other front there are devices. There are pockets of people working on adapting devices for AR, second guessing the hardware companies. Why not get them together with the Intels and Nvidias of the world, and discuss what this device should be able to do. And then compete to make it happen.

Tish: How much luck are you having with this discussion part?

Ori: People are very interested in doing this. We proposed these panels for ISMAR. And I’ve got some key people already on board. They have tons of input, they want to get involved. We’ll see how much we can actually get out of it.

Tish: In virtual worlds it was a while before vibrant opensource communities developed.  OpenSim has I think been the breakthrough community in this regard.

Ori: You have to think about the elements up front. The dream job is to architect the industry. Say we agree on the required pieces. Then we could help the right companies succeed in delivering the pieces. Next, we have to collaborate so that these pieces talk to each other. And eventually these communication methods will become defacto standards and most developers will adopt it.

Tish: So I’m going to put you in the role. You’ve got your dream job. You’re going to architect this community. So what are the key pieces and where would you like to see the open source communities take hold first?

Ori: Open source will not be exclusive. It’s going to live side by side with proprietary technology.

The key pieces? You have the user at the center. And the user interacts with a lens. The lens includes both the hardware and the software. And then the lens senses and interacts with the world, which includes people, things and places. And these people-things-places emit information – about who they are, where they are, what they’re doing, etc  – which is then stored in the cloud.

And then you have the content providers, the people and companies, composers who weave AR experiences through the pieces we mentioned before. These composers need a platform that glues these pieces together. Pieces of the platform will be on the lens, and in the world, and in the cloud. If you manage to remove the frictions, and connect these pieces into an experience that people like – then you have a platform. What the platform does it reduces the overhead and accelerates innovation.

Tish: Another problem virtual worlds faced in their development was their isolation from the world wide web.  Will augmented reality avoid this plight?

Ori:  Yes, I believe the key, like you said before, is not to reinvent the wheel. The cloud is already there.  Take Wikitude for example, all Mobilizy had to do is build  a relatively simple client app, connected to wikipedia, and all of a sudden it offered a wealth of information in your field of view.

I think we can learn a lot from web 2.0. For example, in order to have a ubiquitous experience like Robert Rice and others are striving for, you’ll need to 3d map the world. Google earth like apps are going to help but it is not going to be sufficient. So let’s leverage people. Google became successful in part by making people work with them.  Each time you create a link from your blog to my blog their search engines learn from it.  So let’s find ways to make people create information that can be used for AR.

Ori Inbar directed Wiki Mouse – a WIKI Film co-created by a swarm of movie makers around the world.

HomeCamp is a home hacking, automation and green technology community that will be gathering in London tomorrow, Saturday 25th April 2009, 10am until 6pm BST (GMT + 1), and in an OpenSim event running alongside for virtual participation, to brainstorm new possibilities for distributed sustainability, creative smart meters, monitoring, graphing and visulaizing energy usage.

More details and videos on the blog. The wiki, which includes signup, is the main portal to all the online activity.

As James Governor notes here:

there has been a huge amount of code and applications released focused purely on using technology for home energy monitoring and automation.  We have an active google group and quite a few videos and content showcasing the various applications and hardware currently being used by geeks to save money and live greener.

Now the challenge is to see how this seedling home energy management movement can really grow into widely adopted distributed sustainability solutions that everyone can use, and participate in.

Both Chris Dalby (see here), Andy Piper, James Governor of Monkchips (see here),  and Tom Raftery of GreenMonk (see here), have posted on tomorrow’s HomeCamp event. So I am just going to add some quick notes, especially to highlight some of what will be going on virtually for those of you, like me, who can’t make it to London.

You can tune in either on the live video ustream, or sign up on ReactionGrid and join the OpenSim event. Also, you can keep up on what is happening on Twitter #homecamp. I highly recommend that you catch Tom Raftery’s talk which will be streamed from Spain live into the London meeting, the OpenSim event on ReactionGrid, and Ustream. Tom Raftery, a leading Green technology analyst at RedMonk (see also GreenMonk), will be picking up, in depth, on some themes raised in his brilliant ETech 2009 presentation, “Electricity 2.0: Applying the Lessons of the Web to Our Energy Networks.”

There will be homecampers dropping in to virtual homecamp in ReactionGrid throughout the day, including Chris Hart (the awesome “girl-geek”@dstrawberrygirl), MiketheBee, and Cminion, who has a number of cool projects to demo, including his energy turbines.  Dave Pentecost (pictured above with his Tweetawatt, Pachube Orb) and I (see our presentation for EarthWeek SL here) plan to be at Virtual Homecamp on ReactionGrid between 9am and 10.30am EST. Dave has done a number of cool energy monitoring hacks including a Pachube link to and from OpenSim.

Also keep your eye on Dave’s blog, The Daily Glyph, for what’s new in distributed sustainability. Dave just posted some great links on Sustainable Interaction, design and work by ITP researchers and others in sustainable use of technology.

Sustainable Interaction | Main / Papers

Sustainable interaction design | Sustainable Minds

Design For the Other 90% | Cooper-Hewitt, National Design Museum

If you are in London, look out for Oliver Goh of Shaspa as Oliver will be at Homecamp in London. As I mentioned in my previous post, Oliver will soon be launching both Shaspa commmunity and enterprise hardware and software packages for “Intelligent Energy Management.”

For a bit of homecamp history, James Governor (picture below from Chinposin), recaps  some of the successes of  the first HomeCamp here.

And last but not least, a big thanks to sponsors, CurrentCost, Greenmonk, Pachube, Onzo, and ReactionGrid,  and media partner theattick.tv who are making the London and virtual homecamp events possible.

]]> http://www.ugotrade.com/2009/04/24/homecamp-2-home-energy-management-and-distributed-sustainability/feed/ 2 http://www.ugotrade.com/2009/04/19/sensor-networks-and-sustainability-connecting-real-virtual-mobile-and-augmented-reality/ http://www.ugotrade.com/2009/04/19/sensor-networks-and-sustainability-connecting-real-virtual-mobile-and-augmented-reality/#comments Sun, 19 Apr 2009 06:32:59 +0000 http://www.ugotrade.com/?p=3381

Today, I did a presentation, on connecting real, virtual, mobile, and augmented spaces to support sustainability, for Earth Week SL, with Dave Pentecost and Jim Purbrick, who presented on Carbon Goggles.

Dave and I focused on sensor networks, open data, Pachube, OpenSim, and sustainability from perspective of, “hack local, think global.”  Dave and I will be picking up on some of these themes of sensor networks and sustainability next week in our presentation with Dimitri Darras at ITP,  NYU, Aprl 24th, 6.30 pm to 8 pm – details here.  If you are in New York City, I hope to see you there.

We got some interesting insights into augmented reality from Jim Purbrick whose Carbon Goggles project prototypes how we can use augmented reality to read carbon identity and to combine well organized, verified data from AMEE – a neutral aggregation platform to measure the “carbon footprint” of everything on earth, with crowd sourced tagging and linking.

Shaspa – “the sensor network system that has it all”

We also discussed, recently launched, Shaspa. Shaspa’s energy management packages connect spaces – real, virtual, mobile and augmented.  Shaspa has been blogged  by Maxping and Virtual World News, so you can read all about it, but the Shaspa device kit won’t be available until next week. Some key features of the Home Energy  package are listed on the slide above.  However, this evening, Dave Pentecost and I got a sneak preview of both the Shaspa commmunity and enterprise hardware and software packages from Shaspa founder Oliver Goh. We were pretty impressed.

Dave: “It’s the ultimate hackable device for energy management!”

Oliver: “Bring us any sensor device – with documentation, and within three days we will put a driver into Shaspa.”

Oliver is on the right and Dave on the left in the picture above. The picture below shows Shaspa in OpenSim. Oliver and I will be attending the 3D Training, Learning and Collaboration Conference in Washington, DC, next week.

Links

Here are some of the links that came up in the presentation as many people asked for them to be published. Dave also has them on his blog.

SLIDES on GOOGLE DOCS:
Earth Week SL Presentation, April 18th, 2009 – Google Docs

Pachube, sensor networks

Dave’s blog covering Maya archaeology, jungle ecology, and technology

Maya Frontier, Usumacinta River videos

Collapse

Arduino

Pachube – tutorials

Pachube Apps -

Arduino-SL-Pachube data site

SL to Pachube site

Dave’s Danger Shield – Pachube tutorial

TweetaWatt site (LadyAda)

Dave’s post on TweetaWatt to Opensim/SL

Peter Quirk’s post on Importing Sketchup into RealXtend

Opensim

RealXtend

ReactionGrid

homecamp

cminion -wind turbines in OpenSim

MiketheBee

Is it “OMG finally” for Augmented Reality?

Smart Planet: Interview with Andy Stanford-Clark

Orange Cone – Information Shadows and Things as Services

Nathan Freitas holding a Peek with Oliver+Coady partner David Oliver talking to fans at New York Tech Meetup – Mobile Meets Social

Volunteerism and participation in public life seem to come naturally to Nathan Freitas. Nathan is one of the leading innovators/developers in NYC in mobile strategy/design (for more on his Android development read on). And he is much in demand as speaker who shows others how to realize their mobile experience and design dreams (for upcoming speaking engagements see Nathan’s blog). But also Nathan has spent much of the last ten years working on new ways for causes and non profits to benefit from technology.

Most recently Nathan has started working part time for the NY Senate under, “Albany’s King Geek,” the new CIO Andrew Hoppin:

“The CIO team is organizing training sessions for senators and their staff on social networking platforms and how to pay attention to online feedback. Last week, they hired mobile specialist Nathan Freitas to create new phone applications that will allow citizens to get government news on the go.”

Also, Nathan is currently supporting engineer on, The Extraordinaries, a smart phone application that explores territory “beyond the flattening tendency of online relationships” (see this list from Andy Oram of the Questions on Government participation).  The Extraordinaries is Ben Rigby and Jacob Colker’s prize winning project  – “a smartphone application that delivers volunteer opportunities on-demand.”

Ben’s post, Information Age Volunteerism – Open Sourced! Crowdsourced! and the extensive comments give a detailed analysis and critique of this brilliant and creative new approach to volunteersim in the information age.

Nathan, in my view, is a great example of how to “do well by doing good.” And, I am particularly excited by the work Nathan and his partner in Oliver+Coady, David Oliver, are doing on Android, e.g., Nathan’s new gReporter – opensource, geotagging, media capture report client (you can download the source here).

I first met Nathan when I interviewed him about Cruxy in 2007 (see my post, The Mixed Reality Metarati and “Destroy TV:”  Merging Art, Technology, Politics and Play).  Nathan recently announced that “the fat lady has just uploaded her last song.” Cruxy was an innovative distributed music venture Nathan started with Jon Oakes.  Although, as Nathan explains, Cruxy “never really broke through in the way we hoped.” Nevertheless Cruxy seems to have been a fertile garden for ideas that are coming of age in Oliver-Coady’s current mobile experience endeavors.  As Nathan explains, “the world, including Apple and iTunes, has shifted to embrace some of the ideals we have always had – open formats, more ways to distribute and promote online, more avenues for niche content to be discovered and heard.” Cruxy’s technology platform, built by the incomparable Will Meyer:

“was a great success in my mind, being one of the first to fully embrace Amazon’s cloud and provide a widget-based commerce system that actually worked!”

Nathan has a new company, Oliver+Coady. But Nathan told me that he feels he is over his “start up phase.”

Nathan Freitas: I am just tired of the term “startup.” I’m more interested in being defined as person than a member of a corporation. Also I am more interested in the ideas of cooperatives, and have been working on this idea (see here for more on the New York Creative Cooperative ).

Tish Shute: You do a high percentage of non profit work. Are you still managing to keep the home fires burning in the economic downturn?

Nathan Freitas: There is definitely profit to be made in non-profits because even if you only get paid half of what you get for corporate work, it is worth it in terms of fulfillment, ego, respect, and general contribution back to the planet. However, I’ve also been investing time & energy w/o pay into thinking about how causes can benefit from technology for over ten years. So its not just something you decide to do one day, and suddenly are successful.

Tish Shute: What are some of the highlights of your non profit work recently?

Nathan: Well, The Extraordinaries project is definitely a highlight. It is focused on a whole new approach to volunteering and winning the first prize at the WeMedia Conference for the non-profit tech category was a great validation of the work. I am just a supporting engineer on the effort, which was founded by my good friend Ben Rigby (a longtime non-profit tech guy as well) and Jacob Colker.

Ben wrote this excellent book on mobile tech and organizing, Mobilizing Generation 2.0 He’s done a ton of mobile work with youth voters via his non-profit, Mobile Voter.

The Extraordinaries is really taking all of our joint experience and putting it into a whole new system that is meant to go beyond generic email blasts that just ask you to “send a fax” or “send a link”. it gives people specific tasks they can accomplish on their phone or in their local area using their phone.

Tish: Did you do Twitter Vote Report with Ben too?

Nathan: Oh, no, Twitter Vote Report was with a different group of folks…mostly east coast-based, organized by the TechPresident.com blog. But Ben and I worked on SMS efforts for the 2004 election. We sent 40,000 messages out to SEIU labor members and MoveOn members… really the first time SMS was used in a wide-scale manner to help get out the vote on election day.

Tish: Do you have a new mobilization project planned?

Nathan: Its all about The Extraordinaries right now. We’ve got a big launch coming in June, and are working actively to add more causes that can benefit from volunteers and organizations that have volunteers but don’t know what to do with them.

Tish: I was just looking at your post on Peek too.

Nathan: Yeah… fortunately that is a completely  “for profit” gig.  But I like the company a lot, and think their spirit of providing access to email at a very low cost plays well with the non-profit world.

Tish: So it isn’t just iphone apps that are paying the bills?

Nathan: Nope. iPhone is just an aspect. Everyone is so obsessed with it and how to strike it rich quick, but in the greater scheme of things, there is a huge ecosystem of mobility out there for you to find a niche in, if you are looking.

Tish: Are you able to monetize your work on Android yet?

Nathan: here and there… releasing some for pay apps soon, also including “free” Android ports in some high-profile iPhone apps we hope to have out soon. Some successful iPhone app developers are looking for people to port their apps to Android, as well.

gReporter – opensource, geotagging, media capture report client

Tish: So what are your hopes for Android development in general and your gReporter app in particular?

Nathan: I think Android represents right now what Linux on desktops did in 99 or 00  Though as we all know, cycles of technology seem to speed up. There is huge interest in it at the academic level and there is also a genuine interest in its use by non-profit/development agencies working around the globe.

You have to jump through hoops to get an unlocked, open iPhone w/o contract. Android provides an alternative solution to this, that acts more like a true platform, and not just a consumer product.

Tish: At the moment the Android market place is only for free apps right?

Nathan: No, it now supports paid apps. I just bought one today for $2.99

Tish: What did you buy?

Nathan: An app that allows me to turn my G1 phone into a WiFi hotspot sharing my 3G connection to anyone who connects.

Tish: So what are the most important aspects of Android in your view?

Nathan: There are  two sites to help demonstrate what is really going on with Android that makes it significant

1) Open Intents – this is the ecosystem of developers, all creating services and apps that interoperate, share data, and generally build a very rich Microsoft style platform:
except all these are open-source and built by lots of small developers and not one big corporation.

2) Android on HTC – this is the home for all the efforts to port Android to pre-existing HTC/XDA mobile phone hardware. You can see the status of ports here: http://wiki.xda-developers.com/index.php?pagename=Android_devices  Imagine… taking an old Windows Mobile HTC phone, and then popping in an SD card that reformats it over to Android brand new phone!  For much of Asia, India and Africa, there is huge interest in this.

Tish: Nice! You mentioned earlier that you are thinking of doing SDK for the android sensor API’s?

Nathan: That would be part of the geo report app… expanding it to capture all sensing data and report that when you submit your text, photo or audio report.  Right now it just detects your lat and lon, but no reason it couldn’t also check your compass, altitude and whatever other data the device might offer.

Tish: So what will your geo report do now?

Nathan: It allows you to submit a text, photo or audio report, tagged with geo coordinates, timestamp, and basic user info (name, email, home location, etc) to whatever server it is configured to us. it is the latest release of code used for the TwitterVoteReport and InaugurationReport efforts.

There is also just a lot to learn or use from the code itself, which is available at: http://github.com/natdefreitas/georeport-android

Lots of little lessons learned packaged up into a functioning application

Tish: How many sensor APIs does android have?

Nathan: http://developer.android.com/reference/android/hardware/SensorManager.html

int SENSOR_ACCELEROMETER A constant describing an accelerometer.
int SENSOR_ALL A constant that includes all sensors
int SENSOR_DELAY_FASTEST get sensor data as fast as possible
int SENSOR_DELAY_GAME rate suitable for games
int SENSOR_DELAY_NORMAL rate (default) suitable for screen orientation changes
int SENSOR_DELAY_UI rate suitable for the user interface
int SENSOR_LIGHT A constant describing an ambient light sensor Only the first value is defined for this sensor and it contains the ambient light measure in lux.
int SENSOR_MAGNETIC_FIELD A constant describing a magnetic sensor See SensorListener for more details.
int SENSOR_MAX Largest sensor ID
int SENSOR_MIN Smallest sensor ID
int SENSOR_ORIENTATION A constant describing an orientation sensor.
int SENSOR_ORIENTATION_RAW A constant describing an orientation sensor.
int SENSOR_PROXIMITY A constant describing a proximity sensor Only the first value is defined for this sensor and it contains the distance between the sensor and the object in meters (m)
int SENSOR_STATUS_ACCURACY_HIGH This sensor is reporting data with maximum accuracy
int SENSOR_STATUS_ACCURACY_LOW This sensor is reporting data with low accuracy, calibration with the environment is needed
int SENSOR_STATUS_ACCURACY_MEDIUM This sensor is reporting data with an average level of accuracy, calibration with the environment may improve the readings
int SENSOR_STATUS_UNRELIABLE The values returned by this sensor cannot be trusted, calibration is needed or the environment doesn’t allow readings
int SENSOR_TEMPERATURE A constant describing a temperature sensor Only the first value is defined for this sensor and it contains the ambient temperature in degree centigrade.
int SENSOR_TRICORDER A constant describing a Tricorder When this sensor is available and enabled, the device can be used as a fully functional Tricorder.
float STANDARD_GRAVITY
with a few easter eggs as well
GRAVITY_DEATH_STAR_I
SENSOR_TRICORDER

Nathan: They are all in the API however, there isn’t hardware to support all of them yet… for instance TEMPERATURE is not yet supported
nor is LIGHT.

Tish: and errr what is gravity_deathstar

Nathan: It is a value representing the fictional gravity on the Death Star from Star Wars – geek humour

Tish: That makes me think of this great essay by Julian Bleeker, Design Fiction: A Short Essay on Design Science, Fact and Fiction:

“When you trace the knots that link science, fact and fiction you see the fascinating crosstalk between and amongst ideas and their materialization. In the tracing you see the simultaneous knowledge-making activities, speculating and pondering and realizing that things are made only by force of the imagination. In the midst of the tangle, one begins to see that fact and fiction are productively indistinguishable.“

Picture below is Nathan playing his dream ukulele – designed using the free, open-source Inkscape vector drawing tool (see his open-source Ukulele plans here)
See Nathan’s blog for the whole story of how the Flying V Rockin’ Ukulele Design he posted to Thingiverse a few weeks ago, after being inspired by Bre Pettis’ talk at ROFLThang materialized at the  NYC Resistor “amazing workshop laboratory in Brooklyn where they let anyone come over and hang out at, to learn how to make, build and fabricate pretty much anything. They also have a laser (aka “LAAAZZZOOOR”) which you can think of as an automagic thing cutter-outer!”

so this….

became this …

Nathan and David presented Coovents at NYTM – Mobile Meets Social. They had a large group of questioners surrounding them (see picture below).   I talked to David after the presentation.

David Oliver was a software architect, user experience designer and product manager in the areas of mobile/wireless and electronic payment at IBM for over a decade.  Most recently, he lead the effort to productize a mobile client for IBM’s Lotus Connections enterprise social networking suite.  As a software architect, David was often technical lead for IBM’s business partner relationships with mobile device manufacturers.  Prior to IBM, David was co-founder of the Internet’s first “micropayments” company, Clickshare.

Talking with David Oliver

Tish Shute: How are smart phones are causing us to rethink what networked online relationships are all about.

David Oliver: You know these [mobile] devices are .. there’s a long time we tried to pitch that we’re going to treat them like they’re PC’s, or they’re just like anything else. But they’re really not. It may be the same coding style but the way you think about using them is entirely different. And the way you think about your program. so if you use html, java and that kind of stuff, yes it’s same code type but the way you think about it is entirely different. And to me these little devices make what you said [relationships inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.] a lot more possible than a PC. because in a PC you almost have to sit in front of it and like it controls you. But the device is so little and there’s almost no user interface by comparison. You got to be very smart how you build something so that it’s almost invisible. And of course that’s the beauty of the iphone, Apple will tell you. The idea of ubiquitous computing. Ubiquitous what? Am I really computing? I don’t feel like I’m computing. I feel like I’m interacting or something.

I think twitter is very cool. The real way it’s cool is that there’s no required client. You can access Twitter any way you want.  You can imagine other ways to use it. Tweet Deck happens to be a nice for now. What I like about Twitter is, if you give it a tiny bit of thought, the Twitter network’s complete white noise, just like the internet itself. If you put a probe on the internet it’s all white noise, it’s all unordered packets. It makes no sense. So it’s cool that Twitter is at the level of little bitty conversations, but collectively all white noise. Totally meaningless white noise.  There’s some neat things going on, but I think we haven’t seen barely the first of what you can do with Twitter.

The way I see it is it’s like instant messaging where you don’t instant message to someone you instant message to the network and there are listeners. So normally in the old world of IM like AOL IM I would say Tish let’s talk and I kind of like grab you. Then it’s a narrow pipe you to me. You can add a few people in and make a little group, and that makes a bit of a closed network. But with twitter you just like talk into the air as if I were standing over there and you had a twitter client here, we could have the same interview. Because I would be watching you OH I see Tish’s question. I’d be over there talking  and you’d be picking me up over here. I’ts like you’re talking into white noise, like at this bar. You choose to hear me, this guy is not choosing to hear me right now.

Tish Shute: So what does Android bring to the party?

David Oliver: They have the notion that you have a telephone platform that’s open, and that everybody can use. And it’s got a variety of sensor data – not just location but also accelerometer and compass and more.  So in theory you can almost broadcast that data. It’s connected to a network. It’s easy, open API’s to get at that data. But the question is who are you going to broadcast it to or who are you sending it to. What are they going to do with it? How are you going to control it, and make sure people don’t misuse it? As you heard with the services tonight, there’s a central kind of service necessary to filter and rebroadcast that stuff back out to places that need it, or can use it, or you want to have use it. I think the mobile device is only one piece of this. Nat and I always talk about well we do mobile applications but a portion of it is on the server. And coordinating with the people or the group or the central resource that brings all this data together.

Tish Shute: There seems to be a lot of new location based services – platforms to aggregate location based data being developed (e.g. xtify and viaplace). What do you think about the direction this development is going in?

David Oliver: It’s not conventional wisdom but it’s one of these things where when a crowd of people does something, and that means people themselves are the service providers,  when they all get together the net effect is greater that the individual effect would be. Pooling together makes more sense than doing it individually. Its a little bit like an advanced version of you have to have a password for every single site and you manage your passwords. Location is the same way. If you had to give every single website that you enjoyed your location data or tell them how to get it, what a huge pain. So they’re offering a way to do that in a more general sense. There are humongous privacy issues though. Just like passwords. Would you really trust a place that held all your passwords centrally?

Even with the most basic level of calling. Now that you can call from anywhere. Largely people are getting into a mode where their mobile phone is them. It’s always with them. That’s how you reach me. Forget the home phone, the work phone it’s just a mobile phone. You have an address attached to you, an address I can reach you at that’s location independent. So there some beauty in that and it’s very freeing. It makes your location unimportant, you can call me anywhere. You can text me anywhere, message me anywhere. You can be anonymous. My son told me something recently. “I love going to New York City because I can just walk around and nobody knows me. I’m completely anonymous. That’s the coolest thing”, he says. At one level that is a good thing and a lot of good things can happen that way. But this new thing is sort of the flip side where everybody knows your location. And we haven’t figured out if that’s a good thing yet. But we’re in the throes of that whole changeover happening. And we’ll see. There’ll be some misuse. I’m not an advertising guy, so the fact that everything’s got to be ad supported makes it potentially very creepy and very dangerous. So we’ll see how that evolves.

Is there any model where you can go “Oh this is just like ‘S’”? I don’t see where that’s possible. It’s a new world. Where you’re exposed all the time, potentially. And how do you figure out either as an individual or a larger group, society or whatever, when that works and when that doesn’t. And you know there’s going to be some mis-steps probably. But the tangibility creates some of these interesting opportunities, there are just some amazing things that could happen, really, really good things. But we’re not going to get there in one step.

One of the things that was really a killer for privacy and a killer for in some ways the internet, was during the dot com bust. Prior to the bust, there were web sites that you’d given your name and email, and they said “we promise to preserve this privacy.” But as soon as those companies went bankrupt, their email list was gold. It was value. And a bankruptcy judge, in a court in Delaware, created a legal basis to sell that data. Those things that were formerly private were no longer private – “no no no that’s got value. I’m going to sell it so the shareholders get their money.” So all these web sites who had lists of user names that they promised were private, became public information. That was one of the biggest blows to privacy in the history of the internet. That’s going to happen again and again. Like if MeetMoi goes out of business the likelihood is all your shit’s going to get sold. I’m sorry it’s all going to be sold. It’s all a big joke. And that’s why central services are horrid, and I don’t like anything about a central service.

There are some pragmatic things about the way routing on networks actually works and the fact that the internet has gotten very centralized itself. The core ideas of the early internet which were essentially a survivable telecommunications network, remember it was the defense department that did the original internet? So the original idea of the original internet was survivability. The Russians could bomb the daylights out of the United States, territorial U.S. and we would still have a survivable network. That was the idea. And therefore all the nodes were dispersed and did not count on each other, and could reroute. Well now one company UUNET or whatever they are they own the whole thing. And you can look up all their locations on some internet database. 18 well placed bombs and the whole internet goes down. That’s what happens over time.

Well the whole cloud thing is also kind of a myth. It’s a very neat sounding term, and some aspects of it are different and new. Nate and I do a lot of cloud computing, it’s all on Amazon.

But we’ve always had that. That’s called time sharing. Strictly speaking it’s a thin contractual accompanied by a much much much easier application programming interface. That’s what cloud computing is. It’s a very skinny contract. Timeshare was a  huge contract. Literally it’s legal and a little bit of API ease. It’s just timesharing. But at Amazon and the other ones too, you’re not responsible for your node going down. If it goes down, they push it somewhere else automatically. Your disk goes down. You’re not responsible for backing up your disk, it’s already on 14 copies on 8 continents. They do that. So it’s a higher level of service. Nate and I have this thing called slice host. And we’ll probably build some services on it, and if they get popular, it’s like a vending machine. You just drop in a dime, they give you another slice. No contract at all. It is growth and learning about old ideas. Like this whole idea of software as a service. The company called ADP Automatic Data Processing, who basically in short do payroll for everybody. It’s software as a service. It’s been going on since 1952 or something. It’s more like a reconception using modern tools. It’s like virtual worlds are a different thing. That’s a whole different beast.

Wilfred Pinfold, Director, Extreme Scale Programs for Intel, and the Supercomputing Conference general chair, is working with some Intel colleagues to make a project called ScienceSim the centerpiece of a special workshop event at the SC09 conference (see Supercomputing Conference, an ACM and IEEE Computer society sponsored event).

Recently, I interviewed Wilf Pinfold (see interview below), Mic Bowman (also see my previous interview here), and John A. Hengeveld (see interview below). I wanted to find out what are the underlying goals of this SC conference program?  Why are members of the SC community being encouraged to participate with the ScienceSim environment? What projects are beginning to emerge? And, what are Intel’s goals in giving infrastructure support to further the conversation between high performance computing and collaborative virtual worlds?

The vision of creating new ways to collaborate and interact with big data does seem to be one of the more significant steps we can take at a time when we find many of our most complex systems roiling and threatening total collapse. As Tim O’Reilly has pointed out – from financial markets to the climate, the complex systems we depend on for our survival seem to be reaching their limits.

But,  how can we get from the place we are now – see this example of an n-body simulation in OpenSim, to the point where we can collaboratively steer from our visualizations big data simulations of climate change, financial markets, or the depths of the universe.  The picture opening this post is a:

Frame from a 3D simulation of gravitational waves produced by merging black holes, representing the largest astrophysical calculation ever performed on a NASA supercomputer. The honeycomb structures are the contours of the strong gravitational field near the black holes. Credit: C. Henze, NASA

Wilf Pinfold explained to me part of the reason to begin a dialogue on collaborative visualization at SC ’09 is that super computing communities (that tend to be highly skilled and visionary) have played key roles in internet development in the past. Wilf pointed out,  key browser technology  developed out of these communities in the early days of the internet – see this wikipedia entry that gives  a background on the role of NCSA (National Center for Supercomputer Applications).

The hope is, while there are many obstacles to overcome, the super computing community has both the skills and motivation to find solutions to creating collaborative environments capable of the kind of rapid data movement that scientific/big data visualization needs. Solving the problems of realtime collaborative interaction with big data will  have many ramifications for the way we understand virtual reality, the metaverse, virtual worlds (all these terms are becoming increasingly inadequate for cyberspace in the age of ubiquitous computing, an argument I will make in another post!).

There have already been a number of blogs on ScienceSim (see Virtual World News, New World Notes, Vint Falken, and Daneel Ariantho). There have also been Intel blogs – see this post by John A. Hengeveld (a senior business strategist working with Intel planners and researchers to accelerate the adoption of Immersive Connected Experiences). And Intel CTO Justin Rattner’s post announcing the project this November.

But to blow my own horn a little, I think i was the first to blog the encounter between OpenSim and Supercomputing (an encounter I to some degree provoked by making the introductions) see this post.  So I have been following the ScienceSim initiative with great interest.

Very shortly after N-Body astrophysicicsts Piet Hut and Jun Makino, creators of  – GRAPE (an acronym for “gravity pipeline” and an intended pun on the Apple line of computers) – a super computer that will become one of the fastest super computers in the world (again), met Genkii – a Tokyo based strategic company working with OpenSim, the first N-body simulation appeared in OpenSim.  And in a matter of weeks  this video went up on YouTube – the result of a collaboration between MICA and Genkii.  But the nirvana of being able to create visualizations using real time data from super computers that can be steered from a collaborative environment is still a ways off.

Super computing communities tend to be geographically very dispersed and researchers often find themselves far from simulation facilities so there is both the motivation and skills to pioneer new tools for collaborative visualization. I know that astrophysicists certainly see their value (Piet Hut has some profound ideas on this). Astrophysicist Piet Hut and others  (see here for more) have been pioneering the use of VWs for collaboration.  There are two Virtual World organizations, both founded by Piet Hut and collaborators, that are currently exploring the use of OpenSim for scientific visualizations.  One is specifically aimed at astrophysics, MICA, the Meta Institute for Computational Astrophysics, and the other is aimed broadly at interdisciplinary collaborations in and beyond science, Kira, a 12-year old organization focused on `science in context’.  As of last week, there are two weekly workshops sponsored jointly by Kira and MICA that explore the use of OpenSim, ScienceSim, and other virtual worlds.  One of them is “Stellar Dynamics in a Virtual Universe Workshop” and the other is “ReLaM: Relocatable Laboratories in the Metaverse.”

MICA was founded two years ago by Piet Hut within the virtual world of Qwaq Forums (see the paper “Virtual Laboratories and Virtual Worlds”). The Kira Institute is much older: it was founded in 1997.  Later this month, on February 24, Kira will celebrate its 12th anniversary with a presentation of talks, a panel discussion, and a series of workshops.  See the Kira Calendar for the main event, and the Kira Japan branch for a special mixed RL/SL event in Tokyo.  During both events, Junichi Ushiba will give a talk about his research in which he let paralyzed patients steer avatars using only brain waves.

Other early adopters of ScienceSim include Tom Murphy, who teaches computer science at a Contra Costa College. Prior to teaching, Tom spent 35+ years working for supercomputer manufacturers. Tom said:

it is very natural for me to find significantly new ways to visualize and interact with scientific mathematical models via ScienceSim and the OpenSim software behind it. ScienceSim also allows us to interact with each other and teach students in new ways.

Also Charlie Peck, chair of the SC09 Education Program, (his day job is teaching computer science at Earlham College in Richmond, IN), is working with Wilf Pinfold, Tom Murphy and others “to explore how 3D Internet/metaverse technology can be used to support science education and outreach.”

Cristina Videira Lopes, University of Irvine, is doing very interesting work  on road and pedestrian traffic simulations. Crista is also the creator of hypergrid in OpenSim,

People Meet People Meet Data: A Conversation With Mic Bowman

Screenshot of ScienceSim from Daneel Ariantho

Tish: How does this work on ScienceSim  fit into a wider dialogue on linked data? Where people meet people meet data, and where data meets data?

Mic: Yeah… that’s hard by the way.  Open integration of data (and more interestingly the functions on data) is very hard if it comes from multiple, independent sources.

That’s the people part. For example, if Crista can build a model of the UCI campus somebody else builds an accurate model of several cars and another expert provides the simulation that computes the pollution generated by those cars in that environment…its bringing people together to solve real problems, no matter how far apart physically.

Tish: You mention three different simulations here. Could you explain why it is difficult to integrate data from multiple sources?

Mic: integrating data from multiple sources has always been one of understanding & interpreting both the syntax & semantics of the data. Even relatively simple things like multiple date formats require explicit translation. More complex formats, like the many formats data is represented for urban planning, are barely computable independently let alone in conjunction with data from other sources (each with its own representation for data). Its often the expertise & the collaboration of bringing people (and their bag of tools) together that solves these problems.

Tish: and in this case the bag of tools is high performance modeling..?

Mic: high performance modeling, rich visualizations and data. Its the three that matter… data, function, and interface.

Tish: Some people have a very hard time wrapping their head aropund the fact that anything that seems related to Second Life can do this.  Can you explain more about the difference between SL and OpenSim?

Mic: OpenSim potentially improves data & function because it can be extended through region modules. Region modules hook directly into the simulator to provide additional functionality. For example, a region module could be implemented to drive the behavior of objects in a virtual world according based on a protein folding model.

We need to work on additional viewer capabilities to address the user interface limitations.

Tish: Yes Rob Smart’s (IBM) recent data integrations with OpenSim (see here) are impressive. Re viewers one of the biggest objections to virtual worlds is the mouse pushing and pc tied interface.

Mic: There are great opportunities for improving the interface

Tish: Yes I really like where the Andy Piper’s experiments with Haptic Interfaces for OpenSim lead, see Haptic Fantastic! And I think that we will have cyberspace ubiquitous in our environment, not just stuck on a pc screen, sooner than we think.

Mic: Mic’s opinion (not Intel): until we get souped up sunglasses with HD screens embedded (or writing directly into the eye) there will always be a role for the PC/Console/TV).  But, it isn’t about the device… its about the services projected through the device… sometimes you’ll want a very rich experience… sometimes you’ll want an experience NOW wherever you are.

Tish: I think people are only just realizing that VWs will be a now and wherever you are experience very soon.

Mic: That’s the critical observation the virtual world is not an application you run… its a “place”… and you interact with it where you are or maybe interact through it. Speaking for Intel… it is the spectrum of experiences that are critical to support.

Interview with Wilfred Pinfold

Picture from National Science Foundation – “Climate Computer Modeling Heats Up.”

Tish Shute: I know your day job for Intel is in High Performance computing. Could you explain to me a little bit more about what you are working on in this regard – a mini state of play for high performance computing from your perspective?

Wilfred Pinfold: My title is Director, Extreme Scale Programs. This program drives a research agenda that will put in place the technologies required to make an Exa (10^18) scale systems by 2015. The current generation of high performance computers are Peta (10^15) scale so this is a 1000x increase in performance and this increase will require significant improvements in power efficiency, reliability, scalability and new techniques for dealing with locality and parallelism.

Tish: The nirvana in terms of linking supercomputers to the collaborative spaces of immersive virtual worlds is to be able to create visualizations using real time data from super computers in collaborative VW environments, and ultimately for researchers to be able to collaborate and steer their simulations from their visualizations.  Where are we at now in terms of scientific data visualization in VWs? And what are the current obstacles to using realtime data from super computers?

Wilf: Being able to steer a simulation from a visualization requires both a visualization interface that allows interaction and a simulation that operates at a speed that is responsive in interactive timeframes. For example a weather model that predicts the path of a hurricane would need to operate at something close to 1000x real time. This would run through a day in ~1.5 minutes allowing an operator to run the simulation over several days multiple times with different parameters in a single sitting to understand the likelyhood of certain outcomes?

Tish: Do you see a networked online collaborative virtual world being capable of being a visualization interface that allows meaningful interaction with the hurricane scenario you describe in the near future (next 6 to 18 months)?

Wilf: I was using the hurricane example to explain the usage model not an imminent capability. Hurricane Simulation: Accurate hurricane simulations require multiscale models able to resolve the global forces working on the storm as well as the microforces that define precipitation. We can build useful weather models today that run faster than real time (anything slower is not useful for prediction) but we are a long way from the ideal.
Visualization: There are excellent visualizations of weather systems but I have not yet seen a virtual world that can track a simulation and allow the scientist or team of scientists to see what is going on at both the macro scale and zoom in to see precipitation conditions. Today’s supercomputers are much better at this than they were a few years ago but they are a long way from ideal.

Tish: Open Source Virtual World technologies are pretty diverse in their approaches, Croquet, Sun’s Wonderland and OpenSim are quite different and have different strengths and weaknesses. As you have become more familiar with OpenSim, what have you found about the technology that particularly lends itself to this project – ScienceSim (Mic mentioned Crista’s hypergrid code for example, modularity is another feature often cited).

Wilf: We have found OpenSim’s client server model is well suited to the visualization model and the ability to put the server next to the supercomputer producing the visualization data is critical. We are however very interested in other environments and encourage papers, demonstrations and research on any of these platforms at the conference.

Interview with John A. Hengeveld

Tish Shute: OpenSim’s dependence on Second Life based viewers is sometimes cited as a limitation, and sometimes as a strength. What are your views on this?  What would a strong open viewer project directed at science applications bring to the picture?

John Hengeveld: There may be more than one strong open viewer project required for opensim compatible experiences.  The strength of the Hippo viewer, for example, is availability and its weakness is the size of the client.  We would love a ubiquitous, client.. that runs on all platforms, but each hardware platform brings tradeoff and restrictions of its own.  Today, probably all of the folks innovating in the space can deal with the size of a very fat rich client ap.. they have big computers anyway.  But as we get into more 3D entertainment and augmented reality applications.. virtual mall, collaboration apps.. etc… there is a great deal of room to optimize for the specific experience.  Balancing visual experience with bandwidth and compute performance available .. tying into standard browsers, etc… people have done some of this work.. and I think all of it adds to the usefulness of these worlds.

Tish: Integrating highend game engines and OpenSim opens up new possibilities. But licensing issues have been an obstacle. Could a project like ScienceSim get a non-commercial license on a high end game engine?  What would that bring to the picture?

John: Anything is possible. Game engines can give a great deal of design power for high value experiences, but the programming of these experiences must be simplified.  Mainstream adoption in enterprise can’t be premised on the programming model of studio games… that’s a big step to get over I think.  There are very interesting possibilities when we take that step tho.  Simulation, training, agents of various types (I just finished watching “The Matrix” for like the billionth time… I think agents are cool…)

Tish: Where does Larabee fit into the picture of ScienceSim and next generation virtual worlds?

John: We are all very excited about the Larrabee architecture and its application to work loads like next generation virtual worlds, both in the client.. delivering immersive reality.. and someday potentially in a distributed architecture simulating and producing these worlds.  For Intel CVC is an all play.  Atom will be used in strong mobile clients.  Core will be used in Enterprise PCs, Laptops and Desktops   Xeon will be simulating these environments and handling the data communication, and Whatever we brand Larrabee… will be enabling compelling visual experiences. Oh.. and our software products (Havoc, tools and others) will be building blocks in knitting all this together.  Larrabee is a part, but there are a lot of other pieces in our vision…

Tish: If the kind of rapid data movement that scientific visualization needs is achieved in virtual worlds, this will be quite a game changer for business applications of VWs too. Also it will blurr the boundaries between what we call virtual worlds and mirror worlds. It seems to me this kind of rapid data movement is a vital step towards what Mic described to me as Intel’s vision of CVC: “Connected Visual Computing is the union of three application domains: mmog, metaverse, and paraverse (or augmented reality).” It almost seems to me that if you achieve your goals for ScienceSim you will change how we think about virtual worlds in general? What do you think?

John: I certainly hope so..  Part of our goal is to stimulate innovation in the technology and usage models that will enable broad mainstream adoption of CVC based applications (what we categorize as immersive connected experiences).   By tackling the scientific visualization problem, we hope to find the key technology barriers and encourage the ecosystem to solve them.

Tish: To me virtual worlds and augmented reality should be complimentary and connected experiences. How do you see this connection evolving?

John: We certainly see them as related.  In the long term, there are many common building blocks.. but they aren’t united per se.  Its about the user experience, and in some usages these two are almost identical…  in some.. they don’t look or feel at all alike… the viewer is distinct by a lot.  Our approach is to enable building blocks that people can quickly build out usages that are robust.

Tish: What is Intel’s vision for ubiquitous mobile computing and an internet of objects?  How can high performance computing be an enabler for this vision?

John: Mobile computing is a central part of our life, culture and community in economically enabled economies.  It feeds the data of our decisions, it connects us to entertainment, it is the access point to our soapboxes, pulpits, economy and families.  This creates a massive increase in data, a massive increase in interactions, transactions and visualizations.  While many HPC applications will be behind the scenes (finance, health, energy, visual analytics and others), HPC will emerge as a part of a scale solution to serving some of this increase… particularly that part where interactions and visualizations are complex or compelling.. or where scale enables the usage per se .. I talked about my love of agents earlier.. and some of that comes in here.  Compute working behind the scenes to help managed the data complexity, manage some of the base interactions between ourselves and technology.  The other thing we talk internally about the “Hannah Montana usage” where millions of people use their mobile devices to access and participate (using the sensors in the device) with an interactive live concert.  When Mylie hears the applause of a virtual interactive audience… and can scream back at them.. we’re there.  Access to ubiquitous compute will be mobile, and interactive experiences will be complex.. and HPC can help make that real.  Watch out for the mental trap that HPC is always high end super compute clusters tho… the “mainstream HPC”.. smaller clusters… high threads, etc… will play a key part in all of this as well.

Interesting that John ended on this point as this just came in from Wired.