The prophets of Augmented Reality, Bruce Sterling (best know for his science fiction works, his non-fiction includes “The Hacker Crackdown,” “Tomorrow Now,” and “Shaping Things”), and  Vernor Vinge (author of “Rainbows End” and “The Coming Technological Singularity”) are joining  Will Wright (Legendary game designer of SimCity, The Sims, and Spore), and Jaron Lanier (a computer scientist, composer, visual artist, legend of virtual reality research, and the author of You Are Not A Gadget: A Manifesto) to judge the Auggies at the Second Annual Augmented Reality Event, 2011, Santa Clara, Ca., May 17th, May 18th!  Pictures top row, Bruce Sterling, Vernor Vinge, Will Wright, 2nd row, Jaron Lanier, Blaise Agüera y Arcas, and Frank Cooper III.

Augmented Reality has been prototyped in science fiction writing and films for many years now.  But Augmented Reality Event, the first global event dedicated to the emerging industry of AR,  is your chance to be part of a momentous transformation of science fiction into science fact.   Bruce gave a seminal keynote last year at ARE2010, “At the 9am of the Augmented Reality Industry…” But we are edging closer to high noon this year – see John Markoff’s recent New York Times post, “Augmented Reality Comes Closer to Reality,” and Bruce Sterling’s opening keynote at ARE2011 will raise the bar for us all again.

Bruce Sterling  and Vernor Vinge will wrap up the show with fireside chat.  Jaron Lanier will be keynoting at are2011 for the first time and Blaise Agüera y Arcas (a leader in Microsoft’s mobile strategy), will return to top his Ted talk demonstrating innovations in Bing Maps and Augmented Reality.  Frank Cooper III, who serves as Senior Vice President and Chief Consumer Engagement Officer of PepsiCo, will give, an up to now, missing perspective of  the big brands’ on Augmented Reality (AR).

“The main stage at ARE 2011 will be blessed with science fiction visionaries, ground breaking scientists, mind blowing technologists, and legendary game designers and more!”

Come prepared to Augmented Reality Event.  Read as much Bruce Sterling and Vernor Vinge as you can, and follow @bruces blog on Wired, Beyond the Beyond.  With only a few weeks to go until Augmented Reality Event, myself and co-chairs Ori Inbar, and Chris Grayson, have been working hard on a preliminary schedule ( by the time you click on this link it should be updated again).  You can follow us on twitter for updates as they come in – @tishshute, @comogard, @chrisgrayson,  and please follow @arealityevent and join our ARE facebook posse.  And, of course, don’t forget to register soon while we are still offering discount codes.

Readers of this post can use the code TISH295 for $100 of the already sweet price if $395.

If you joined us for Augmented Reality Event last year, you will know that the”Auggies” – an award for the best AR demo presented live with commentary American idol style,  is a chance to join the best and brightest in AR as they write the next chapter for AR in the sharp and often amusing reparte between judges and contestants.  Submit your demo proposal here under the “Auggies” track.

The picture below is the Auggie Award for the best AR demo designed by Sigal Arad Inbar.  (See, Ivan Franco recounts the team’s ARE 2010 experience, and winning the event’s first-ever Auggie Award, and the video shot at the YDreams booth by Bruce Sterling. “The Hotness” – YDreams rocking it at ARE2010 from brucesflickr)

ARE2011 will include more than 90(!) speakers in 30 sessions organized into 3 tracks: business, technology & programming, and production & design.

It will feature special activities such as a Startup launch pad (submit your proposal here under track: “Startup Launch Pad”), and an ARt Gala with live performances.  There will be displays by top AR artists of AR art projects and live performance art by the world’s best AR artists featuring: Helen Papagiannis, Amir Baradaran, Sander Veenhof, Ina Centaur and more to be announced.

The exhibition area will be open throughout the 2 days of the event and will include the latest product demos by the leading AR companies, as well as a career fair to help grow the fledging industry.  A small number of booths is still available in the exhibition hall ($995 for 10′x10′).Grab them while they last!

A press conference will kick start the event, as a vehicle for AR companies to launch new products and services. Submit your proposal here and add: “Press Conference” in the title. The event wrap up will feature Marco Tempest with a live AR Magic show

I have been so busy working on are2011 (and another project yet to be named -  hint I am focused on social augmentation and the world as a platform for a geo-situated social interest graph), that I have had no time to blog since I was on The Potential for Augmented Reality in Education Panel at SXSW, 2011. But I have posted my slide deck here for my talk, “Enchanted Objects and People: Data Driven AR.”

We are entering a new era of collective engagement possibilities for augmented reality, consumer to consumer brokerages which could unleash the the visions of the collaborative consumption movement and tap into our collective energies in totally new ways.  As Seth Praebatsch, Scavngr, suggested in his SXSW keynote, The Game Layer on Top of the World, the power of communal game play may even help us address even the most intractable problems like failure in education and global warming!!

One of my favorite new AR ventures exploring the potential of new forms of social engagement through AR is Suwappu.  We are working on how to get them across the pond and a few time zones to are2011.

“Dentsu London are developing an original product called Suwappu. Suwappu are woodland creatures that swap pants, toys that come to life in augmented reality. BERG have been brought in as consultant inventors, and we’ve made this film. Have a look!”

Click here or on the image above to watch the video.

“Suwappu is a range of toys, animal characters that live in little digital worlds. The physical toys are canvasses upon which we can paint worlds, through a phone (or tablet) lens we can see into the narratives, games and media in which they live.”

Many new players in Augmented Reality will join the pioneering AR companies, Layar, Metaio, Occipital, Ogmento, Total Immersion, YDreams, and renowned innovators, Mark Billinghurst, Steven Feiner, Blair MacIntyre and the Georgia Tech team and many others will return to surprise and amaze us this year – see the still growing list of speakers here.   Qualcomm, who announced an SDK for vision based augmented reality, and $200,000 developer challenge last year, will also be back with several of their team at are2011!

Visual Search and OCR augmented reality apps have flourished in the last year and they will have a strong presence at are2011.  Long anticipated and oft fretted over face recognition apps are coming to a phone near you soon.  Viewdle, working locally on the phone, is taking a new approach to face recognition.  Google has so far said that they will not use or release the face recognition technology that they apparently already have.  But Shailesh Nalawadi, Google Goggles, will return to are2011 to discuss building out a computer vision search database and 3D Classifiers to solve AR challenges and show us what Google Goggles has been working on recently.  Aurasma, out next month, is perhaps the next big thing in this space, so we are all eager to try it out and see what they bring to ARE2011.

My interview with Anselm Hook, Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, is still a great reference point for important questions in this space.

Wordlens, wowed us all with OCR done locally on the phone, and they will be  showing off new features at ARE2011. Pulling out WordLens to translate at subway poster for the first time is a magic moment, and guaranteed to impress the person sitting next to you too!    Tactical Information Systems, who Fred Wilson said had the potential to be a Shazam for faces at Web 2.0 Expo, NYC, will show us what they have been up to since then. 

Data driven AR is still in the early stages, but it is data that will unlock augmented experiences of the world around us in many powerful ways.   Also, we are seeing exciting new efforts to put people at the center of their data.  Jeremie Miller who ushered in the Twitter era with his invention of XMPP has brought us a new real time protocol, TeleHash and the Locker Project, that will unleash the full potential of communication at the edge of the network.   And don’t miss Ben Cerveny and Bloom,  at ARE2011, with “pop-cultural instruments for data expression and exploration” and “where data visualization meets game design,”  I highly recommend Ben’s talk!

Also I am very excited that Mike Kuniavsky will be speaking at  ARE2011 for the first time.  He  pointed out in 2007 that Magic is a powerful core interaction metphor for Ubiquitous computing [and AR].  Offices. libraries, and the desktop provided us with common metaphors that unleashed the power of the PC through files, desktops and trash cans, browsing, etc.,  but metaphors of magic – “enchanted objects and people,” are keys to augmented reality experiences.   And, if Magic is a core interaction metaphor for AR, kinect hacks have shown us that gesture will be the soul of the AR experience.  Jaron Lanier’s keynote will be a must see and make sure you catch Chris Arkenberg’s presentation on the Future of Hands Free AR.

This post is just a glimpse at some of what will be going on at Augmented Reality Event.  I cannot do justice to everything in a single post, so I hope to see you there!  And, if by any chance you have a few moments to daydream between now and then, feel free to ponder your own script for a Green Lantern movie:  “In brightest day and darkest night, no evil will escape my sight.”  I will too.  As @brendanscully noted, “Green lantern’s power is Augmented Reality!”

The video above, The Imawik commercial, is a collaboration between In The Can Productions and Paige Saez for Makerlab

“The Imawik (ImageWiki) is a visual search tool for mobile devices. It allows for the ability to turn images into physical hyperlinks, conflating visual culture with a community-editable universal namespace for images.”

Paige Saez is an artist, designer and researcher.  In 2007 she founded Makerlab with Anselm Hook, an arts and technology incubator focused on civic and environmental projects.

Paige and Anselm (see my interview with Anselm Hook here, Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook) have been asking a very important question:

“Who Will Own Our Augmented Future?”

But most importantly, they have been actually developing applications (again see my interview with Anselm for more background on this), to allow people to play with, hack and explore and create with the physical world platform, and to imagine new possibilities for physical hyperlinking and augmented realities.  This is pretty important stuff, and kudos to Paige and Anselm for beginning this work before the big players – Google Goggles, Point and Find, and SnapTell came hurtling into the field of visual search and physical hyperlinking  – see this demonstration of translation and optical character recognition in Google Goggle’s.  Also check out Jamey Graham’s (Ricoh Research) Ignite presentation at Tools of Change, 2010 – Visual Search: Connecting Newspapers, Magazines and Books to Digital Information without Barcodes, for more see ricohinnovations.com/betalabs/visualsearch.

We are only just beginning to get a glimpse of how contested the social commons of the physical world platform is going to be – see the Yelp controversy.

As Paige points out:

“The lens that you are actually looking through was as important as what you were looking at. And democratizing that lens became the most important thing that we could possibly do.”

I am in total agreement.  One reason I have so much enthusiasm for ARWave (note: if you are interested in following the developer conversations there are several public Waves) is I see this open framework playing an important role in the democratization of our augmented views, by creating an open, distributed, and universally accessible platform for augmented reality that will allow the creation of augmented reality content and games to be as simple as making an html page, or contributing to a wiki.

Federation, real time collaboration, linked data – ARBlips that contain metadata that is usable for semantic searches, and modified wave servers that can listen to and respond to  SPARQL HTTP requests properly (see Jason Kolb’s many interesting posts on XMPP and Wave).  These are just some of the reasons why ARWave could revolutionize augmented reality searches and more! (see my presentation at MoMo13 – video here)

For more on real time social augmented experiences see our panel, The Next Wave of AR: Exploring Social Augmented Experiences at Where2.0 2010, and don’t miss the Where2.0 conference which has been the crucible for the emergence of location technologies.

Augmented realities, proximity- based social networks, mapping & location aware technologies, sensors everywhere, linked data, and human psychology are on a collision course in what Jesse Schell calls the “Gamepocalypse”   See Jesse Schell’s Dice 2010 talk here, and check out his Gamepocalypse Now blog.  As Bruce Sterling’s notes in his post here:

*Another precious half hour out of your life.  However: if you’re into interaction design, ubiquity, social networking, and trendspotting, in the gaming biz or out of it, you’re gonna wanna do yourself a favor and listen to this.

And don’t forget to register now for Augmented Reality Event (ARE2010 in 2-3 June, 2010 – Santa Clara, CA).

Bruce Sterling, Will Wright, and Jesse Schell will be keynoting, and there is a totally awesome line up of AR innovators and industry leaders, including Paige and Anselm!

And:

You are in luck!

Here is a discount code for the first 100 folks to register to the event (before the end of March). Go to the registration page, type in code AR245 and “you’ll be asked to pay only $245 for 2 full days of AR goodness.”

“Watching AR prophet Bruce Sterling, and gaming legend Will Wright, visionary game designer Jesse Schell deliver keynotes for this price – is a magnificent steal.  And on top, participating in more than 30 talks by AR industry leaders will turn these $254 into your best investment of the year,” as Ori  put is so well on Games Alfresco!

If you want a preview of just how exciting it is to be involved in augmented reality right now check out Ori Inbar’s great round up on our latest monthly Augmented Reality Meetup NY (or as, Ori notes, we fondly like to call it ARNY.)  There is lots of video up now (much thanks to Chris Grayson, who  live streamed it).  Augmented Reality Magician, Marco Tempest, is an absolutely must see.  (developers note this is an awesome use of open Frameworks and OpenCV).   The video of the show includes a rare explanation of how it all works  – see here.

Talking with Paige Saez – “Software is candy now!”

Tish Shute: What interests me about ImageWiki is that you have thought about physical hyperlinking beyond the obvious of where to get your next good hamburger and beer, right?

Paige Saez: Right. It was interesting for me in just thinking about the two things. How do you design a tool to work in a way that people are getting value from it? And also, how do you make it work in a way where people can explore and hack it? I think the most interesting technologies, and this is probably something somebody else said sometime, are the ones that disappear, that we don’t see, instead we see through. They become just the intermediaries.  They don’t interfere with what we are trying to do.

It’s a struggle whenever you are developing a new way for people to get information or make something happen, because you are playing with magic a little bit. And you have to make it vanish the way a good magic trick makes an experience a magical one. But at the same time you also need to reveal just enough that you let people in and they can see how to change it and make it their own. That is the interesting tension for this space right now, the idea of augmented reality begins to lead the idea of a social commons for physical things. The Imagewiki project was a locus of just this tension. Tish you and I have previously discussed how difficult it was to even get people to understand the two concepts independently.

Tish Shute: Right, until recently most people hadn’t even heard the term augmented reality and I am not sure that a particularly high percentage of people would recognize it now despite the recent interest in smart phone apps.

Paige Saez: It’s very difficult to get people to understand the two concepts, and now you are adding in the third level of participation as well. So I don’t think it is impossible, but I do think it requires narrative. It is interesting that you were talking about the stories you heard this morning from the creatives at the event [Tish mentioned David Curcurito, Creative Director, Esquire gave an excellent presentation at Sobel Media event NYC] because it’s narrative and the attention to telling a story that help you walk through all of the ways you can understand how completely expansive this area is right now.

So I think we have to play with it, play with the space and the tools. I think we need to have an idea of what we want people to use the tool for, and we need to not only introduce them to the tool and the technology, but also introduce them to the concepts as well. So I see it as a three part process.

I’m really excited to be there with people, helping them do that. I think we need to do this face to face. I don’t think this can be only through a social network. The ImageWiki website is like one quarter of the entire picture, you know? The website is the resource center and the place where you can see people adding images, but what value is it to you to see an added image? It is more valuable for you to be interacting with the image or interacting with the object in the real world.

Designing for the experience of using the ImageWiki got very complicated very fast. I was trying to figure out the main thrust of the design for the UI for the ImageWiki and at a certain point I had to take a step back and say “Okay, this has to be good enough for now because we can lay it out and prototype as long as we want on the Web or mobile UI. What we need to be doing is going outside and actually aggregating and putting images into the database in order to see what exactly happens when we are adding.”  It’s not just like you are taking a picture of something and adding it to Flickr. Using the tool is very context specific and the information is context specific, and you can’t necessarily make that all happen at the exact same time. I think these are really fascinating spaces to be struggling in and I’m so glad to be working in this space.

Images by Chris Blow of unthinkingly.com

Tish Shute: Could you explain why we need ImageWiki? I mean I think I have ideas on this, but perhaps you can explain to me from you point of view why we need an ImageWiki, as opposed, to say, extending the image space of Wikimedia or something added on to Flickr.  I mean maybe something leveraging the geotagged photos sets and APIs we already have?

Paige Saez: Yes, definitely. It’s a really good question, I mean it really is. Like, do you need an entirely new place to be holding images outside of the places that we are already holding images? That’s a huge question; enormous. Especially when you take a look at the problems around that. Its’ exhausting for an end user. Who the heck wants to go and reload everything into yet another place, right?

Tish Shute: Right.

Paige Saez: Moreover, who is going to really bother? Another problem would be what happens to the existing datasets that people have already committed to? And then of course there is the problem of authority and explanations why….Gaining interest and authority in a space when nobody even understands why that space should exist in the first place. And those are just three, you know, off the top of my head problems with that idea.

And yet at the same time, I don’t actually know how else to go about thinking about the ImageWiki unless I think about it as it’s own thing. Then you start thinking about models of large independant image databases that exist already, examples of this from a product standpoint- references to consider. The Getty Foundation comes to mind. There are many other historical centers that have huge resources and images that are licensed out and used. So here we have a working example of people already doing this. But succesfully? I don’t know. We do have a ton of intellectual property rights and copyright issues and ownership and use issues with images currently. As a working artist these issues for me were a major red flag to consider. Working on the social commons for augmented reality starts paralleling issues found in digital rights management and intellectual property.

Tish Shute: But one good thing about Wikimedia, why I focused on Wikimedia, is Flickr and Wikimedia already use a creative commons licensing, right?

Paige Saez: Creative commons, you know they have their own resource center, too. But you know they haven’t been successful as great databases for images so far.

Tish Shute: What would you like to see that they don’t have? Like say maybe start with Wikimedia, right?

Paige Saez: There’s just still a lot of issues with how to encourage people to want to contribute. It’s hard to show the value to someone who doesn’t already understand the value for some reason. At least for me personally this is something I have run into frequently. I don’t know if it is necessarily what Wikimedia doesn’t have, I think it is a lack of understanding of what creative commons really means. And there is still a very strong sense of ownership and concern about creative property rights. Being paid to be creative is a tremendously difficult thing to do. People fear losing their livelihoods. They think this is possible. Is it? I dunno.

For example : Look at me, I take a photograph of something, I can sell that. And there’s a question about whether or not, as an artist, I want to have my photographs in a pool of images that is open and accessible when I could be making money on it instead. Now that is just an example. Me personally, I can see the value. But that is a common concern. The gist of the question being, ‘what value does it bring to give something away versus holding on to it?’ A hugely popular discussion right now.

This is the same crux of the problem we are dealing with when we talk about thinking about images in the social commons for the real world. It’s a conversation about ownership. It’s about, who does this belong to really? If I take a photograph of a Levi’s billboard, does that photograph belong to me or does it belong to Levi’s? We know the boundaries of that. But when the image becomes a living image, an image capable of transmutation; an image that provokes an action or hyperlinks to a product, experience, information….where are the boundaries in that?

Tish Shute: But how is ImageWiki handling that differently from Wikimedia, I suppose is my question.

Paige Saez: We haven’t solved the problem.

Tish Shute: Yes, I suppose it is not like we have fully solve the problem of a creative commons for images on the internet let alone the issues of a social commons for the real world! So neither one has solved the problem, right?

Paige Saez: Exactly. To be honest, it made my head spin. I realized we were building a web application and a mobile tool doing augmented reality, real time feedback on the world and suddenly we weren’t. Suddenly we were dealing with DNS and talking about physical hyperlinks and ownership and property. And basically at that point you just have to sit and really start looking at catching up on IP issues and figuring out how to deal with that space in a much more wholistic way. It became so important that we had to take a step back and go

“Oh my god I think we have really uncovered a real problem here.”

At the point when we were building out the tools we realized something was really going on with our project. Here we were thinking that this was just a beautiful experience of learning about the world around us. We really…Anselm and I both just really wanted this tool to exist. It was something that we both just really wanted to happen in the world, something that we felt really just thrilled to make. And we looked at and used it and realized that instead of it just being a beautiful experience, it was a fundamental shift in how we understood everything. That it impacted our world in the same way the Internet impacted our world. It was a fundamental shift in understanding. A sea-change.

So I put down the prototype and went back to researching, read a ton of books on IP and went and presented to friends, family, schoolmates and co-workers trying to explain the project and then the larger conceptual framework that had emerged from the project. I began using the metaphor of thinking about Magritte’s “Ceci n’est pas une pipe.” Thinking about a pipe that isn’t actually a pipe.

Tish Shute: Oh, yes!

Paige Saez: ..to try to help explain to people that the image that you see is actually not, you know, it’s not an image of a thing. It’s an image. And that image has a tone and that image has a voice, and that image was chosen. And there were decisions that were made through the interface of the camera, specific decisions that defined the view of what you were looking at. And that that wasn’t being acknowledged and that that was a fundamental part of what the ImageWiki was aiming to do. The lens that you are actually looking through was as important as what you were looking at. And democratizing that lens became the most important thing that we could possibly do.

Tish Shute: So the emphasis for you on ImageWiki was in fact the lens, even though you found obstacles to creating the interface, right?

Paige Saez: Yes. Definitely. That’s what I fell in love with first. I really wanted to be able to use my phone to learn about what kind of tree this was or to buy tickets for the band on the poster I just saw, or see a hidden secret. For me it was very much a story, a narrative experience that I just thought was magical. And that is how I fell in love with it, which is not where I ended up. Where I ended up was realizing it was a fundamental shift in not only my own understanding of how to use the world around me, but in our understanding of looking at the world.

Tish Shute: It would be pretty scary if an image DNS was basically in the hands of either one or very few people, right? I mean even ImageWiki would be stuck with this problem, that if you set up a bunch of servers, you are going to be holding a very, very large image database. I mean, whatever your motivation, right? I think at the minute that is why I am very into seeing everything through the lens of federation, I see that unless we have federation, these giant central, databases are inevitable aren’t they?

Paige Saez: Essentially, yes. I mean I wasn’t able to walk through it as quickly as that. It kind of just overwhelmed me. Looking back on it, it seems perfectly obvious. I was just like “Oh my god, what have we done? Like what is going on?” Particularly for me because so much of my life has been spent in art, it was really easy to immediately understand the connection between the view, the viewer, and what’s being viewed as all just different layers of ownership and understanding that it is a gaze. Right? We know that we are never able to look at something without passing judgment on it, but to see that become a part of the interface in a real-time fashion just blew my mind.

Tish Shute: Yes.

Paige Saez: I think you are right. Getty Images, Flickr images, no matter what you are always holding on to something and you have to be responsible for it. Right? So how do you deal with the responsibility but don’t take on too much ownership? Where is the boundary with that?

Tish Shute: And for me, the simple answer to that is loosely connected small parts, distributed systems and federation. Because there is only one way to be able to utilize these things is to have them distributed so that no one holds all the cards. Right?

Paige Saez: Definitely and I personally agree with you wholeheartedly. However, the idea of distributed power is a concept that most people just don’t know how to deal with.

Tish Shute: And it’s easier said than done because actually the root problems that you are talking about aren’t got rid through federation, because if someone really holds the, sort of, all the good image databases just because they have the potential to be federated, they may not choose to open them up on many levels.

Paige Saez: And even then you have to think about, sort of, like the next level of it, which is we want it to be all open and accessible, but everything is owned by somebody. Like, what really is public anymore, in general?

Tish Shute: And what is interesting though, regardless of what we speculate conceptually on this, we already set off down the road. I mean we have already several large…they are all in beta I suppose, Google Goggles, Point and Find, right? But we have applications that are beginning to implement this. They are beginning to implement search on it, and it is geo-located even if it’s not in an augmented view, right? So it is proximity based.

Paige Saez: Right, right. I mean maybe the solution is that if we follow that line of thinking then Flickr will be partnering with Google Goggles. And then my images would stay under my ownership through the authority of Flickr. And I would use Flickr as my place to add images and they would just be responsive via my devices via AR.

Tish Shute: That’s very interesting.

Paige Saez: Definitely I think so. It is also the shortest distance between things.

Tish Shute: Yes, and as Anselm kept pointing out, basically it is going to happen in the simplest way possible, really, regardless of the implications of that. But OK, getting back to ImageWiki. As you say neither Wikimedia nor Flickr were really designed to take this role, right?

Paige Saez: Right.

Tish Shute: With ImageWiki, you’ve had these ideas and a concern with the social implications of physical hyperlinking in your mind since it’s inception. Are there any design ideas you’ve come up with that you know, as opposed to sort of, as you say, connecting Flickr to Point and Find, or who knows, Google Goggles. How is ImageWiki going to be different, do you think? Is that a hard question at this point?

Paige Saez: It is, and it’s a great question, and it’s a question I really love to think about. I think we have to introduce the politics with the tools. It has to be acknowledged that it’s not just a place to hold information, that’s what I feel in my heart.

At the same time, is that too much for people to really grasp at one time? In my experience it really has been, so the design of the experience needs to allow for an understanding of the power of the tool and the level of authority that the tool offers, while not getting in the way of it; just using it. Because ultimately, at the end of the day, nobody will use anything if it isn’t valuable to them. And so I could talk for miles and miles and miles about how important it is that corporations don’t own all of the rights to all of the visual things in my life, right? For the rest of my life I could talk about that. The idea that advertising is dominating all of our views of anything in the world around us is horrifying. It doesn’t matter unless I can show somebody why it matters to them or how it affects them. It’s just that that is a tremendously difficult thing to explain through a user interface.

And I actually think that it’s great that tools like Google Goggles and Nokia Point and Find are here to do a lot of the hard work of showing people how it works. Recently somebody explained to me their experience of using Google Goggles. They went through this process of saying how the Google Goggles took a picture and then did this really complicated visual scanning thing over the image and it took a full minute.

And I said, “Well of course they did it that way.” And they said, “Well what do you mean?” I said, “Well, what they are really doing there when they are doing all these fancy graphics, is they are showing you how it works.” And even if it isn’t actually related at all to how it functionally works, algorithmically, that’s not the point. The point is that this gesture of the time taken to make it look like it’s scanning an image and going back and forth with pretty colors is giving people the time to process that as an experience. That’s a metaphor for what’s really happening. And these kinds of metaphors are crucial with user experience design. We have lots and lots of examples of them and how they work, and many of them aren’t necessary. Like you know, for example, the bar that shows you the time it’s taking for something to process.There is no relationship between that and reality. But it is really important.

Tish Shute: Yes those bars often have no relationship between the actual time..

Paige Saez: And that’s the thing. Like the idea of time versus our perceived understanding of time. Right? The length of time it takes for your Firefox browser to open and load your last 30 tabs, versus the reality of what’s actually happening. When you are doing that sort of research you are actually accessing millions and millions of places and points of interest all over the world, so we need more of that. We need more of the process shown. Anselm and I worked with a film maker named Karl Lind from In the Can Productions here in Portland to try and make a video about the ImageWiki. We made this little video and I can try to show it to you or send it to you if you want.

Tish Shute: One of the issues with this kind of visual search is that it is inherently dependent on large databases, regardless of where they are federated, are going to be very large. Right? I mean someone is going to have something big, and aggregated there. I suppose someone will figure out the challenges of federated search eventually but that is quite a big challenge!

So I suppose I am still trying to understand what ImageWiki can offer that we can’t get with any other existing service? How will their be a social commons and even a social contract for the world as a platform for computing and physical hyperlinks?

Eben Moglen brought up something when I talked to him about virtual worlds, he said we need code angels to let us know what was going on in the virtual space – who was gathering data and how, for example.

Paige Saez: Tell me more about that, I want to hear more about that.

Tish Shute: Eben suggested this metaphor for when I was asking him about privacy in virtual worlds. The fact that people just didn’t know that when they were pushing avatars around virtual worlds what metrics were being gathered on their behavior. And he basically said that what we need is code angels when we enter these spaces because having the rules of the game buried in a TOC was ridiculous.

Paige Saez: That is a really interesting idea.

Tish Shute: Maybe ImageWiki needs to be our code angel to navigate the augmented world. I mean that’s what I want to see it as. And when I hear you talk, what I hear is you talking in broad categories about what a code angel might be in the space of images and image links to the physical world. I mean that is what I hear from you.

Paige Saez: Yeah. No, I definitely agree with that. It is interesting. In that sense, it is kind of a protection layer. Is that what you are thinking?

Tish Shute: Yes, I suppose because we can’t be navigating a lot of complicated opt-ins and opt-outs just to get around our neighborhood safely (in terms of privacy (also see Eben Moglen’s definition of privacy here…) We will need a code angel that is sort of keeping up with you in real time!

Paige Saez: Right, right. I wonder how that would work in regards to images, though. That is a really interesting thing to try and put on an image. I guess why I am having such a hard time being specific about it, is I am just trying to work it in my head, thinking of a specific use case, like what would be an example of that?

Tish Shute: Well I suppose the example, and this is a crude one, is when you point your Google Goggles to the book jacket, the code angel, this is very crude, would say “You are right now drawing images from the Amazon database – they are collecting data such and such data from your search.

And then of course the ability to have crowd sourced tagging and corrections..

There was a wonderful book that came out last year on how we can have commercial intelligence -Dan Goleman’s new book: “Ecological Intelligence: How Knowing the Hidden Impacts of What We Buy Can Change Everything”…

how corporations various different stakeholders, including their customers will drive corporations to do the morally right thing because they will lose the commercial support of customers who won’t support them unless they are more green, fairer, do the things we would like them to do whatever that happens to be – physical hyperlinking and tagging I guess would be a big part of this.

Paige Saez: Sort of a transparency issue. And that almost becomes a page rank algorithm in and of itself. I mean now we are really talking about search more than anything, and what tool becomes the dominant search tool. Anselm and I talked a lot about one platform… I mean eventually we will have a unified platform. It will…No matter what, for the Internet and for physical objects and visual objects in the real world. It will just be a matter of, literally, who can find the best and most valuable, most relevant information on a thing. Currently we just have it very proprietary.

Tish Shute: Yes.

Paige Saez: That definitely won’t last. It just can’t, because of the exact problem that you are raising. And we already know too much about resources and information as they pertain to products for us to ever go back to a time where we are not considering other ways of getting information about it anyway. Right?

Like I have the same concerns nowadays when I look at fruit. I look at a piece of fruit in the store. I would never just assume that the person who put the sticker on that fruit, anymore, is the ultimate authority necessarily. I would always assume at this point I could go online and go find out more information about a company. Issues about like eco-footprint or how much toxicity, or pesticides or whatnot are now totally accessible already.

So I am thinking when you look at that piece of fruit and that sticker for Google, say what you are describing, do we just go immediately to the company’s website, or is it even more specific? Do we know that the sticker on that piece of fruit is going to tell us specific information about that? Or are we just getting back the nutritional resources, or are we getting a listing of all of the different options out of a page rank algorithm that shows us, “Well this is the website for the fruit. Here is the nutritional information. Here are the last 15 comments on it.” It’s basically just a basic search.

Have you heard of Good Search?

Tish Shute: you mean http://en.wikipedia.org/wiki/GoodSearch

Paige Saez: Right.

Tish Shute: A code angel interface would have to give you options, wouldn’t it on possible views available?

Paige Saez: Yes. You are then talking about filtering your view. Then it really gets really interesting, of course. I don’t even know if we have a choice in that. I think we are really kind of hitting a wall with who owns the space and the platform. Is it just a basic search because we are already familiar with search? If you had an option to choose, say, “I want to look at this apple sticker and I only want to get…programmatically only looking at my friend’s opinions of this company.”

Or I have a safety valve on it that only shows me certain information based on what the code angel knows about me, my preferences, my age, things like that. Then that gets really, really interesting, because we are trying to do all that work right now just with social media and the Internet. We are already overwhelmed with too much information. It is already past the point of comprehension. So to think that we would actually drill down even more specifics is very interesting.

Tish Shute: That was a point Anselm made about the fact that once you are into this mobile, just in time, one view kind of situation, it is quite different than the Internet where you can bring up all these different screens and go to another website.

Paige Saez: Well yes, mobile is a different level of engagement. Very contextual. Much less information. Much more about timeliness. I don’t want to look an apple and get back a Google search. Oh my God no. That’s the last thing I want. I would love to be able to look at an apple and my phone already knows exactly what I want, information-wise, to get back from that apple. But I don’t know. It’s all contextual and personal. So I think the code angle concept you are talking about is really interesting because you still need to think about who is the person that is adding or creating those level filters- is it you, a filtered friend network, an algorithm? How much work is too much work? Where do we draw the line? How much of this are we willing to let the machine do for us?

Tish Shute: Right.

Paige Saez: And then of course once you have those filters in place, you need control over them. You will need to dial them up and dial them down, be able to choose and add new ones, so on and so forth. It becomes very modal at that point. For example, I want to change my view: To walk into a grocery store and instead of finding out information, I’d want to see where the hidden Easter egg puzzles were that my friends left last week because we’re playing a game.

I’m still really attracted to the creative opportunities with the ImageWiki. I’m really attracted to changing this experience from being a one-to-one relationship (from Corporation to Consumer) to an open-ended relationship (From Person to Person). If I look at a book jacket, sure I can find out where to buy the book, but that’s boring. Who cares? I’d like to find out a link to a story or an adventure or a movie or something unthought-of before.

How do we build that in? How do we encourage serendipity? Mystery? I think the ImageWiki is the space for building that in, actually. Not how, that would be the one place, right? That’s my really big fear is that this relationship just stays one-to-one. Click an image of consumable object, get back objects retail value. How completely dull. We have to do better than this.

Additionally, what if I want to take a photograph of a book, an apple, or something and I don’t want to pull back data. Instead, I want to pull back music, or I want to pull back a video, or I want to pull back a song, or lyrics, or a story, or another image. It’s just a hyperlink at the end of the day, you know? That’s all we’re really doing. Hyperlinks can pull back so many different things.

Tish Shute: And that’s one of the reasons I’m into mobile social interaction utility building, because without that, if we don’t have that way to do that in mobile technology…that’s very available on the Internet, as we’ve seen, with Twitter. These applications are very easy to do on the Internet. They’re not easy to do natively in a mobile application..

hey, I’m just promoting AR Wave again. I should shut up.

Paige Saez: Oh, no. I think it’s a fascinating concept, I really do. I totally agree. As we’ve talked about it before, it’s amazing that marketing and advertising are helping push forward AR, and it’s great. It’s fantastic.

But it’s also the worst possible thing that could ever happen because it is such a singular way of looking at an overall ubiquitous computing experience. There are other ways.

The best experience I ever had was trying to explain to people about physical hyperlinks. I had to walk them through it. Good interactive isn’t something you present or show, it’s something you do. Nothing beats just walking around and showing people with a device or a tool or something else.

I mean, God forbid it always stays in our computers and our phones. I really hope we don’t have to be stuck living our entire lives with these horrible interfaces. But for the time being, we will. Having an AR app show you a puzzle, or a mystery, or a game, or an adventure is a magnificent experience, totally overwhelming, and people get it right away. There’s no question; they totally understand.

Tish Shute: Yes, I agree.

Paige Saez: You walk them through the experience with a physical hyperlink and then you say, “Here, I could use this device and I could show you where to buy this thing, or I could use this device and we could start playing a game.” Then everybody gets it.

Tish Shute: So then I have a question, because one of the things Anselm said to me when he wanted me to refer back to you is that he feels that the direction for ImageWiki should be perhaps to focus less on the technology and more on just the actual, I suppose, gathering of the images, how they’re going to be annotated, the metadata, right? But my question to him was, the problem if you do that, without the platform, there’s no experience or motivation for people to do that. Right? Is there?

Paige Saez: Yeah, I agree with you on that one. I’m curious what his…I think the reason why he wants to do that is he wants to be able to show people examples via the resources. Like to be able to show someone a library, essentially, which I think makes sense with some people. I definitely think that some audiences would really relate to that. For me, it doesn’t make sense because I’m just very experiential. I need to do it and I need to show other people how to do it and I need to grow that way. I think that at the end of the day, those are great ways to go about doing it. It’s just it’s a huge thing to do in either direction.

What Anselm’s really thinking on, I believe, is more about exemplifying how we read and understand images culturally. Then you’re really getting into Visual Studies and Critical Theory which is what I did for my Masters at PNCA. I worked on the ImageWiki while I was in grad school, it was something I was doing for fun. Independently of my studies, the project lead to issues on democracy and objects and property and I ended up right smack in the middle of what I was studying; the nature and cultural analysis of images Questions like, ‘what exactly do we get out of images?’ and how all these different things are happening in an image, and people get tons of totally different things out of an image depending on many factors.

The questions I began to ask myself got very philosophical. Questions like “Is this apple red? Is this apple red-orange? Is this a small apple? What’s my understanding of small versus your understanding of small?”

Because you supposed that you needed a text backup to the search, how would I be able to search for an apple? Because what if my understanding of apple is red and your understanding of apple is green. And so if I’m looking for a green apple, am I looking for the same green apple as you? It’s all semantics, sure. But at the same time, it gets bigger and bigger, and it’s fascinating.

Tish Shute: Google Goggles seem to work best on book jackets, basically.

Paige Saez: But book jackets are actually perfect for this. Book jackets are perfect for this problem, because book jackets are specifically designed art. So at the end of the day, we are still talking about creative works, artistic works, that have been designed as a communication tool. But that is not something that people can own. Creative works that are designed are a communication tool, with varying levels of skill to be sure, but still something anybody can do. What we need to do is we need to be using that language. We don’t need to be trying to reach as far as facial recognition. We need to develop our own logos, our own brand, our own…I mean not brand. Brand is a bad way of saying it. Another way of saying it would be like, just use it. Develop a visual language that we can use that is as effective and as well utilized as book jackets or the movie posters or something.

Tish Shute: What are some of the use cases for ImageWiki you would like to develop first?

Paige Saez: My dream…I have like four or five use cases that I want to see happen. One of them is I walk down the street and there is a new poster for my favorite band. And I can just go up to the poster and I use my device, whatever it looks like, and I download the latest album. It’s transactional. I am able to just plug in my headset and walk down the street and the transaction is done. I saw something I wanted. It was beautiful. I was able to get it and I was able to move on in my life. And that is totally possible.

Another one would be I walk down the street and there is a piece of graffiti. And I am able to use my device to find out who the artist was that made it and to give them props, and to point my other friends to the fact that the piece is there and it will most likely be there only for a short period of time- information retrieval and socialization.

Or, use my device to find an Easter egg, to find a narrative puzzle that ends up going on for weeks, and everybody is involved, and we are all playing this game together. Adventure-based, non-linear experiences. I want playfulness, not just purchases.

Tish Shute: Did you think of piggybacking on the Flickr API for geo-tagged photos as a way to work with those databases or not?

Paige Saez: Yeah, we definitely thought about that.

Tish Shute: And why did you decide not to, for any reason or…?

Paige Saez: Ultimately, we just…we were such a small group, we just had to tackle certain things at a certain time.

Tish Shute: Right. And you were so prescient, you were working slightly before we had the mediating devices, weren’t you? You were just before the mobile devices really got adequate for this.

Paige Saez: Yeah. We started on it…I believe it was January…No. December 2007. Basically, the iPhone had just launched like maybe six months prior or something like that.

Tish Shute: But not 3G and not 3GS, right?

Paige Saez: Not 3GS. It was the first generation iPhone. We built the ImageWiki before the App Store existed.

We knew that the App Store was coming out. And we knew that the App Store was going to be the biggest thing in the whole world. I remember getting into multiple fights with friends about how revolutionary the iPhone and the App Store were going to be and people thinking I was totally crazy; people just thinking I was absolutely nuts for being so excited about it.

It sucks that it is a closed proprietary system, but the App Store has done something for software that nothing has ever done in the whole world. Software is candy now. It’s candy. It is like when you are waiting at the grocery store at the checkout line and you are stuck behind somebody, and you have got all these little tchotchka’s, candy bars, magazines, nail-clippers and things. That is the equivalent of software now. It’s become an impulse buy, which is amazing. Nobody would ever have thought…that is actually revolutionary. That’s huge.

Tish Shute: Steven Feiner, who is one of the founding fathers of augmented reality said to me during a conversations at the ARNY meetup that one reason that augmented reality, despite the hype, is manifesting very differently from how virtual reality burst onto the tech scene is that it is about affordable apps on affordable readily available hardware.

Mobile social communication is beginning to take center stage as the internet moves to real time communications. The recent explosion of interest in augmented reality is part of a wider concern to orchestrate a new landscape of contextually relevant information linked to location/place/time and mobile social connectedness.

The picture above, “Having an iphone has completely changed the way I poop,” is a slide from Gentry Underwood’s workshop at Web 2.0 Expo, NYC, “Social Interaction Design a Primer.”

Last month, I attended  three events starting with MoMo #13, Amsterdam, where I presented on, “The Next Wave of AR: Mobile Social Interaction, Right Here, Right Now!.  Then I caught the last two days of the Web 2.0 Expo, NYC, and finally, Real Time Crunchup SF (which I watched online).

New forms of real time mobile, social connectedness were central themes on all three occasions.

But, in terms of doing stuff that matters with mobile real time technologies, at the moment, we are still at the “hello world” demonstration (see my conversation below with Anil Dash and Mark Drapeau at Web 2.0 Expo below).

As Anil Dash noted,  “I think everybody starts with a train schedule…”

“I remember five years ago when Adrian did Chicagocrime.org. It was a revelation but I mean, that was five years ago.  And people still keep making that app over and over.”

Anil Dash announced at the Web 2.0 Expo that he will be the director of Expert Labs, a new nonprofit that will take the dot-com incubator model and apply it to new digital tools for the federal government:

“For me, in starting Expert Labs it’s been great just to tap into the desire people have to help and serve and to take the idea that you can work for your country without having to work for your government. What can you do to participate?”

The Gov 2.0 movement is attracting the best and the brightest, if you need inspiration check out Carl Malamud’s Gov 2.0 Summit presentation, By the People…..   Jennifer Pahlka is leaving her long time post as co-chair of Web 2.0 events for TechWeb to concentrate on Code for America. And Mark Drapeau is co-chair of the Government 2.0 Expo next May, that O’Reilly and TechWeb are also producing.  You can submit ideas about Gov 2.0, ICT, and cities (or other topics) to the upcoming Gov 2.0 Expo.  Mark says he will welcome them! Note there is a Free Gov 2.0 Online conf. Thursday, Dec. 10th

Tim O’Reilly has committed to Gov 2.0 work and “doing stuff that matters” with missionary zeal (see his keynote Web 2.o Expo, War for the Web).   Tim O’Reilly’s talk, also the article,  War for the Web, are a stark reminder of how the centralization and privatization of large parts of our communications infrastructure threatens the open web.  But “doing stuff that matters,” as it turns out,  is one of the best ways to win the war for the open web.

“Level playing Fields, Open access, Open APIs, Controlling our data, being able to move with it” (Anil Dash)

Slide above from Anil Dash’s presentation at the Web 2.0 Expo, NYC, “Listening to the Experts”

The Gov 2.0 movement is still in the idea and initiative phase, but the ideals and scope of the movement are a natural antidote to the fox in the social network chicken coop business model du jour (see the latest antics of Zynga).

Anil Dash notes the intrinsic bond between Gov 2.0 work and the open web:

“Because government has an inclination to creating openness by its nature. Right?  We don’t have an entirely toll system of federal highways in the states. We understand that the broadcast airwaves are a public good. And so government is inclined to think about creating public goods. It would be ridiculous to spend tax payer dollars on funding proprietary platforms.”

The Power of Government as a Platform for citizen involvement is just beginning to emerge from initiatives like Data.gov  “a collection of federal data housed on the www.data.gov Web site that’s open to public access.”

One of the most challenging aspects of creating in context mobile applications that do stuff that matters is the data curation.

Sean White, explained to me at ISMAR 2009, the challenges of data curation behind this beautiful example of augmented reality doing something that matters (pic below) -  a pollution meter, that “shows carbon monoxide levels projected over New York City.  The height of each ball reflects concentrations of the pollutant” (developed at Columbia University Graphics and User Interface Lab where Steven Feiner is Director).  Note Sean White and Steven Feiner will be at AR DevCamp NYC this weekend at The Open Planning Project office (TOPP) – see below for more information.

Open Data combined with Open Architectures of Participation are a Powerful Combination.

Scott Yates commented in his very insightful post on RT Crunchup SF that a “literny of fixes” for a broken web were “presented as  the state of the art”  in a “series of presentations from companies that have solutions that fix some subset of all the long list of annoyances” (annoyances arising from finding data and friends locked into a variety of different walled gardens).

And, Scott Yates writes:

“There have been presentations from companies who hope to be the future of socially connected communications, but not one of them has the economic or intellectual heft to be considered a true vision for the future.”

If you have been following my recent posts, you will already know that I agree with Scott Yates when he concludes: “Wave really has an opportunity to fix so much of what is broken in communications.”

I have been working on a project to create an open distributed augmented reality/mobile social communications framework based on the Wave Federation Protocol.

This Saturday Dec 5th there will be AR DevCamps held in Mountain View, New York City, Wave and Skype. There will be sessions on many aspects of open augmented reality, including Wave enabled AR.

AR DevCamp

I will attend AR DevCampNYC at the NYC location, The Open Planning Project office (TOPP) penthouse in Manhattan.  This will be an awesome opportunity to meet some of the key augmented reality thought leaders and innovators, including Sean White, Steven Feiner,  Steve Henderson, and many others (see the sign up list here).  Ohan Oda will demo the open Goblin XNA platform.   Thomas Wrobel will answer questions on writing AR Blips to PygoWave Servers and Sophia Parafina (@spara), Joe Lamantia (@mojoe) and I will be on hand to discuss the open distributed framework for AR project -  Wave enabled AR.   The PyGoWave crew will participate via skype (they will be introducing some of their latest work ).  Ori Inbar of Ogmento and Games Alfresco will lead a brainstorming session on the “Big AR NY Game”: The first location-based, social, augmented reality game designed for New York by New Yorkers.

We will continue the interesting discussion led by Marco Neumann (@neumarcx ) on the Semantic Web and Augmented Reality at the Semantic Web Meetup.  John C. Havens will introduce the Outernet Guidelines Initiative.  And Matthew Snodgrass @mattsnod, Lippe Taylor will lead a session on the future implications of AR.  Noah Zerkin, will share his brilliant work on AR software and hardware interfaces and exploring the idea of an AROS.  And Kate Chapman, from Washington, DC,  @wonderchook, and a bevy of local NYC geo geniuses, including organizer Sophia Parafina (@spara ), will explore ways to visualize government data through AR.  I am hoping we will have some projects for the upcoming Gov 2.0 Expo at http://gov2expo.com.

And there will be much, much more – keep checking and adding to the wiki. See you there!

“Hacking Human Behavior Within a City”

Picture from inspiring cities.org -shows some Amsterdam bicycle trends

At Web 2.0 Expo, Gentry Underwood, IDEO, gave a great presentation on how software changes community and communities change software from an ethnographic perspective – “Designing Web 2.0: Here Come the Anthropologists.”

And Baratunde Thurston’s,“There’s a #hashtag for That, was an inspired, brilliant romp through the “mini-grass roots movements” of hashtags - which are “quickly assembled/demolished malleable fun!” or “great ways to mess with people,” that reminded us the power of grass roots movements when it comes to “hacking human behavior.”

But my visit to MoMo #13 preceeding the Web 2.0 Expo showed me clearly “hacking human behavior within a city” is on home turf in Amsterdam, where smart phones and bicycles are the vehicles for the MoMoesque lifestyle.

Thanks to the foresight and generosity of the MoMo organizers, who make sure that the experience of  the speakers together goes beyond the few hours of the event, I had a three day, three night intensive on the future of mobile social interaction – living, thinking, and breathing mobile social connectedness, often into the wee hours, with Dennis Crowley, CEO of FourSquare (see his great MoMo 13 presentation here), Ted Morgan, SkyHook (a must see presentation on what SkyHook is doing with data), the MoMo crew, and many of Amsterdam’s enthusiastic Four Square community. 

And yes, Four Square really is an awesome way to enjoy a city and meet new people.  MIA in this particular pic are key MoMo organizers -  @samwarnaars @mdbraber and @vangeest

But what do fun times in Amsterdam and FourSquare have to do with doing stuff that matters?

Marc Fonteijn, MoMo chair and co-founder of 31Volts points out: “foursquare looks and feels like a game but what it’s actually doing is changing behavior in a playful way”

And Yuri van Geest, who co-founded not only Mobile Monday Amsterdam but also TEDx Amsterdam, added:

“in Holland we are working on using the FourSquare API for mHealth purposes also we see that smart venue owners reward all mayors/lead users/visitors with free meals/drinks/privileges/perks etc. and smart advertisers to boost their co-marketing deals based on FourSquare targeting capabilities of key influencers”

Dennis Crowley, seemingly immune to lack of sleep and jet lag, followed up his MoMo #13 talk with a presentation at Web 2.0 Expo.   I was sitting just behind Mark Drapeau, and I managed to catch up with Mark after Dennis’ talk.

Mark listed Foursquare in his big takeaways from the Web 2.0 Expo, pointing out the potential new forms of mobile social interaction have for “hacking human behavior within a city.”

“I had always been a little leery of trying FourSquare because I have a certain level of privacy I try to keep up. But listening to the CEO of Foursquare talk about it… I knew what it was. I have friends that use it.. but thinking about it as hacking human behavior within a city and social engineering of peoples’ behavior and what they can do, and really understanding what citizens are doing within cities, or other areas, and how they interact with each other.   I think could be incredibly valuable for government 2.0 and government understanding people better.”

And Anil Dash concurred:

“I think Foursquare is a good model in terms of having a game dynamic, being mobile from its default, having a great social experience, leveraging existing networks like Twitter and Facebook instead of trying to compete with them by building their own. I think those are all really, really smart leanings.

I think about if I were a government agency trying to meet those same goals, could I earn badges in Foursquare by doing things that help my community. Right? So when I volunteer at a soup kitchen is that one way to earn an exclusive badge? Is that going to earn me a discount at the bar? Those are all dynamics that we can set up very, very easily and I think that model…maybe it is a public-private partnership. That’d be great.”

Talking with Mark Drapeau and Anil Dash

Tish Shute: I was in Amsterdam speaking at MoMo#13 and I had a lot of fun hanging out with the MoMo crew and Dennis, CEO of Four Square.  I got to meet people and hang out  with Amsterdam’s new Four Square community. But unfortunately I missed the first two days of Web 2.0 Expo.

Mark Drapeau: I got here yesterday too. Yeah. So some big takeaways.

I had always been a little leery of trying FourSquare because I have a certain level of privacy I try to keep up. But listening to the CEO of Foursquare talk about it… I knew what it was. I have friends that use it but thinking about it as hacking human behavior within a city and social engineering of peoples’ behavior and what they can do, and really understanding what citizens are doing within cities, or other areas, and how they interact with each other, I think that could be incredibly valuable for government 2.0 and government understanding people better.

Also I really wanted to hear Tim O’Reilly interview Beth Noveck. I thought the most interesting thing about the interview were the questions and not the answers (also see Mark’s Posterous ).  I thought a lot of the answers were disappointing and political and vague.

But I thought Tim really got some important issues about how do people in the web 2.0 community, the audience of Web 2 Expo, interact in reality when you have a system that we nicknamed Gucci Gulch, where you have lobbyists and lawyers and special interest and councils and all these things that…developers and app builders are not really a part of.  So how do you break in?  I didn’t really hear good answers for that.

I really liked the presentation by the IBM researcher, if I can get his name. Forgive me. Ching Yun-Lin. Talking about putting a value on how many friends you have, how many connections you have and the fact that IBM can actually put a monetary value on the number of connections you have to managers. The number of email accounts you have in your inbox. Or your address book.

I thought that was just fascinating and that’s something I’m very passionate about is social networking for the sake of social networking and not merely for collaboration but making connections among diverse communities and using that to help your business or help your government agency.  Those are my big takeaways this morning.

Tish Shute: I was lucky enough to attend Gov 2.0 Summit.  I think a lot very important areas for Gov 2.0 were defined there, transparency, open data, getting developers into the public sector loop, and citizen-government interaction.  In what areas are we seeing progress and where are we stymied and why?  How do you see this Web 2.0 community connecting to the ideals and plans for action of Gov 2.0?

Mark Drapeau: I think there’s a lot of unanswered questions about Government 2.0 because there’s a lot of good talk and a lot of good ideas and initiatives but there’s still a long way to go before people in this audience, in this community who want to help the government or be a part of policy making or technology in the government can really in a meaningful way, interact with the government processes that, for the most part, are not going away.

And the people that are also part of the system, like giant contractors, they’re not going away. There’s a place for everyone. The question is how do the smaller people break in and I don’t think there are really great answers for that.

Tish Shute: What is the plan for the next Government 2.0 event?

Mark Drapeau: So part of the reason I’m here is to learn and be inspired as the co-chair of the Government 2.0 Expo next May that O’Reilly and TechWeb are also producing. And so there’s increasingly at the Web 2.0 events that they host, there are technologies and people relevant to government missions, or the public sector missions.

And so I think some of the speakers here will carry over in different ways to the Gov 2 Expo in May.

Tish Shute: Is the call for proposals for the Gov 2.0 Expo still open?

Mark Drapeau: There’s still an open call for proposals. Or people, if they know me they can talk to me directly.

(Anil Dash arrives and Mark introduces  me (Anil and I met briefly at Gov 2.0 Summit) – see Anil’s post, New York City is the Future of the Web I do agree but, of course, NYC is my hometown!).

Tish Shute: Anil you are moving into Gov 2.0 work full time now after being a key thought leader in Web 2.0.

Anil Dash: My perspective is probably unique in that I am very strongly from the Web 2.0 world, and new to the Gov 2.0 world and I think it’s telling that you can make the leap. I think that the profound thing is that these worlds are converging and it’s not where it was.

Five years ago the government technology was a bike with the training wheels on it. It was very much somebody’s old hacked up version of Drupel and crossed fingers.  And it looked a little homely and you thought, “well this looks like a run down office kind of thing.”

Now we have have institutions that have wonderful physical presences. You can’t stand in front of the Capital Building or the White House or Supreme Court and not say “that’s a majestic building.” We should have online institutions that reflect the scope and the scale of what they do.

For me, in starting Expert Labs it’s been great just to tap into the desire people have to help and serve and to take the idea that you can work for your country without having to work for your government. What can you do to participate?

Tish Shute: You and Mark had very interesting journeys  into Gov 2.0 didn’t you?

Mark Drapeau: Like the Hunter S. Thomson of Government 2.0

Tish Shute: I like that!

Anil Dash: Can you say that about yourself?

Mark Drapeau: I did the other night and people seemed to buy it, so

Anil Dash: People were feeling it..

Mark Drapeau: That’s right!

Tish Shute: And even if it’s controversial it’s good too!  I see Mark (and perhaps I am wrong with these characterizations) as coming to this via an interest in the social narratives of government and, Anil, you have come to Gov 2.0 work, as you point out, from a deep immersion in the cultures of technology and Web 2.0….

Anil Dash: And it’s also a little bit, I’ve had the privilege of seeing blogs and social media develop from the start and what I learned from it is cultural change and [not just] technology change. This is the same thing happening in government.

We’re calling it Government 2.0 and it makes it seem like it’s a version upgrade and it’s a software thing but it’s cultural change. And the interesting thing is many of the key players have a willingness to go through that cultural change, which means that the technology, therefore, has the opportunity to succeed.

Tish Shute: What did you think about Tim O’Reilly’s keynote and the warning he gave re the open web?

Anil Dash: The war for the web! He’s absolutely right. Honestly, before Expert Labs had started and I’d come on board, my initial plan for a talk at this event was exactly the topics Tim covered in the War for the Web.  That the centralization of vast parts of our communications infrastructure around privately owned, venture funded companies is a risk to innovation in some ways.

We have to make sure to set up our incentives for those companies, the Facebooks and Googles and Twitters of the world, to align with what our goals are as a society, as a culture, as entrepreneurs, and all those other goals.

So I think it’s good to have a voice like Tim’s articulating that threat and that danger so that we can respond to it. I agree completely that we are in the next phase of the battle between open and closed platforms that we went through ten years ago with AOL.

There was a time when AOL dominated more of the dial up internet, one-third of all dial-up users in the US were coming through AOL. People now say “oh, the iPhone is dominant.” The iPhone has 2 percent market share of all phones or something like that, and yet people are doing all their innovation on their platform.

Well, people used to do all their innovation on AOL’s platform and then they ended up having to rewrite it all for the open web.

This pattern is going to repeat. The choice is whether people want to encourage it or fight it or hope it goes away and ultimately there’s no great business that was built entirely within the walls of AOL’s garden. I doubt there will be a great business built entirely within the walls of Apple’s or Facebook’s or anyone else’s garden.

That’s not to say those companies couldn’t evolve to be open, I hope they do, but as it stands right now you would be foolish to bet your business either from a promotional standpoint, from a start-up standpoint, from a new technology standpoint, on any closed platform that you don’t control.

Tish Shute: I was mentioning to Mark that I thought it’s sort of ironic that we now understand how important the architecture of participation of the internet can be to government just as we are on the verge of another big battle to keep the web open…, a moment when walled gardens are seeming to dominate..will this be an obstacle for Gov 2.0?

Anil Dash: No I think actually they’ll get to skip the closed era.

You know I look at the rather famous example in India of never having landlines. They went directly to satellite phones, skipped directly to the wireless generation so they never had an old infrastructure to rip out.

I think you are going to see the same thing with government tech adoption is they are going to start in the era of recognizing the threat of closed platforms and move directly to open platforms.

Because government has an inclination to creating openness by its nature. Right? We don’t have an entirely toll system of federal highways in the states. We understand that the broadcast airwaves are a public good. And so government is inclined to think about creating public goods. It would be ridiculous to spend tax payer dollars on funding proprietary platforms.

Mark Drapeau: Universal accessibility for citizens.

Anil Dash: Right. It’s a fundamental tenant of government and we have an incredible history including the Internet itself, of embracing open standards to solve government problems in a way that helps all of society.

Tish Shute: So people who have championed open participatory architecture of the internet and open source approaches now have even is more incentive to team up with government 2.0!

Anil Dash: Yeah it is an advantage. But also, I mean candidly, open source is almost incidental to it. I mean I think we have come to the point where open source is assumed as some element of any new tech venture. It is much more about level playing fields, open access, open APIs, controlling our data, being able to move with it, that I think is key.

And I go back to that AOL example, there was a moment where they opened their email gateways to standard Internet email. And so instead of the AOL users only being able to email each other, they could email anybody on the web and this is the moment in which all the value was created. You start to have email marketing companies, and open exchanges and open mailing lists happen when anybody could email anybody else, that is the sort of thing that government catalyzed just by being the example.

Tish Shute: Mark talked about Four Square and how that could be really interesting as part of a Gov 2.0 project. But mobile has followed a course with many complications re an architectural participation – I am thinking about the control exerted by the carriers and now Apple for example?

Anil Dash: No, I think it has revealed complications that have always been there. Right? There’s always been multiple platforms. There have always been user agents and web browsers that have different capabilities. There has always been a digital divide. Mobile is making clear that those realities existed.

But I keep saying this, like, I think if I am designing an application today, you design for mobile first, for a number of reasons. One, the digital divide is much less pronounced on mobile devices. Two, you are much more likely to have an experience that scales well from a small device to a larger one than vice versa. Three, you are able to target international markets or other developing markets where mobile is the default computer platform. And you become aware of constraints in bandwidth, in accessibility, in user experience, in general experience with computers, that a lot of people in the technology industry just completely ignore.

And you know you go to Silicon Valley and people think it’s normal to have a six hundred dollar phone that has a thousand dollar a year data plan. And without blinking they designed for devices like that. It’s myopic and ridiculous to think that people can live with that level of privilege all the time on all the devices that they have and that they have a brand new computer. And so that will be its own undoing.

Right? It’s the people that are thinking about open platforms and working with any device and I think FourSquare candidly is doing a good job of this because they did start with the assumption of iPhones and this thing but their initial target audience of hipsters in the east village probably did have those. But now they have an open API, anybody can access it, that’s the right evolution. And I think they’re smart enough, that was always on their plan.

Tish Shute: But in terms of mobile social interaction we basically have really a structure all of lots of different wall gardens?

Anil Dash: For now..

Tish Shute: How do you see mobile developing more interoperability and social interaction capabilities?

Anil Dash: By using the web. I think it doesn’t have to be full-fledged Ajax-y, html applications on the phone. But if we simply rely on the capabilities of the web as it stands today instead of developing for proprietary mobile platforms we can make a lot of amazing things happen. It’s a good constraint. We should embrace our constraints.

It’s not conventional wisdom yet that mobile applications should be developed for the web. But that’ll change in the next year.

Tish Shute: There is a lot of exciting new real time technologies coming to the Web, Pubhubsubbub, Google Wave Federation Protocol. How will these change mobile development?

Anil Dash: RSS cloud. I mean there’s a ton of real time technologies that are coming out together.

Tish Shute: What are your favorites in the real time area?

Anil Dash: I wrote a post about this .. called The Push-Button Web where I actually go into this…

Tish Shute: Oh yes great post!

Anil Dash: I don’t pick a favorite. I think all of them together will work. I think it’s similar to how the web itself evolved.

We have a tangle of different related technologies that get abstracted away when you use a browser. You don’t know if it’s a gif image or a jpeg image when you browse a page. You just know it’s showing an image in line.

I think we’re going to see the same thing happen to real time web. We’re going to very, very quickly settle into a stack of technologies that let us do real time. As a developer you might have to be aware of the subtle differences. As a user your experience is going to be, “I have real time and it works on whatever device I’m on.”

Tish Shute: Mobile seems like a vital part of government 2.0 because it can connect people and their government to their context/public infrastructure/environment that is a shared concern. The open data movement has shown that being able to mash up data and get that delivered in context is a very powerful kind of technology for government 2.0. Right?

Anil Dash: I don’t know. I think that’s there’s just been the “hello world” demonstration. I think everybody starts with a train schedule…

Mark Drapeau: I was just going to say that everyone is starting with the very low hanging fruit. The transportation, the crime. It’s not exactly clear where it’s going to go but I think it’ll go –

Anil Dash: I remember five years when Adrian did Chicagocrime.org. It was a revelation but I mean, that was five years ago. And people still keep making that app over and over.

I remember at the time I had just become friends with Craig Newmark and I said, “Craig’s List should show the crime around the neighborhoods where you have an apartment listing.” And he said, “Well, if I do that then neighbourhoods that are getting better, that historically had more crime, will never improve because people won’t rent apartments there.” And he came back with that answer immediately as soon as I suggested the idea and revealed one, why Craig’s List is the success that it is. But two, what the implications are of releasing data and having to think about the social implications of that.

Mark Drapeau: Well, it’s like Gentry from IDEO said that, “Social software changes the community, which changes the software.”

Anil Dash: Right. Exactly. We have to think about the social implications of the tools and technology we create.

That means that the reason…one of the reasons we have only had these, frankly, unambitious obvious applications of open data is because the people that have had access thus far have been people that are not socially oriented. Like geeks are very inwardly focused.

Tish Shute: Oh. Okay. Well Mark’s changing this..

Anil Dash: They’re in a very insular community.

Mark Drapeau: I think there’s a number of people that are trying to change that.

Anil Dash: Yeah. It’s starting to change but I’m saying that’s why we’ve seen that symptom in the past.

Mark Drapeau: I get a lot of mileage out of the fact that I’m neither sort of a career govie type that’s getting into the 2.0 stuff. Nor am I a lifelong techie who’s getting into the government and stuff. I’m sort of…I’m interested in these anthropological, psychological, animal behavioral, ecological questions about human behavior and networking. And that’s where I kind of come into this.

Anil Dash: And I think we’re going to need an ethnographic approach to looking at how people work with this data in their real lives. People are using this data already and don’t realize it. You know, when we grab a map in an unfamiliar city you’re using government data. We just don’t think of those behaviors as doing so, and we need to understand that to build applications that really solve people’s problems.

Tish Shute: So can you speculate on the next generation you’d like to see?

Anil Dash: I think Foursquare is a good model in terms of having a game dynamic, being mobile from its default, having a great social experience, leveraging existing networks like Twitter and Facebook instead of trying to compete with them by building their own.

I think those are all really, really smart leanings. I think about if I were a government agency trying to meet those same goals, could I earn badges in Foursquare by doing things that help my community. Right? So when I volunteer at a soup kitchen is that one way to earn an exclusive badge?  Is that going to earn me a discount at the bar?  Those are all dynamics that we can set up very, very easily and I think that model…maybe it is a public-private partnership. That’d be great.

Mark Drapeau: Or even doing things to help your internal community. Key people at work or within your agency or things like that. From my vantage point it does seem like local Government 2.0 types are thinking much more about mobile than the Federal government types. The reality is government employees all have BlackBerries and they’re running around all the time. But they’re in terms of government 2.0 type stuff they’re thinking about the Dell desktop they have and the Microsoft Windows system and whenever I mention something like mobile or pervasive videos…..people aren’t really there. They’re worried about cyber security on the traditional systems. They’re worried about desktop applications on a Dell.

Anil Dash: They’re still five years ago.

Mark Drapeau: Yeah. They’re still five years ago and so I think these kind of O’Reilly-Tech Web events, Gov 2.0 Expo, Web 2.0 Expo, etc., are really starting to get at these questions that are now and not five years ago.

ISMAR 2009 -  was an extraordinary mix of  high geek, academic eminence, gungho Dutch Cowboy entrepreneurial spirit, German engineering and industry, brilliant artistry, and invention, all fueled by a sense, and a very active presence in the case of Diamond Sponsor – Qualcomm, that the big technology players are waking up to augmented reality.

In the picture sequence above (click on photos to enlarge),  Noora Guldemond, Metaio, demonstrates Junaio (coming to an iphone near you Nov 2nd) to Dirk Groten, CTO of Layar (top left photo).  One of the nice social features of Junaio is that users can share the 3D augmented scenes they have created.  Noora is demoing this capability to Dirk, and as you can see he cracks up when he sees the  scene Noora has stored on her phone.  Dirk and I both recognize that this cute little dinosaur augmentation (close up above on bottom left) must have been created by Peter Meier, CTO of Metaio, during the Interoperability and Standards workshop earlier that day.  Metaio it seems were discussing standards while enjoying some 3D augmented back chat.

Both Dirk and I were active participants in the workshop too.  But little did we know that Peter Meier had introduced his little 3D dinosaur into our discussion while we diligently, and sometimes heatedly, debated the merits of XMPP, Wave Federation Protocol,  KML, ARML, VRML, X3D, and  more!  The photo I took is on the bottom right of the four pics above. It was probably taken very shortly after Peter’s augmented Junaio scene.  Of course there is no little dinosaur in my pic of  Dirk Groten with Joe Ludwig and Markus Tripp of Mobilizy who were discussing AR standards oblivious to Peter’s virtual pet in our midst.

I must say I had noticed an impish look on Peter Meier’s face (see photo above on the left – Peter is wearing glasses and holding a phone).  And Markus Tripp, of Mobilizy  revealed a little bit of gaming of his own, when he let out that, in part, ARML is a provocation.  But Peter was clearly unfazed and enjoying himself.  Dirk, tasked to summarize our discussion, stalwartly maintained an optimistic but serious tone fitting for a standards discussion:  “There is a willingness to standardize….,” he began (pic above on left – click to enlarge and read text).

But it was a little 3D dinosaur that, perhaps appropriately, had the last laugh. Fitting, as I am not sure whether anything anyone says about AR standards at the moment will hold up.  But, as Ori commented in his great post – an epilogue for ISMAR 2009, the vibe was “Peace and Love” in AR Browser land (although Chetan Damani of Across Air was not in the standards discussion because he attended the UX/content? workshop instead).  But as they say, “all’s fair in love and war.”  And it is my feeling the games have barely begun!  There are many players (virtual pets included) waiting in the wings. I met some at ISMAR, and they are just itching to join the frey.

Ori Inbar, Ogmento and Robert Rice, Neogence Enterprises, both founders of the AR Consortium, made great efforts to set our young industry off on the right foot -  in the  spirit of coopetition (a neologism coined to describe cooperative competition). See Curious Raven for Robert’s conference observations, and Ori’s post on Games Alfresco for more about Mobile Augmented Reality at ISMAR 2009.  The Mobile Augmented Reality Workshops were driven by an indomitable spokesperson for the new AR industry, Christine Perey.  Christine not only helped motivate discussion on the issue of oxygen to the system, i.e. business value, but also she was a very generous connector at the conference.

What’s Next From Augmented Reality’s Top Chefs?

As Ori pointed out, Kent Demaine, oooii (pic above is from the oooii web site), Minority report VFX designer was hanging out at ISMAR 2009 and he came to the panel I was on: “Augmented Reality in Sports,  Entertainment and Advertising.”  We chatted afterwords about instrumented environments and how this is such a key to development interesting augmented experiences.  Also I mentioned how back in the day I was involved in some of the early development of motion control software.  And it was great to hear Kent say they were still finding motion control cool at oooii.  As Ori notes, he is the “guy with the most enviable AR credentials in the world (the guy who designed VFX for minority report),” and oooii is busy and hiring.

One of the highlights of the Arts, Media and Humanities track for me was meeting Jarrell  Pair. He really brought the best out in panelists with his well tuned questions.  The recording of ISMAR was comprehensive and videos should be up next week.  I will post the slides on Ugotrade of my presentation:  “The Next Wave of AR: Shared Augmented Realities and Remix Culture.”.

“Mixed and Augmented Reality: ‘Scary and Wondrous’” – Vernor Vinge

“Imagine an environment where most physical objects know where they are, what they are, and can, (in principle) network with any other object. With this infrastructure, reality becomes its own database.  Multiple consensual virtual environments are possible, each oriented to the needs of its constituency.  If we also have open standards, then bottom-up social networks and even bottom up advertising become possible. Now imagine that in addition to sensors, many of these itsy-bitsy processors are equipped with effectors.  Then the physical world becomes much more like a software construct.  The possibilities are both scary and wondrous.” (Vernor Vinge -  intro to ISMAR 2009)

Vernor Vinge’s short intro to ISMAR 2009 (which can be downloaded with the ISMAR 2009 schedule here) captures the essence of the “Scary and Wondrous” dawn of the age of ubiquitous computing and mixed and augmented reality.  It is definitely worth a moment to download.  The future of augmented and mixed realities, as Vernor Vinge points out, is tied up in a “tension between centralized and distributed computing” that “will continue long into the future.” One of  my fascinations with Wave is that it offers a tantalizing opportunity to explore augmented reality in an open distributed architecture.

At ISMAR, I talked with as many people as possible about the AR Wave project – see my post here for more about Wave enabled AR.  Many people were very enthusiastic to join the AR wave and the only thing I really lacked was about 100 invites to hand out!

“Everything, Everywhere – making visible the invisible”

Some of the areas that I would have liked to see given more attention on at ISMAR were sensor networks, data curation, and user experience.  Not that these areas were entirely neglected with Pattie Maes, MIT as a keynote speaker, and Mark Billinghurst presenting on some fascinating work on social augmented experiences and user experience.  I highly recommend catching up on these and other ISMAR presentations when the videos go up.

And, I was very happy to meet and talk to Sean White whose work at Columbia University is one of my inspirations (for more about Sean’s work see here or click image above):

“the confluence of powerful connected mobile devices, advances in computer vision and sensing, and techniques such as augmented reality (AR) enables exciting new opportunities for interacting with this hidden network of dynamic information and shifts the locus of interaction from the desktop computer to the world around us”

And I had several very interesting conversations  at ISMAR about developing social augmented experiences that connect us to a physical world that is becoming “much more like a software construct” (Vernor Vinge).  Dirk Groten, CTO of Layar mentioned a few interesting projects Layar has up their sleeves, including something  Layar may be cooking up with The RoomWare Project.

The picture above is of RoomWare’s Social RFID Installation for Media Plaza in Utrecht (read more here).

Demos Galore!

In the demo rooms, Noah Zerkin (pic below left) pretty much single handedly carried the AR flag for a growing community of augmented reality Makers and Hackers.  But his presence was much appreciated, and he tirelessly demoed The Zerkin Glove. See the first of what may be several posts from Noah on ISMAR here.

And I got to try out the Vuzix goggles (picture above on right).   This was my first experience playing an AR game that was smart about real world gravity. It’s  “an augmented reality marble game that uses gravity as a game controller” – see Ori Inbar’s write up here.  It was a very compelling experience and I have to say I didn’t really notice the shortcomings of the Vuzix goggles while I was absorbed in the game. And  I turned out to be quite good at the game too. It is intuitive unlike the kind of  rule based games I never have time to learn properly.  But what is so special about this project is the tools that it is built with are open, and available for all, and affordable (see this list on Games Alfresco).

It was a great pleasure to meet Prof. Steven Feiner (picture on below the left) who heads Columbia University’s brilliant AR research team at The Columbia University Graphics and User Interfaces Lab.

Ori Inbar (pic below on right) also spent a lot of time in the demo room showing off Ogmento’s lovely AR learning game that delighted attendees, “Put a Spell: Learn to Spell with Augmented Reality.”

For a round up of  what’s next for augmented reality head mounted displays check out, Games Alfresco here, and Thomas Carpenter’s excellent review of the head mounted displays.

Ori Inbar on Games Alfresco asks is “Microsoft – the new big player to watch?”   “Georg Klein, inventor of PTAM-on-an-iPhone (and the smartest Computer Vision guy on the block)” has joined Microsoft to make Mobile AR.

The picture on the left above shows Georg trying out ARhrrr with Blair MacIntyre.   And on the right Blair is demoing his marker card pack to Senior Vice President of Cypher Entertainment, David Elmekies.  Yes ISMAR was abuzz with demos. See this post from Gail Carmichael for more video demos.

Next Year ISMAR 2010 in Korea!

At the banquet, I managed to find a seat at a table with Sean White (at left in photo above with Christine Perey to his right) and the Columbia University team.  The banquet culminated with the “Past and Future of ISMAR” Panel chaired valiantly by Jay Wright of Qualcomm.  We were asked to offer our input for ISMAR 2010.  I offered up an idea that I have been nurturing for a while now -  to stage a “Green Tech AR Competition.”  Perhaps, I suggested, we could base the competition around a conference (ISMAR 2010 in Korea?) and set up a target rich, instrumented environment for the occassion.  I think the Arduino open hardware community and AR developers have a synergy that is just waiting to be explored!  And, if we add the innovators of data curation to the mix, e.g., Pachube, AMEE, and Path Intelligence…(Markus Tripp left ISMAR to speak on a Web 2.0 Summit panel, “Humans as Sensors,” which also included Path Intelligence, Deborah Estrin on “participatory sensing,” and the brilliant work of Di-Ann Eisnor, Platial, on “Transactional Cartography”).  Anyway a big Green tech AR competition could get people working together across the broad spread of AR terrain on some of the sticky problems of user experience.  And, with a high level of support from Smart Phone companies, HMDs manufacturers and the chip makers we just might come up with some extraordinary magic.

The devil of course will be in the details.  But a competition like this could not only motivate key players to come together in the spirit of coopetition but also be an opportunity to show the world the power of AR to make visible the invisible ecosystems that are so important to the health of our planet.

One of the notable presences at ISMAR 2009 was the Qualcomm team.   Jay Wright’s presentation (an exclusive for ISMAR) not only outlined AR for 2012, but Jay also talked about some “close to the metal” innovation that we will see from Qualcomm very, very soon!  I had some time in the press room with Jay and his team prompted by MoMo’s Yuri van Geest.  When I twittered about Qualcomm’s presentation at ISMAR, Yuri replied:

vangeest “@tishshute: good stuff, hopefully you will integrate the neat new solutions and ideas in your talk in November ;)”

I will be presenting at MoMo #13 on AR, open AR, future of AR and GeoWeb,  and hopefully will bring some good news from Qualcomm too.  Anyway Jay seemed to like the idea of a Green Tech AR Competition, even though I did stress that I thought it needed some serious sponsorship and BIG prizes.

Where’s the beef? Tracking and Mapping at ISMAR 2009

On the flight from NYC to Orlando and ISMAR’o9 I dozed (I had been up late preparing my presentation) and I watched the Dew Tour Pro Skateboard competition and Top Chef on the Food Channel.  In this particular episode of Top Chef, the aspiring chefs were all given a brown bag of ingredients by an already famous chef who then judged whether the contenders managed to make a delicious meal with their allotment which was notably lacking in key ingredients of haute cusine.

This metaphor of  trying to cook up a great meal while perhaps missing the staples is apt for the current early stage of commercial augmented reality.  And when I arrived in Orlando, not only were the Dew Tour pro skateboarders staying at the same hotel as ISMAR, but ISMAR itself felt remarkably like an Augmented Reality Top Chef Coopetition.

Much of ISMAR was dedicated to the task of  providing the meat and potatoes of Augmented Reality, solutions to mobile tracking, mapping and registration, particularly in the Science and Technology track.

Industrial and Military Augmented reality solutions I found out, typically, solve the tracking problems by using fixed mounts which clearly wouldn’t translate well into the AR everywhere with everything mobile consumer culture expects.

In the picture on the left Fabian Doil stands by the VW engine that provided some of the outdoor targets for the ISMAR tracking competition.  On the right is a picture from the VW’s presentation on their research and development of AR.

I followed the tracking contest, organized by Daniel Pustka and Fabian Doil of Volkswagen, quite closely. And I learned a lot in the process. While  it is clear there has been progress in AR mapping and tracking, we still have a ways to go.

But hanging around the Tracking Competition was a good way to find out the state of play of this crucial part of the AR dream.  For example,  a little tidbit I learned is that Michael Goesele who has been reconstructing “high-quality geometry models from images collected from the internet (so called community photo collections, CPC)” is soon to be at the Institut Graphische Datenverarbeitung where top contenders in the tracking contest – Harald Wuest  and Folker Weintipper (in the foreground of the photo at the left and right respectively) are also to be found. [update Harold and Folker were the winning team see here for details of scoring and results!] Otto Korkalo and Tuomas Kantonen of VTT, Finland, Augmented Reality team are in the background. They have been working on the joint IBM, Nokia and VTT project that brings, Avatars and People Together for Virtual Meetings in Physical Spaces.

The picture on the right is another team that were doing very well. If my notes serve me well (and please forgive me if they don’t. I came back with my card wallet overflowing!) the photo on the right showsChristian Waechter (on the left) and Peter Keitler (on the right) of the Technische Universitat Munchen.

Germany is certainly leading the way in industrial AR. And I learned how small businesses like Metaio get to work with top research institutions and big companies like VW, thanks to very strong German funding program for AR and VR. The current iteration of a series of funding programs is  called Avilus.  Avilus  is putting 42 million Euros into AR and VR this year alone (click on the slide below to see more about Avilus ).

I wish we had the equivalent of Avilus here in the US.  But there is no equivalent to Arvilus for AR here, and  no AR is  being developed by the US car industry either it seems.  But look at the slide above to get a taste of some of the cool stuff Metaio and other small AR and VR businesses do for VW through the Avilus project.

I also got to meet many people from one of the world’s most important AR hubs -  The Department of Informatics, Technische Universitat Munchen, including Prof. Gudren Klinker on the far right in pic above.  And from left to right, Fabian Doil (VW, co-organizer of contest), Sebastian Lieberknecht , Selim Ben Himane (Metaio), Tobias Eble (Metaio).  Prof. Klinker is the engine behind much of German innovation in AR.

Metaio was one of the few teams to rely mainly on markerless tracking which in this contest was very challenging because of the very different light conditions (see pics below) between the windowless interior and dazzling Florida sunshine outside (pic on the right shows targets under ideal lighting conditions).  Many people in the US may be  familiar with Metaio’s consumer applications, like Junaio,  but thanks to Germany’s efforts to nurture augmented and virtual reality they are also respected software developers in industrial AR.  And I suspect that Metaio will spearhead markeless tracking in consumer AR too.

This post as usual has already expanded to something much longer than I originally attended – pretty typical for me! There is much I have not been able to cover including some of the interesting contributions by augmented reality artists at ISMAR – again I recommend the upcoming videos.

But I cannot end without a hat tip to, Oriel, Nate et al. who won the best student paper award for AR Sketch – again please see Games Alfresco for more on this (pic below from Games Alfresco). AR Sketch, Ori notes, is featured “in our top post and popular video.” And

“Their work is revolutionizing the AR world by avoiding the need to print markers – or any images whatsoever.”

Above is an image above  from Total Immersion’s augmented reality experience developed for the “Networked City” exhibition in South Korea, – “a fun scenario created for a u-City’s infrastructure and city management service”

“To the naked eye, the exhibit looks like a bare bones model of a city. But when visitors put on the special AR goggles a whole new world unfolds – as graphics overlaid on the city model.” (Games Alfresco)

“The Networked City,” is a large scale augmented virtuality of a scenario for a networked city. But my guess, reading the     Korea IT Times, is the plan is to move from an augmented virtuality to an augmented reality as Incheon Free Economic Zone  (IFEZ) realizes its vision to become a leading u-City – where reality is turned “inside out” (see Inside Out: Interaction Design for Augmented Reality ).  If you are not familiar with South Korea’s u-Cities, check out this post for a short primer (and note Google Trends search on Augmented Reality shows  South Korea leaving everyone else in the dust).

Ubiquitous computing and augmented reality are like adenine and thymine – a DNA base pair.

A sky view of Incheon Free Economic Zone (from Korean IT Times). For more on the IFEZ vision to become a leading u-City see here.

Korea IT Times writes about the u-city concept:

“Korea began using the term u-City after accepting the concept of ubiquitous computing, a post-desktop model of human-computer interaction created by Mark Weiser, the chief technologist of the Xerox Palo Alto Research Center in California, in 1998. There have been a lot of research in this field since 2002. As a result, many local governments in Korea have applied this concept to various development projects since 2005 based on a practical approach to it.”

The back story to many of my recent posts, including this one, is an understanding of a relationship between ubiquitous computing and augmented reality that emerged, for me, in a February conversation with Adam Greenfield, Towards a Newer Urbanism: Talking Cities, Networks, and Publics with Adam Greenfield.  In cased you missed it, here is the link again because I think it holds up very well considering the rapid developments of recent months.  Also, importantly for this post, it includes a discussion of  moving on from Weiserian visions.

Adam Greenfield’s Speedbird is one of my key sources for understanding “networked urbanism,” and the list he makes of the elements of networked urbanism here (also see the comments) – is my mantra for thinking about the DNA base pair relationship of augmented reality and ubiquitous computing.

Adam Greenfield’s, “summary of what those of us who are thinking, writing and speaking about networked urbanism seem to be seeing” is:

1. From latent to explicit; 2. From browse to search; 3. From held to shared; 4. From expiring to persistent; 5. From deferred to real-time; 6. From passive to interactive; 7. From component to resource; 8. From constant to variable; 9. From wayfinding to wayshowing; 10. From object to service; 11. From vehicle to mobility; 12. From community to social network; 13. From ownership to use; 14. From consumer to constituent.

Augmented Reality – Making Visible the Invisible

The screenshot above is one of  the coolest “making visible the invisible” AR applications. It was developed at Columbia University Graphics and User Interface Lab where Steven Feiner is Director (see the deep list of projects from the lab here).  This app “shows carbon monoxide levels projected over New York City. The height of each ball reflects concentrations of the pollutant.” Credit: Sean White and Steven Feiner  (via Technology Review).

The recent emergence of “magic lens” augmented reality apps for our smart phones – Wikitude, Layar, Acrossair, Sekai Camera, and many others now, have given us a new window into our cities. But we are yet to realize the full potential of the AR/ubicomp base pair that can “make visible the invisible” and give us new opportunities to relate to the invisible data ecosystems of our cities, not merely as a spectator experience,  but as an interactive, in context, real time opportunity to reimagine social relations.

Mark Shepard says in his curatorial statement for, “Toward the Sentient City:” (Much more soon on this very significant exhibit which runs from Sept. 17th to Nov. 7th, 2009.)

“In place of natural weather systems, however, today we find the dataclouds of 21st century urban space increasingly shaping our experience of this city and the choices we make there.”

Augmented Reality, as Joe Lamantia points out, is becoming the great “ambassador of ubiqitous computing.” AR is. “…mak[ing] it possible to experience the new world of ubiquitous computing by reifying the digital layer that permeates our inside-out world,” and we are only just glimpsing the razor thin end of the wedge in this regard.

I am still working on my Gov 2.0 Summit write up  and, amongst other things, I will talk about how an emerging new social contract around open data, here in the US,  will put augmented reality  apps center stage  – “doing stuff that matters.” At Gov 2.0 Expo Showcase Tim O’Reilly tweeted:

Tim O’Reilly Really enjoyed @capttaco (Digital Arch Design) @ #gov20e: “Augmented Reality could be a new public infrastructure” http://bit.ly/18iCx

Also see Tim O’Reilly and Jennifer Pahlka on Forbes.com discuss the The “Web Squared” Era -  “the Web Squared era is an era of augmented reality arriving (like the sensor revolution) stealthily, in more pedestrian clothes than we expected.… …our world will have “information shadows.” Augmented reality amounts to information shadows made visible.”

Again there is back story to how I came to think about Information Shadows in relation to augmented reality.  So in case your missed it the first time, here is the link to a conversation that began in a hallway meeting between Tim O’Reilly, Mike Kuniavsky, ThingM, Usman Haque, Pachube, and Gavin Starks, AMEE, at ETech earlier this year,  “Dematerializing the World, Shadows, Subscriptions and Things as Services: Talking With Mike Kuniavsky at ETech 2009.”

Slide from Augmented City Lab @ Picnic ’09

So What’s Next for Mobile Augmented Reality?

These videos from Daniel Wagner’s team from Graz University of Technology showing Realtime Panorama Mapping and Tracking on Mobile Phones and Creating an Indoor Panorama in Realtime, as Rouli from Games Alfresco points out,  indicate that there is a lot in store for us at ISMAR09.

We may not be so impressed by directory style/”post it” AR anymore, as these applications have become common place so quickly!  But while these early mobile AR apps may be disappointing in relation to some futurist visions of AR – merely AR/ubicomp appetizers,  there are still good implementations of this model coming out (see new comers to the app store Bionic Eye and RobotVision). And Layar, always on the ball, has upped the ante for the new cohort of AR Browsers with Layar 3D.

But as Bruce Sterling notes here:

*In AR, everybody wants to be the platform and the browser, and nobody wants to be the boring old geolocative database. Look how Tim [creator of RobotVision] here, who is like one guy working on his weekends, can boldly fold-in the multi-billion dollar, multi-million user empires of Apple iPhone, Microsoft Bing, Flickr, and Twitter, all under his right thumb

(watch video here)

But if  you looking for something more from AR, you probably won’t have to wait too long.  The two pioneering companies in AR, Total Immersion – founded in 1999, and Metaio – founded in 2003 are both coming out with “mobile augmented reality platforms” in a matter of weeks (see press releases here and here).  And both companies, it seems, will deploy much more sophisticated AR rendering and tracking than we have seen to date.

I approached Bruno Uzzan, founder and CEO of Total Immersion, for an interview as part of my look at the new industry of augmented reality through the eyes of the founding members of the AR Consortium. These consortium members are some of the first commercial augmented reality companies.

The interview below with Bruno began early this summer and then we both went on vacation and it picks up after the announcement of the partnership between Total Immersion and Int13.

The significance of this announcement is that Total Immersion is now positioned to take the augmented reality experiences they have developed for a number of top brands onto multiple mobile platforms with, “Int13′s very clever embedded solution that allows our [Total Immersion's] solutions to work across many [mobile] platforms,” while Int13 gets to extend their reach.

Total Immersion has a 50 person R&D team and their two main focuses have been, firstly getting:

“Augmented Reality to work with as many platforms as possible – PC, Mac, Mobile, Game Consoles, all those are the platforms that we are targeting. We are currently doing lot of work in the R & D team in cross platform compatibility….”

and, secondly:

“Our R&D guys are working on the real world interacting more with the virtual world.  And I have started seeing some results which are pretty much crazy and this will be ready for next year.”

Pandora’s Box – Shared Augmented Realities

Spes or “Hope”; engraving by Sebald Beham, German c1540 (see Wikipedia article on Pandora’s Box)

There are many weaknesses to the mobile smart phone AR experiences we have now, and the lack of near field object recognition (to date), and difficulties with accurate positioning aren’t the only ones.  Note re solving positioning problems in mobile AR, we are yet to see AR  leverage public libraries for analyzing scenes like Flickr’s geo tagged photos, see Aaron Straup Copes’s work on “The Shape of Alpha.” And for more on this my post here.

But, as Joe Lamantia points out:

“One of the weakest aspects of the existing interaction patterns for augmented reality is their reliance on single-person, socially disconnected user experiences.”

In my view, The Pandora’s Box of Augmented Realities is an open, distributed, multiuser augmented reality framework, fully integrated with the internet and world wide web.

As Yochai Benkler has pointed out many times, and argues again in, Capital, Power, and the Next Step in Decentralization, it is “open, collaborative, distributed practices that have been at the core of what made the Internet.”  We have to try to make sure that open, collaborative, distributed practices are at the core of mobile augmented reality.

Can Google Wave be the basis for an Open, Distributed, Multiuser Augmented Reality Framework?

I have been exploring the idea of using Google Wave protocol as the basis for a distributed, multiuser open augmented reality framework with a small group of AR enthusiasts and developers. And I am happy to say the proposal is beginning to get fleshed out a little.  New collaborators are welcome both for “gear heady” input and use case suggestions (but re the latter, you can’t just say everything you see in Denno Coil..!).

This effort started with Thomas Wrobel’s  proposal for an Open AR Framework prototyped on IRC – see here, and click to enlarge the image above of, “Sky Writer: Basic Concept for an Open Multi-source AR Framework.”

But recently we began looking at the Wave Federation Protocol.  And, if you check out this post, and this post, you may get a glimpse of why Google Wave protocol might be a good basis for an open, distributed, AR Framework.  You will notice, if you study what Google Wave has done with the XMPP protocol, that many of  the elements of networked urbanism that Adam Greenfield describes resonate strongly with what is being attempted in Wave.

But enough said for now!  Regardless of the details of implementation,  Google Wave or an AR protocol built from scratch (phew! the latter does seem like a lot of work) -  an open, distributed, multiuser AR framework integrated with the internet and web would explode the potential of AR, creating new possibilities for data flows, mashups ,and shared augmented realities.

And we are excited by Google Wave because, as Thomas puts it:

“The really great thing wave does ….(aside from being an open standard backed by a major player…hopefully leading to thousands of worldwide servers )….is that it allows anyone to create any number of waves, set precisely who can view or edit them, and for them to be able to be updated quickly and continuously (and even simultaneously!) Better yet, changes will (if necessary) propagate to all the other servers sharing that wave. It does all this right now. From my eyes this does a lot of the work of an AR infrastructure already.

I cant see any other protocol actually doing anything like this at the moment, although correct me if I’m wrong, as alternatives are always welcome :)”

Also, Thomas notes, “even the playback system (that is, the ability to playback the changes made to a wave since its creation) …this could give us automatically some of the ideas Jeremy Hight has mentioned in his visionary work here,  and here on “the geo spatial web, interlinked locations and data, immersive augmentation and open source geo augmentation.”

One of the many reasons why an Open, distributed AR Framework would be so cool is it would open up all kinds of possibilities for GeoAR by providing the over-arching standard protocol for communication of updates necessary for the substandards that will facilitate GeoAR.

Also important to note is the  Wave Federation Protocol allows anyone:

“to run wave servers and become wave providers, for themselves, or as services for their users, and to “federate” waves, that is, to share waves with each other and with Google Wave. – “the federation gateway and a federation proxy and is based on open extension to XMPP core [RFC3920] protocol to allow near real-time communication between two wave servers.” See Reuven Cohen’s blog for more here and here, “HTTP is Dead, Long Live the Real Time Cloud.”

Still some people have expressed concern that an AR Framework using Google Wave protocol would give Google disproportionate influence.   Will Google-specific functionality be an issue?  How much stuff is Google specific just because no one else is using it (yet)? And how much is Google specific because it holds no value to anyone else but Google? These are some of the questions that have come up.

You are going to see a variety of suggestions for standards and specs for open AR coming out out in the next few months which as, Robert Rice of the AR Consortium points out is: “a good thing, we need that competition early on to settle down on best case.” Recently,  Mobilizy have offered up an ARML (“an augmented reality mark-up language specification based on the OpenGIS® KML Encoding Standard (OGC KML) with extensions”) for consideration see here.

So it is, perhaps, also important to note, that an Open AR Framework should be neutral/transparent to techniques of  “reality recognition,”  and methodologies of registration/tracking, allowing various ones to work on the system as new techniques evolve, and to support as many evolving standards as possible.

Augmented Reality developers, like Total Immersion and others with powerful rendering/tracking AR software, should be able use an Open AR Framework to exchange the data which their tracking will use. And the tracking/rendering problems they and other researchers have solved are much harder than figuring out data exchange on on a standard infrastructure or protocol!

So I pricked up my ears when I heard Bruno Uzzan, CEO of Total Immersion -  the first and currently the largest augmented reality company, with a 50 person R&D team in France and offices in LA, where Bruno himself is now based, say: “Total Immersion is  only months away from launching shared mobile augmented reality experiences using near field object recognition/tracking across multiple platforms” (for more details read my conversation with Bruno Uzzan below).

I was happy when I asked Bruno about the possibilities for developing an open, distributed, multiuser augmented reality framework fully integrated with the internet and world wide web (possibly using Google Wave protocols), and he replied:

“I think this is feasible. I think that’s doable, that’s just  in my opinion. I mean some people might have another kind of opinion but I think that that’s definitely doable.”

Total Immersion – working with the “symbiosis between augmented reality and brands”

Total Immersion has created many of the best known and most ambitious augmented reality experiences for major brands to date, including Mattel’s new AR toys to be released in conjunction with the James Cameron film Avatar, and AR baseball cards for Topps, video here (or click screenshot above), and the UK’s first augmented reality books.

Bruno founded Total Immersion 10 years ago when he was just 27. And the kind of conviction it took to survive as an augmented reality business in the decade before augmented reality captured the world’s attention is remarkable.

AR’s first steps out into the world after 17 years as predominantly a lab science maybe “wobbly” (what new technology isn’t), and sometimes gloriously kitsch – check out this riotus video of the 3D Interactive Live Show Total Immersion produced in Korea (also see the Total Immersion Augmented Reality Blog for more on the TI’s turn key  Interactive 3D Live Show Solution).

As Lamantia points out here, ” projecting mixed realities into public, common, or social spaces makes them social by default.”

However, the potential for shared location based augmented reality experiences is as yet untapped.  So I see the entry of the most experienced commercial augmented reality company into mobile as pretty interesting.   While  smart phone AR still has significant limitations, and it certainly does differ from some of the futurist dreams of AR (see Mok Oh’s post here on his disappointment in this regard), it is significant that Total Immersion is committing to becoming a leader in mobile AR.

Our smart phones, the powerful networked sensor devices that so many people carry in their pockets, have proved themselves a “good enough for now”  mediating device for early manifestations of the ubiquitous computing and augmented reality base pair.  And now AR and ubicomp is mixed in the  rich, messy soup of everyday life, commerce, business, marketing, art, entertainment, and government, we should get ready to see these technologies grow up fast, and unfold in some surprising ways that lab science didn’t necessarily predict.

And, perhaps, the new dialogue between scientists and entrepreneurs may spur both communities to outdo themselves.

Particularly, as Joe Ludwig notes: “It seems to me that the biggest disconnect between the academics and the entrepreneurs is that they disagree on how far we are from the finish line.”

See the comments’s on Ori Inbar’s post, Augmented Reality Entrepreneurship: Natural Evolution or Intelligent Design?, for  a courteous but spirited discussion on the potential benefits and frictions of the newly expanded AR community of  researchers and  entrepreneurs.

As Blair MacIntyre (see my long conversation with Blair here) notes:

“not all academics and researchers are only interested in the traditional models of impact. Case in point: I wouldn’t be building unpublishable games, nor investing so much time talking to the press, entrepreneurs and VCs if I did not believe strongly in the value of the impact I am having by doing that — and I know others with the same attitude.”

In this vein, check out the Marble Game (video here) developed by Steve Feiner and his team at Columbia U. It’s enabled by Goblin XNA, an open source AR framework built on top of Microsoft’s XNA, which powers XBox live games, Zune games, and some Windows games. For more about Goblin XNA and AR from Columbia U see here.  (Hat tip to Brian Jepson for this link)

While we are still waiting for the kind of sexy AR specs – nothing totally game changing in Gigantico’s AR eyewear rounup (maybe note this Apple patent), that might get wide adoption. But at least researchers are not afraid to explore the possibilities of AR Goggles.

But how far are we now, with or without sexy goggles,  from a fuller expression of the base pair DNA of ubiquitous computing and augmented reality?

We may have a LAN of things before we have an Internet of Things

The picture above is a workshop I attended at Conflux last weekend – Fish ‘n microChips, with Natalie Jeremijenko. We are at the site of the Amphibious Architecture project (a commissioned work for Toward the Sentient City) and “a collaborative project with xClinic, The Living and other intelligent creatures.”

We are probably as far off some grand futurist visions of ubiquitious computing as we are some of the futurist visions of augmented reality. But as it turns out that may not be a bad thing! Recently, @mikekuniavsky noted in a tweet:

“Another argument for the LAN of Things before the Internet of Things: http://tinyurl.com/lgp9uq”

Bert Moore, in the article Mike linked to points out, the grand vision of an “internet of things” with everything connected to everything  can “distract people from thinking about the benefits of RFID in smaller, more easily implemented and cost-justified applications.”  The same argument I think applies to sensor networks and augmented reality.

In New York City, a series of commissioned works for the Architectural League of New York’s exhibit, “Toward the Sentient City” are giving us the opportunity to dip our toes into the ocean of a “networked urbanism.”   For only a small budget, two of the five commissioned works, Amphibeous Architecture and Natural Fuse demonstrate how sensor networks can allow us to explore new kinds of communities – connecting people to environments in interesting ways to create new forms of social agency.

“Amphibeous Architecture” -  from The Living Architecture Lab at Columbia University Graduate School of Architecture, Planning and Preservation (Directors David Benjamin and Soo-in Yang) and Natalie Jeremijenko, Environmental Health Clinic at New York University, uses a skillfully built (electronics and water are notoriously hard to mix) array of partially submerged sensors to pierce the blinding, reflective surfaces of the rivers  surrounding Manhattan and to create a new two way relationship with the ecosystem below – the water, our neighbors the fish and even a beaver that lives in the water surrounding Manhattan.

Image from Toward the Sentient City

In a similar spirit, “Natural Fuse” – Usman Haque, creative director, Nitipak ‘Dot’ Samsen, designer, Ai Hasegawa, designer, Cesar Harada, designer, Barbara Jasinowicz, producer, creates a network of people and electronically assisted plants to explore what it takes to work together on energy consumption and to experience the consequences of “selfish” and “unselfish” behavior interactively before it is too late to modify our actions.

The “Greedy Switch“ from Natural Fuse on the left. On the right “The System” – click to enlarge.

Much more to come in another post on these works, and “Toward the Sentient City.”  Also an update on how Pachube – an important part of both these projects and a very important contribution to ubiquitous computing because it creates the opportunity to connect environments and create mashups from diverse sensor data feeds – has matured since my interview with Pachube founder, Usman Haque, “Pachube, Patching the Planet,” in January this year.

In the picture above Natalie Jeremijenko, and Jonathan Laventhol give the Amphibious Architecture sensor array a last look over, as it will soon be lowered into the East River. Jonathan is on a busman’s holiday to help out at the pre launch of Amphibious Architecture, nr Manhattan Bridge, NYC.

I was very happy to get  a chance to talk to Jonathan Laventhol - more on our conversation in another post. Jonathan Laventhol is CTO of Imagination – one of the world’s leading design, events, and branding agencies. We talked about the importance of  Pachube, which Jonathan called the “The Facebook of Data,”  and  how the symbiosis between brands and augmented reality, and healthcare applications, would  be key to augmented reality emerging into the mainstream.

Natalie Jeremijenko’s workshop at Conflux on the social negotiation of technology and how “everyware” can give us the chance to experience new forms of agency and connection was a totally inspiring.  And I will cover this too in another post.  I have so much awesome stuff  to write about at the moment!

None of the projects in, “Toward the Sentient City,” included a mobile augmented reality, or “magic lens” component, but they all pointed to why “enchanted windows into our newly inside-out reality” are going to be so important. And why the DNA base pair of ubicomp and augmented reality can really do stuff that matters.

Shangri- La – “Transfigured City”

Screenshot from Shangri-La episode Transfigured City

In my AR Consortium founder member interview series, I have found that, understandably, the visionary founders of these first augmented reality companies are a little reticent about sharing their full vision.  They are basically on stealth mode in this regard.  So as you will not, from my interview with Total Immersion founder and CEO, Bruno Uzzan, get a fully drawn scenario of his vision for a next generation of shared augmented reality experiences, here’s a really interesting anime episode from the anime Shangri La called, Transfigured City, to mull over instead.

As you can tell from this rather long and circuitous intro to my my conversation with Bruno Uzzan, I  have been investigating shared augmented realities pretty intensively recently. And Mike Kuniavsky pointed me to Shangri-La, and Transfigured City, in a conversation with Mark Shepard, after Mark’s presentation at Conflux, Sentient City Survival Kit.

Mike Kuniavsky with Tod E. Kurt is founder of ThingM, a ubiquitous computing device studio. Also Mike Kuniavsky researches, designs and writes about people’s experiences at the intersection of technology and everyday life – see Mikes blog Orange Cone.  And I interviewed Mike at Etech- see here.

In Transfigured City, the “Metal Age” group has to figure out how to share and communicate in a city transfigured by augmented realities/virtualities, where no-one sees the same place in the same way.  Only one character can figure out from her previous experience of the city the relationship between the transfigured city and how it used to be.

The conversation I had with Mike Kuniavsky on The Transfigured City continued at a picnic in Washington Square Park the next day with Elizabeth Goodman, who I met at Etech when she gave a brilliant presentation, Designing for Urban Green Space.  We covered so many areas at the picnic related to ubiquitous computing and augmented realities that this conversation probably deserves a post of its own (my writing to do list is growing longer!).

The Plot Synopsis for Shangri La:

“In the mid-21st century, the international committee decided to forcefully reduce CO2 emission levels to mitigate the global warming crisis. As a result, the economic market was transferred mainly into the trade of carbon. A great earthquake destroys much of Japan, yet the carbon tax placed on the country is not lifted, so Tokyo is turned into the world’s largest “jungle-polis” that absorbs carbon dioxide. Project Atlas is commenced to plan the rebuilding of Tokyo and oversee the government organization, which the Metal Age group opposes due to its oppressive nature. However, Atlas is only built with enough room for 3,500,000 people and most people are not allowed to migrate into the city. The disparity between the elite within Atlas and the refugees living in the jungles outside of its walls set up the background of the story.”

Talking With Bruno Uzzan

Tish Shute: We won’t have fully opened the Pandora’s Box of Augmented Realities until we have ubiquitous, shared augmented realities, will we?

Bruno Uzzan: Yes. The most important for augmented reality is the experience we want to share. Now we are working on the cell phone, we can potentially do some marketing components that we already have developed now on cell phone. Done. It’s working.

But the most interesting part of it is how these new components [cell phone AR] will be used for marketing campaigns by brands. And we are also pretty much well positioned to transform some of the AR that we currently have working on Mac and PC and to transform these to applications working on mobile devices.

Tish Shute: We haven’t really experienced yet what it means to actually share mobile AR experiences?

Bruno Uzzan: It’s hard — we did a Facebook app. It’s a first try, it has a way to go.  But to go more and more into social, is the way forward for us – to share and expand AR experiences. But yes, I mean what you’re seeing is how two people on two different applications can share that same expanse.  For sure we are going in that direction. We are currently working on those kind of solutions. How people can share and experience together at the same time. That’s how we start creating excitement in augmented reality, and it’s coming up.

It’s a new market and there’s so much more in store for augmented reality. You know, some people are telling me, don’t you believe that augmented reality is a gimmick? It will be a trend for a few weeks or a few months and then gone? I say, you’re kidding me. This is only the beginning. I mean I can assure you that the applications that are on the market today are one percent of what we will have five years from now.

Tish Shute: I agree.

Bruno Uzzan: And I’m sure that augmented reality will be a part of a lot of components that we are currently using today – GPS, web browser, glasses, I mean there are so many applications that will come up shortly. This is only the beginning. I’m completely convinced that augmented reality will be in three years from now what virtual reality is today, which is a billion dollar market.  I know that it’s not just a gimmick of a few weeks or a few months, because so many brands are jumping into it, spending money, exploring solutions.  I know that it’s not just short term -what they are willing to do and we are willing to do, but also middle and long term. And that’s what makes this adventure pretty much unique and what makes creating a cutting edge technology, very, very much exciting for us.

Tish Shute: First could you explain more to me about your partnership with Int13. I am not sure I understand what is in the arrangement from Total Immersion’s POV. I mean what happens re your own mobile software development? Haven’t you only been licensed the Int13 SDK for a limited period of time and have limited access to all it’s power? Stephane from Int13 said to Ori on Games Alfresco, here, “we have licensed the SDK4 for two years,” and then Ori asks, “but you have basically kept the power to yourselves, right?” So if they are the only ones that can enhance it and develop the software, where will  TI be in two years in mobile if you haven’t really had the chance to develop your own software .

Bruno Uzzan: Actually it’s a real win-win situation. Int13 is a very small company and they have so many requests they can’t possibly fulfill them all. So  this is a way for both of us to be, as quickly as possible, the first mobile provider for all the requests we have. Also they give us exclusivity so nobody else can use INT13 SDK for such applications.  I think that it is a good partnership,

And concerning our own mobile application… First of all we have currently some mobile applications working. But with Int13 we have a mobile solution that can work on many different devices. That’s a fact and that’s working. And, believe me you will hear from us a lot more about this soon. We are fully independent on our mobile development. The reason we closed the partnership with Int 13 is  to be able to deploy mobile in a broad way.

I mean you know that the difficulty with AR mobile is that each separate device needs some customization. Working on the iPhone is different from working on the Nokia, different from working on the Palm; it’s different from working on the Samsung. Each of them have their own operating system inside and so we were interested in Int13′s very clever embedded solution that allows our solutions to work across many platforms.

The reason we are working with Int13 is that we are able to work on so many mobile devices, thanks to Int13. And in the mobile AR race that we are currently in, the next two years will be extremely important to us…

Tish Shute: OK, that definitely clarifies it a lot. So Int13 has done an embedded solution to allow TI developed AR solutions to work easily across many devices?

Bruno Uzzan: Yes  they have kind of an embedded solution, a way to address extremely quickly new cell phone… But, currently on our side, we are in discussions with a mobile company… and that only refers to some very specific mobile devices.  And what they have is also a way to embed deeper our technology into mobile, so that we can have quicker… applications that work on a large number of cell phones.

Tish Shute: So, basically it means you don’t have to go through some complicated negotiations with each of the cell phone companies, is what you are saying?

Bruno Uzzan: Not only negotiations, but also hard development. You know? Working on the Windows mobile is completely different from working on the Palm OS. You know, that’s different! Its a big work, to have a mobile application working on many other devices. So, INt13,  provides us a way for us to save some time and some development cost too.

Tish Shute: And Int13 doesn’t have powerful AR development tools like D’fusion right?

Bruno Uzzan: Right! That’s right. That’s why we say it’s a true win-win solution. They can benefit from our work too. And we can benefit from their work, in order to deploy quicker and faster mobile solutions.

Tish Shute: Now, the second thing is… there is a lot of debate and disagreement about how far mobile augmented reality is from delivering something more that the “post it” approach that has been much publicized in recent months, via all the AR browser apps.

But from my understanding from the conversation we had earlier this summer (see below), Total Immersion is targeting a much higher level of mobile augmented reality than we’ve seen to date?

Bruno: Yes the browser apps we have seen are a kind of augmented reality, but not exactly the way we see it. Let me explain you why. With this kind of application it’s true that you can overlay 3D-information and video. That’s a fact. So, in a sense, that’s augmented reality. But the way that they are working on the position of the 3D on that video is that they are using compass and GPS-information.. so it means that this AR solution will work only on some building and some physical objects that are FIXED. In a fixed and known position.

So you want to go to a theater?

The theater is here, for sure it will not move, so you know the position of the theater, and that’s a fact that you can superimpose an object on the theater. That’s what can be done currently. What we are achieving and what we are doing on mobile is more than that. We want to be able to port our solution with trading cards, with brands, into a smart phone.

I’m assuming that you want a can, a drink can, to be able to trigger an experience. The only way you can do it is to be able to understand what the can, it is. And the current solutions that are out there can’t do that, it’s impossible.

Tish Shute: Right, yes. There’s no near-field object at all in these early browser apps.

Bruno Uzzan: And the solution we have is that we can recognize a can and then — in a very, very precise way and that activates geo-location, so we can superimpose 3D. I mean in that case, it opens up all the applications that we currently have, so they could work on mobile.

Tish Shute: So for example, if you’re working with a soft drink company, people can trigger that experience wherever they see that can?

Bruno Uzzan: Correct.

Tish Shute: Yes. Yes, I assumed that was what you’re doing

Bruno Uzzan: We believe — and maybe that’s not the case, but we believe that our marker-less tracking technology is pretty much unique on the mobile devices.

I haven’t seen yet, from anyone, a full augmented reality mobile solution working.

I really see AR being part of the Web 3.0 next generation. I mean the vision I have is that, you know — today, when you want to have information, you go on a website and then you find your information. AR — and the future is that I think it will be part of the opposite. You want to have information about a product, you just show it to your computer and the information will automatically pop up. I see here a new way to market some key messages, a new way to get information is that some physical product by themselves could be a way to get information, and you don’t have to search anymore for them, it’s coming out to you.

AR is definitely for me, one of these components. Another thing that AR is a solution, another thing that AR itself will create these kind of results in how information is being displayed. But I’m seeing  here a way that could be part of a new way to have access to information. And that’s part of the vision I have. Whatever, if it is through mobile phone or web or PC, Mac, whatever, I really believe that now this kind of new generation of receiving information will come shortly and could be a kind of a new — could be part of the new 3.0 generation of the web.

Tish Shute: My friend Gene Becker did an interesting post recently on some of the current limitations of mobile AR where he pointed out the problem of:

“Simplistic, non-standard data formats – POIs, the geo-annotated data that many of these apps display, are mostly very simple one-dimensional points of lat/long coordinates, plus a few bytes of metadata. Despite their simplicity there has been no real standardization of POI formats; so far, data providers and AR app developers are only giving lip service to open interoperability. Furthermore, they are not looking ahead to future capabilities that will require more sophisticated data representations. At the same time, there is a large community of GIS, mapping and Geoweb experts who have defined open formats such as GeoRSS, GeoJSON and KML that may be suitable for mobile AR use and standardization.”

Bruno Uzzan: That’s interesting. I mean — I know exactly what his is referring to. He is mainly referring to a localization and how you can have a quick, accurate localization.  If you look at current solutions, and you look at this 3-D superimposing on the video, the 3-D is shaking a lot. I don’t know if you see that in some of these early efforts.

It’s hard to use because the 3-D, you know, is  part of the magic of augmented reality, that is when the 3-D is being inserted in a very easy way and smooth way in your solution. Here, when you see this overlay, 2-D or 3-D overlaid on the video, it’s shaking a lot. One reason for this is that the GPS compass is not accurate enough to coordinate the perfect location of the user. And here, what Gene says is interesting. I think we are addressing this localization issue in a pretty smart way.

But to be frank with you, I don’t believe mobile augmented reality in the extremely short term — I’m talking about three weeks, one, two months is mature enough for good AR applications.  It will be shortly.  But for now it is more proof of concept than a true and easy application to use.

But we are starting to see a lot of new application coming out, but I really believe that marketing and entertainment are the two key markets for AR right now.

I’ve been working ten years in augmented reality. And, eight years ago, when I was talking about augmented reality, I was E.T., you know? Nobody understood what I said, and I thought it was crazy. And now, today, yes it’s completely different.

Tish Shute: The Pandora’s Box of Augmented Realities, in my view, is an open, universal and standard, distributed, multiuser, augmented reality framework fully integrated with the internet and world wide web. I have been looking into Google Wave protocols as a basis for this would you be interested in this? Do you think it is feasable?

Bruno Uzzan: I think this is feasible. I think that’s doable, that’s just  in my opinion. I mean some people might have another kind of opinion but I think that that’s definitely doable.

Tish Shute: Yes I suppose an open AR Framework involves cooperation and collaboration, it is more about business and politics than technological problems.

Bruno Uzzan: Yes!  Actually the Web is politics. Business is politics.

Tish Shute: I would be interested if anyone in your R&D team would be interested in looking at some of the ideas that are emerging in our little discussion of Google Wave and an Open AR Framework  to offer feedback. it is an interesting time now to input on the Wave Federation Protocol docs because nothing is set it stone right now.

Bruno Uzzan: Just shoot me an email, I’ll try to put you in touch with the right person and, and a team member that can input on this.

Tish Shute: For mobile augmented reality the best thing we’ve got now is the phone, right?

Bruno Uzzan: Right.

Tish Shute: And the only way we can use the phone is by holding it up, right?  Isn’t this a bit of an an obstacle as you introduce better object recognition and tracking?  People are going to have to stop moving to use their phone. What do you feel about that experience? Isn’t AR eyewear and essential part of a tightly registered AR experience?

Bruno Uzzan: We don’t do hardware and we don’t have the current solution for eyewear that would do all we need for a good mobile AR experience, so I guess we don’t have the current answer for that.  But we are beginning to see the next generation of this — of these glasses.

Tish Shute: But you’re happy enough with the mobile experience of augmented reality on smart phones that you’re investing in this next generation of software for this.

Bruno Uzzan: Yes, I know. We know that some application will not work on the iPhone. And yes, whatever you do, you still need to hold the iPhone, so it means that you can’t play with your hands anymore. So we know that partially, some AR solutions  we have on other platforms will lose the magical effectivities on just the iPhone.

But I’m starting to see on the market some glasses that could perhaps be not too expensive — that’s a challenge!  And easy to use — that’s another big challenge. And, that could fit on anybody’s faces and head — there’s another big challenge. So yes, I’m starting to see that, but so far AR glasses are only applicable for some very, very specific application, like design or theme park or, you know, some specific location where it makes sense to move forward with glasses.

I don’t believe that kids will use glasses for — in our toys and for games in the next months or maybe othe next one or two years. But maybe something will come out shortly and that could be a big breakthrough, and enable us to think another way. But  from what we have seen so far and from what we know in this hardware market, I don’t believe that currently there is a workable solution.

Note: The following section of the interview took place earlier in the Summer.

Tish Shute: You are the first commercial AR company  – you started in 1999 right?

Bruno Uzzan: Yes you are right. We started the extremely early in this augmented reality market. We were the first company worldwide to start doing augmented reality and to start promoting augmented reality. So it’s true, we are pretty old players although the market has been getting bigger and bigger for the last year and a half. So for a long time we were only in the market, and the market was not really there.

But for the past 8 months, the company has been growing really fast.

Tish Shute: Yes I’m sure. Congratulations for hanging in there long enough to get the pay off!

Bruno Uzzan: You know, my background is Financial. So I have been driving the company for many years in a very cash efficient way. So we have been waiting for the markets to reach maturity before starting make some investments. So that’s the reason we are still here, and that’s the reason I think we managed pretty smartly the cash that we raised for the company.

Tish Shute: Yes there is a saying that when a market takes off you can tell a pioneers because they are the ones with the arrows in their backs. But I am glad you are dodging the arrows!

Bruno Uzzan: You know, I’ve always driven the company with revenue. And because revenue was not there at the beginning I was extremely cautious about the cash. So now that the company is getting some revenue, for sure we are making more and more investments, and taking advantage of our situation as a worldwide leader of augmented reality.

This situation is not easy as it appears today but it’s now getting better, as you can see, AR, Augmented Reality, has very good momentum and we are benefiting a lot from all this momentum for augmented reality right now.

Tish Shute: You’ve been very involved in researching developing augmented reality tools. Are you still as active in the research area, or are you too busy keeping up with work for hire now, to be working on research and building new technology for Augmented Reality?

Bruno Uzzan: Both. First of all, we are part of lot of projects either directly with clients like Mattel or with some partners that are using our technology to promote and develop other AR projects. From what we he have seen, many, many, many, projects augmented projects have been done currently with our solutions.

To continue with your previous question. So we are being perceived as this leader in that space, and we  have some pretty heavy demand for our services. But we are coming up with new technology, of course, still connected to Augmented Reality.  But, our R & D is working in two different directions, which of course also bind together.

The first one is platform developments. So we want Augmented Reality to work with as many platforms as possible – PC, Mac, Mobile, Game Consoles, all those are the platforms that we are targeting. We are currently doing lot of work in the R & D team in cross platform compatibility.

Tish Shute: Robert Rice said recently, “markers and webcams equal Photoshop page curls…”

Bruno Uzzan: Yes. There are so many concerns with markers. The quality is extremely bad. As soon as you hide a part of the marker, a slight part of the marker, you’re dead. You can’t track any more of the object. So compared to our solution where I want to say play with cards or where you are going to play with a Mattel toy, even if you hide a part of the toy, it’s still working.

Tish Shute: But you haven’t offered the public an SDK to your engine right? Basically the way people get access to your tools is working in a partnership with Total Immersion right?

Bruno Uzzan: Correct.

Tish Shute: Do you think in the future you might open your SDK? Are you considering that?

Bruno Uzzan: Yes, it would be interesting.

Tish Shute: So that is something we can see coming soon?

Bruno Uzzan: Maybe, because it’s true that Total Immersion is starting to be mature enough for these kind of tools. The only thing is that we have to respect good timing for that.  It’s a big decision. You know what I mean?  It is a big, big decision. We would then compete with others using our technology.

Tish Shute: Oh I know, it is a big decision when you have so much skin in the game! But it would be nice to have your SDK being THE platform for AR, wouldn’t it?

Bruno Uzzan: It is a really big decision that we can’t just take like that, you know.  There are a lot of friends who told me you have to be extremely careful about timing. This timing is pretty much connected to the maturity of the market. For sure, we see the market being more and more mature. But, there are a lot of low hanging fruits we still want to address

To get the best value possible for all the publicity we have and all the clients we have now.

Tish Shute: Yes, I know. You’ve been in this game so long. Now, there is an interesting question here though about tools and platforms because you know, A.R., augmented reality has already expanded  beyond its kind of original purist definition. And when I talk to people  about augmented reality, there are actually lot of different ideas and priorities of where the tools should go right now. You know, obviously we have these kind of browser-like applications, but these browser like applications are not dealing with recognizing near field objects yet.  What are your priorities for tool development and what are your priorities for AR development in the future? What areas are you going to focus on? Oh dear that is a rambling question!

Bruno Uzzan: [laughter]  So, one of our first priorities is we need to create our software with one development, one installer, one software that can be spread on different platforms. The same application, the same software can be used either on a PC, Mac, phone or console. So that’s a lot of work, because that means that our platform has to address many many different devices and that’s a big priority for us because we received this request from our clients. We want to be able to use one application on many different platforms and devices. So, that’s the first one.

And the second one is to add more and more interactivity between the real and the virtual world. So, we are working on some improvements to add some real components that will interact with virtual, and that also part of our big strategy and direction and these two worlds can more and more be bridged together, linked together so they can interact  one with the other.

Our R&D guys are working on the real world interacting more with the virtual world.  And I have started seeing some results which are pretty much crazy and this will be ready for next year.

There are so many different directions for interaction between the real world and virtual world to develop.  I’m sure ten years from now you’re going to have AR applications everywhere.  Its not just temporary fashion stuff or a gimmick for few months. I mean we are getting there, its getting stronger and stronger and we are getting a good adoption rate from our consumers. They like it, they test it, they play with it and brands wants more, people want more and its getting bigger and bigger.

Tish Shute: Yea and I totally agree, its not a gimmick because the interaction between “virtual” and “real” enhances the magic of both. Another question about you RandD operation. Is your R&D still in France or have you moved totally out to LA.

Bruno Uzzan: We are 50 people in France and I started this LA office two years ago and I moved permanently two years to LA. So I’m now permanently located in the US to take care of the US office, knowing that revenues are really getting bigger and bigger in the US. So it means that we are getting a lot of traction, working with large company and now I’m currently located in the US.

Tish Shute: My sister lives in Paris. Could I visit your R&D lab at some point? I’d love to visit!

Bruno Uzzan: Yeah sure sure sure. I mean if you want to go. You won’t have access to all the research. But if you want to go out and meet all the team please do.

Tish Shute: I’d love to.

Bruno Uzzan: No problem. Shoot me an Email you and I will introduce you to Eric Gehl, COO, he is the COO of the French team. And he can definitely take care of that.

Tish Shute: That would be fun. Thank you!

Recently, AR browser applications have really caught the imagination of the web community, eg., Layar and Wikitude?  Where do you think the most important market for AR is at the moment, entertainment,  green tech, business, education?

Bruno Uzzan: I think that all that you mention will be important. The first one that did grab my attention is entertainment particularly dual marketing, because they always searching for new ways to interact with players or the consumers.  But it’s just the tip of the iceberg, you know, I mean medical applications could be huge using augmented reality. Education, and edutainment is definitely using more and more augmented reality components.  And I will just be submitting with big companies – that are considering using augmentation for education. Museums are very important too. Also augmentation as a kind of free sales tool, you know there are so many applications, design, architecture – so many directions that it’s hard to say today which one will take the lead.

But I do believe that on the short term the ones that are really really moving fast are the entertainment business and the digital marketing business.

Tish Shute: What do you think are the biggest shortcomings with current augmented reality and what are the obstacles that no one has solved yet?

Bruno Uzzan: I think the cell phone is not fully ready for augmented reality – a lot of people are working on that but there are still a lot of constraints to get the augmented reality working on a cell phone and I think that from what I heard a lot of manufacturers and a lot of companies are working from direction that are going to help us a lot to develop some great cell phone applications.

And I think that’s one of the biggest part of the game. All the applications that you see on cell phones so far are just gimmicks – the next big key is how to transform some gimmick cell phone application to a real, industrial, robust application that’s going to work on a cell phone. So I think that’s a big challenge for this year.

Most of what we see now is just matching and overlaying some 2d components in a video. This is not what I call AR.  You’re far away – with this kind of application, you are far away from doing the registration that we need to do – you can’t do it. So here’s the challenge: “how can you get a Topps is an application working on cell phone. That’s the big challenge  How we can make that work!” You can’t today get a real AR Topps application working on cell phone because there’s no cell phone  that’s actually ready. But we are working on it and the first one that can make that work, it’s going to be huge.

When you are working with good AR components you need a lot of CPU and GPU programs. So today new cell phone have started to be more and more ready for augmented reality but you need a really good cell phone to make it work. You can’t choose an old cell phone to make it work because you have some recognition, you have some tracking, you have some rendering, so you can’t choose a Nokia cell phone two years old to make that work. For sure the newest iPhone is the one that can make it work, but that’s it for now. There is a lot of research – from large cell phone companies – to get more CPU and GPU into their cell phone.  But so far we are also waiting for these devices to be released to consumers.

Tish Shute: And the current economic climate has put a damper on MIDs hasn’t it. But who can tell? It depends what price points some new MID came out at right?

Bruno Uzzan: Correct.

Tish Shute: Yes,I agree. But basically what’s interesting, the interesting thing is, the iPhone can deliver so much of what is necessary and even if Apple hasn’t given access to the full power of the iphone to AR developers yet, there is really no going back now – the mobile augmented reality cat is out of the bag!

Bruno Uzzan: You’re right, you’re fully right.