Planetary from Bloom Studio, Inc. on Vimeo.

It is just over a week until Augmented Reality Event, and I know there are a lot of people, including me (full disclosure I am co-chair and co-founder) who are totally psyched to see what unfolds there this year.   Bruce Sterling, Vernor Vinge, Blaise Aguera Y Arcas,  Jaron Lanier, Will Wright, Marco Tempest and Frank Cooper will join 107 speakers from 76 augmented reality companies on a single stage (see my previous post) to tell a momentous story of a technology of our time (also see my previous post here).

As Bruce Sterling points out, Augmented Reality is “truly a child of the twenty-teens, a genuine digital native,” and one visible indication that:

..the Internet really could look like a “legacy.” The Legacy Internet as an old-fashioned, dusty, desk-based place best left to archivists and librarians, while the action is out on the streets (see the full interview below).


(photo by Jasmina Tesanovic)

Opening this post is a video of Ben Cerveny’s Planetary app, which “turns your music into a universe,” and enchants all who try it.  Planetary shot into #3 on the Top Ten Free ipad app list soon after its release.

Ben Cerveny’s talk at Augmented Reality Event will be one of the must attend talks (see the full schedule for Augmented Reality Event here, and note my discount code for Augmented Reality Event, TISH295, is still good, if you want to register).

Planetary, while it is not an AR experience,  points the way for AR to take us out of the old-fashioned, “Legacy Internet.”

“Planetary is just the sort of science fiction experience you expect when using an object from the future like iPad,” developer Bloom Studio writes on the app’s iTunes page. ( from Mark Brown’s Wired post).

In his interview on cnet Daniel Terdiman, Ben describes how popular computing will evolve beyond those, “dusty, desk-based place best left to archivists and librarians,” (Bruce Sterling).

Ben points out:

“The tablet is a total disruption of how we understand popular computing. The next era of experiences will be driven by visceral gesture-based input, and rich fluid responsiveness in native graphics contexts. I see the potential for Bloom to help define a “killer pattern” for application design. Because apps have been deconstructed into discrete tasks that flow across devices….”

Bruce Sterling had some interesting comments on the Bloom app:

I’m a big fan of Ben and his good works in infoviz — and urban informatics, too. I admit I’m not sure the I entirely need the metaphor of a solar system in order to play a few Texas blues tracks. But I could be persuaded. Ben Cerveny is a significant thinker and a very well-spoken guy.

I’m not saying that happens tomorrow; I’m just saying, why not? Why not aspire to that?

I too am a huge fan of  The Bloom team, Ben Cerveny, Tom Carden, and Jesper Sparre Andersen (also see my post here about Fizz, the Bloom team’s app used by The Locker Project for their Strata demo).  And, if you haven’t already heard about The Locker Project and Telehash – get on it!  This is one of the most important projects of our time – an infrastructure for a better future!

Interview with Bruce Sterling by Tish Shute and Ori Inbar

Tish Shute: As you so memorably put it, “AR is a technovisionary dream come true – something really rare, and you have to be really patient for those….”

What is best and worst, in your view, about the way Augmented Reality technovisionary dream is coming true and emerging to flourish in the wild?

Bruce Sterling: The best part is that AR is truly happening and is a lot of fun, and the worst part is that it’s happening in a Depression. If AR had broken loose in the dotcom days when cash flew around like soap bubbles, man, that would have been psychedelic.

AR that is even more of-our-time than “social media.” AR has arisen directly from modern technical factors that just didn’t use to exist. It’s made from shiny new parts, and is truly a child of the twenty-teens, a genuine digital native. It’s a little kid and it has to walk before it can run, but it’s great to see it walking.

Tish Shute: As Jesse Schell pointed out last year at ARE2010, “The whole point of AR is to see things from a different point of view…How can there be a more powerful art form than one that actually changes what you see?” What do you feel will be the most impactful application of AR in people’s everyday lives?

Bruce Sterling: I’m all for impact, but it’s pretty clear that the people who would weep for joy to have Augmented Reality are people whose reality is already damaged. People who need reality augmented as a prosthetic, in other words, so that they can achieve an “everyday life.” This is like the impactful but underappreciated role of the Internet in the lives of people who’ve been shut-in. If you’re laid-up in a hospital bed, a laptop is a revolution in convalescence.

But that kind of “impact” doesn’t sound too exciting or too profitable. My guess would be that the biggest arena for “impactful AR” would be augmenting cityscapes for foreign people who can’t speak the local language, can’t read the signs, and lack time to learn the local reality. Imagine, say, the Brazilian overlay for Moscow. You could show up, read your native Brazilian overlay of that city, do your business, eat, sleep, buy, leave, and scarcely “be in Moscow” at all. Constructed right, the AR Brazilian Moscow might even be a better Moscow — a Moscow that Russians themselves would pay to visit.

Tish Shute: You pointed out last year, in your opening keynote for ARE2010, that less immersive forms of AR have their own merits. We are still not seeing much “head mounted display weirdness” yet, but many other forms of AR are emerging – mobile, webcam, projected video, sonic augmented reality, even sticky light. You noted, practically everything that AR is involved in is a transitional technology. But since you spoke last year at ARE2010, which of these transitional technologies have shown the most promise for AR?

Bruce Sterling: It’s got to be handsets. Smartphones. The stats there are just amazing. The smartphone biz makes the personal computer business look like a Victorian railroad. When I read a guy like Tomi Ahonen, who talks about transitioning out of the old-fashioned “Legacy Internet,” that idea is startling. But AR is one visible indication that the Internet really could look like a “legacy.” The Legacy Internet as an old-fashioned, dusty, desk-based place best left to archivists and librarians, while the action is out on the streets.

Tish Shute: This year we have seen gestural interfaces go mainstream. What are the most interesting directions for gestural interfaces that you have seen emerge in recent months?

Bruce Sterling: To me, the most “interesting” part is seeing people do gestural stuff in public. William Gibson, my fellow author, observes that cellphones have stolen the gestural language of cigarettes. There’s lots of fidgeting, box tapping, ash-swiping, slipping boxes in and out of pockets… People quickly learn to do that without thinking twice, and they forget how weird it looks. It’s “design dissolving in behavior,” as Adam Greenfield puts it.

The gestural hack scene for the Kinect has been amazing. It’s like watching 1950s Beatnik dancing go mainstream.

Tish Shute: You have observed that Augmented Reality is Glocal which not only gives us different flavors of augmented experience but is “a departure from earlier models of tech startups, where you usually have like three hippies in a local garage. Now you’ve got German-American-Korean outfits like Metaio, and Total Immersion has a Russian affiliate. They’re inherently multinational, both inside the company and out.” What flavors of glocalness do you hope/expect to see at Augmented Reality Event this year.

Bruce Sterling: I’d be pretty happy to see some AR input from Brazil, India, and South Africa. I seem to be picking up a lot of followers in my Twitter stream from those locales. If I saw some Augmented Bollywood Reality, that would pretty much make my day.

Ori Inbar: What sessions will you go to at ARE this year? Who do you want to meet at ARE 2011?

Bruce Sterling: I make it my business to hang out with artists, but I’m hoping to drill down more on the technical aspects. For instance, where exactly are the bottlenecks in building animated augments? It looks like we’re about a sneeze away from jamming some crude Hanna-Barbera cartoons into real spaces. But the devil is in the details there.

Ori Inbar: Your commentary about the evolution of the AR industry over the years had significant focus on style. Is the AR industry dressed to kill yet? Any glimpses of promise in that direction?

Bruce Sterling: I’m not “pro-style” in every possible aspect of life, but as an Augmented Reality critic, it’s clear to me that if you claim to “augment” reality, then you should work hard to augment it — struggle to make it better. Otherwise you might as well call yourself “Defaced Reality,” or even “3D Spam.” When I see that kind of crudity and carelessness in AR, I’m gonna call people out on it. I know there will be the AR equivalent of cheesy billboards and gang graffiti, but I never much cared for those, either.

The industry’s videos have improved radically in the past year and a half. It used to be all about “look at my grainy, shaky handheld video of my cool new AR hack,” but nowadays the biz has really pulled its socks up.

If AR is about “experience design,” as I think it basically is, then eventually, as a matter of intellectual consistency and professional pride, everything you create will be considered part of “the experience.” That’s the industry’s way forward — that’s what it would do if it was grown-up.

AR people already look better than most similar geeks in the gaming business, and some day, I really do believe that augmentation people will become glamorous. They won’t be supermodels, but they’ll be about as chic as, say, professional set designers. Because AR is set design, in a way; it’s real-time interactive set-design for three-D spaces.

Ori Inbar: In the Layar Launch in 2009 you said “it’s the dawn of AR…”, at ARE 2010, you followed up on the theme saying “it’s 9am in the AR industry.” What time is it now?

Bruce Sterling: I’d be guessing it’s around 9:30 AM, but come on, that’s just a metaphor! ARE we all gonna blow off at 4:30 PM and have a beer, or is AR one of those cruel tech startups where nobody ever gets a personal life?

Ori Inbar: Are you reading any new fictional literature about AR that inspires you? And/or What interesting design fictions for AR have you come across recently?

Bruce Sterling: Well, I’m always interested in creative people who just plain make stuff up. Because that’s why I commonly do myself. The stuff that “inspires” me is usually stuff that I just didn’t expect to see. But when I don’t expect it, that usually means I wasn’t paying enough attention. I plan to pay a lot of attention to AR this year.

I’m not sure it makes a lot of sense to write fiction nowadays “about AR,” because it’s no longer a fictional topic. It’s become like writing fiction “about cinema.” You can write good fiction about someone who works in cinema, but not fiction about cinema itself. AR is not sci-fi “Augmented Reality” any more, it’s become a real-world phenomenon, a new industry of real augmentation.

With that said, I must remark that I sit up straight whenever I see Marco Tempest do stuff. Magicians are all about mystery and wonder. You wouldn’t see a magician, say, using AR to work an assembly line, or re-order library books, or find a pizza joint in Barcelona. And that’s great. Marco is always gonna do something freaky and out-there, and even though he’s a tech magician, it’s never about the tech first. It’s always about his ingenuity in finding new ways to employ new tools in creating a magical experience for his audience.

Marco’s not an entrepreneur, he’s not gonna revolutionize people’s daily lives or invent Web 4.0, but even if AR becomes “old hat” some day, it’s never going to be old hat when he’s doing it. The guy is a pro, and I’m quite the fan.

Magic Projection Live @ TEDxTokyo 2010 from Marco Tempest on Vimeo.

The Missing Manual for The Future (or The Future: The Missing Manual)

O’Reilly Media, is famous for is producing  “missing manuals” for new technologies, but thinking of O’Reilly as just a publisher of books would be like saying Facebook is just a website (this came up in the discussion at Media Round Table at Web 2.0 Expo, NY, 2010).   In recent weeks, I managed to catch Tim O’Reilly at several events, Maker Faire, Web 2.0 Expo, Hadoop World, and the free webcast Tim did with John Battelle on The Battle for the Internet Economy (although Tim spoke several other times during this period!).

It occurred to me, as I immersed myself in the depth and breadth of innovation showcased and discussed at these events that Tim O’Reilly, and the O’Reilly team, are creating, The Missing Manual for the Future.

As Tim puts it, we are “changing the world by spreading the knowledge of innovators.” Tim uses a quote from William Gibson to illuminate what is at the heart of the O’Reilly project:

“The Future is here, it is just not evenly distributed yet.” (William Gibson).

But Tim O’Reilly makes another point about the future when he speaks.  The future unfolds unexpectedly – so we must invent for an unknown future not a known future, or as Alex Steffen put it so well in his post, Why Our Bright Green Futures Will Be Weirder Than We Think, – “The world we need is one we’ve never yet seen.” The magic of attending an O”Reilly event is that it gives you a chance to work on this koan in interesting ways, and to take more responsibility for how things turn out.

Tim O’Reilly also urges that we think more deeply about what we are doing.  His keynote for Hadoop World , NYC, billed as, “The Business of Data” turned towards “The Consequences of Living in a World of Data.” The 900 strong crowd at Hadoop World was probably one of the most savvy crowds in the world about the business of data, so this was a nice turn.

Web 2.0 Expo with the theme, Platforms for Growth, was a deep dive into the business of innovation.  Tim O’Reilly’s keynote at Web 2.0 Expo,  “Thinking Hard About The Future” (or rather “thinking a little bit creatively or differently about the future)  – see video here, developed the call he made at Web 2.0 Expo 2008, to “work on stuff that matters,” into a Four Cylinder Engine for Innovation.   The first of the four cylinders in the firing order is, “Having Fun!” But,  at Maker Faire, Web 2.0 Expo, and Hadoop World I got an inside look at the workings of all four cylinders, and there is more to come, I am sure, as the other O’Reilly events unfold over the coming months including,  Web 2.0 Summit, Strata (a new O’Reilly conference on The Business of Data), and Where 2.0, 2011.

In a free webcast, last week (recording here), previewing Web 2.0 Summit, John Battelle and Tim O’Reilly discussed the Points of Control Map which is developing into a fun and useful tool to examine a very serious topic, “The Battle for the Internet Economy,” and how the “increasingly direct conflicts between its major players” could effect “people, government and the future of technology innovation.”   In my previous post, Platforms for Growth and Points of Control for Augmented Reality, I had a great conversation with Chris Arkenberg using this map as a springboard.  More on Points of Control later in this post.

The Four Cylinders of Innovation

click to enlarge

From Jet Ponies to Jet Packs: The First Cylinder of Innovation – “Have Fun”

The “maker” energy and its spirit of play, and the courage to create, hack, reinvent and re-purpose everything and anything, is a quintessential example of the first cylinder of innovation firing big.  Many “maker” projects also go on to fire on all four cylinders.   But the Maker forte definitely is in the first cylinder zone (and safety third as some of the rides, including Jet Ponies, warned).  The photo opening this post by Marc de Vinck – for more pics see here, is of Tim riding The Jet Ponies at Maker Faire which took  the New York Hall of Science by storm in late September – see The New York Times coverage here.  The ride was “built by the dastardly danger-hackers at the Madagascar Institute.“ See this wonderful interview with Hackett on his work to design “our specific jets from a patent that was filed in 1960s by a Mr. Lockwood, for Valveless Pulse Jets.” Hackett points out:

“Louder than god, glowing white-hot and looking like the trombone of the Apocalypse, pulse jets are also really shitty, inefficient engines,”

But, he adds:

“I have always wanted a jetpack, and one of the reasons I learned to build these things was to further that goal.”

This grand vision behind the Jet Ponies is a key to firing, The Second Cylinder of Innovation,  “Hey, we can change the world!”

But Jet Ponies, as a stepping stone to jet packs, also really struck a chord for me as I have been devoting a lot of time lately to the emerging Augmented Reality industry, a technology which was lumped in the same category of sci fi chimera as jet packs until very recently.

Data is the Gasoline

“The faces are coming from the sky.  The locations are coming from the sky.  All these apps depend on something, somewhere up.  And that, to me, was always the heart of Web 2.0.  And I am so delighted that people are finally getting it.  Because for a long time, people thought, ‘Oh, Web 2.0, it’s about lightweight advertising supported in a web start up.’  So I went, ‘No, no, no. It’s about the fact that we’re building these giant database subsystems in the sky that are going to drive applications.’  And now, of course, the same application is on your PC, it’s on your phone, it’s on you iPad.  And clearly, the applications are just sort of an interface to something that is being driven from the cloud, and that is fabulous.  That’s the difference.  People get it now.” (Tim O’Reilly, said this as part of a response to the first questioner at the Media Round table Web 2.0 Expo)

Answering questions about the importance of “Having Fun” to innovation doesn’t look quite as fun as riding Jet Ponies! Photo above from luca.sartoni’s Flickr stream

“ the data that is generated by the sensors and the applications that use that data is going to be where people are going to be innovative.” (Tim O’Reilly)

During the Media Round Table, I had a chance to ask Tim more about the role of bottom up innovation in a world where big data is the gasoline for increasingly sophisticated engines – platforms integrating machine to machine intelligence and real time analytics.

Tish Shute: You brought up Maker Faire in your keynote, and again now.  I was there, which not many people in the audience were  [not too many hands went up when Tim asked during his keynote].  But I think one of the things that struck me was the jet ponies – they were just earthshaking to stand near.  They made the ground tremble; they made the world shake.  Yet, most of your keynote, and most of what’s on our minds here, at Web 2.0 Expo, is extracting intelligence from the big data [in the sky], and algorithmic intelligences are the jet engines of the internet.  And of course, not to be forgotten, as we are here in New York City, where the trading markets are creating the air we breathe  [although we probably don't realize it until we lose our mortgage or something] and these algorithmic economies or “robot casinos” as Kevin Slavin put it, are all about speed – it’s not just real-time, issues of latency are  so critical that co-location is key to winning the game of the markets.  [Kevin Slavin brilliantly unpacks this in his talk, "Loitering on the Motherboard." For more in this see my conversation with Kevin Slavin below].

So my question is, who’s making the jet ponies for the algorithmic economies in the sky that you just described?   How can we make a play from the bottom up?  I always feel Ushahidi is one of the jet ponies of the data algorithmic space [because of their great work to bring human and machine intelligence together to solve problems in crisis situations].  But who do you think is doing exciting work and how can we ensure that this powerful world of data and algorithmic intelligences does not become hidden in a closed black box [only really accessible to elite players like the NYC trading markets]?

Tim O’Reilly: “Well, I think there’s certainly a lot of interesting things happening in, say, the financial services that a lot of, kind of, the Internet folks are kind of blind to.  I think that there are companies like Next Jump which are really good with data and good with algorithms. But kind of speaking specifically to the maker side of this, that whole sensor enabled world which is going to produce data is in its infancy.  What we have that I think is so powerful right now is we have the first portable sensor platform.  I said in my talk the other day, you know, your phone has ears, it has eyes, it has a sense of where it is.  And these are all available to application developers. You know, you can compare, say, Dodgeball to Foursquare, you can see how different… Dodgeball is Foursquare in the tele-type era.  Foursquare is now possible because there are so many more capabilities on the phone.

And I think that we are going to see a lot of other areas that are revolutionized by the sensors in the device.  It could well be that some of them will come explicitly out of the maker kind of projects, or it could just be that make is sort of a proxy for them.  So yeah, Arduino is this great maker sensor platform, but hey, here’s a consumer sensor platform [holding up phone]. Maybe we vaulted past the maker stage already and we just didn’t know it.

And that’s not entirely true, because Arduino is building a whole economy of special purpose devices.  But it feels a little bit like the days when people rolling their own PCs coexisted with the rise of Dell, who was a kid in his college dorm room who made his own PCs and sold them on the net, but figured out how to scale it pretty quickly and get good at it. But there were still a lot of garage shops, you know, ‘I’ll make a PC and sell it to you’ people for probably a decade before there was really a clue that that was a commodity industry.  In fact, I do think the sensor platforms are going to become a commodity industry.  And the data that is generated by the sensors and the applications that use that data is going to be where people are going to be innovative.”

The internet operating system is a data operating system and it is happening in real time (Tim O’Reilly)

click to enlarge the image above  – a slide from Mike Olsen’s  (CEO of Cloudera) keynote at Hadoop World

Not only do we have a portable sensor platform in our pockets  but developers also have powerful platforms and tools to make sense of data that fuel our apps.   Opensource Hadoop makes available, to anyone with some data munching chops, the power to work with giant unstructured databases and do the kind of real time analytics previously only available to giants like Google.  Big players like Yahoo, Facebook, and Twitter use Hadoop (Jonathon Gray from Facebook noted they add 10TB a day).   But, as this great roundup of Hadoop World points out, while Hadoop gets the press for handling petabytes of data , Mike Olsen (CEO of Cloudera) noted, the fastest growing area of users are working with clusters smaller than 10TB and over half of the Hadoop clusters were under 10TB in size.

Four Square: A Platform for Growth with an ecosytem built on top of data that exists in the real world

As an augmented reality enthusiast it is not hard to guess that one of my favorite platforms for growth is Four Square.  See Dennis Crowley’s keynote at Web 2.0 Expo here.  The Four Square API has been available to developers since November 2009,  and there are already a number of  interesting applications, and there will be many more to come.  The screen shot above is of geopollster – see the gallery of Four Square apps here.

@dens tweeted recently  “Politics + @Foursquare = @GeoPollster” http://geopollster.com <- I love love love that people are using 4SQ to think about election tools

As Kati London pointed out in her keynote, Four Square is the “kind of augmented reality that is aimed at shifting or changing a person’s social reality, e.g. the mayor badges in Four Square that change my relationship to the people and the place I am in, and augment engagement and reputation through socially driven consumer tie ins.” We are already see augmented reality developers beginning to work with the Four Square API – see here, Foursquare + Augmented Reality + Virtual Graffiti = ARstreets.

As augmented reality development tools mature, Four Square will, increasingly, become an important platform for creative AR developers interested in integrating the power of this platform for augmented engagement and reputation with “device aided augmented reality that can shift visual experiences of situated geolocal experiences.” With the Qualcomm vision based augmented reality SDK now available for download, and Wave in a Box soon? to be released, and an ARWave client working on Android (almost!), I have been exploring the Four Square API in my non existent spare time!!

The Four Square API also offers some interesting possibilities for exploring games that take the complex economy of Four Square – not personal data but aggregates of behavior, as their subject matter (for more on this see my conversation with Kevin Slavin later in this post and in an upcoming post).

I took this picture of Dennis at Where 2.0, 2009 at the beginning of Four Square’s phenomenal growth (they are at 4 million plus users now).

Pachube (Patch-Bay): a web service for storing and sharing sensor, energy and environmental data

Eighteen months ago, I interviewed Usman Haque (architect and director, Haque Design + Research) and founder of Pachube – see Pachube, Patching the Planet.   Usman pointed me to this wonderful evocative image from T.R. Oke’s book, “Boundary Layer Climates” (original photo source Prof. L. E. Mount’s The Climatic Physiology of the Pig).  “It’s the same piglets, in the same box, but on the right hand side the temperature has been increased. This small change in how the space is “programmed” has dramatically changed the way the ‘inhabitants’ relate to each other and how they relate to their space.”

The Challenge of Connecting people and environments.

At Web 2.0 Expo, I got the opportunity to talk with Usman Haque again.   Pachube, is becoming an established platform now, Usman explained.  They have a development team of eleven and robust back end.  And, they will now be spending some more time on the front end, including a redesign of the website, making “it a lot easier to widgetize the entire website so that you will be able to take almost any element and embed that into your own website.” And, as Usman mentioned in his presentation, they are working on an augmented reality interface, Porthole, for facilities management and, “as a consumer-oriented application that extends the universe of Pachube data into the context of AR – a ‘porthole’ into Pachube’s data environments..  Usman is also contributing to the AR standards discussion and on the program committee now for the W3C group on augmented reality.

Click to enlarge the image above from Chris Burman’s paper for the W3C, Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual”

Pachube, is sometimes described as the Facebook for Data or an analogy Usman prefers, a Twitter for Sensors.  At Web 2.0 Expo, I had an amazing opportunity to hear from Twitter and Facebook about their strategies as platforms for growth.  This gave me lots of fuel for questions about Pachube’s approach to developing their platform.  Simplicity was a theme that Facebook  and Twitter both affirmed as a key.  One of Pachube’s challenges will be to deliver ease of use, and the equivalent of Facebook’s “like” and  Twitter’s “follow” to gain mass appeal.

Here is a brief excerpt from my upcoming conversation with Usman:

Tish Shute: So as a platform you see Pachube as having more in common with Twitter – a Twitter for Sensors. In what ways is Pachube similar to Twitter?

Usman Haque: Well we are the Twitter of sensors, devices & machines in the sense that, really, the API that enables all this communication is important, much more so than the website itself. It is where, basically, most of the millions of our hits actually go, is to the backend. And we’ve now got dozens of applications built on top of the system, a little bit like Twitter’s applications; you know, all the apps are the important part.

But we are actually going to be doing some quite exciting things with API keys that we haven’t really spoken that much about in public. But we have come up with a pretty innovative solution to make almost every resource have granular privacy options on it, now discussed here.

At Hadoop World, Tim O’Reilly also raised some interesting broader questions that are very relevant to Pachube’s vision to “patch the planet”, e.g, the problem of digital identity in the age of sensors? (Smart phones already know their users by the way they walk!) And, “How should we think about privacy in a world where data can be triangulated?”

Usman talked about Pachube’s approach to both the technical aspects of how to build a massively scalable system, and the conceptual aspects of how people connect to each other, and what they might do with these new opportunities to connect environments and sensor data  (see my earlier talk with Usman, Pachube, Patching the Planet, for a detailed explanation of some of the concepts behind Pachube).

I look forward to posting this conversation. Pachube is growing, and Usman always goes beyond the familiar tropes of connecting human and machine intelligence.

2nd Cylinder of Innovation: “Hey Can We Change the World!”

The possibilities for reimagining of the role of data in healthcare produced some of the most powerful “Hey Can We Change the World” moments for me at both Web 2.0 Expo and Hadoop World.  The slide above is from Esther Dyson’s brilliant Ignite presentation, “What you can and can’t learn from your genes?” are here,   Tim O’Reilly also brought up the powerful role real time data analytics can play in improving healthcare in his Hadoop World Keynote.  Also see Alex Howard’s post, 10 Lessons for Gov 2.0 from Web 2.0 for some more great, “hey we can change the world moments” at Web 2.0 Expo.  The keynote from Lukas Biewald of CrowdFlower and Leila Chirayath Janah of Samasource (screen shot below) in particular, is a provocative exploration of the future of work in the new ecologies of human and machine intelligence.

Changing the World When Our Lives Are Increasingly Shaped by Forces Invisible To Us?

Click to enlarge

Mike Olsen, CEO of Cloudera, noted that “the largest area of data growth does not come from humans interacting with machines; rather, it’s from machines interacting with each other” (see here in Minor Technical Difficulties).   One of the most interesting presentations at Web 2.0 Expo was Kevin Slavin’s, “Loitering on the Motherboard,” which, as Tim O’Reilly pointed out in his keynote at Hadoop World, is a talk that raises all kinds of questions about a system where big players are gaming the data for their own ends.

Kevin Slavin, a founder of Area/Code, notes the operating system of our mortgage, life insurance, the operating system of currencies and gold is now governed by machine to machine intelligence and algorithimic economies outside of human cognitive processes.  The markets are now legible only to bots in an algorithmic arms race with bots surveilling bots, and throwing off false information in a bid for counter-surveillance.  He showed some slides of the eery but beautiful visualizations of traces of the trading bots created from the Nanex API.

The screenshot above is from the Nanex: Crop Circle of the Day – Quote Stuffing and Strange Sequences.  “The common theme with the charts shown on this page is they are all generated in code and are algorithmic. Some demonstrate bizarre price or size cycling, some demonstrate large burst of quotes in extremely short time frames and some will demonstrate both…” This one is a zoom of the NSDQ “Wild Thing.”  Wild price/size repeater from NSDQ running at 1,000 quotes per second, effecting the BBO along the way (I love the great names Nanex gives the different patterns and traces produced by the trading bots).

Nanex supplies a real-time data feed comprising trade and quote data for all US equity, option, and futures exchanges. They have archived this data since 2004 and have created and used numerous tools to “sift through the enormous dataset: approximately 2.5 trillion quotes and trades as of June 2010.” May 6th 2010 (day of the flash crash), had approximately 7.6 billion trade, quote, level 2, and depth records.

Kevin points out that our lives are being shaped by criteria invisible to us and the old hackneyed tropes of machine to machine intelligence such a robots reading HUDs in English are long worn out.  The latter point is, perhaps, something for us augmented reality geeks absorbed in ideas of “making the invisible visible” to chew on.

Changing a world shaped by forces that are, increasingly, invisible to us presents a huge challenge.

But I had the glimmer of a, “Hey Can We Change the World” moment, when I attended Kevin Slavin founder of Area/Code’s presentation and had a conversation with him after his talk.  Could games take these complex economies as their subject matter?  The economies of  Farmville and games like WoW are not opaque at all, and these are environments with complex economic behavior, “where you can actually have enough data to understand what it is” – “it’s not so much about personal data.  It’s more about, like, aggregate behaviors.” “Games that can really model those, and play with those, and take those as the subject the way that Monopoly takes Monopoly as a subject could be really interesting.” Kevin made many fascinating points – more to come on this topic.

Photo by James Duncan Davidson, of Kevin Slavin speaking at Web 2.0 Expo NY, 2010, from the O’Reilly Conferences Flickr stream

Here is the beginning of our conversation:

Talking With Kevin Slavin

Tish Shute: You began your talk today about visibility and where some of the algorithmic masters of disguise went to work, after they had solved the math behind stealth bombers.  I thought perhaps you were leading into ideas about a reverse surveillance society.

But you surprised me, as I felt you made visibility itself kind of a non-issue by the end of your presentation and that counter surveillance became basically a time and speed issue.  Now I am not sure quite how to imagine a counter-surveillance society, something I try to think about…

Kevin Slavin: Well, let’s see.  There’s a couple ways to think about it.  I think one point is just that when we talk about counter-surveillance, we usually locate that as something that comes from  the bottom up, something that comes from the population. Think about the way the plane spotters discovered the CIA black rendition flights.

I think in general, when people talk about counter surveillance, or sousveillance, they imagine it as an inversion of the traditional relationship between the people and the state.

But that’s what’s interesting. What’s happening now, is that there are forms of surveillance and counter-surveillance that are in play beyond any human perceptual horizons. These forms are at their most sophisticated in financial services, in the markets.

If you were a bot, and could read the market legibly (which humans cannot), what you would see, effectively, are bots that are surveilling bots. Then you have bots that are throwing off false information in a bid for counter-surveillance. Many of the bots are, themselves, surveilling other bots; each one of them is trying to figure out what all the other ones are going to do. In essence, it’s an algorithmic arms race, and game theory has become concrete, since the theories are code, the code is action, and the action affects, let’s say: your mortgage.

And so, basically what you have is you have this series of algorithms that are all looking to discern each other, while also trying to prevent themselves from being discerned. I think of the tunnels under the trenches in WWI, tunnels to surveil the trenches, and then, later, tunnels to surveil the tunnels. There’s a few examples of this kind of thing.  But It’s especially strange when it’s computer code, and at the magnitude we’re seeing today.

All of it, as noted in the talk, accounting for 70% of all the trades in the market. 70% of the market trades are never touched by human hands or even seen by human eyes; they don’t move through a conventional cognitive process.  And that’s why you get things like the Credit Suisse algorithm, it was buying, selling 200,000 shares of stocks to itself over and over and over again. It was a bug and it slowed the market to a crawl.

Credit Suisse was fined, in essence, for failing to control an algorithm. Maybe that’s the first time an algorithm was treated like a human, in a way. As if the algorithm broke the law, and Credit Suisse was responsible for letting it do so. For me, that feels like a threshold event.

It’s not that humans never made mistakes when trading on the market. But when algorithms err, they err with magnitude.

The idea that we now have bugs in the United States market economy is really worth looking at.  If Apple can’t keep code bugs from the most simple iPhone apps in a closed and regulated ecosystem, I’m pretty certain we’ll have a lot more Credit Suisse type bugs in the future.

And that will be pretty interesting. There will be viruses, and the operating system they will operate on will be the operating system of the United States. The operating system of your pension, your house, your life insurance. The operating system of currencies and gold.

Tish Shute: I was hard-pressed by the end of your talk to think of like, “Well, what would be the equivalent of, sort of a people’s uprising to create a better fairer society in this kind of world where, really, the things that affect the key aspects of lives most are going on beyond human perception at an algorithmic level?”  But you made a pretty radical suggestion at the end…

Kevin Slavin: Well I think increasingly the markets have become delaminated from anything meaningful. First from goods, then from fundamentals, and now finally from homo sapiens. So that’s hard to fight.

It’s the race towards abstraction that makes it impossible to simply “resist.” The latest version in the long series of fiscal catastrophes was based on Wall Street finding goods that could be rolled up and sold with false valuations, but goods that would take a long time to fail. Mortgages are handy like that. It’s the tradition of extending the abstraction as long as possible, until finally the bill arrives and the banks fail. I don’t know if that’s something to rise up against or not. It’s like a rally against evil.

But really, I think the point is that it won’t be the people that rise up. It will be the financial services themselves that rise up. They’ll just detach completely.

That was harder to do with cotton or with wheat, with simple futures; they keep financial services tied to the ground.  So what we’re doing is creating increasingly complex financial instruments that are further and further removed from anything you can touch.  Like the way a mortgage is abstract. But, of course, the bottom line is that at the end of that mortgage lies someone’s home.

It’s said that Wall Street is now moving onto life insurance, because that’s going to take even longer to fail.  They’re doing the exact same thing. The word is that they are rolling up CDOs made out of crap life insurance policies, same way they rolled them up with crap mortgages a few years ago.

And those will probably take, I don’t know, 15 or 20 years to unwrap and unravel.  But what you see in the meantime, is that they are looking for things that are increasingly abstract, intangible, removed as far as possible from the experience of everyday life.

So maybe this is good. Maybe that’s financial services rising up. Lifting off. I think best case scenario now is that they actually leave humans alone altogether.  That, someday, they are just trading, effectively, completely arbitrary goods, the stocks could be anything at all, maybe for crops that no longer exist, and I’m just saying that then these bots would no longer affect what we do and what we are, it would just be a robot casino, an invisible paradise in the air.

People are the platform: How Games Can Be Engines of Innovation in Our Lives

See the video of Games that Know Where We Live here (screen shot above)

Kati London, Senior Producer, Area/Code, in her keynote showed how games that know where we live can shift players perspectives – from device aided augmented reality that can shift visual experiences of situated geolocal experiences to a kind of augmented reality that is aimed at shifting or changing a person’s social reality, e.g. the mayor badges in Four Square that change my relationship to the people and the place I am in, and augment engagement and reputation through socially driven consumer tie ins.

Area/Code has recently developed two games for the Knight Foundation that take people as the platform.  Macon Money, uses very simple games dynamics (for more see the video of Kati’s keynote) in a game designed to help “Knight’s continuing efforts to support revitalizing Macon and creating a vibrant college town.”

The other game that Area/Code has designed with the support of the Knight Foundation  is for the Biloxi and Gulf Coast community, a game called Battlestorm.  “The game’s purpose is to increase awareness about natural disasters and change the way people prepare for them.”

3rd Cylinder of Innovation: Build products, business models and entire industries.

Glympse – real-time, private location tracking

Julianne Pepitone, Yahoo! Finance, nailed the essence of Web 2.0 Expo, NYC, this year in her post, Web 2.0 Expo startups are big on neighborhoods, storytelling.  She writes:

“At the Web 2.0 Expo in New York City this week, executives from big sites like Facebook, Twitter and Pandora all spoke about industry trends. But the showcase of 27 startup tech companies stole the show.”

Listen carefully to Tim O’Reilly and Fred Wilson, Union Square Ventures, question their picks from the startup showcase at Web 2.0 Expo.  Also see this video of Fred and Tim discussing their conversations with all the start ups.  This  is one of the clearest public windows onto both how to present your company to VC, and how to figure out what are the most important questions for you as an entrepreneur  building a business in a world of data.

Glympse successfully pitches their “jet pony” strategy for a location based business, and is Fred’s pick.  They hold up well under pressure and answer Tim and Fred’s hard questions about how their start up will not get overtaken by an encumbent player with resources and market share before they can gain traction.   food52 responds to Tim’s probing about their strategy for business data analytics that he points out are vital if they want to survive with the small margins of ecommerce.  There is a list of all the participants in the start up showcase in Brady’s post here. hour.ly was the audience pick.

Shazam for Faces!

My favorite start up was a biometric service doing face, iris, and finger print matching, Tactical Information Systems.

Tim and Fred also liked them, and they have an interesting discussion about the merits or not of approaching your platform through a narrow first application as Tactical Information Systems are with WanderID -  an application to help identifying lost Alzheimer patients.  As Fred pointed out, they are potentially the Shazam for faces, so why start so small?

I  had asked TIS the same question when I met them in the “speed dating” session.  This is just their first toe in the water as they are a two person company at the moment. Their vision for their platform is big.  Mary Haskett and Dr Alex Kilpatrick, the founders of this quintessential jet pony for the algorithmic economies in the sky, are not only a partnership with the credentials to do a  Shazam for faces – see their bios here, they are the people I would want to be running a Shazam for faces!  They really get the consequences of living in a world of data – check out Dr Kilpatrick’s absolute killer Ignite talk, “Defeating Big Brother.” (screenshot below)

How Can Augmented Reality Add Value to the Real Time Internet/Data Operating System?

planefinder.net – an augmented reality app that lets you find information about planes by pointing your phone at the sky, “including flight number, aircraft registration, speed, altitude and how far away it is” (via MacLife).

The new opportunities in the algorithmic economies in the sky were center stage at Web 2.0 Expo and there are some interesting AR apps for the real time internet/data operating system emerging, like planefinder.net.  But Augmented Reality was still pretty low profile at Web 2.0 Expo (except that NVidia augmented reality demo attracted a lot of attention at the sponsors expo).  However, everyone working in the emerging industry of AR should recognize that apps big on “neighborhoods and story telling” are heading right up the AR street, and that platforms like Four Square and Pachube present enormous opportunity to explore the possibilities of AR.  And if augmented reality enthusiasts are not already paying attention to real time data analytics, and Hadoop, they should be (see this post for an excellent round up on Hadoop World).

At Hadoop World, Tim O’Reilly referenced the great tagline from the  IBM commercial:

“Would you be willing to cross the street — blindfolded — on data that was five minutes old? Five hours? Five days?”

As I have noted in several earlier posts – see here and here for starters,  we are just seeing the tools  for developing near field, vision based, mobile, social AR become widely available to developers, so there should be a new level of AR apps emerging through 2011.  There is a wonderful discussion in the comments of this post by Mac Slocum, “How Augmented Reality Apps Can Catch On,” between Mac, Raimo one of the founders of Layar, and Chris Arkenberg on what constitutes a platform for growth for augmented reality.

Mac’s post, the comments and Chris Arkenberg’s post on the latest edition of the Gartner Hype Cycle, that rather curiously placed Augmented reality almost at the peak of inflated expectations. really got me excited about exploring an idea I have been thinking about for a while, which is to get the AR community to discuss the Points of Control map.    See my discussion with Chris Arkenberg here, Platforms for Growth and Points of Control for Augmented Reality. The recording of  John Battelle’s and Tim O’Reilly’s webcast on Points of Control is posted here.

The interactive Points of Control map is an amazing tool to think with! Check it out in movements, territory and movements, acquisition mode.  There is a competition for the most interesting comment and most interesting acquisition suggestion.  The prize is a ticket to Web 2.0 Summit!

What is the Future of Social?

The recent “defection” from Google to Facebook – see Lars Rasmussen, Father Of Google Maps And Google Wave, Heads To Facebook,  is as MG Siegler of TechCrunch points out, “the biggest one since Chrome OS lead Matthew Papakipos made the same jump in June” (TechCrunch also notes “current Facebook CTO Bret Taylor was heavily involved in the launch of Google Maps”).

These moves have drawn my particular attention as did Bret Taylor’s response in his conversation with Brady Forrest at Web 2.0 Expo to Brady’s question, “How soon until we get the Facebook firehose?”

If you have been reading Ugotrade you already know how important I think an open, distributed, standard for real-time communications such as the very innovative Wave Federation Protocol could be for AR development  -  see ARWave and my presentation at MoMo13, Amsterdam last year, The Next Wave of AR: Mobile Social Interaction Right Here, Right Now!

The anticipated release of  Wave in a Box, has raised hopes in the developer community that  WFP will soon become easier to work with, and hopefully more widely adopted.  Like many others, I wonder what will happen to Wave in a Box now?

But the innovation of Wave is deep and broad (and as many have pointed out hugely ambitious).  Perhaps the boldest attempt yet to innovate both at the low level of architecture (where Google is so powerful) and at the high level of the Mark Zuckerberg, “big idea,” which as Tim O’Reilly notes is, “What is the future of social?” MG Siegler noted Facebook Groups Is Sort Of Like Google Wave For Human Beings.

But I deeply hope that the open, distributed standard part of the Wave big idea is not lost in the mix here.

Fourth Cylinder of Innovation: Keep the Ecosystem Going, Create More Value than You Capture

The Points of Control map is interactive, so please click here or on the image above for the full experience.

Tim O’Reilly points out that there is a worrisome dark side to the Points of Control Map – see Tim’s keynote here.  To paraphrase some or his points:

There are companies on the map that are forgetting to think about creating a sustainable ecosystem.  Rather than growing the pie, they are trying to divide up the pie and that threatens to cause the fourth cylinder of innovation to misfire.  This fourth cylinder is essential to the ecosystem.

Tim O’Reilly looks back to the lessons of the personal computing industry which was incredibly vital and creative, and lots of people made money until a couple of big players “sucked all the air out of the ecosystem” and innovation had to go elsewhere.

The Power of Platforms is to create value not just for your company but for other people.   Create value for yourself by creating value for other people.  Tim O’Reilly used the wonderful example of  Henry Ford inventing the weekend so that there would be enough people with time and money to buy his mass produced cars.  Think about building the ecosystem that will support the future your are going to build.  Grow the pie rather than cut up the pie.  This will be the vital fourth cylinder of innovation in a Web Squared world.

Tim O’Reilly has long proposed that  Web 2.0 is all about harnessing collective intelligence,  But as Gartner predicts, “By year end 2012, physical sensors will create 20 percent of non-video internet traffic.” Yet another previously unevenly distributed future is going mainstream, and if you haven’t read it already, now is the time to read this paper by Tim O’Reilly and John Batelle, Web Squared: Web 2.0 Five Years On.

The Consequences of Living in a World of Data

To bring this very long post to a close!  Here are just a few of the key questions re The Consequences of Living in a World of Data that Tim O’Reilly raised during his keynote for Hadoop World:

“How would we solve the problem of digital identity in the age of sensors? (Our smart phones are able to know their users by the way they walk – their gait!)

“How will we input data when our devices are smart enough to listen on their own?”

“How should we think about privacy in a world where data can be triangulated?”

“We are moving to a world in which every device generates useful data, in which every action creates information shadows on the net.”

“Shouldn’t we regulate the misuse of data rather than the possession of it?”

“How do we avoid a data arms race?”

“Create more value than you capture.”

Today Metaio is holding Inside AR in Munich, Germany.   Metaio (the picture above shows Metaio co-founders Thomas Alt and Peter Meier), is behind some of the best known commercial and industrial AR experiences of recent years.  But as important as the many AR projects they have executed are the AR tools that Metaio has made available to developers.  Metaio’s AR products and tools have played an important role in bringing AR to a wider public, and given many developers the opportunity to explore AR.

Inside AR is a great opportunity to see what these AR pioneers  will be up to in the coming months.   I could not make it to Munich this year.  But, fortunately, I had the opportunity to talk with Thomas Alt, recently.   In this conversation – see below, I got a chance to discuss what was going on inside AR with Metaio.

The Fall season is always jam packed with great events, and I wish I could be in two places at once.  But this week, I will be in my home town, NYC, attending Web 2.0 Expo which, reflecting the heat in the NYC tech community, is a sold out event with a very exciting schedule this year (more on some of the presentations that I will be attending later in this post).  If you missed out on tickets to Web 2.0 Expo, all Keynotes will be Streamed Live: TUES 9/28 to THURS 9/30, and keep your eye on @w2e and #w2e on twitter.

Meanwhile, I am missing Inside AR, which had some great speakers lined up, including fellow New Yorker, John Swords, partner and Ringleader at Circ.us.  Hopefully, Swords will share his experiences at next month’s ARNY Meetup which will be “joining forces with another vibrant community – NY Gaming – for an unforgettable night of Augmented Reality Games” on Tuesday, Oct 19th, 6:30 PM at AOL Ventures in New York, NY.

At the most recent ARNY @swords gave a brilliant talk on the possibilities for AR Game development on the newly available opensource Parrot ARDrone platform.  It was great to hear from social game guru @swords on his plans for Parrot ARDrone games, and more.  The picture below of an ARDrone camera view is from John Swords Flickr set.  Swords was flying it inside his garage because the winds outside were too strong.

Also, I kicked off what will hopefully be an ongoing discussion on, “Story Telling with AR and the Big ARNY a collaborative AR Game for NY,” with a few slides.  I have opened up the presentation document for collaboration, so please ping me if you would like to be added as a contributor/editor, and are interested in getting involved.

Ori Inbar, CEO and co-founder of Ogmento, Games Alfresco, ARNY and my co-chair on Augmented Reality Event 2010, suggested The Big ARNY – A Collaborative AR Game Development Project modelled after A Swarm of Angels last year at the First ARNY Meetup – so let’s make it happen!  I will be catching up with Ori in October about what Ogmento has been up to since they became the first VC backed AR Game company!

“Games allow us to see each other, for a moment, in a way that living in a city prevents” – Kevin Slavin

I believe that, AR, to get beyond the stage of “interface du jour” needs to offer us new ways to relate to each other and the world around us so that we can actually improve  and deepen our engagement with reality not just create experiences that are primarily optical  (see James Turner’s interview with Kevin Slavin “Reality has a gaming layer” on not letting “the pleasure of a game and the meaning of a game and the experience of a game rest primarily in the optics.“  And see my recent post, Urban Augmented Realities and Social Augmentations that Matter: Talking with Bruce Sterling, Part 2).

Two of the most inspired creators of urban games,  Kevin Slavin and Kati London of Area/Code will be speaking at Web 2.0. Expo tomorrow.   And you can be sure I will be at both these sessions. Loitering on the Motherboard, Kevin Slavin, is 2:35pm Tuesday, 09/28/2010, and Games that Know Where you Live, Kati London – is a keynote that will also be live streamed – 4:55pm Tuesday, 09/28/2010

Recently Kevin Slavin was interviewed by James Turner, on O’Reilly Radar.   This, Reality has a gaming layar, is a must read piece about a “world where games shape life and life shapes games”  (see @timoreilly). 

Interview with Thomas Alt

Tish Shute: Perhaps you could just start with your background Thomas because I think there’s a lot of newcomers to AR but you are really one of the first movers in commercial AR. How long you’ve been involved in this?

Thomas Alt: Actually I’m an ex-researcher in augmented reality. I started with me actually after getting my master’s work in engineering from the Technical University of Munich working for a big company called Volkswagen. And at that time,1999, we got a research grant for researching how augmented reality could change manufacturing processes in the automobile industry.

And from the research work there, I basically went back to school, did my PhD about augmented reality. And while speaking at a conference, I met Peter Meier who is the co-founder of the company who was also a master’s student writing his thesis about augmented reality. That was in 2002.

And so it really was in the very early days of augmented reality. And both Peter and myself we got really excited about the technology; we saw endless possibilities. We said, ‘OK. Let’s just found a company. We actually founded the company in early 2003 with virtually no money. As a matter of fact the founding capital of the company was 25,000 Euros and this 25,000 Euros were won in a case competition in Germany – a business plan competition.

Tish Shute: So you won 25,000 Euros on this case competition and that’s where Metaio started….

Thomas Alt: Exactly. And to legally found a company in Germany it takes exactly 25,000 Euros so that was the founding capital. We started pretty much like good old SAP started.  It wasn’t in a garage though it was a very small office and we basically built up the business through work, so we don’t have any investors or whatever. Right now we are 66 people located in Munich where our headquarters have been for five years. We have some presence in the US, and we have a venture company in Seoul, South Korea.

Tish Shute: Awesome. I just noticed how fast you’ve been growing. So right now, I’m going to ask a couple of questions about where you see the technology and the emerging industry going.

First, what are the platform of choice for Mobile Augmented Reality at the moment?

Thomas Alt: Obviously in the cellphone hardware space there’s a fierce competition going on. It’s yet to be defined what will be the prevailing platform right now, obviously it’s the iPhone is big now, right? But Android is catching on very, very fast.

Tish Shute: You have pioneered bringing a cross platform SDK for vision assisted AR to a wide community of developers with Junaio and with your partnership with Kooaba – a visual search company from Europe.

Thomas Alt: Yes, yes, and this is how we would, also in the future like to position ourselves with Junaio.  Junaio will be a platform, a technology platform, which will allow users to do whatever they want to do in augmented reality. The API of Junaio is huge in the sense you can do anything from outdoor gaming, to visual search, to normal, uh, lay out style, you know, find the next burger king a mile away kind of super impositions.

Tish Shute: The only licensing you pay is for unifeye right? When you want to use your tool kit right?

Thomas Alt: Yes and this is how we’re distinguishing it. Junaio is our consumer brand targeting newbie AR developers, with limited programming skills, while the Unifeye platform is really our B to B platform where B to B customers can create their individual AR experiences.

Tish Shute: Yes which is what my friend Patrick O’Shaughnessey, Patched Reality, did for the Ben and Jerry’s app he created using Unifeye.

Thomas Alt: exactly…

Tish Shute: It is a lot of work developing for so different mobile platforms isn’t it. Junaio is on Android and iphone but you havent moved Junaio to Symbian?

Thomas Alt: To be honest with you right now its a matter of priorities we have other things we want to do first. And from analyzing the user base, iphone was a big step Android was a big step and now we are pretty much seeing what is happening next. As you know Nokia going into different directions as far as their smart phone operating system goes, and so on and so forth.  There are also capacity constraints. And right now obviously the most – potentially not the most possible users, but the users most inclined to do AR on a day to day basis are the ones using the iphone and android devices. But obviously there are a lot bigger cellphone manufacturers out there.  But just you know even the mobile web users aren’t as strong as the users on the iphone and android devices?

Tish Shute: So what do you think the iphone 4 has that brought to the AR picture?

Thomas Alt: Very fast camera access, very good for marker recognition. If you go to the Metaio site you’ll find a movie where we show on the iphone 4 app for a real augmented reality Leggo peice – this is something which is very nice

Tish Shute: Yes I see that, yes that is nice, yes, yes very nice. The Unifeye SDK is really putting markerless AR into the mainstream.

Thomas Alt: Yesterday we launched the first, a err very nice shopping… shopping solution for , but that’s completely external.

Tish Shute: Oh yes – the augmented reality shopping for seventeen.com, i was going to ask you about that, because it is the first augmented reality online shopping using natural feature tracking.

Also I am very excited to see the gestural interface, awesome!

The seventeen augmented reality shopping app is a PC experience but are you working on developing gestural interfaces for mobile AR?

Thomas Alt: We are continually pushing the envelope of what’s possible with AR. Gestural interfaces for mobile AR is certainly the next step in taking what we’ve done on the PC and making it more portable by using the mobile platform. One thing to keep in mind here is the limitations of mobile platforms and size of the screen needs to fit and make sense for the user experience.

Tish Shute: I know you started off as an AR researcher (although as you mentioned earlier you have been working in commercial AR and building Metaio for a long while now.

So in addition to how we are progressing towards gestural interfaces for augmented reality, I would like to ask some questions about AR eyewear. We won’t really have hands free AR without eyewear. What is your projection on when we will see consumer AR eyewear? And, Do you have a any comments on those speculating that we will not see AR eyewear go mainstream for 20years?! What is Metaio doing to move eyewear technology along?

Thomas Alt: Well as you know, it’s always, you know on the technological roadmap, and we’re still doing research projects, in our AR research department. We have worked on things like calibrating eyewear for augmented reality. There is some nice development there.

But really, commercially, the whole thing with eyewear has never caught on on a level which would make it a valuable avenue, business avenue, at least for Metaio. So, I guess as an ex-researcher, it’s still a very interesting, a very good technology. And it would definitely change the marketplace radically when available. But as per right now, there are very few commercial applications.

Tish Shute: Are the obstacles to AR eyewear technological obstacles or is it just a question of a a business model. I mean is it realistic to see eyewear in the next 3 to 5 years at a price point affordable to consumers, where you really, truly can have eye tracking? You know, the whole problem there was with virtual reality and eyewear giving people a headache. How far have we come in terms of the technology?

Thomas Alt: Well, it’s not so much technological factors ’cause all fundamental problems are solved. It’s more a rather large corporation, I guess, would have to step up to the plate and say okay, do, let’s get all the state of the art in electronics and develop just a perfect HMD.

Tish Shute: Something Yohan Baillot’s company Simulation 3D is doing at is looking at is hooking up eyewear to smart phones.

Thomas Alt: Yes exactly that would be even better. Metaio has done a strategic move into this HMD space for augmented reality about a year ago by acquiring a bankrupt company. I mean, we had considerable IP around it from our research base but in the long term we still believe in it and we did a move about a year ago in buying what was left over from a bankrupt company including a lot of IP, which basically goes into the direction of mobile augmented reality but also mobile augmented reality in connection with head mounted displays.

There’s actually a press release about this but that’s about a year ago…

I know that the whole HMD thing… I mean, I’ve seen companies come and go. Metaio has worked previously, very closely, with Microvision of Seattle. We have worked with a German company, doing HMDs and we have worked with Vuzix.  We are still working with Vuzix, so we’re still consider it very valuable.  But right now, I mean, it’s just not a big part of our commercial pipeline, to put it that way.

Tish Shute: It was interesting what Bruce Sterling said in his keynote at ARE 2010.  He actually made a strong case for why smart phone augmented reality may be more interesting because it’s less immersive. I mean, he raised the question of the fact that if you really truly had AR eyewear and HMDs you’d re-enter the world of virtual reality or as he called it AR’s Gothic step sister VR would rise from the grave….

Thomas Alt: Yeah, well, that’s a cultural or even a philosophical question and we have discussed it a lot, especially in the industrial domain. Also will the deployment of HMDs come about from end consumers using it in their spare time, or from professional users using the idea in their work time?

Tish Shute: Do you think it surprised people who have been working in augmented reality research how much people have engaged with the idea of smart phones as the mediating device for AR, and that rather than having the always on experience that eyewear would give us, we use smart phones as a magic lens of a smart phone when we need to or want to. Some people were skeptical that anyone would want to hold up a little window to look at augmentations of the world – a magic lens. I mean, it wasn’t self-evident that that would be an experience people enjoyed, and it turns out that it was.

Thomas Alt: That’s actually a very good analogy. And also in my view, I mean, certain behaviors just change also, right? I mean, this is exactly what Apple’s trying with the iPad, right? You’re taking the iPad, and all of the sudden you’re not constrained to a laptop or whatever. And it’s truly a companion of the couch, in-bed Web, in the kitchen, and so on and so forth. So digital usage with the iPad, which is a different market, and I’m aware of that but as an example, the iPad has changed our behavior. And obviously, the augmented reality guys are hoping that something similar happens with AR.

Tish Shute: Which of course brings up the question, I’m assuming that some of the next generation of slates/ipads are going to have front and back cameras, GPS, and compass, right?

Thomas Alt: Actually we know that.

Tish Shute: You know that, yes. I assume that you know that, because are you working on some prototypes, and have you got some plans?

Thomas Alt: You have to understand that I cannot talk as I’ve talked as a researcher. It’s the rules, so I have to be a little bit careful about what I say. We very much think that a webpad, or whatever pad, you would want to call it is on some occasions very good device for AR.

Tish Shute: Yeah. But it is an interesting point with holding up a larger device, because you hands aren’t free, but the neat thing about the phone for augmented reality is that you really can do a lot with your thumb, as we’ve found and just the position of the phone. But, how will this work it with the two hands on the larger device?

Thomas Alt: Keep in mind, everyone’s talking about mobile augmented reality, but really where the case for augmented reality, at this point, is the strongest is in the installation business, it’s in the web business… Not necessarily only commercially, but also use case-wise. There are tons of museums out there which are using our augmented reality system in an installation fashion, and to communicate products better, and more efficiently, and so on, and so forth.

So, I know that the hype is clearly on the mobile augmented reality side, but there are many examples augmented reality experiences where holding up a larger device is not a big obstacle.

Tish Shute: Well this brings me to some questions about the future of mobile AR. My interview with Jay Wright focused on how we are now in a new period for AR bringing together computer vision, visual search into a mobile stack that is really optimized for AR. What do you see emerging in mobile AR as we move beyond compass, GPS, camera, accelerometer based AR into markless image-based AR. What will the new use cases and where will we see mainstream users getting in AR. Will AR games be the first mainstream AR experiences?

Thomas Alt: My partner is actually, first of all, one of my best friends, second of all, very emotional, third of all, very intelligent, and he said the other day something I think very valuable in this area. He said, basically think about Mobile Augmented Reality, Thomas. There’s really a very limited number of use cases which you can do if you look at these classical Point and Find applications, ok? But there are almost unlimited number of use cases when Augmented Reality becomes a day to day companion, ok? So what he meant is, ok, I’m looking at my normal day… I’m looking at the city, I’m walking throughout this, I’m coming home, I’m having dinner basically. I can deploy Augmented Reality in a pure POI search fashion perhaps not even once. Ok when I’m travelling it’s a different story, but in an ordinary day I might use a POI search not even once.

But where this ultimately leads, you know, is even in the 15 minutes I’m having breakfast, I’m using AR – looking at the cereal box with my cell phone, I’m taking part in a sweepstakes or whatever. So from that we draw the conclusion that as a general strategy for Junaio, we should basically throw as much technology as possible into Junaio, make it halfway self-explanatory, and just give people the possibility to come up with ideas on how to deploy augmented reality continuously.

We have actually got a creative team from an Art School working on that, and just, you know, with very little programming skills coming up with things you can do with Augmented Reality on a day-to-day level. And it could be a scavenger hunt game, in the city, with monsters flying around, it could be the normal POI routine, it could be marketing purposes, and so on and so forth. And I think that’s really the roadmap, and this is a little bit similar on a more technical level, to what Qualcomm is doing, ’cause they’re floating out possibilities or capabilities I want to call them, and Metaio is doing that, but on a higher level [re the tools] meaning on a Junaio level.

Junaio is a capability platform. It is also a way for Metaio to demonstrate the capabilities of our technology. We will offer all the possibilities for AR and more that we have already demonstrated on PC augmented reality.

Tish Shute: What is the business model for Junaio? Are you encouraging developers to develop business on your platform ?

Thomas Alt: Junaio is our end consumer platform and our business model is similar to the way Google structures its business model. We work with OEMs, content partners, brands, and developers to offer free content to our end users. Where we do charge is on the advertising side, more specifically contextual and location based advertising. At the current stage, we are focused on building the content base, fostering our developer community, but in the near future, we will be introducing advertising channels.

First of all you have to have very good use cases in the platform basically. And then to put a business model on top of that from a technology stand point is not hard – its a pay channel. Its all prepared for this.

Tish Shute: You have quite a broad base as a company don’t you – you do everything from industrial AR, marketing to technology licensing and more?

Thomas Alt: Basically, there’s a lot of things people don’t see us. There is also the Unifeye PC SDK and we have a client base and partners who are sourcing software from us, and we are doing great pieces. I mean the hype of augmented reality is really coming to a peak. There are lots of pieces that are not even talked about any more. China’s GQ magazine launched with AR from Metaio, the biggest AR campaign anywhere – there are a lot of potential readers in China. And um, so that’s our business model… we have our IP, our patents and so on. And on this we can move on onto the mobile platform whenever it’s advisable or feasible.

Tish Shute: Right. Yeah. I mean uh, you’re very fortunate to have this base built on uh, years of developing IP. What are the most important areas of AR that Metaio holds IP and patents in, in your view?

Thomas Alt: there’s sleepless nights in that too…

So far we’re extremely excited about what’s going on with Junaio, it’s one of our big, big success stories. But we are sensible and trying to experiment because, you know, analogies from the past won’t really work in my view for Augmented Reality in a sense, that, you know, you better bring for a new system to fly, for a new technology to fly, you better bring a very concrete use case to the table, ok? And a well-defined use case. And we are, right now, with Junaio, in a state where we are checking out what could be such a very defined use case.

Tish Shute: So how many users does Junaio have now?

Thomas Alt: Let’s put it that way, we are, especially in the last 2 months, we are very satisfied. But we are not disclosing that, because users, and we’re seeing that from the competitive landscape, always needs 1 page of description what exactly a user is.

You understand what I’m saying? So this is why, ’cause we don’t want to up or downplay things, we are very careful saying, with users. Because I mean we have people who are actually also commercially very interested in Junaio…  We go through with them and discuss what exactly a user is. Cause there’s more then…a download is not a user. An app or something on your phone is not a user, basically, in my opinion.

Tish Shute: I am still waiting to see someone do something with AR and the Four Square API, or now the Facebook Places API. Do you see an interesting potential in the marriage of the rapidly emerging location based social networking space and augmented reality?

Thomas Alt: Definitely. Augmented reality offers a way for users to find information around them easily. Adding in the social networking component such as geo-tagging, rating, commenting can enhance the user experience and create engagement beyond just viewing the information. For example, within junaio, an average user can create their own personal channel and geo-tag photos or leave text messages at different locations. They can create a virtual tour of San Francisco and share it with their friends. By connecting the social side with good content, the augmented reality experience becomes more fun and interactive.

Tish Shute: And, of course, there’s the Junaio API. Are you beginning to see developers use that?

Thomas Alt: Yeah exactly, I mean if you go to Junaio.com, you can get a login, you have an API description. And the way it works is, that you bascially set up a call, which contains the information you would like to have in your individual channel. You submit it to us, it will get checked for profanity and other things, basically. And then we admit it to Junaio basically. The API is huge.   You can also use Junaio indoors.

This is very relevant. And there’s a tool chain for that, and so on and so forth. You can do mission-based search with Junaio. It’s in there, it’s called Junaio Glue. And there will be another very interesting feature coming up in a couple of weeks. And you can just do it, you can do a scavenger hunt, a game, normal POI search, and so on and so forth. And it’s all active. And that’s, what’s sometimes difficult for us to communicate, is it’s really a capabilities platform, but on the other hand it’s obviously very good to developers. And I mean on the developers side there’s huge interest.

Social Augmented Experiences leveraging geoawareness and human and machine intelligence to create real time information brokerages, combined with an augmented reality view, can create a new opportunities to reimagine our relationships with each other and our environment.

This Summer, I have been on a blogging hiatus, which has meant I haven’t been sharing as frequently and, unfortunately, the second half of two conversations I had earlier this year, both of which have much influence my thinking on social augmented reality, have languished in private mode -  part 2 of my talk with Bruce Sterling (see Interview with Bruce Sterling, Part I: At the 9am of the Augmented Reality Industry, are2010, and part 2 of my conversation with Anselm Hook - Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, Part 1. Time to get caught up on some blogging!  The lightly edited transcript of Part 2 of my conversation with Bruce Sterling is posted in full below.

Bruce Sterling has been blogging all the key developments in augmented reality (amongst other topics of interest!) on his Wired Blog, and he brought my attention to Boskoi the Ushahidi based app for Android phones, augmented foraging pictured in use above – for more pics see fightthegooglejugend.

Augmented Reality and Real Time Information Brokerages

Picture above is the path the “nomads” took through the Westhaven cryptoforest with Pieter Bol,co-auteur of the book Biological Globalisation and Theun Karelse of Urban Edibles Amsterdam “who presented his ‘augmented foraging’ app Boskoi.“  For more see, The Cryptoforests of Utrecht and, Westraven Psychogeography, 6 June 2010. Note: Cryptoforests: 1) Urban forests hidden from view 2) Urban fallows that might or might not be considered as forests 3) Gardens gone wild)

My interest in the Ushahidi family of ideas was already fired up by a conversation with Anselm Hook early this year.  We discussed a number of Ushahidi related projects, Swift, Crisis Filter and Anselm’s project Angel, Augmented Reality, and my own keen interest in an open, real time, distributed platform for augmented reality – ARWave.

The Ushahidi platform and the related project Swift has pioneered the real time brokerage of information with people acting in curatorial roles or matchmaking roles coevolving with machine assisted matching to connect wants to haves.  Ushahidi uses multiple gateways including SMS, and Twitter.  But the Ushahidi family of ideas is extremely interesting when combined with augmented reality and suggests many new possibilities for social augmented experiences, as Anselm pointed out, for human to human communications, human to civilization communication, and human to environment communications (e.g., perhaps, how machine intelligence can help bridge the difference in time scale that Kate Hartman explores in her, Research for Glacier-Human Communication Techniques).

Ushahidi, which means “testimony” in Swahili, is a website that was initially developed to map reports of violence in Kenya after the post-election fallout at the beginning of 2008. It is now an open platform with a wide range of applications and growing developer community.  See What is the Ushahidi Platform? from Ushahidi on Vimeo.

Swift - a project that emerged from the Ushahidi dev community, is a human sensor/real-time brokerage for dealing with emergencies, enabling the filtering and verification of real-time data from channels such as Twitter, SMS, Email and RSS feeds.

Boskoi – augmented foraging is the first app,  I have seen, to begin linking Ushahidi with augmented reality – although I don’t think there is a full augmented view for Boskoi developed yet?

“The whole point of AR is to see things from a different point of view…”

Click to enlarge poster from upcoming ARWave demo at Software Freedom Day – for more see below

I am often asked what augmented reality brings to the table with respect to location based social networking, which is on the verge of going mainstream in smart phone apps like Four Square. While the first part to my answer is usually to explain what is unique to augmented reality.

As Bo Begole notes, the full vision of AR requires machine perception technologies to detect the identity and physical configuration of objects relative to each other to accurately project information alongside/overlaid with a physical object (see this post on the PARC Blog by Bo Begole on the difference between AR and ubiquitous computing – thank you Rouli for bringing my attention to this).

But it is only in recent months that we have begun to see the kind of tools that make this possible become freely available to developers – see my interview with Jay Wright of Qualcomm here.   Also see this post on Bundler: Structure from Motion for Unordered Image Collections an open source system that allows the creation of 3D point clouds from unordered image collections, e.g. internet image collections.  We now have many tools available to move mobile augmented reality beyond the recent crop of apps relying on GPS and compass alone for positioning into a new era of vision assisted AR apps that will increasingly bring the full vision of AR into our daily lives.

Further, the integration of visual search applications like Google Goggles and Kooaba which can detect the identity of particular objects will add another vital tool to machine perception technologies enabling AR “checkins” on potentially anything in the physical world around us, and more fuel to the Gamepocalypse (e.g. it would be easy to turn every trash can in the city into a basketball hoop as we discussed at the ARNY meetup last month).  And soon, the Pandora’s Box of  facial recognition (Google Goggles have the capability though it is not released to the public yet) will open up.

Jesse Schell described the importance of AR in a nutshell in his keynote for are2010:

“The whole point of AR is to see things from a different point of view…How can there be a more powerful art form than one that actually changes what you see?”

But how AR matures as a social experience will be the key to Jesse’s suggestion that:

“Augmented Reality will be one of the things that fundamentally define the 21st century”

There are many interesting forms of AR that are not reliant on a tight registration between media and physical objects – several are put forward by Bruce in the convo below.  And, it is likely we will see AR eyewear as an occasional useful accessory to a smart phone long before we have the sexy, affordable augmented reality eyewear worn that we wear throughout the day.   These speech to text glasses would be a very useful and viable accessory to a smart phone right now for the hearing impaired.

For the moment, as Bruce notes, some of the most interesting and useful augmented experiences to date have not been in the cell phone space:

“There are other aspects of AR besides the cell phone space. There’s Total Immersion’s big display screens. There’s the web-based fiduciary stuff. And there’s projection mapping. And then there’s experience design just for people who need their reality augmented for whatever personal or social reason.”

On of my favorite social AR experiences is this SMS Slingshot.

But I have been excited for a long while about the intersection of mobile social augmented reality, real time communications, and ubiquitous computing see Total Immersion and the “Transfigured City:” Shared Augmented Realities, the “Web Squared Era,” and Google Wave.  And I have described in many places why I think real time, open, distributed communications for AR are so important to developing social augmented experiences – see the slides for my talk at Augmented Reality Event here, here and here for starters.

ARWave at Software Freedom Day 2010, September 18th 2010

Thomas Wrobel and Bertine van Hovell will demo the first ARWave Android client at Software Freedom Day this weekend!

A number of people have asked me, (including Bruce), What will be the future of ARWave now that Google Wave is no longer a stand alone application?  Yes, the recently announced release of Wave in a Box (see here and here) is very exciting for the ARWave team.

The ARWave Android client is the first open AR client built on an open, real time, distributed platform -  based on a server that anyone can download and set up, currently the “FedOne” server but Wave in a Box, hopefully, will be even easier to deploy.  Wave in a Box seems perfect for ARWave’s needs – for more see the WiaB Google Group here.  And for more information on the ARWave client -  click to enlarge the poster below, see the ARWave concept video here, and for more, and how to get involved see arwave.org.   Props to Thomas Wrobel and Bertine van Hovell (posters below from demo for Software Freedom Day), Mark Evin, Davide Carnovale, and Markus Strickler, for all their hard and brilliant work on ARWave.  Also to JCPT the open Android 3D engine that has saved a lot of work!

click to enlarge slide

Social Augmented Experiences that Matter

My ideas on the future of social augmented experience have been deeply informed by the the conversations I had with Bruce Sterling and Anselm Hook this year.

Bruce Sterling notes in the conversation below, location based social apps like, Four Square, are interesting because they are not “urban geography like Google’s satellite stare from above,” but rather “groups of citizens are doing portraits of their own region.” Augmented Reality, with its of lauded power to make the invisible visible is, of course, is the ideal tool for “citizen portraits”  to the next level.  Cory Doctorow described to me three years ago (see here) an “inverse surveillance society,” enabled by an augmented view  – “where all the data from the positional and temporal characteristics of all the objects that we own were in aggregate visible and available so that we can mix and match them remix them understand them and have more agency in the world.”

It is very cool to go back to reread this conversation now that it is becoming possible to build the kinds of apps Cory described, and Bruce Sterling envisioned in Shaping Things (see Amazon.org  page 111).

click to enlarge

My  conversation with Bruce earlier this summer (see below) took place on the heels of are2010 – Augmented Reality Event.   See the video of Bruce’s keynote, “Bake a BigPie,” here,  and the final keynote, “Seeing,” by Jesse Schell (see video here) in which Jesse riffed on AR and the man with the X-ray eyes.  Both these awesome talks are still fresh in my mind.  Bruce noted how we should pay attention to augmentations for people and situations that could really use some augmentation… and not get too fixated on the coming of AR Goggles.  He elaborated on this in our conversation (again full transcript below):

“Well, it’s a matter of deciding whose reality it is that you’re trying to augment. I’m not trying to be a bleeding heart about it, but obviously there are people in our society right now with reality that could really use some augmentation. They are mostly disadvantaged people. They are vision impaired, or maybe they have autism. They might be senile and just can’t remember where they put their shoes. These are people who could really use some help, right?”

So, start with people who really need sensory or cognitive help. Before you turn our geeks into Superman, why don’t you try turning some people who are harmed into more functional individuals? Then you’ll be able to learn how to do that. Then maybe you can ramp it up to these Nietzschian heights of the superb Man With the X-ray Eyes. Whatever.”

What will make AR interesting and useful long before and long after we see the full vision of AR eyewear manifest is its social aspects.  Bruce points out:

“My argument would be that if you want people to be more sensitive toward certain, say, issues and problems, it’s better to find the people who are already sensitive to those issues and problems, and give them a bigger stake in your augmentation system.”

“Say that I am really worried about public health. Well, if I have a lot of nurses that are using my system, people who are aware of my issues, then I could be walking around and I’ll see a lot more tags saying, “This is where he got food poisoning!” “In this shooting gallery, many people have caught AIDS!” Or, you know, “Tuberculosis has been spotted over here in this building.”

At that point, I could simply share their knowledge and get some social intelligence. As opposed to trying to amp the basements of my little hacker-mind and drag stuff up that’s escaped my conscious attention.”

Finding new ways to broker information – bring together needs with haves and different participants, empowered and disempowered  is., as Anselm discussed with me, one way to change our view of human to human, human to environment and human to civilization communication (particularly in light of this  “sobering account of how open data is used against the poor in Bangalore” that as @timoreilly noted recently OpenData Empowering the Empowered).

The key idea in a crisis filter, Anselm noted,  was to break up the participants into different kinds, to connects wants with haves:

“There are people who are in  situation.  We call them citizens.  And then there are reporters, people who report situations back to Twitter.  And then there are curators, people that canvas Twitter looking for important Tweets.  And then there are first responders, people who take the curating collection of responses and then act on them.”

This kind of brokerage between people acting in a curatorial role or matchmaking role with each other can be extended into and coevolve with machine assisted matching as Anselm explains.

It is also a vital part of creating social augmented experiences that matter.

One of Anselm Hook’s projects, which is called Angel is the the most radical expression of connecting wants with haves in that the idea is that “you have a situation, you broadcast that situation, and help magically appears.  You don’t even sign up for a service.  You just get help …

As Anselm explains this is the same idea of a brokerage for dealing with emergencies, but applied to the long tail of crisis response.  As Anselm describes it:

“I am interested in personal crisis.  ‘I lost my cat.  Help.  I can’t find where my kid is.  I am out of gas.  I have a flat tire.  My house is on fire.  My aunt is trapped in the bedroom.’  The kind of personal crisis that is just as important, but is not enough to get a national movement to help you…

I will publish this conversation with Anselm in full in an upcoming post.

Zorop – an ARG for World Peace

If you want to be part of a really exciting experiment to reimagine our relationships with each other and can be in San Jose this weekend, I highly recommend exploring this “rabbit hole”.

Thank you Gene Becker, Lightning Laboratories and Ubistudio for sending me this invite:

“Ken Eklund (@writerguygames) is developing a wonderful game for the 01SJ Biennial called ZOROP, aimed at creating World Peace(!). Some of you might know Ken from his work on the amazing ARGs EVOKE and World Without Oil. Anyway Ken, along with his collaborator Annette Mees, are furiously working to get ZOROP ready to go for the Sept 17th premiere at 01SJ.

Are you intrigued? I thought so, and here are your next steps down the rabbit hole:

> Check out http://zorop.org to learn about the game

> Follow @ZoropPrime to watch it unfold: http://twitter.com/zoropprime

> ‘Like’ ZOROP on FB for a different view: http://www.facebook.com/pages/Zorop/141140772593618

> Become one with the game; consider volunteering as a Zoropathian: curious@zorop.org

> Head down to San Jose on the 17th, play the game, and ride the ZOROP Mexican Party Bus. Seriously.”

Interview with Bruce Sterling

Click on image above to see video clip from from brucesflickr

[Note the first part of this interview is here and I broke in anticipation of Part 2 just as I started experimenting with an idea Joshua Kauffman – an advisor and entrepreneur working on design in the public sphere gave me for an interview technique – the All Souls College one-word question interview.  Although apparently they recently scrapped it and I am not very good to sticking to a single word!]

Tish Shute: We were talking about these proximity-based social work networks like Foursquare and Gowalla and how they may influence the emergence of social augmented experiences.

So Joshua’s suggestion for the first word was “territorialization” e.g. how do these new mobile social experiences like Foursquare, and the observation that actually rather than breaking down territorialization – which would be a good thing, tend to support territorialization…

Bruce Sterling: Yeah, they’re re-intensifying it in a very odd, electronic fashion.

Tish Shute: Yes.

Bruce Sterling: It’s not true of projection mapping or the webcam fiduciary display stuff. But with the handheld stuff, and especially the urban informatic stuff, it really can’t help but take on a local flavor. Layar is like “Augmented Dutch Reality.”

And TonchiDot is “Augmented Japanese Reality.” It’s hard to imagine a Layar interface going gangbusters at Tokyo. Whereas the TonchiDot interface, which is so clearly influenced by Anime and cartoon graphics…. Maybe it could find some niche of hipsters in Amsterdam hash bars…

Stuff that’s socially generated by people on the ground, as with Foursquare and Gowalla, is bound to take on a regional influence. Right? It’s like the New York hipsters who were early adopters of Foursquare. They’re not mapping New York! They’re mapping Hipster New York.

It’s all about Williamsburg and places where 24-year-olds go to drink… They found a demographic niche there. These guys are building the service for them. They’re people who are willing to work for Foursquare for free, because they want to wear the little king hat.

Tish Shute: I got the far far away badge ‘cos I live on the Upper West Side!

Bruce Sterling: But that’s not urban geography, right? I mean, that’s not like Google’s satellite stare from above. That’s a group of citizens doing a portrait of their own region. You’re going to see interesting things happen because, of course, people who use Foursquare elsewhere are going to check into New York, and they’re going to look at the “New York Foursquare.” They’re going to be aliens who interact with Foursquare people in New York and annotate what they’re seeing.

Tish Shute: Oh! Yes. Good point.

Bruce Sterling: That Foursquare community has a certain émigré soul. It’s different from the normal émigré soul of simple tourists on New York. So you’re friend there is right about the territorialization.

Tish Shute: Yes, Joshua Kauffman is a smart guy! Yes I am interested to see what interesting kinds of deterritorializations proximity based social networks and the hyperlocal view of augmented reality might bring, not just the new territorializations.

Bruce Sterling: It’s not the intense kind of territorialization, like gangs putting down graffiti markers and beating people up. It’s an inherent regional character that comes with using peer production to build your database.

Tish Shute: We were discussing whether AR could break down the walls between people – people who share the same physical space but actually inhabit different territories even if they are sitting on the table next to you.

Bruce Sterling: You know, I just wrote an article for my Italian magazine column. I think I mentioned this to you – a report about ARE 2010. I titled it, “Chicks Dig Augmented Reality.”

Tish Shute: [laughs]

Bruce Sterling: There is a very heavy social element to AR, and a phone based element. So the question is: Why would a woman wear a fiducial marker? Like our Metaio speaker at ARE2010 who had a fiducial marker on her lapel pin.

Tish Shute: Right. Lisa!

Bruce Sterling: Why would a woman go out in public with her Facebook profile on her body?

Tish Shute: Well I can think of some reasons…

Bruce Sterling: So that men will approach her, of course.

Tish Shute: Yes the core of all successful social networks is always a form of dating app.

Bruce Sterling: You do it a social icebreaker. It’s like: I’m a woman, I’m sitting here alone, and you can sort of glide by and, you know, take a snap of me. Then you retreat and have a beer with your friends and you work up the courage, and then you come and say, “So! Susan! I understand you like bicycling! And, boy, me too!” Right?

Tish Shute: There are all kinds of social barriers between people in cities that AR might be helpful in breaking down. An extreme example is the dilemma you actually quite often face as a New Yorker as you walk around a city. There are people asleep on the pavement and you don’t know if they’re dead or alive.

Bruce Sterling: Right.

Tish Shute: And you sort of like have this awful ethical dilemma of like, “Am I walking by someone I should be shaking by the shoulder, right, to wake them up so they don’t die, right?”

Bruce Sterling: Yes.

Tish Shute: You said in your keynote that we should pay attention to augmentations for people and situations that could really use some augmentation..

Bruce Sterling: Right. There actually is such an app in Britain right now. I posted about it: two Augmented Reality schemes for rubbish and hobos.

Tish Shute: Right. Yes I saw that!

Bruce Sterling: “Any sufficiently advanced technology is indistinguishable from garbage and hobos.” You don’t need to personally find out whether this hobo is worth your help. What you need is a good way to report the hobo to a hobo check-up service. They come in, and they look on their own database or supply a database to you, or a facial recognition unit, whatever. The service says: “Oh, well. That’s Fred. He’s a paranoid schizophrenic. He always sleeps in that alley. Let him be.”

The same goes for the rubbish — although I don’t want to compare rubbish to hobos. In fact, people do go out with their AR kits and take pictures of abandoned garbage bags and broken glass. They upload them with geolocated tags for the local garbage guys. Guys who are sitting around doing pretty much nothing because they don’t know where the rubbish is.

And they will come out and get the rubbish! I mean, they just deputize guys to go out and follow these alerts. Right?

But nobody predicted — least of all me — that you were going to have a high-tech Augmented Reality system that consisted of removing rubbish and derelicts. Right? But rubbish and derelicts always go profoundly under-reported. It’s just hard to get people’s attention.

But it’s very easy to set up a system so that, if you get ten reports on the same piece of rubbish, that’s going to work its way to the top of the stack. That’s why I was trying to get AR people away from the romance of the hottest app for the shiniest machine. More toward a design stance that’s more user-centric.

Where are the actual problems about stuff that we perceive? Stuff we can’t do anything about? Or people whose mechanisms of perceptions are harmed. They could be doing good work, being more participative, if they didn’t, basically, walk around without their glasses on.

Tish Shute: Well this leads well into the second word, Joshua suggested was interesting spring board – sensitivity.

On the one hand we can do these things for people who maybe need the augmentation because they have difficulty with one or another sense, e.g., their eyes are not functioning, or their ears are not functioning. But on the other hand, we can’t cross the social bridge to communicate with people who are temporarily disempowered in relation to the rest of society e.g. hobos and people who sleep on the streets of New York City.  And even though Augmented Reality could potentially be helpful it can even be more disempowering to the already disempowered.

Bruce Sterling: Right.

Tish Shute: But re “sensitivity” – does augmentation increase or decrease our sensitivity? This is a problem that Will Wright brought up [see video of Will Wright’s keynote at are2010], e.g, the problem of parking HUDs getting in the way of your intuitive parallel parking skills. The Lexus that takes driving control from you when you look back, ‘cos it knows that you’re looking at the road, and it starts to brake. Right?

Bruce Sterling: Right.

Tish Shute: The fact that the problem with technology is that it makes us less sensitive, right, augmentations sometimes get in our way?

Bruce Sterling: I suppose that’s true. But I’ve heard that said about practically every medium. Especially television.

Everybody wants to blame machinery for their lack of morality. It’s hard to top something like the Kitty Genovese killing in New York. This sort of legendary New York horror story from the 1960s. A woman is stabbed to death in public, no one does anything.

Tish Shute: Right.

Bruce Sterling: I don’t think that our media is making us any less humane or more callous.

Tish Shute: All right. Oh no! I see what you’re saying. Perhaps I misrepresented what Will was suggesting by putting it that way. The question is perhaps more how do we get the sensitivity into the technology. Human bodies are fantastically sensitive and sensory.

Bruce Sterling: Right.

Tish Shute: And we have these like sensitivities. For instance, How could augmentations of reality be like a blush ? You definitely want an interaction that’s not just this data being pushed at you. But what is the data that counts, right? Will shows a slide often of an iceberg with the tip of the iceberg which is the conscious mind.

Bruce Sterling: Oh, I see. Yeah.

Tish Shute: And underneath it is all the preconscious stuff that really counts, right? Any thoughts on that?

Bruce Sterling: I did take interest in that. Will has obviously been spending a lot of time studying cognition.

Tish Shute: Yes.

Bruce Sterling: I’m not convinced that AR has got a lot to do with that. There is certainly a trend there. There are a lot of people who want to do body hacks and brain hacks. I can imagine AR being used for that purpose, but it seems like a niche application. What is the point of our accessing even more stuff that’s outside of our consciousness?

Tish Shute: One of the things he is talking about is game dynamics, is it? The role of the imagination in play. For example, he shows the high dynamic range photos that make the world magical. Something you want to engage with playfully. This he points out increases a sense of agency because you are encouraged to engage and to play with the world.

Bruce Sterling: Well, I’m a literary guy. Italo Calvino did a lot of writing about this. He talked about the classics of literature. Why do we read the classics? Calvino said we do not read, but reread the classics. And the reason we do that is that, at first, we read a classic book and we think, “Boy, this book is really good.” Then, five years later, we read it again and we think, “Boy, this is a really good book, and it’s got so much more in it than I thought it had when I was 18.” Then we read it again at 28, and it’s like, “OK, now I really seem to understand this book, and it means something to me now that I didn’t know when I was 18 and 25.”

What you are doing through that access is learning something about yourself. So Will is arguing is what I really need is like a better augmentation. So that I can go in there and sop up the book all at once. I can grab every cultural nuance in it, instead of the stuff that’s sliding past me because I’m 18 and kind of young and hasty. Maybe I could have certain words and phrases helpfully underlined, that are like, “OK, well, this part is problematic for you.” In some sense, that’s not allowing me to be 18.

I’m never going to have the experience of my own maturation against this text, because I’ve devoured it all in one gulp.

My argument would be that if you want people to be more sensitive toward certain, say, issues and problems, it’s better to find the people who are already sensitive to those issues and problems, and give them a bigger stake in your augmentation system.

Tish Shute: Yes the social augmented experiences are going to be the most valuable.

Bruce Sterling: Say that I am really worried about public health. Well, if I have a lot of nurses that are using my system, people who are aware of my issues, then I could be walking around and I’ll see a lot more tags saying, “This is where he got food poisoning!” “In this shooting gallery, many people have caught AIDS!” Or, you know, “Tuberculosis has been spotted over here in this building.”

At that point, I could simply share their knowledge and get some social intelligence. As opposed to trying to amp the basements of my little hacker-mind and drag stuff up that’s escaped my conscious attention.

Tish Shute: Interesting that seems to bring us to another kind of repetitive theme in AR, the people tend to pigeon hole it as “merely” a visual interface. But actually, it’s the intersection, isn’t it, of social intelligence and augmentation.

Bruce Sterling: Well, it depends entirely on how you design the system. If I’ve got a military augmented reality, I would expect that to be mostly about urban fighting. It’s going to be about kicking in a door and shooting terrorists. If I pull that helmet off my head and put that on the head of an emergency worker or a cop, I’m going to get a militarized cop or a militarized emergency worker.

Tish Shute: Well the histories of the two great mass media of the twentieth century – TV and the atomic bomb were intertwined, and I suppose the evolution of ubiquitous media, augmented reality and urban warfare is already intertwined too.  So how can we encourage augmented realities to move beyond military roots that is common to much technology and into more peaceful urban realities?

Bruce Sterling: Well, it’s a matter of deciding whose reality it is that you’re trying to augment. I’m not trying to be a bleeding heart about it, but obviously there are people in our society right now with reality that could really use some augmentation. They are mostly disadvantaged people. They are vision impaired, or maybe they have autism. They might be senile and just can’t remember where they put their shoes. These are people who could really use some help, right?

So, start with people who really need sensory or cognitive help. Before you turn our geeks into Superman, why don’t you try turning some people who are harmed into more functional individuals? Then you’ll be able to learn how to do that. Then maybe you can ramp it up to these Nietzschian heights of the superb Man With the X-ray Eyes. Whatever.

Tish Shute: Did you notice that a couple of apps actually like TagWhat have apps geared towards people with disabilities – I haven’t had a chance to check it out.

Bruce Sterling: I’m sorry, I wasn’t looking at their tags.

Tish Shute: I was discussing this with Joshua who mentioned Zachary Lieberman’s Eye Writer, which is for people with locked-in syndrome. Do you know that?

Bruce Sterling: Sure. And people appreciate that because the poor guy, he’s laid up with Lou Gehrig’s Disease. Now they’ve given him a way out. AR is like a spark of new hope that gives his life meaning. What’s wrong with that?

Tish Shute: Yeah. And Tim Byrne using Sixth Sense for Autism is interesting.

Bruce Sterling: Let’s consider it the other way. Let’s say this graffiti writer there, instead of him being sick and weak, let’s say he’s an athlete. So I want to make him into a super-human graffiti writer. I want him to run around graffiti-tagging the entire town before dawn. Is that a good idea? Do we need that? Super human, super taggers? What if he’s going to spray up stencils of Nietszche? I kinda wonder whether the game is worth the candle.

Tish Shute: Yes I suppose it is not a great social scenario to be always augmenting the lives of the elites! Hmm, the third single word interview question is “homophily,” and earlier were you’re saying that we’ve kinda got to accept this is very much part of AR – as how it works, because hyperlocal experiences gets created by local communities – that up to know have tended to be homophilies.

Bruce Sterling: Well, I think that’s easily handled with some design thinking. You’ve got to do some user observation and show some sympathy with the user, and to be aware that you’re designing for the user and you’re not designing for yourself.

In a field as young as this, it’s mostly geeks building cool stuff for geeks. In a lot of ways, it’s a “can you top this” contest. That’s OK, but it’s not good design to be your own client all the time. It’s like writing novels to amuse yourself, or sitting on the porch singing the blues on your own guitar with only yourself to hear.

Tish Shute: What will it take for AR mature out of this “geeks building cool stuff for geeks” phase do you think?

Bruce Sterling: It’s necessary to master some of the tools first. I think of the way the web has developed over the years. When the World Wide Web first appeared, it was just for physicists, and was all line commands and quite unstable and difficult. Then you got usability studies, and things like Ajax and so forth. It’s a very painstaking thing.

We’re not best at building interfaces for the best computer scientists. Web 2.0 was built from things like watching people cry while they were trying to fill out insurance forms. “Well, why are you so upset?”

“Well, I got to the end of the webpage, and then it said I took too long, and it cut me off and now I have to start all over!” Jesse James Garrett, right? Benefactor of mankind.

If you’re experienced, you think: “Why don’t I build a little module here, and kind of move the form over here, then I’ll periodically update it with some asynchronous Java and XTML.” And people are like, “Gee, how odd.” But that really works for real people. It comes from studying what people want to do. Whereas, the current AR approach to a problem like the insurance form would be like, “I will give you the ability to record the entire insurance form, and it will flash before your eyes!” OK great, that’s a cool hack, but I don’t really need X-Ray Eyes to fill out my insurance form. What I need is a more user friendly interface.

Tish Shute: Well it seems like we are moving into the terrain of Joshua’s fifth word “ventilation,” – if I understand it rightly – it is at least partially the antidote to territorialization because it’s this idea that a place needs air so we come out of our hermetically sealed boxes of the way we relate to a place and what kind of augmentation would bring more oxygen to that space.

There was an interesting moment in the Auggies because when Maarten Lens-FitzGerald presented the guerrilla shopping Layar and basically Mark Billinghurst and Jessie Schell who spoke first didn’t seem too impressed. They didn’t want to walk to shopping – that was what web shopping did, it saved us from walking to shop… but I felt, to me you picked up on something which might have some bearing on “ventilation” in that this AR shopping Layar was kind of squatting Prada – a favela chic AR shopping thing?

Bruce Sterling: I wasn’t sure if I was interpreting what Maarten had in mind by that. But I think Maarten sees his structure accurately as an experience thing rather than a mapping thing. I think he’s proudest of things like the Berlin Wall app on Layar, as opposed to Layars that help you go get a hamburger. It’s like…so when Layar inserts parasitic augmented shopping over other people’s real shopping? That was rather a subversive thing.

I think the key there is that his client is called “Hostage T-shirts,” right? I mean it’s actually kind of a transgressive little hippy T-shirt store that Layar can dump anywhere in the world. Layered right over, say, Versace and Prada. I don’t know what becomes of that effort. And I’m not sure about the term “ventilation,” because that’s a term of art I haven’t heard much.

Tish Shute: Maybe it’s like in a cafe. Ventilation would mean we were able to communicate with all these different categories of people that we normally would be unable to connect to, even though we might be sitting only a few feet apart.

Bruce Sterling: So it means ventilation in the bottles of our homophilies.

That’s not a personal problem for me. I commonly live in foreign cities and, you know, and spend a helluva lot of time talking to strangers at conferences. So I don’t think I’d have that particular tight little social island problem.

Tish Shute: Of the three judges at the Auggies, you seemed most enthusiastic about the Layar entry.

Bruce Sterling: It may be they’re not as familiar with the business models of locative AR as I am, and as Maarten is. It was kind of a subtle in-joke he was making about Layar’s own business model there.

Tish Shute: How do you explain that?

Bruce Sterling: Well, you know, Layar’s in the business of selling software to make mapping and urban structures into ecommerce.

The ideal way to do that obviously would be to move the richest customers into the most expensive shops in the most rapid way possible. Or at least distribute them in the directions they want to go, a la Google. Whereas this app that Maarten was talking about puts big barnacles in the way that are selling punk t-shirts.

Tish Shute: Right! Right!

Bruce Sterling: The Dutch are a bit subtle in their humor. I rather imagine there’s a lot of discussion in Layar’s inner circle about exactly what they want developers to do with their platform. They’re going to have considerable political difficulty deciding who can have a Layar key and how you discipline people when they start doing weird stuff. “The Oakland Medical Marijuana layar.”

Tish Shute: Well, finding nudists is one of the top layars at the moment.

Bruce Sterling: You know, obviously so. And finding narcotics in Amsterdam, or a prostitution layer. I warned them nine months ago this was bound to happen. I’m sure they’re aware of it. I don’t think Layar wants Google’s style of cool, technocratic detachment.

Tish Shute: But that’s pretty difficult to do in current augmented reality because we don’t have all the mathematical voodoo for full on AR search yet, do we?

Bruce Sterling: Well, you can hire it out. Somebody’s going to do it, if they get interested enough. There’s Nokia-Yahoo. Nokia-Yahoo! just did a big corporate deal…involving Nokia’s mapping system and Yahoo’s localization. So the Nokia-Yahoo! mash-up is called Nooo! Or could be called Yahno. Yakia! Unfortunately ridiculous names.

Tish Shute: It’s interesting because you mentioned the spiders’ mating problem at Google. They’ve got all the pieces to make this kind of level of AR obviously right now. But they actually haven’t done it yet.

Bruce Sterling: There must be at least some discussion in Google, but the same goes for Microsoft. I’m frankly baffled by Microsoft, because it’s just full of insanely brilliant people. What the hell are they doing in there? Name one serious innovation that’s come out of their labs in five years. They make Integral Research look dynamic. It’s really kind of sad.

Tish Shute: It’s a very curious situation with AR though, because AR more than any new technology relies on these big hordes of data particularly for the mapping, right? And only the big four have the data – although we are beginning to see upstarts, Earth Mine, Simple Geo… Did you get a chance to meet Di-Ann Eisnor from Waze – real-time maps and traffic information based on the wisdom of the crowd.  Waze is a very interesting project that is a potential giant killer.

Bruce Sterling: No, I didn’t talk to them. I’ve seen people speculate that Earthmine and Apple are going to make an allegiance. I guess if you’re thinking that urban informatic mapping is a super big thing for AR, that must be true. But I’m not convinced that’s necessarily the case. People have pointed out that you can just use Google Maps, and you don’t have to walk around with a little visor. There are other aspects of AR besides the cell phone space. There’s Total Immersion’s big display screens. There’s the web-based fiduciary stuff. And there’s projection mapping. And then there’s experience design just for people who need their reality augmented for whatever personal or social reason. [dog barking]

Tish Shute: Right. Oh, I’m in the middle… My son’s come. What a good hair cut!

Bruce Sterling: Hi, there.

Tish’s Son: Hi.

Bruce Sterling: How’s it going, sir? Good to see you…

Tish’s Son: Good.

Tish: [laughs]

Bruce Sterling: Yeah. Nice looking shirt. I like the back of it.

Tish Shute: That’s from the American Shaolin Temple. [laughs]

Bruce Sterling: All right. Awesome. Kung Fu geek shirt.

Tish Shute: Yup he is a bit of Kung Fu Geek. He and his dad did an iPhone app on it for Yu-Gi-Oh, for Yu-Gi-Oh scoring.

Bruce Sterling: Awesome. Plenty of Pokémon-style combat in Yu-Gi-Oh.

Tish Shute: Yeah. Well, it’s interesting because you’ve talked about this aspect. That all of this, the Pokémon aspect of AR hasn’t kicked in yet. But it’s obviously a match made in heaven to some degree, isn’t it?

Bruce Sterling: One would think so, yeah. The whole little kid gaming thing. What does that have to do with Google or Bing? You don’t need a massive database for stuff like that.

Tish Shute: Yeah, you’re right. But good tracking, mapping and registration requires a lot of mapping…

Bruce Sterling: Well, our current tracking, mapping and registration requires that. Maybe there’s some other way to hack it that we don’t know about yet.

Tish Shute: That’s a very interesting point. We always have to stretch the way we think about mapping… perhaps its a real-time understanding of the location you’re in…perhaps the map is being negotiated through several social processes?

Bruce Sterling: There are maps, and then there are maps. There’s a kind of artillery map where you need to know the precise location of target spaces. And then there’s the kind of social map where I’m really looking for the IN-N-OUT Burger where my sister went last Tuesday. That’s a different system.

Tish Shute: And I think AR, at the moment, we’re getting the most out of the social maps certainly. And the other [machine perception technologies to detect the identity and physical configuration of objects relative to each other to accurately project information alongside/overlaid with a physical object] is still kind of the big dream, isn’t it?

Bruce Sterling: They say that men never ask for directions and women never read maps. Clearly, the genders have different ways of navigating the world. Who’s to say what manner of augmenting our experiences is hottest? I’m not convinced that today’s rather rigid geolocativity is really what our society wants from that particular service. Maybe what we want is something more folksy. Some useful nudge in the right direction as opposed to grids with 200 meters here and instructions to turn such-and-such.

Besides, there’s other hacks we haven’t considered. We’re very dependent on GPS, but just suppose all those satellites are blown out of the sky in a solar storm. Would we really want to give up mapping? Wouldn’t we just come up with some other nifty hack? Radio beacons, let’s just say. Atomic clock timers in towns. Or maybe just little QR codes on lampposts that give you the exact location of that lamppost, and just click the thing and have it calculate where you are.

Tish Shute: Yes the NextHope OpenAMD project had a clever way of triangulating location indoors.

Bruce Sterling: Well, GPS is there and people all want to use it. It’s got good API, so of course you want to. And the guys who are good at doing it are real geolocative freaks. But the mere fact that we can do it this way, and that you can make it pay, doesn’t mean that it’s the ultimate way to provide that service to people. It’s like saying that Egyptian hieroglyphics must be the greatest way to write, because we’ve got a lot of them and they’re hard to learn. What if somebody comes along with an alphabet? It’s going to be a little embarrassing.

Tish Shute: Yeah, that’s a very good point. Now, this is a more simple ordinary question about the event. YDreams went off the map in the Auggie voting, and walked away with The Auggies. No one doubted that that was the most…

Bruce Sterling: I don’t know. I thought those Occipital guys with the panoramic painting…. That was hairy. I would have been tempted to give them the prize myself, actually.

Tish Shute: And what did you like best about that? Because I agree. I love Occipital.

click to enlarge

Bruce Sterling: I thought it was a more technically difficult stunt than the hand registration thing. Using a hand as a 3-D cursor is hot, but not like painting a panorama in 3-D in real time. That was an impressive technical feat.

Tish Shute: And they hinted at the 2.1.1 AR, more AR version of that. What do you see coming out of that as possibilities?

Bruce Sterling: Well, I’d heard of YDreams, so I wasn’t stunned. But I’d never heard of those guys. I wonder what else the heck they’ve got in the attic.

Tish Shute: very cool stuff…

Bruce Sterling: Well, more power to them. But clearly YDreams was the popular favorite. And who couldn’t like it? It was just so AR.

Tish Shute: It was so AR and so gorgeous.

Bruce Sterling: It was pretty, actually.  Except for their ugly menu button and poor font choice.

Tish Shute: Oh, yes. You didn’t like that, did you? [laughs] But with the Occipital panorama, what do you see the next stage of that?

Bruce Sterling: Well, obviously quicker and faster. Quicker and faster and more accurate in a network. Let’s just say I’m in New York and you’re in New York and I’m calling you for help. And you say where are you? I just whirl around like this and I mail it to you on a Google Wave. And you whirl around like that, and then we compare the two panoramas and do an instant triangulation. And you say: I’m over here on this red dot of your screen.

Tish Shute: Yeah, exactly.

Bruce Sterling: We’re navigating with panoramas by having two connected panoramas and considering the difference.

Tish Shute: Yeah, very interesting…

Bruce Sterling: Not shabby, right?

Tish Shute: Not shabby at all.

Bruce Sterling: If you could do it in real time.

Tish Shute: Then the other thing I missed because I was going to meet Will was I missed the Launch Pad competition. Did you catch that?

Bruce Sterling: I didn’t see it either. I thought of another app though.

Tish Shute: Oh!

Bruce Sterling: You’ve got a panorama maker in your home office, and it just scans the office 24 hours 365 and tags anything that moves, right? OK, where’s the clipboard?  At 8:15 it was over here. Now it’s vanished. Now another object is viewed over here. So, logically, ping, you hit it with a sticky light and there it is, right?

Tish Shute: Oh, that’s cool also knowing what has changed in any environment would be a big enabler for a lot of AR visions.

Bruce Sterling: I’m sure there are many other things you could do with panoramas.

Tish Shute: My jet lag is beginning to kick in big time – so many ideas to pursue from are2010 – those panoramas are very exciting though.

Bruce Sterling: Oh, well, it’s all right. We can augment reality! I’ve got three heads and six hands!

I have been so busy working on Augmented Reality Event, Jun 2nd, 3rd, Santa Clara, CA, in recent weeks that I have barely had time to post!  But it’s getting hard to contain my excitement about ARE2010.  If you haven’t already seen the sneak preview of the schedule  – see here.

Augmented Reality Event will be an unique opportunity to see how the complete vision of AR is emerging, one that will include visual recognition of real life objects, sensors to enable interaction with physical objects, and the long anticipated, comfortable, sexy, AR eyewear.  Six companies with be presenting and exhibiting AR glasses and HMDs at Augmented Reality Event.

Keynote luminaries will include: Bruce Sterling, “AR Prophet,” Will Wright (Spore, The Sims & now AR and The Stupid Fun Club), Jesse Schell (who will give the AR follow up to his now viral Dice talk), and Blaise Aguera y Arcas (Microsoft, Bing), whose talk on augmented reality maps was one of the most popular talks at TED this year.

Over 40 AR companies from all over the world are represented, and there will be speakers from Qualcomm, Google, Microsoft, Nokia, Intel, and Nvidia.  Over 80 augmented reality experts and entrepreneurs are presenting, including a AR titans, Total Immersion and Metaio.   Mobilizy, Layar, Ogmento, TagWhat, Tonchidot, and Zenitum (just to name a few of the stellar augmented reality start ups coming – for more, and a list of speakers see here) will demonstrate AR is not just a vision, but actually drives significant businesses today,

Five Augmented Reality industry start ups will be on the hot seat in a Launchpad competition with $10,000 prize for the winner, thanks to the generosity of  Qualcomm, a leading developer and innovator of advanced wireless technologies, and the featured sponsor of Augmented Reality Event.  Zenitum, a leading augmented reality company, with a technology portfolio that includes both location and advanced vision-based sensory input is the Gold sponsor for ARE2010, and Adobe are sponsoring the ARt Gala & Reception.  Many thanks to the sponsors for making ARE2010 possible.

Readers of this post can use my discount code TISH245 to register here for a $245 price for the whole 2 day event.

But just to add some balance to my total enthusiasm for the rocket like trajectory of augmented reality in recent months, this post will also look at an area that has been, to date, disappointing for AR developers.   While Nokia will have a couple of speakers at ARE2010, and certainly they have a very impressive history in augmented reality research, they have not delivered the magic brew to attract the augmented reality developer community.   Will this change with the release of the N8 (see video opening this post)?  I wanted to find out if AR developers see a light at the end of the tunnel with N8 as a platform for AR.  So I asked some AR developers and entrepreneurs, “How attractive is the N8 for the AR community?”

Nokia’s apparent nonchalance about augmented reality is in sharp contrast to the bear hug shown in this Verizon Droid Augmented Reality Ad featuring Layar!  As Bruce Sterling notes,

*Hollywood-up for Layar by cooking up some o’ those big-time sci-fi production values.
All magic, all the time, Mr. Roboto.

And it is clear that everyone has high hopes that iPhone SDK 4 delivers fully for AR developers this time! For more on this listen to @chrisgrayson’s 3rd video on Mobile AR here.

But before I get back to my question about the AR community’s thoughts on N8 as a platform for augmented reality, let me crow just a little more about the upcoming awesomeness of Augmented Reality Event.  As the chair of the technology track, ARE2010 is on my mind night and day at the moment!

The tools for the kind of augmented reality we have always dreamed of  – the zero click interface to a heads up, hyperlocal view, bringing you computing anywhere, anytime, with anything, are evolving fast.  Powerful algorithms for search, and machine learning, combined with cooperating cloud data services, will soon be bringing apps that learn by context accumulation to your AR view (see Siri).  Google Goggles, who will have a speaker at ARE2010, have already announced their intent to open APIs, putting a vital tool of visual search in developer’s hands.

A bunch of technologies are maturing these days and helping bring AR to consumers.  Mobile devices that pack all the ingredients for AR, location-based services, see thru video goggles, visual search, innovations in 3D mapping, Simple Geo’s innovative a approach to aggregating geo data, and the next generation of “Street View” such as Earthmine, and, very importantly, open standards and software for AR that will bring many new opportunities for creativity, and monetization.   These are just some of the aspects of this energetic technology convergence that will be represented at  Augmented Reality Event.

In the area of open standards, I am especially excited about the debut of Polaris at Augmented Reality Event.  Polaris is an AR browser, from Blair MacIntyre and the Georgia Tech team, based as much as possible on existing standards and tech.  More on this and ARWave in another post!  Also, I am excited to be demoing ARWave at the Com.Geo 2010 conference, June 21 – 23, in Washington.  Again more on this later.

But for now back to my question opening this post:

“Are Nokia back in the AR?”  and “How attractive is the N8 to the AR community?”

First I asked, my co-chair on Augmented Reality Event, Ori Inbar of Ogmento – one of the augmented reality start ups who’s star is on the rise, and Ori (also known for his writing on Games Alfresco) is one of the key thought leaders in the emerging augmented reality industry.

Ori Inbar:

“Well, the hardware spec is impressive – but that’s not what makes it a good AR device. What’s really important is reach, distribution, and ease of development – areas in which the N8 still has a lot to prove.

- Will the N8 become as prevailing as the iPhone or lead Android phones or will the Nokia landscape remain fragemented with too many variants of phones to support?

- Will the Ovi store pick up steam and become a lucrative chanel for selling apps?

While we wait for these questions to be answered – we’ll keep developing games for Android and the iPhone “

So there you go.

Ori is right that augmented reality developers like the hardware specs.  Here are some comments from Patrick O’Shaughnessey of Patched Reality, Peter Meier, Metaio, and whurley of Chaotic Moon and Robert Rice, Mirascape.

Patrick O’Shaughnessey:

“Looks like it has everything one would want to make both looking glass and magic mirror AR, and built-in face recognition software to boot!  I wonder if they allow you to use both cameras at once making AR video conferencing possilbe?”

“Dedicated graphics processor with OpenGL 2.0 enables 3D graphics” nice!

“Looks like it also does Flash (Flash Lite 4.0, anyway). Not sure if Flash Light provides access to the camera. Could be a nice way to do quick and dirty POC’s on the device”

Peter Meier:

“Metaio thinks the N8 is a very promising device for AR. Especially in Europe we expect a lot of reach.  Symbian is hard to develop for, but on the other hand is much more efficient in terms of real time performance than Android and less restrictive on APIs than iPhone.”

whurley:

“It looks really great and perfect for ar in many regards.”

“clarity of a 12 MP camera with Carl Zeiss optics will give the Nokia N8 some clear advantages in the AR world.  The clarity of this camera will allow for more advanced applications of augmented reality such as image/object recognition”.

“At first glance this is definitely very impressive from a hardware specification. I am surprised though, at the lack of buzz about the device. I have heard virtually nothing about it in the usual sources I monitor, and I wonder about the marketing.

Given the technical superiority and the difficulty Nokia has had penetrating at least the North American market, maybe something catchier like the GR-8 (Nokia Great) or M8 (Nokia Mate) might be a bit more user friendly and easier to market. I particularly like “Nokia M8″…take it with you wherever you go, can’t live without it, etc.

Anyway, as awesome as the specifications are, I would like to echo some of the other comments here…tools, community, ease of development, etc. are critical parts of the ecosystem that would make this ideal for AR development, above and beyond the pure hardware specs. I think Nokia has to really re-think some of its strategy to engage consumers, attract the developers, and establish a new beachhead in North America.”

Talking with Michael Halbherr at Where 2.0

I talked to Michael Halbherr briefly at Where 2.0 before the release of details on the N8, so, of course, he was unable to be as explicit as he might have been about the N8.  Below is a lightly edited transcript.  At Where 2.0, Michael Halbherr  gave a clear explanation of  Nokia’s emphasis on a global strategy.   And, as this blow by blow account on the Google Earth Blog noted, “They’re doing some cool stuff with their handheld maps. In particular, their Ovi Maps (turn-by-turn navigation) are now completely free and becoming very popular.”

But the question remains:  Where is AR is on the Nokia roadmap?

Michael Halbherr (Nokia) on stage at the O’Reilly Where 2.0 Conference 2010 in San Jose, California. Photo by James Duncan Davidson. See Where 2.0 Flickr set here.

Tish Shute: Can I ask you a few questions about Augmented Reality?

Michael Halbherr: So for me augmented reality and mixed reality is essentially a way you search.  So when you look at where we come from, as the company that acquired Gate 5, we believe the phone is essentially the gate that basically combines or connects your virtual and your real world in both ways.  So we can have virtual overlays on top of reality, and you can also record and push it back.

But when you look at the way we do it, I think it is actually an extension of mapping, right?  So you will have the map, which is an abstraction of reality, and then basically with a flick you can go to real reality.  And I think to really understand what you see, you need to actually really know what you see, and an image doesn’t do that.  But if you have compass, and GPS, and 3D models, which is what Navteq is capturing, we can actual do line of sight calculations and you can really say, “OK.  He is looking at that building.” And you can click on the building because you have that understanding.  And you can combine that.  And that leads to a completely new way for how people interact with their environment.

So the here and now, and me here and now, and what next is a key innovation in our focus hereafter.

Tish Shute: I suppose for AR enthusiasts we were all disappointed when we saw the N900 with Maemo, which was very exciting as an operating system, didn’t have a compass.

Michael Halbherr: Yeah, but you know, that is just a matter of timing.  At the end of the day, Meego is our, I think, high-end push.  You will see this Mego device have what you need it to have.  So I wouldn’t worry too much about that.  We are building all the  phones, and everything has everything.

I think it is not just Meego.  It is the whole thing around it.  It is the QT, it is the web run time, it is the whole Meego system.

Tish Shute: Yes we have used QT for some development on the ARWave project, it is very good.

Michael Halbherr: Yeah, and the web run times, it’s open…So Nokia is about open and emotional.  Others are maybe open and more utilitarian or closed, and we are more open and more emotional.  And I think that is what we want to basically do with this platform.

And we are working very hard.  And so whatever I say has been built almost a year ago.  So there is a lot that we are building that we don’t talk about before we announce.  And we try to do this as a global offer, so we can’t launch a little service in the US, which maybe is a problem for us.  But we are basically building complete solutions for global markets.

Tish Shute: Yes.  And when I heard you talk just now on Nokia’s global strategy just now, it all fits in.  But from the point of view from someone who lives here in the US, it seems Nokia is disappointing us.

So is Nokia enabling augmented reality in any ways that noone else is?   I mean if you are interested in augmented reality, the backend and data is vital, e.g. the Navteq part.  And Apple has excited us with the amazing performance of their hardware, and they have taken steps like controlling the manufacture of their chips, powerful cpu and gpu is very important for augmented reality experiences too? Is there a need for a chip set geared to augmented reality specifically?

Michael Halbherr: I’m not sure you have to go that far.  I think at the end of the day it is the data, right?  The map data and the 3D data that is actually extremely important.

Tish Shute: Yeah, I agree with you that the data is vitally important.

Michael Halbherr: Because at the end of the day, when you have a real abstract view, it is really important, because the camera or the view is only a viewer, right?  And people don’t really know what they see.  It is just they plaster on it believing that the person sees what he sees.  So I think real augmented reality or mixed reality is really basically having a real picture, but having, actually, the understanding of the 3D logic of the picture.  And for that, you need to capture the data which Navteq is doing.

Tish Shute: The original dream of augmented reality is 3D media and graphics tightly registered to the physical world.  But actually, it is the data…being able to integrate the streams of data and sensor fusion techniques that will make this interesting and possible right?

Michael Halbherr: Again, it is overlaying virtual elements on real elements.  It is like you are taking your glasses and now you can see information attached.  And of course, you need to make sure it is not overloaded, and that everybody works with it, and it is precise.  So that is basically it.  So I think that is why we are investing a lot in the precision of the maps and the 3D elements of the maps.

At this point we had to leave our seats has the set was being broken down for the next event at Where 2.0.  Check out all the videos for Where 2.0 here,  it was a watershed event for all things hyperlocal, geo, and augmented!

Augmented Twitter

Presenting Augmented Twitter (see video and slides above) at Jeff Pulver’s 140 Characters Conference (#140conf ) was super fun, and great video makes this a conference that you can enjoy catching up on after the fact.  Jeff Pulver does an excellent job of keeping people to a challengingly short format.  Even I managed to bring my talk in under 5 mins!

#140conf is a real time mobile social crowd, and pretty attuned to Augmented Reality.  Everyone had heard of Augmented Reality in the audience, and while most had never tried an AR app, nearly everyone used a mobile social app like, Four Square, Gowalla, or Latitude.   As Dan Harple (@dharple) – Executive Chairman, GyPSii, said in his  interesting presentation, Evolution of Location and Places,  “everyone get’s connection, and that connection in real time is the thing if we can get it, and that real time connection is innately mobile.”

ARwave aims to push mobile, social, real time connection even further with augmented reality.  As Anselm Hook puts it so brilliantly in his presentation at EComm, “AR is about publishing “verbs” – interactive, actionable, digital agents not publishing 3D models.”  I have some mega posts brewing on this topic.  Augmented Reality will need to support publishing game like behavior, and digital agents that can embody a set of actions and reactions.

This need for augmented reality to publish behavior, and to share and integrate, in one view, multiple real time data streams are just some of the reasons AR Wave uses an open federated protocol.  Federation is also particularly important for augmented reality because, as Anselm pointed out at WhereCamp,  AR will certainly demand very efficient distribution of state change at the systems level –  to move the computation to its lowest latency.

The only other cloud over our Augmented Reality party at #140conf  was that #ashtag kept our co-panelist and panel chair from joining us.   Rita J King, @ritajking, who is Innovator-in-Residence at IBM’s Analytics Virtual Center, the “General of the Imagination Age,” and Dancing Ink Productions, and Joshua Fouts, @josholalia, “Cultural Attaché,” and Chief Global Strategist of Dancing Ink, were on a 5 day trek out of #ashcloud, and, sadly, not there for our panel.

Bu Twitter, once again, was a life line in a time of crisis connecting them to TEDxVolcano, an impromptu unconference with must see presentations from Rita and others, see Rita’s blog for more.

So the two of us carried the flag for  Augmented Twitter.  Myself and Jerry Paffenfdorf, futurist, artist, entrepreneur and swell guy – the co-inventor of the most famous real time social web system you have never heard of (actually I tried and loved it in alpha testing, before it was quote “shut down by blood thirsty investors”).

Now Jerry lives in Detroit Michigan where he works on the Loveland Micro-real estate project which is the simplest, cheapest, funnest way to become a land owner.  At a dollar a square inch it mixes video games and real estate, like Farmville for urban development.

Joshua and Rita, our very virtual panel mates, are the first and largest inchvestors, and creating their own micro city within the project.  Jerry is one of the most creative and original thinkers on the planet, so treat yourself to glimpse of what is on his mind in the video above – Loveland, 3D mailbox, canned augmented reality, and the relationship of virtual worlds to the real time social web.

Jerry also hat tipped one of the most captivating projects and presentations of the conference, Alon Nir’s, The Story Behind @TheKotel, “Tweet Yr Prayers!”  What a great story about the power of Twitter to reach out into the world, and beyond!  I got a chance to chat with Alon at #140conf, and I found out he is brother of augmented reality guru, Rouli Nir, @augmented.  Rouli is known for his sharp and comprehensive AR commentary on Augmented Times and Games Alfresco.  Cool family!

Before I close this post, I want to mention @AndyDixn’s talk on the prison sysetm, A conversation with Andy Dixon: What the prison yard & twitter have in common.  This conversation, I think, is a great example about what makes #140conf special.  As @nwjerseyliz pointed out, we, “hear few voices from those who’ve experienced that side of the issue.”

Thank you @jeffpulver for creating such a cool staging for so many diverse voices.

And before I close here is what the only slide I didn’t have time to show said!

If you liked “Augmented Twitter”
Don’t miss Augmented Reality Event!

2 days, 3 tracks, 40 AR companies, 76 SpeakersArt! Magic! Competitions! Awards!Bruce (the Prophet) Sterling, Will (The Sims) Wright, Jesse (Gamepocalypse) Schell, Blaise Aguera y Arcas (Microsoft Bing) and You! The sneak preview of the schedule is here.

Register today at Augmented Reality Event.com

Discount code for @140 attendees, (and readers of this post!) TISH245 activates $245 price for full conference.

See you there!

“The game is about the world not dragons,” Will Wright, Founder and Chief Executive  Stupid Fun Club, Creator of Spore and The Sims.

I had a brief chat with Will Wright after his talk at Engage!, and I was delighted to hear that augmented reality is high on his agenda at the moment:

“a lot of our stuff is kind of in the experimental format right now, but definitely one of our strong interests is AR.”

Will Wright will be coming to speak at Augmented Reality Event, Santa Clara, CA., June 2nd, 3rd,  2010.  But, for now, here are a few hints at some of the directions that are intriguing him, e.g., the game potential of 3D mapping like Blaise Aguera y Arcas’s  demo of augmented reality maps at TED -  see the full conversation below.

There has been a vital shift, Will Wright points out.  Before the Wii,  immersive was understood as  how much we were pulled into the world of the game.  Now immersive is how much the game pulls us deeper into our world, e.g., our relationship with the people we are playing with as in Rock Band, or engaging with other people’s crazy antics when playing Wii games.

“Computers are imagination amplifiers and toys are imagination constructors.”

The slide above is from Will Wright’s talk at Engage!

Will Wright’s talk was extraordinary, dense, layered, and deeply thought provoking.

I have picked out a few samples from Will Wright’s vast tome of slides here.  They are just a glimpse of the many insights he offered.  If you are still wondering what will transform augmented reality into a mainstream experience, I suggest studying this talk carefully (I think the audio will be posted on the Engage! web site).  Also watch Will Wright’s, Games For Learning Institute talk at NYU, February 17th, 2010, archived here.

Will Wright and Stupid Fun Club are getting ready to takes us to the next level of imagination amplification and construction.

“Smart” things can make us dumber by overriding our instincts

Just one of the many wonderful anecdotes Will Wright told  was the story of his experiences with a new “smart” car (he bought this car with the intent of exploring the pinnacle of the “smart” car experience). The slide above is from Will Wright’s talk at Engage!

Increasingly, artifacts are being designed to send us more and more data, and this car was endowed with an array of sensors supplying data aimed at assisting parallel parking – a notoriously challenging aspect of driving.  But the car  failed miserably in helping.   While parallel parking had been easy for him prior to being deluged with all this data, Will Wright pointed out, ironically, he had to learn to ignore this stuff to park the “smart” car.

Instinctively, we learn to filter the information necessary for parking to the relevant stuff.  This kind of pre-conscious filtering is a key challenge for augmented reality, and one that Will Wright, as a game designer, has given great deal of thought to.

As Will Wright pointed out, a  lot of our ideas about augmented reality, and sensor enabled artifacts, are rooted in trying to give us more data, to “take over our instincts.”   Not only do these artifacts attempt to give us more data, which as in the case of the HUDs for parallel parking can get in the way of our own highly effective intuitive instincts.  But, as Will Wright also noted, these artifacts also have more data which they can deploy independently to override our instincts, e.g., the car detecting your head has turned back to talk to a passenger and applying the brakes!

“Toys Encourage Agency”

Toys can be the antidote to instinct blocking “smart things.”  In contrast to “smart” data spitting cars that “take over” our instincts, toys encourage agency.  Will Wright gave the example of  high dynamic lighting photographs that make the world “toy like” and encourage us want to reach in and play with it (photo above from HDRCreme).

“What Computers are really good at is harvesting human intelligence”

Another key insight that Will Wright explored in depth in his talk was the significance of  crowd sourced intelligence (the slide above is from Will Wright’s talk at Engage!).  If the crowd is training the filter, he suggested to me, this might build the kind of context we need to build meaningful augmented reality experiences (for more on this see the conversation below).

Talking with Will Wright at Engage!, NYC, 2010

Tish Shute: I was very interested by the idea you put out that this deluge of information gathered by sensors is not necessarily a kind of nirvana for augmented reality, in fact it can be just the opposite.  In the embryonic world of augmented reality, we have two streams it seems at the moment – one is the idea of a kind of like hyper local nirvana imagined for AR, in which we get information relevant to us, when and where we need it.   But you talked about some of the problems in realizing this, didn’t you?  The other strand  is the emerging stream of play which you are exploring..

Will Wright: Right.  I think part of it is like what I was talking about-the way our senses are set up to know how to filter out 99% of what is coming into them.  That is why they work, and that is what is beneficial.  I think that is why AR needs to focus on…

You look at what I can find out on Google or whatever, the amount of information is just astronomical.  The hard part, the intelligent part, is how do you figure out that one tenth of 1% that I actually care about at this given second?

Tish Shute: Yes.  Have you seen any examples of AR beginning to do that?

Will Wright: No, not at all.  I think that you have to have a contextual understanding of where I am at, where my mindset is, what my situation is, what my goal state is in a moment by moment basis.  And then it is still a complex task.  But the very first thing we need is more context for building a filter.  See, that filter is changing every few minutes, you know, what I am filtering into my senses is changing, and my context is changing moment to moment.

Tish Shute: I really liked your emphasis on crowd sourced intelligence as the key power of a networked world, is this the seed..?

Will Wright: Well, you can imagine crowd sourcing that filter…it would affect a million people and get a sense of what mental context that they were in and what filter they turned on.  And so, in a sense, the crowd is training the filter.

Tish Shute: Yes.  The problem with projects like SIRI, that is driven by the big DARPA AI project, CALO, is it is centralized – although I am not sure what they intend to do in terms of crowd source corrections?  But if it was all open and we could crowd source as well that would be interesting.  But in the end we need a framework for AR that is as open as the internet, don’t we?

Will Wright: Right.  I think the technological infrastructure needs to be much lighter so that it can be grounded in more like a Twitter feed or something.

Tish Shute: Yeah.  I’m actually working on a project using the Wave Federation protocol as the basis for an open communications framework for augmented reality, AR Wave – not the Wave user interface, just the real-time federation protocol.  But, of course,  for it to become an open framework that could be a vehicle for crowd trained augmented reality it would need good take-up!

Will Wright: Right.  You really want a million people involved.

Tish Shute: Yes our dream is that the creation of augmented reality content will be as open, accessible and  simple as making an html page, or contributing to a wiki.

So in terms of AR games what is interesting on the horizon, presumably games also have to solve the problems of  delivering a hyper local experience.  The car that you described in your talk tried hard to use augmented reality to solve the problem of parallel parking and ended up making it harder.  So giving us the information we need, where we need it, when we need it, and specific to who we are is going to be a big challenge.  But I mean in terms of games, what kinds of hyper local experiences will be most fun and what have you seen that is interesting in terms of augmented reality games up to now?

Will Wright: I’ve not actually seen much at all.  I’ve seen people doing interesting stuff with like Google Maps.  They aren’t really entertainment oriented, but I think you can start thinking about…

I mean I think for a lot of people, Google Street View is entertainment.  But I haven’t really seen something that was really leaning into an entertainment application using existing technology and data that is already out there.

I mean I have seen some cool experiments-people playing Pac-Man in Washington Square and stuff like that, but nothing really serious.

Tish Shute: Yeah.  of course I think one of the missing links is that the barrier of entry is way to high for creating social augmented experiences for smart phones, and as you point out in your talk it is the social implications of the game is what makes it compelling.

Will Wright: Also, I think using them [smart phones] as data aggregation devices rather than just data consumption devices…so that people out there are using their phone, cameras, microphones, or whatever to gather data and get an experience where they are rewarded for gathering data.

Tish Shute: Like foursquare where you get the badges, and people can become the mayor of like a cafe or something.

Will Wright: Right.  Yeah, you can imagine people using their phones to actually kind of pull information…

Tish Shute: A Dutch developer/artist/game designer, Thomas Wrobel,  Lost Again, came up with the original concept for the AR framework we are building on the Wave Federation protocol.  Thomas and his partner Bertine van Hovell design alternate reality games, amongst other things they do…so they are deeply immersed in the potential of the world as game.

Will Wright: Yeah, one of my programmers actually works in Amsterdam….there is a whole sub-community…
Well, yeah.  The possibilities are tremendous.  And Wii is actually training us that way [to be as much engaged with the other players in the physical space as the virtual game], so it is going to happen.

Tish Shute: What are the most exciting things you see at the moment, and for the next 12 months for augmented reality?

Will Wright: Gosh.  I mean I just think there is cool stuff happening in mapping, in general.

Tish Shute: Like Blaise Aguera y Arcas’s  demo of augmented reality maps at TED?

Will Wright: Yeah, I thought the 3-D mapping with Microsoft…I think like the next level of that is going to be really compelling.

Tish Shute: You see game potentials in that?

Will Wright: Yeah.  You start overlaying really cool game potential on top of that.

Tish Shute: Might you get interested and do something?

Will Wright: Oh, yeah.  I mean in terms of games, that is one of my biggest interests, is AR.

Tish Shute: Are you allowed to talk about anything specific at all?

Will Wright: Not yet, no.  I mean a lot of our stuff is kind of in the experimental format right now, but definitely one of our strong interests is AR.

Tish Shute: Yeah, absolutely.  We are over being tied to our desks to use computers -we want to be doing it anywhere, anytime, with anything…

Will Wright: Now the game is about the world instead of about dragons.  I love that.

Images  from Mitsuo Iso’s Denno Coil (Click to enlarge), the game “Metroid Prime,” and Terminator.

Thomas Wrobel, Sophia Parafina, Joe Lamantia, Matthieu Pierce, and I will lead a  session tomorrow for AR DevCampNYC introducing the AR Wave Project.  Thomas, Joe and Matthieu will be participate via skype (10am to 11.30am EST), and Sophia Parafina and I will both be at AR DevCampNYC at the The Open Planning Project office (TOPP).  The PyGoWave crew will be introducing PyGoWave via LiveStream.

At 1.30pm EST to 2.30pm EST there will be a shared PyGoWave/AR Wave session with Mountain View (if bandwidth permits).

The skype conference will be at ardevcampnyc .  To participate in Wave,  please join the public Wave,  AR Wave: AR DevCamp Session.  There is also a AR Wave Wiki up now – see here.

Dimitri Darras (avatar Dimitri Illios) is working on streaming the AR DevCampNYC sessions into Second Life,  SLURL here.

Thomas has done a very nice introduction and FAQ below.  This should help people new to this project to get up to speed quickly.

There are already several Waves that show the history of this project including: AR Wave: Augmented Reality Framework Development,  AR Wave Use Cases, PyGoWave AR Tech Discussion,  AR Wave Augmented Reality Wave Development, AR Wave / Muku Organization and Admin.

Also I have several posts for people interested in more of the background, including: The Next Wave of AR: Mobile Social Interaction Right Here, Right Now!, AR Wave: Layers and Channels of Social Augmented Experiences, Total Immersion and the “Transfigured City:” Shared Augmented Realities, the “Web Squared Era,” and Google Wave.

Thomas uses the term Arn (augmented reality network) which is one of the candidate names for the project, Muku (crest of a Wave) is another suggestion.  Thomas’ intro and FAQ below can also be found here.

What is the AR Wave Project?

In simple terms its a protocol for storing geolocated data on Wave servers that’s currently being developed.

We believe this will help lay the foundations for an open, universally accessible, and decentralised system for shared augmented reality overlays which various clients can connect to and use.

This AR Network should spark a lot more rapid adoption of AR technologies, give existing browsers more functionality, and provide the network infrastructure, allowing many of the fictional depictions of AR to become a reality one day.

The AR Network.

When we speak of a future AR Network, we mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.

Where people can just point their phone, webcam, or pair of AR glasses anywhere where a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.

The Arn should be an inclusive and open platform where any number of devices can connect to, and anyone can make and host their own location-specific models or data.

It should allow people to communicate both publicly and privately, and not have their vision constantly cluttered with things they don’t want to see.

This is our vision, and we think a Wave protocol will help it become a reality.

Why Wave?

Wave allows the advantages of both real-time communication, as well as the advantages of persistent hosting of data. It is both like IRC, and like a Wiki. It allows anyone to create a Wave, and share it with anyone else. It allows Waves to be edited at the same time by many people, or used as a private reference for just one person.

These are all incredibly useful properties for any AR-experience, more so Wave is open. Anyone can make a server or client for Wave. Better yet, these servers will exchange data with each other, providing a seamless world for the user: a single login will let you browse the whole world of public waves, regardless of who’s providing or hosting the data. Wave is also quite scalable and secure: data is only exchanged when necessary, and will stay local to just one server if no one else needs to view it.

Wave allows bots to run on it and thus allowing blips in a waves to be automatically updated, created or destroyed based on any criteria the coders choose. Wave even allows the playback of all edits since the wave was created.

For all these reasons and a few more, Wave makes a great platform for AR.

How?

In basic terms, we will diverse a standard way to geolocate a bit of data and store it as a Blip within a wave.

This data could be a 3d mesh, a bit of text, or even a piece of audio.

Then various clients on various devices could logon, locate, interpret and display this data as they see fit.

Click on image above to enlarge.

A typical example of this might be holding up your phone and seeing messages written by your friends and family in the locations which they are relevant.

You could see an arrow hovering over the café your meeting a friend at, notes above their flat saying if they are in or out, or messages by shops telling you to pick up the particular brand of cereal they like.

This data would be personal to just yourself and whoever you invite to share that wave with.

Other forms of data could be public, like city-maps, online games, or historical landmarks being recreated. Custom views of the world with data for entertainment, commercial, environmental or informative purposes.

The possibilities with geolocated data are endless, as are the various ways to display and make use of them.

One of the things I’m most passionate about is people being able to see many different types of data, both public and private at the same time and from many different sources at once.

For instance, if your playing a AR game, why shouldn’t your chat window be viewable at the same time?

If you have skinned your environment with a custom view of the world, why shouldn’t you also see mapping or restaurant recommendations?

The ways to present these layers of data and toggle them on/off in the most intuitive and flexible ways would be a task for the client markers, and I’m sure we will see many innovations in those areas.

But by using Wave it at least provides the framework for having multiple information sources controlled by many different people yet accessible, and user-submittable, via the same protocol.

Who?

This idea first sprouted from a paper I route focusing on the potential for IRC to be used for AR;

http://www.lostagain.nl/testSite/projects/Arn/AR_paper.pdf

I suggested near the end Wave might be a better alternative (using Google Wave was an idea Tish Shute, Ugotrade, brought up in response to the Arn prototype design on IRC), and it quickly became apparent that Wave was a very suitable medium.

Since then, there was a lot of interest, and numerous people have offered to help.

In particular, recently, the PygoWave team is helping us out, as they have an existing server supporting c/s protocol, which is currently being actively developed.

Where?

You can join the general discussion here;
Augmented Reality Wave Development

Technical side here;
Augmented Reality Wave Framework Development

When?

There’s lots still to do, and we are at an early stage.

Our current targets: (last updated 11/12/2009)

• Getting reading/writing of prototype ARBlips to the PygoWave sever. (the PygoWave team have already made a standalone client and have the protocol for this sorted!)
• Establishing a minimal spec for ARBlips to be later expanded.
• Writing a very simple prototype online client showing how to store/retrieve the data.
• Expanding client to work for some use-cases.
• Establish a logo/branding for the project.

Other FAQs.

Where’s the catch?

While we believe Wave is highly suitable for development, it has the drawbacks of being a new system with just a few servers worldwide, which (at the time of writing this), have not yet been federated together yet.

Naturally, as a new technology, its likely to have some growing pains. And building a new technology on other new technology will multiply that somewhat. The first pain is the lack of a standard client / sever protocol. PygoWave have stepped in to the rescue a bit here, by being not just one of the most developed Wave server other then Google, but also leaping ahead with support for Json based c/s interaction. Google has stated they want community to take the lead on on a c/s protocol, so we are hoping they will adopt a Json variant, or a XMPP one and add it to the spec. We hope in much the same way as POP3/IMAP have been a standard for email server interaction, a similar one will develop for Wave.

In the meantime we plan to keep the code for writing ARBlips somewhat abstracted so as to make it easy to adapt in future.

As for the newness of Wave and other potential problems it will bring, we aren’t that worried as its built on XMPP, which has proved reliable already.

The other catch is we are unfunded, which slows development down considerable as we have to fit it around our other jobs.

I’m making my own AR Browser, and am slightly interested in maybe supporting you.

We are naturally very keen for support, and particularly for those with skills and visions to give feedback on the proposed protocol. Specifically: what do you want stored in a blip?

That’s what’s important at this stage.

We don’t see the Arn as a replacement for existing browser systems at the moment. We don’t want to restrict innovation or development in this fast developing market as we are very impressed at what’s been achieved so far. In many ways our task is small in comparison to what’s already accomplished.

However, we do believe the Arn will make a good addition to existing browser systems. It will allow users contribute data and have social features without having to worry about accounts or hosting.

It will still be quite some work to support; new GUIs will need to be developed to make it easy to submit data from the devices, as well as to login to waves.

However, we hope over time to build a set of example libs to make the read/writing of ARBlips as as easy as possible to implement in your software.

Perhaps a good way to think about it is existing AR Browsers are like word-processors, supporting the Arn will be like adding support for *.txt, but doesn’t limit what you can do with your own format.

Eventually we do hope ARBlips hosted on Wave will become the majority of AR data, and its functionality will be analogous to the internet is today. We truly believe in the long run a standard is essential.

But for now we think merely getting a baseline format established for how AR data can be communicated will increase user-ability, usefulness, and help the market grow.

Can I help?

Sure.

We particularly need people with technical skills in relevant fields. (both gwt/javascript web programming and c++(/qt)standalone programming help very welcome!).

But we also welcome people just with vision to help focus use-cases and to conceptualise what we want to be able to do with the system.

Please either join the relevant AR Waves or Wiki

We are especially interested in those with JSON and Comet experience. Specifically those with the abilities to make standalone applications to read/write to a sever using these methods.

What type of data will a AR Blip store?

This is still actively being decided, but essentially its a physical hyperlink.

A connection between a physical location (or object, see below) and a piece of data.

Specifically, we are thinking about the following fields;

Location in X,Y,Z,
Coordinate System used for the above,
Orientation,
MIMEType [the type of data stored]
DataItself [either a http link for 3d meshs and other larger data, or an inline text string if its just a comment]
DataUpdateTimestamp [so clients know if its necessary redownload]
Editors [the user/s that edited/created this blip]
ReferanceLink [data needed to tie the object at a non-fixed location, such as an image to align it to an object in realtime],
Metatags [to describe the data]

Are you purely tying stuff to fixed geolocations?

Certainly not
As part of of the spec we wish to be able for people to be able to link data to dynamically moving objects, trackable by image or other methods.

The idea being that one day someone could link a piece of text or 3d mesh to an image on a t-shirt they are wearing, or perhaps link a dynamically updating twitter feed, or perhaps provide information on a product (based on its logo).

There’s a large number of possibility’s for image-based linking alone, and that’s not even considering possibilities like linking RFIDs, or other forms of less precise but invisible binding data.

We need a lot of feedback from those companies already doing markless tracking. What types of images do you need, idly to link a mesh to an object? is one enough?

Summary of AR Wave Work to Date

Purpose: To provide an open, distributed, and universally accessible platform for augmented reality. To allow the creation of augmented reality content to be as simple as making an html page, or contributing to a wiki.

Specific Goal: To establish a method for geolocating digital data in physical space (or linking it to physical objects) using wave as a platform.

(For justification as to why we are using Wave see: our faq )

Wave as a platform

We are developing on the PyGoWave server at the moment but the goal is to be compatible with all Wave servers

PyGoWave has already achieved an important aspect in enabling the project in being a waveserver with a working and well documented server protocol. This allows both standalone and webbased clients to interface with it already.  See - The PyGoWave Qt-Based Desktop Client

This is one of the reasons why we have chosen to develop for the Pygo server at this stage.

However, the overall goal of AR Wave is to have a framework compatible with all servers using the Wave Federation Protocol. As more wave servers get c/s protocols then ARblips (the data needed to geolocate objects) could be posted and retrieved from various servers using the same client software. For this a standard should emerge. Just as websites don’t have to be hosted on specific servers, neither should AR data need to be hosted on specific wave servers.

In order to reach our goal, there are a few very achievable steps involved – see below.

Feedback

We are still actively seeking feedback, so feel free to join the Wave discussions, and see the history of how the specifications of the protocol evolved. You can also read the justification for some of the choices already made. Note a new discussion for AR DevCamp will be begin at AR Wave: AR DevCamp Session

This will, of course, only be the first draft of the specification, and it is sure to develop much in future.
The important thing now is to make working prototypes while maintaining flexibility.

So what do we need to do?

Steps :

* Establish the overall method – Done.

Each Wave will be a layer on reality which an individual or a group can create.  Each Blip in this Wave refers to either a small piece of inline data (like text) or a remote piece of larger data (like a 3D mesh) as well as the data needed to pin-point it in either relative or absolute real space.
We call these blips: ARblips. They are simply blips that stored the data necessary to augment a single object onto a specific bit reality.

It is up to the clients how they interpret and display the data. They could interpret it as a simple 2d list of nearby objects, or as an advanced 3D overlay, whereby multiple waves from different sources could to be viewed at once. What’s important is that there is a standard way to link the digital data to the real world space.

* Establishing the specification for the ARblip – In progress
We have a good idea of what’s needed to be stored in an ARblip, and we have hammered out a rough format.
The data might be stored as blip-annotations, but this has yet to be finalised.
A rough outline of the type of data stored can be seen in this c++/qt header for ARblip data can be seen at the end of this document.

* Storing and retrieving these pieces of ARblip data on the PyGo server – In progress.
The Pygowave team has made some excellent libraries that should make reading and writing data on the PyGoWave server very trivial for those with c++ skills.
This, however, is a real critical step, so more developers with C++ skills are very welcome!

* Making the above client mobile, and using a devices gps device to place the data. – Not started.
The next step would be to port the code to a mobile phone and use it’s gps-input  to post geolocated data and view what others have posted. This would be a fairly simple and not to useful app in itself. However, it would mark the first time anyone could post AR data and anyone could view it, all using open-source infrastructure.
As a bonus, because we are using wave infrastructure, the updates to any ARblip should appear in near realtime.

* To continue with the proof of concept, we would like to have simultaneous wave input from a PC
and mobile phone at the same time. – Not started.
For example, someone could post a pin on Google maps API and have that data posted to a ARBlip in a wave. Someone logged into that wave on their mobile device would then see the data posted appear.
More so we hope that when the Google map pin is dragged about, the mobile phone viewer, with just a few seconds lag, will see its location updated in real time.

We hope to make a modest yet practical app at this stage.

* After all this, we can go onto the interesting things:
3D data, camera-overlays, data fixed to objects and many more.  There’s plenty of existing software using these features (such as Wikitude, Layer) and some that are even open source software (like Gamaray and Flashkit). The open source code can give us a leg-up. However, we prefer to establish the protocol first. So naturally, these fancy features aren’t a priority for us. Rather we think our energy is better spent establishing the protocols and infrastructure so that other people can build more advanced bit of software easier.

However, once our primary goals are established, we will look to make a open source augmented reality browser ourself which will surely feature many of these features.

Overall, we hope once we have a simple proof of concept, there will be many groups, both existing and new, wanting to use this Wave system for their own apps, games and data.

Conclusion:
Really it’s now all about growing the community. We hope as soon as we show how great Wave can be for augmented reality, that lots of individuals and teams will start making their own clients to read/write geolocated data.
Overall we don’t think anything we make will be that impressive in itself. That’s not our goal.
We instead hope that our project will enable AR-content to be made as easily as web content. That games, information and apps will be able to be created without the creators having to worry
about the infrastructure behind it.

Technical information -

Current ARBlip header file

(below is a c++/qt header file for an ARBlip object that should illustrate the data being stored)

{

public:

arblip();

~arblip();

arblip(QString,QString,double,double,double,int,int,int,QString);

QString getDataAsString();

QString getEditors();

QString getRefID();

QString getXAsString();

QString getYAsString();

QString getZAsString(); bool isFaceingSprite(); 

private:

//ID reference. This would be a unique identifier for the blip. Presumably the same as Wave uses itself.

QString ReferanceID;

//Last editor(s)

QString Editors;

int PermissionFlags = 68356;  // default 664 octal = rw-rw-r–

//Location

double Xpos;   // left/right

double Ypos;   // up/down

double Zpos;  // front/back

//Orientation

// names, ranges and directions are taken from aeronautics.

// If no orientation is specified, it’s assumed to be a facing sprite.

// Roll: rotation around the front to back (z) axis. (Lean left or right.)

// range +/- 180 degrees with + values moving the objects right side down.

int Roll;

// Pitch: rotation around the left to right (x) axis. (tilt up or down)

// Range +/- 90 degrees with + values moving the objects front up. (looking up)

int Pitch;

// Yaw: rotation around the vertical (y) axis. (turn left or right.)

// range +/- 180 degrees with + values moving the objects face to its right.

int Yaw;

bool FacingSprite; //if no rotation specified, this should default to true

//if set to true when a rotation is set, then it keeps that rotation relative to the viewer

//not relative to the earth.

//Data format

QString DataMIME;

QString CordinateSystemUsed; //The co-ordinate system used. This should be a string representing a Open Geospatial Consortium standard. This could be earth-relative for gps co-ordinates, or in some cases relative to the viewer, for data to be displayed in a HUD like style.

//Data itself

QString Data;

QString DataUpdatedTimestamp; //Time the Data was updated changed

//Note; A seperate timestamp should be used for updates that dont effect the data itself.
//(such as if a 3d object moves, but its mesh isnt changed)

//Data metadata QMap<QString, QString> Metadata;

};

Mobile social communication is beginning to take center stage as the internet moves to real time communications. The recent explosion of interest in augmented reality is part of a wider concern to orchestrate a new landscape of contextually relevant information linked to location/place/time and mobile social connectedness.

The picture above, “Having an iphone has completely changed the way I poop,” is a slide from Gentry Underwood’s workshop at Web 2.0 Expo, NYC, “Social Interaction Design a Primer.”

Last month, I attended  three events starting with MoMo #13, Amsterdam, where I presented on, “The Next Wave of AR: Mobile Social Interaction, Right Here, Right Now!.  Then I caught the last two days of the Web 2.0 Expo, NYC, and finally, Real Time Crunchup SF (which I watched online).

New forms of real time mobile, social connectedness were central themes on all three occasions.

But, in terms of doing stuff that matters with mobile real time technologies, at the moment, we are still at the “hello world” demonstration (see my conversation below with Anil Dash and Mark Drapeau at Web 2.0 Expo below).

As Anil Dash noted,  “I think everybody starts with a train schedule…”

“I remember five years ago when Adrian did Chicagocrime.org. It was a revelation but I mean, that was five years ago.  And people still keep making that app over and over.”

Anil Dash announced at the Web 2.0 Expo that he will be the director of Expert Labs, a new nonprofit that will take the dot-com incubator model and apply it to new digital tools for the federal government:

“For me, in starting Expert Labs it’s been great just to tap into the desire people have to help and serve and to take the idea that you can work for your country without having to work for your government. What can you do to participate?”

The Gov 2.0 movement is attracting the best and the brightest, if you need inspiration check out Carl Malamud’s Gov 2.0 Summit presentation, By the People…..   Jennifer Pahlka is leaving her long time post as co-chair of Web 2.0 events for TechWeb to concentrate on Code for America. And Mark Drapeau is co-chair of the Government 2.0 Expo next May, that O’Reilly and TechWeb are also producing.  You can submit ideas about Gov 2.0, ICT, and cities (or other topics) to the upcoming Gov 2.0 Expo.  Mark says he will welcome them! Note there is a Free Gov 2.0 Online conf. Thursday, Dec. 10th

Tim O’Reilly has committed to Gov 2.0 work and “doing stuff that matters” with missionary zeal (see his keynote Web 2.o Expo, War for the Web).   Tim O’Reilly’s talk, also the article,  War for the Web, are a stark reminder of how the centralization and privatization of large parts of our communications infrastructure threatens the open web.  But “doing stuff that matters,” as it turns out,  is one of the best ways to win the war for the open web.

“Level playing Fields, Open access, Open APIs, Controlling our data, being able to move with it” (Anil Dash)

Slide above from Anil Dash’s presentation at the Web 2.0 Expo, NYC, “Listening to the Experts”

The Gov 2.0 movement is still in the idea and initiative phase, but the ideals and scope of the movement are a natural antidote to the fox in the social network chicken coop business model du jour (see the latest antics of Zynga).

Anil Dash notes the intrinsic bond between Gov 2.0 work and the open web:

“Because government has an inclination to creating openness by its nature. Right?  We don’t have an entirely toll system of federal highways in the states. We understand that the broadcast airwaves are a public good. And so government is inclined to think about creating public goods. It would be ridiculous to spend tax payer dollars on funding proprietary platforms.”

The Power of Government as a Platform for citizen involvement is just beginning to emerge from initiatives like Data.gov  “a collection of federal data housed on the www.data.gov Web site that’s open to public access.”

One of the most challenging aspects of creating in context mobile applications that do stuff that matters is the data curation.

Sean White, explained to me at ISMAR 2009, the challenges of data curation behind this beautiful example of augmented reality doing something that matters (pic below) -  a pollution meter, that “shows carbon monoxide levels projected over New York City.  The height of each ball reflects concentrations of the pollutant” (developed at Columbia University Graphics and User Interface Lab where Steven Feiner is Director).  Note Sean White and Steven Feiner will be at AR DevCamp NYC this weekend at The Open Planning Project office (TOPP) – see below for more information.

Open Data combined with Open Architectures of Participation are a Powerful Combination.

Scott Yates commented in his very insightful post on RT Crunchup SF that a “literny of fixes” for a broken web were “presented as  the state of the art”  in a “series of presentations from companies that have solutions that fix some subset of all the long list of annoyances” (annoyances arising from finding data and friends locked into a variety of different walled gardens).

And, Scott Yates writes:

“There have been presentations from companies who hope to be the future of socially connected communications, but not one of them has the economic or intellectual heft to be considered a true vision for the future.”

If you have been following my recent posts, you will already know that I agree with Scott Yates when he concludes: “Wave really has an opportunity to fix so much of what is broken in communications.”

I have been working on a project to create an open distributed augmented reality/mobile social communications framework based on the Wave Federation Protocol.

This Saturday Dec 5th there will be AR DevCamps held in Mountain View, New York City, Wave and Skype. There will be sessions on many aspects of open augmented reality, including Wave enabled AR.

AR DevCamp

I will attend AR DevCampNYC at the NYC location, The Open Planning Project office (TOPP) penthouse in Manhattan.  This will be an awesome opportunity to meet some of the key augmented reality thought leaders and innovators, including Sean White, Steven Feiner,  Steve Henderson, and many others (see the sign up list here).  Ohan Oda will demo the open Goblin XNA platform.   Thomas Wrobel will answer questions on writing AR Blips to PygoWave Servers and Sophia Parafina (@spara), Joe Lamantia (@mojoe) and I will be on hand to discuss the open distributed framework for AR project -  Wave enabled AR.   The PyGoWave crew will participate via skype (they will be introducing some of their latest work ).  Ori Inbar of Ogmento and Games Alfresco will lead a brainstorming session on the “Big AR NY Game”: The first location-based, social, augmented reality game designed for New York by New Yorkers.

We will continue the interesting discussion led by Marco Neumann (@neumarcx ) on the Semantic Web and Augmented Reality at the Semantic Web Meetup.  John C. Havens will introduce the Outernet Guidelines Initiative.  And Matthew Snodgrass @mattsnod, Lippe Taylor will lead a session on the future implications of AR.  Noah Zerkin, will share his brilliant work on AR software and hardware interfaces and exploring the idea of an AROS.  And Kate Chapman, from Washington, DC,  @wonderchook, and a bevy of local NYC geo geniuses, including organizer Sophia Parafina (@spara ), will explore ways to visualize government data through AR.  I am hoping we will have some projects for the upcoming Gov 2.0 Expo at http://gov2expo.com.

And there will be much, much more – keep checking and adding to the wiki. See you there!

“Hacking Human Behavior Within a City”

Picture from inspiring cities.org -shows some Amsterdam bicycle trends

At Web 2.0 Expo, Gentry Underwood, IDEO, gave a great presentation on how software changes community and communities change software from an ethnographic perspective – “Designing Web 2.0: Here Come the Anthropologists.”

And Baratunde Thurston’s,“There’s a #hashtag for That, was an inspired, brilliant romp through the “mini-grass roots movements” of hashtags - which are “quickly assembled/demolished malleable fun!” or “great ways to mess with people,” that reminded us the power of grass roots movements when it comes to “hacking human behavior.”

But my visit to MoMo #13 preceeding the Web 2.0 Expo showed me clearly “hacking human behavior within a city” is on home turf in Amsterdam, where smart phones and bicycles are the vehicles for the MoMoesque lifestyle.

Thanks to the foresight and generosity of the MoMo organizers, who make sure that the experience of  the speakers together goes beyond the few hours of the event, I had a three day, three night intensive on the future of mobile social interaction – living, thinking, and breathing mobile social connectedness, often into the wee hours, with Dennis Crowley, CEO of FourSquare (see his great MoMo 13 presentation here), Ted Morgan, SkyHook (a must see presentation on what SkyHook is doing with data), the MoMo crew, and many of Amsterdam’s enthusiastic Four Square community. 

And yes, Four Square really is an awesome way to enjoy a city and meet new people.  MIA in this particular pic are key MoMo organizers -  @samwarnaars @mdbraber and @vangeest

But what do fun times in Amsterdam and FourSquare have to do with doing stuff that matters?

Marc Fonteijn, MoMo chair and co-founder of 31Volts points out: “foursquare looks and feels like a game but what it’s actually doing is changing behavior in a playful way”

And Yuri van Geest, who co-founded not only Mobile Monday Amsterdam but also TEDx Amsterdam, added:

“in Holland we are working on using the FourSquare API for mHealth purposes also we see that smart venue owners reward all mayors/lead users/visitors with free meals/drinks/privileges/perks etc. and smart advertisers to boost their co-marketing deals based on FourSquare targeting capabilities of key influencers”

Dennis Crowley, seemingly immune to lack of sleep and jet lag, followed up his MoMo #13 talk with a presentation at Web 2.0 Expo.   I was sitting just behind Mark Drapeau, and I managed to catch up with Mark after Dennis’ talk.

Mark listed Foursquare in his big takeaways from the Web 2.0 Expo, pointing out the potential new forms of mobile social interaction have for “hacking human behavior within a city.”

“I had always been a little leery of trying FourSquare because I have a certain level of privacy I try to keep up. But listening to the CEO of Foursquare talk about it… I knew what it was. I have friends that use it.. but thinking about it as hacking human behavior within a city and social engineering of peoples’ behavior and what they can do, and really understanding what citizens are doing within cities, or other areas, and how they interact with each other.   I think could be incredibly valuable for government 2.0 and government understanding people better.”

And Anil Dash concurred:

“I think Foursquare is a good model in terms of having a game dynamic, being mobile from its default, having a great social experience, leveraging existing networks like Twitter and Facebook instead of trying to compete with them by building their own. I think those are all really, really smart leanings.

I think about if I were a government agency trying to meet those same goals, could I earn badges in Foursquare by doing things that help my community. Right? So when I volunteer at a soup kitchen is that one way to earn an exclusive badge? Is that going to earn me a discount at the bar? Those are all dynamics that we can set up very, very easily and I think that model…maybe it is a public-private partnership. That’d be great.”

Talking with Mark Drapeau and Anil Dash

Tish Shute: I was in Amsterdam speaking at MoMo#13 and I had a lot of fun hanging out with the MoMo crew and Dennis, CEO of Four Square.  I got to meet people and hang out  with Amsterdam’s new Four Square community. But unfortunately I missed the first two days of Web 2.0 Expo.

Mark Drapeau: I got here yesterday too. Yeah. So some big takeaways.

I had always been a little leery of trying FourSquare because I have a certain level of privacy I try to keep up. But listening to the CEO of Foursquare talk about it… I knew what it was. I have friends that use it but thinking about it as hacking human behavior within a city and social engineering of peoples’ behavior and what they can do, and really understanding what citizens are doing within cities, or other areas, and how they interact with each other, I think that could be incredibly valuable for government 2.0 and government understanding people better.

Also I really wanted to hear Tim O’Reilly interview Beth Noveck. I thought the most interesting thing about the interview were the questions and not the answers (also see Mark’s Posterous ).  I thought a lot of the answers were disappointing and political and vague.

But I thought Tim really got some important issues about how do people in the web 2.0 community, the audience of Web 2 Expo, interact in reality when you have a system that we nicknamed Gucci Gulch, where you have lobbyists and lawyers and special interest and councils and all these things that…developers and app builders are not really a part of.  So how do you break in?  I didn’t really hear good answers for that.

I really liked the presentation by the IBM researcher, if I can get his name. Forgive me. Ching Yun-Lin. Talking about putting a value on how many friends you have, how many connections you have and the fact that IBM can actually put a monetary value on the number of connections you have to managers. The number of email accounts you have in your inbox. Or your address book.

I thought that was just fascinating and that’s something I’m very passionate about is social networking for the sake of social networking and not merely for collaboration but making connections among diverse communities and using that to help your business or help your government agency.  Those are my big takeaways this morning.

Tish Shute: I was lucky enough to attend Gov 2.0 Summit.  I think a lot very important areas for Gov 2.0 were defined there, transparency, open data, getting developers into the public sector loop, and citizen-government interaction.  In what areas are we seeing progress and where are we stymied and why?  How do you see this Web 2.0 community connecting to the ideals and plans for action of Gov 2.0?

Mark Drapeau: I think there’s a lot of unanswered questions about Government 2.0 because there’s a lot of good talk and a lot of good ideas and initiatives but there’s still a long way to go before people in this audience, in this community who want to help the government or be a part of policy making or technology in the government can really in a meaningful way, interact with the government processes that, for the most part, are not going away.

And the people that are also part of the system, like giant contractors, they’re not going away. There’s a place for everyone. The question is how do the smaller people break in and I don’t think there are really great answers for that.

Tish Shute: What is the plan for the next Government 2.0 event?

Mark Drapeau: So part of the reason I’m here is to learn and be inspired as the co-chair of the Government 2.0 Expo next May that O’Reilly and TechWeb are also producing. And so there’s increasingly at the Web 2.0 events that they host, there are technologies and people relevant to government missions, or the public sector missions.

And so I think some of the speakers here will carry over in different ways to the Gov 2 Expo in May.

Tish Shute: Is the call for proposals for the Gov 2.0 Expo still open?

Mark Drapeau: There’s still an open call for proposals. Or people, if they know me they can talk to me directly.

(Anil Dash arrives and Mark introduces  me (Anil and I met briefly at Gov 2.0 Summit) – see Anil’s post, New York City is the Future of the Web I do agree but, of course, NYC is my hometown!).

Tish Shute: Anil you are moving into Gov 2.0 work full time now after being a key thought leader in Web 2.0.

Anil Dash: My perspective is probably unique in that I am very strongly from the Web 2.0 world, and new to the Gov 2.0 world and I think it’s telling that you can make the leap. I think that the profound thing is that these worlds are converging and it’s not where it was.

Five years ago the government technology was a bike with the training wheels on it. It was very much somebody’s old hacked up version of Drupel and crossed fingers.  And it looked a little homely and you thought, “well this looks like a run down office kind of thing.”

Now we have have institutions that have wonderful physical presences. You can’t stand in front of the Capital Building or the White House or Supreme Court and not say “that’s a majestic building.” We should have online institutions that reflect the scope and the scale of what they do.

For me, in starting Expert Labs it’s been great just to tap into the desire people have to help and serve and to take the idea that you can work for your country without having to work for your government. What can you do to participate?

Tish Shute: You and Mark had very interesting journeys  into Gov 2.0 didn’t you?

Mark Drapeau: Like the Hunter S. Thomson of Government 2.0

Tish Shute: I like that!

Anil Dash: Can you say that about yourself?

Mark Drapeau: I did the other night and people seemed to buy it, so

Anil Dash: People were feeling it..

Mark Drapeau: That’s right!

Tish Shute: And even if it’s controversial it’s good too!  I see Mark (and perhaps I am wrong with these characterizations) as coming to this via an interest in the social narratives of government and, Anil, you have come to Gov 2.0 work, as you point out, from a deep immersion in the cultures of technology and Web 2.0….

Anil Dash: And it’s also a little bit, I’ve had the privilege of seeing blogs and social media develop from the start and what I learned from it is cultural change and [not just] technology change. This is the same thing happening in government.

We’re calling it Government 2.0 and it makes it seem like it’s a version upgrade and it’s a software thing but it’s cultural change. And the interesting thing is many of the key players have a willingness to go through that cultural change, which means that the technology, therefore, has the opportunity to succeed.

Tish Shute: What did you think about Tim O’Reilly’s keynote and the warning he gave re the open web?

Anil Dash: The war for the web! He’s absolutely right. Honestly, before Expert Labs had started and I’d come on board, my initial plan for a talk at this event was exactly the topics Tim covered in the War for the Web.  That the centralization of vast parts of our communications infrastructure around privately owned, venture funded companies is a risk to innovation in some ways.

We have to make sure to set up our incentives for those companies, the Facebooks and Googles and Twitters of the world, to align with what our goals are as a society, as a culture, as entrepreneurs, and all those other goals.

So I think it’s good to have a voice like Tim’s articulating that threat and that danger so that we can respond to it. I agree completely that we are in the next phase of the battle between open and closed platforms that we went through ten years ago with AOL.

There was a time when AOL dominated more of the dial up internet, one-third of all dial-up users in the US were coming through AOL. People now say “oh, the iPhone is dominant.” The iPhone has 2 percent market share of all phones or something like that, and yet people are doing all their innovation on their platform.

Well, people used to do all their innovation on AOL’s platform and then they ended up having to rewrite it all for the open web.

This pattern is going to repeat. The choice is whether people want to encourage it or fight it or hope it goes away and ultimately there’s no great business that was built entirely within the walls of AOL’s garden. I doubt there will be a great business built entirely within the walls of Apple’s or Facebook’s or anyone else’s garden.

That’s not to say those companies couldn’t evolve to be open, I hope they do, but as it stands right now you would be foolish to bet your business either from a promotional standpoint, from a start-up standpoint, from a new technology standpoint, on any closed platform that you don’t control.

Tish Shute: I was mentioning to Mark that I thought it’s sort of ironic that we now understand how important the architecture of participation of the internet can be to government just as we are on the verge of another big battle to keep the web open…, a moment when walled gardens are seeming to dominate..will this be an obstacle for Gov 2.0?

Anil Dash: No I think actually they’ll get to skip the closed era.

You know I look at the rather famous example in India of never having landlines. They went directly to satellite phones, skipped directly to the wireless generation so they never had an old infrastructure to rip out.

I think you are going to see the same thing with government tech adoption is they are going to start in the era of recognizing the threat of closed platforms and move directly to open platforms.

Because government has an inclination to creating openness by its nature. Right? We don’t have an entirely toll system of federal highways in the states. We understand that the broadcast airwaves are a public good. And so government is inclined to think about creating public goods. It would be ridiculous to spend tax payer dollars on funding proprietary platforms.

Mark Drapeau: Universal accessibility for citizens.

Anil Dash: Right. It’s a fundamental tenant of government and we have an incredible history including the Internet itself, of embracing open standards to solve government problems in a way that helps all of society.

Tish Shute: So people who have championed open participatory architecture of the internet and open source approaches now have even is more incentive to team up with government 2.0!

Anil Dash: Yeah it is an advantage. But also, I mean candidly, open source is almost incidental to it. I mean I think we have come to the point where open source is assumed as some element of any new tech venture. It is much more about level playing fields, open access, open APIs, controlling our data, being able to move with it, that I think is key.

And I go back to that AOL example, there was a moment where they opened their email gateways to standard Internet email. And so instead of the AOL users only being able to email each other, they could email anybody on the web and this is the moment in which all the value was created. You start to have email marketing companies, and open exchanges and open mailing lists happen when anybody could email anybody else, that is the sort of thing that government catalyzed just by being the example.

Tish Shute: Mark talked about Four Square and how that could be really interesting as part of a Gov 2.0 project. But mobile has followed a course with many complications re an architectural participation – I am thinking about the control exerted by the carriers and now Apple for example?

Anil Dash: No, I think it has revealed complications that have always been there. Right? There’s always been multiple platforms. There have always been user agents and web browsers that have different capabilities. There has always been a digital divide. Mobile is making clear that those realities existed.

But I keep saying this, like, I think if I am designing an application today, you design for mobile first, for a number of reasons. One, the digital divide is much less pronounced on mobile devices. Two, you are much more likely to have an experience that scales well from a small device to a larger one than vice versa. Three, you are able to target international markets or other developing markets where mobile is the default computer platform. And you become aware of constraints in bandwidth, in accessibility, in user experience, in general experience with computers, that a lot of people in the technology industry just completely ignore.

And you know you go to Silicon Valley and people think it’s normal to have a six hundred dollar phone that has a thousand dollar a year data plan. And without blinking they designed for devices like that. It’s myopic and ridiculous to think that people can live with that level of privilege all the time on all the devices that they have and that they have a brand new computer. And so that will be its own undoing.

Right? It’s the people that are thinking about open platforms and working with any device and I think FourSquare candidly is doing a good job of this because they did start with the assumption of iPhones and this thing but their initial target audience of hipsters in the east village probably did have those. But now they have an open API, anybody can access it, that’s the right evolution. And I think they’re smart enough, that was always on their plan.

Tish Shute: But in terms of mobile social interaction we basically have really a structure all of lots of different wall gardens?

Anil Dash: For now..

Tish Shute: How do you see mobile developing more interoperability and social interaction capabilities?

Anil Dash: By using the web. I think it doesn’t have to be full-fledged Ajax-y, html applications on the phone. But if we simply rely on the capabilities of the web as it stands today instead of developing for proprietary mobile platforms we can make a lot of amazing things happen. It’s a good constraint. We should embrace our constraints.

It’s not conventional wisdom yet that mobile applications should be developed for the web. But that’ll change in the next year.

Tish Shute: There is a lot of exciting new real time technologies coming to the Web, Pubhubsubbub, Google Wave Federation Protocol. How will these change mobile development?

Anil Dash: RSS cloud. I mean there’s a ton of real time technologies that are coming out together.

Tish Shute: What are your favorites in the real time area?

Anil Dash: I wrote a post about this .. called The Push-Button Web where I actually go into this…

Tish Shute: Oh yes great post!

Anil Dash: I don’t pick a favorite. I think all of them together will work. I think it’s similar to how the web itself evolved.

We have a tangle of different related technologies that get abstracted away when you use a browser. You don’t know if it’s a gif image or a jpeg image when you browse a page. You just know it’s showing an image in line.

I think we’re going to see the same thing happen to real time web. We’re going to very, very quickly settle into a stack of technologies that let us do real time. As a developer you might have to be aware of the subtle differences. As a user your experience is going to be, “I have real time and it works on whatever device I’m on.”

Tish Shute: Mobile seems like a vital part of government 2.0 because it can connect people and their government to their context/public infrastructure/environment that is a shared concern. The open data movement has shown that being able to mash up data and get that delivered in context is a very powerful kind of technology for government 2.0. Right?

Anil Dash: I don’t know. I think that’s there’s just been the “hello world” demonstration. I think everybody starts with a train schedule…

Mark Drapeau: I was just going to say that everyone is starting with the very low hanging fruit. The transportation, the crime. It’s not exactly clear where it’s going to go but I think it’ll go –

Anil Dash: I remember five years when Adrian did Chicagocrime.org. It was a revelation but I mean, that was five years ago. And people still keep making that app over and over.

I remember at the time I had just become friends with Craig Newmark and I said, “Craig’s List should show the crime around the neighborhoods where you have an apartment listing.” And he said, “Well, if I do that then neighbourhoods that are getting better, that historically had more crime, will never improve because people won’t rent apartments there.” And he came back with that answer immediately as soon as I suggested the idea and revealed one, why Craig’s List is the success that it is. But two, what the implications are of releasing data and having to think about the social implications of that.

Mark Drapeau: Well, it’s like Gentry from IDEO said that, “Social software changes the community, which changes the software.”

Anil Dash: Right. Exactly. We have to think about the social implications of the tools and technology we create.

That means that the reason…one of the reasons we have only had these, frankly, unambitious obvious applications of open data is because the people that have had access thus far have been people that are not socially oriented. Like geeks are very inwardly focused.

Tish Shute: Oh. Okay. Well Mark’s changing this..

Anil Dash: They’re in a very insular community.

Mark Drapeau: I think there’s a number of people that are trying to change that.

Anil Dash: Yeah. It’s starting to change but I’m saying that’s why we’ve seen that symptom in the past.

Mark Drapeau: I get a lot of mileage out of the fact that I’m neither sort of a career govie type that’s getting into the 2.0 stuff. Nor am I a lifelong techie who’s getting into the government and stuff. I’m sort of…I’m interested in these anthropological, psychological, animal behavioral, ecological questions about human behavior and networking. And that’s where I kind of come into this.

Anil Dash: And I think we’re going to need an ethnographic approach to looking at how people work with this data in their real lives. People are using this data already and don’t realize it. You know, when we grab a map in an unfamiliar city you’re using government data. We just don’t think of those behaviors as doing so, and we need to understand that to build applications that really solve people’s problems.

Tish Shute: So can you speculate on the next generation you’d like to see?

Anil Dash: I think Foursquare is a good model in terms of having a game dynamic, being mobile from its default, having a great social experience, leveraging existing networks like Twitter and Facebook instead of trying to compete with them by building their own.

I think those are all really, really smart leanings. I think about if I were a government agency trying to meet those same goals, could I earn badges in Foursquare by doing things that help my community. Right? So when I volunteer at a soup kitchen is that one way to earn an exclusive badge?  Is that going to earn me a discount at the bar?  Those are all dynamics that we can set up very, very easily and I think that model…maybe it is a public-private partnership. That’d be great.

Mark Drapeau: Or even doing things to help your internal community. Key people at work or within your agency or things like that. From my vantage point it does seem like local Government 2.0 types are thinking much more about mobile than the Federal government types. The reality is government employees all have BlackBerries and they’re running around all the time. But they’re in terms of government 2.0 type stuff they’re thinking about the Dell desktop they have and the Microsoft Windows system and whenever I mention something like mobile or pervasive videos…..people aren’t really there. They’re worried about cyber security on the traditional systems. They’re worried about desktop applications on a Dell.

Anil Dash: They’re still five years ago.

Mark Drapeau: Yeah. They’re still five years ago and so I think these kind of O’Reilly-Tech Web events, Gov 2.0 Expo, Web 2.0 Expo, etc., are really starting to get at these questions that are now and not five years ago.

ISMAR 2009 -  was an extraordinary mix of  high geek, academic eminence, gungho Dutch Cowboy entrepreneurial spirit, German engineering and industry, brilliant artistry, and invention, all fueled by a sense, and a very active presence in the case of Diamond Sponsor – Qualcomm, that the big technology players are waking up to augmented reality.

In the picture sequence above (click on photos to enlarge),  Noora Guldemond, Metaio, demonstrates Junaio (coming to an iphone near you Nov 2nd) to Dirk Groten, CTO of Layar (top left photo).  One of the nice social features of Junaio is that users can share the 3D augmented scenes they have created.  Noora is demoing this capability to Dirk, and as you can see he cracks up when he sees the  scene Noora has stored on her phone.  Dirk and I both recognize that this cute little dinosaur augmentation (close up above on bottom left) must have been created by Peter Meier, CTO of Metaio, during the Interoperability and Standards workshop earlier that day.  Metaio it seems were discussing standards while enjoying some 3D augmented back chat.

Both Dirk and I were active participants in the workshop too.  But little did we know that Peter Meier had introduced his little 3D dinosaur into our discussion while we diligently, and sometimes heatedly, debated the merits of XMPP, Wave Federation Protocol,  KML, ARML, VRML, X3D, and  more!  The photo I took is on the bottom right of the four pics above. It was probably taken very shortly after Peter’s augmented Junaio scene.  Of course there is no little dinosaur in my pic of  Dirk Groten with Joe Ludwig and Markus Tripp of Mobilizy who were discussing AR standards oblivious to Peter’s virtual pet in our midst.

I must say I had noticed an impish look on Peter Meier’s face (see photo above on the left – Peter is wearing glasses and holding a phone).  And Markus Tripp, of Mobilizy  revealed a little bit of gaming of his own, when he let out that, in part, ARML is a provocation.  But Peter was clearly unfazed and enjoying himself.  Dirk, tasked to summarize our discussion, stalwartly maintained an optimistic but serious tone fitting for a standards discussion:  “There is a willingness to standardize….,” he began (pic above on left – click to enlarge and read text).

But it was a little 3D dinosaur that, perhaps appropriately, had the last laugh. Fitting, as I am not sure whether anything anyone says about AR standards at the moment will hold up.  But, as Ori commented in his great post – an epilogue for ISMAR 2009, the vibe was “Peace and Love” in AR Browser land (although Chetan Damani of Across Air was not in the standards discussion because he attended the UX/content? workshop instead).  But as they say, “all’s fair in love and war.”  And it is my feeling the games have barely begun!  There are many players (virtual pets included) waiting in the wings. I met some at ISMAR, and they are just itching to join the frey.

Ori Inbar, Ogmento and Robert Rice, Neogence Enterprises, both founders of the AR Consortium, made great efforts to set our young industry off on the right foot -  in the  spirit of coopetition (a neologism coined to describe cooperative competition). See Curious Raven for Robert’s conference observations, and Ori’s post on Games Alfresco for more about Mobile Augmented Reality at ISMAR 2009.  The Mobile Augmented Reality Workshops were driven by an indomitable spokesperson for the new AR industry, Christine Perey.  Christine not only helped motivate discussion on the issue of oxygen to the system, i.e. business value, but also she was a very generous connector at the conference.

What’s Next From Augmented Reality’s Top Chefs?

As Ori pointed out, Kent Demaine, oooii (pic above is from the oooii web site), Minority report VFX designer was hanging out at ISMAR 2009 and he came to the panel I was on: “Augmented Reality in Sports,  Entertainment and Advertising.”  We chatted afterwords about instrumented environments and how this is such a key to development interesting augmented experiences.  Also I mentioned how back in the day I was involved in some of the early development of motion control software.  And it was great to hear Kent say they were still finding motion control cool at oooii.  As Ori notes, he is the “guy with the most enviable AR credentials in the world (the guy who designed VFX for minority report),” and oooii is busy and hiring.

One of the highlights of the Arts, Media and Humanities track for me was meeting Jarrell  Pair. He really brought the best out in panelists with his well tuned questions.  The recording of ISMAR was comprehensive and videos should be up next week.  I will post the slides on Ugotrade of my presentation:  “The Next Wave of AR: Shared Augmented Realities and Remix Culture.”.

“Mixed and Augmented Reality: ‘Scary and Wondrous’” – Vernor Vinge

“Imagine an environment where most physical objects know where they are, what they are, and can, (in principle) network with any other object. With this infrastructure, reality becomes its own database.  Multiple consensual virtual environments are possible, each oriented to the needs of its constituency.  If we also have open standards, then bottom-up social networks and even bottom up advertising become possible. Now imagine that in addition to sensors, many of these itsy-bitsy processors are equipped with effectors.  Then the physical world becomes much more like a software construct.  The possibilities are both scary and wondrous.” (Vernor Vinge -  intro to ISMAR 2009)

Vernor Vinge’s short intro to ISMAR 2009 (which can be downloaded with the ISMAR 2009 schedule here) captures the essence of the “Scary and Wondrous” dawn of the age of ubiquitous computing and mixed and augmented reality.  It is definitely worth a moment to download.  The future of augmented and mixed realities, as Vernor Vinge points out, is tied up in a “tension between centralized and distributed computing” that “will continue long into the future.” One of  my fascinations with Wave is that it offers a tantalizing opportunity to explore augmented reality in an open distributed architecture.

At ISMAR, I talked with as many people as possible about the AR Wave project – see my post here for more about Wave enabled AR.  Many people were very enthusiastic to join the AR wave and the only thing I really lacked was about 100 invites to hand out!

“Everything, Everywhere – making visible the invisible”

Some of the areas that I would have liked to see given more attention on at ISMAR were sensor networks, data curation, and user experience.  Not that these areas were entirely neglected with Pattie Maes, MIT as a keynote speaker, and Mark Billinghurst presenting on some fascinating work on social augmented experiences and user experience.  I highly recommend catching up on these and other ISMAR presentations when the videos go up.

And, I was very happy to meet and talk to Sean White whose work at Columbia University is one of my inspirations (for more about Sean’s work see here or click image above):

“the confluence of powerful connected mobile devices, advances in computer vision and sensing, and techniques such as augmented reality (AR) enables exciting new opportunities for interacting with this hidden network of dynamic information and shifts the locus of interaction from the desktop computer to the world around us”

And I had several very interesting conversations  at ISMAR about developing social augmented experiences that connect us to a physical world that is becoming “much more like a software construct” (Vernor Vinge).  Dirk Groten, CTO of Layar mentioned a few interesting projects Layar has up their sleeves, including something  Layar may be cooking up with The RoomWare Project.

The picture above is of RoomWare’s Social RFID Installation for Media Plaza in Utrecht (read more here).

Demos Galore!

In the demo rooms, Noah Zerkin (pic below left) pretty much single handedly carried the AR flag for a growing community of augmented reality Makers and Hackers.  But his presence was much appreciated, and he tirelessly demoed The Zerkin Glove. See the first of what may be several posts from Noah on ISMAR here.

And I got to try out the Vuzix goggles (picture above on right).   This was my first experience playing an AR game that was smart about real world gravity. It’s  “an augmented reality marble game that uses gravity as a game controller” – see Ori Inbar’s write up here.  It was a very compelling experience and I have to say I didn’t really notice the shortcomings of the Vuzix goggles while I was absorbed in the game. And  I turned out to be quite good at the game too. It is intuitive unlike the kind of  rule based games I never have time to learn properly.  But what is so special about this project is the tools that it is built with are open, and available for all, and affordable (see this list on Games Alfresco).

It was a great pleasure to meet Prof. Steven Feiner (picture on below the left) who heads Columbia University’s brilliant AR research team at The Columbia University Graphics and User Interfaces Lab.

Ori Inbar (pic below on right) also spent a lot of time in the demo room showing off Ogmento’s lovely AR learning game that delighted attendees, “Put a Spell: Learn to Spell with Augmented Reality.”

For a round up of  what’s next for augmented reality head mounted displays check out, Games Alfresco here, and Thomas Carpenter’s excellent review of the head mounted displays.

Ori Inbar on Games Alfresco asks is “Microsoft – the new big player to watch?”   “Georg Klein, inventor of PTAM-on-an-iPhone (and the smartest Computer Vision guy on the block)” has joined Microsoft to make Mobile AR.

The picture on the left above shows Georg trying out ARhrrr with Blair MacIntyre.   And on the right Blair is demoing his marker card pack to Senior Vice President of Cypher Entertainment, David Elmekies.  Yes ISMAR was abuzz with demos. See this post from Gail Carmichael for more video demos.

Next Year ISMAR 2010 in Korea!

At the banquet, I managed to find a seat at a table with Sean White (at left in photo above with Christine Perey to his right) and the Columbia University team.  The banquet culminated with the “Past and Future of ISMAR” Panel chaired valiantly by Jay Wright of Qualcomm.  We were asked to offer our input for ISMAR 2010.  I offered up an idea that I have been nurturing for a while now -  to stage a “Green Tech AR Competition.”  Perhaps, I suggested, we could base the competition around a conference (ISMAR 2010 in Korea?) and set up a target rich, instrumented environment for the occassion.  I think the Arduino open hardware community and AR developers have a synergy that is just waiting to be explored!  And, if we add the innovators of data curation to the mix, e.g., Pachube, AMEE, and Path Intelligence…(Markus Tripp left ISMAR to speak on a Web 2.0 Summit panel, “Humans as Sensors,” which also included Path Intelligence, Deborah Estrin on “participatory sensing,” and the brilliant work of Di-Ann Eisnor, Platial, on “Transactional Cartography”).  Anyway a big Green tech AR competition could get people working together across the broad spread of AR terrain on some of the sticky problems of user experience.  And, with a high level of support from Smart Phone companies, HMDs manufacturers and the chip makers we just might come up with some extraordinary magic.

The devil of course will be in the details.  But a competition like this could not only motivate key players to come together in the spirit of coopetition but also be an opportunity to show the world the power of AR to make visible the invisible ecosystems that are so important to the health of our planet.

One of the notable presences at ISMAR 2009 was the Qualcomm team.   Jay Wright’s presentation (an exclusive for ISMAR) not only outlined AR for 2012, but Jay also talked about some “close to the metal” innovation that we will see from Qualcomm very, very soon!  I had some time in the press room with Jay and his team prompted by MoMo’s Yuri van Geest.  When I twittered about Qualcomm’s presentation at ISMAR, Yuri replied:

vangeest “@tishshute: good stuff, hopefully you will integrate the neat new solutions and ideas in your talk in November ;)”

I will be presenting at MoMo #13 on AR, open AR, future of AR and GeoWeb,  and hopefully will bring some good news from Qualcomm too.  Anyway Jay seemed to like the idea of a Green Tech AR Competition, even though I did stress that I thought it needed some serious sponsorship and BIG prizes.

Where’s the beef? Tracking and Mapping at ISMAR 2009

On the flight from NYC to Orlando and ISMAR’o9 I dozed (I had been up late preparing my presentation) and I watched the Dew Tour Pro Skateboard competition and Top Chef on the Food Channel.  In this particular episode of Top Chef, the aspiring chefs were all given a brown bag of ingredients by an already famous chef who then judged whether the contenders managed to make a delicious meal with their allotment which was notably lacking in key ingredients of haute cusine.

This metaphor of  trying to cook up a great meal while perhaps missing the staples is apt for the current early stage of commercial augmented reality.  And when I arrived in Orlando, not only were the Dew Tour pro skateboarders staying at the same hotel as ISMAR, but ISMAR itself felt remarkably like an Augmented Reality Top Chef Coopetition.

Much of ISMAR was dedicated to the task of  providing the meat and potatoes of Augmented Reality, solutions to mobile tracking, mapping and registration, particularly in the Science and Technology track.

Industrial and Military Augmented reality solutions I found out, typically, solve the tracking problems by using fixed mounts which clearly wouldn’t translate well into the AR everywhere with everything mobile consumer culture expects.

In the picture on the left Fabian Doil stands by the VW engine that provided some of the outdoor targets for the ISMAR tracking competition.  On the right is a picture from the VW’s presentation on their research and development of AR.

I followed the tracking contest, organized by Daniel Pustka and Fabian Doil of Volkswagen, quite closely. And I learned a lot in the process. While  it is clear there has been progress in AR mapping and tracking, we still have a ways to go.

But hanging around the Tracking Competition was a good way to find out the state of play of this crucial part of the AR dream.  For example,  a little tidbit I learned is that Michael Goesele who has been reconstructing “high-quality geometry models from images collected from the internet (so called community photo collections, CPC)” is soon to be at the Institut Graphische Datenverarbeitung where top contenders in the tracking contest – Harald Wuest  and Folker Weintipper (in the foreground of the photo at the left and right respectively) are also to be found. [update Harold and Folker were the winning team see here for details of scoring and results!] Otto Korkalo and Tuomas Kantonen of VTT, Finland, Augmented Reality team are in the background. They have been working on the joint IBM, Nokia and VTT project that brings, Avatars and People Together for Virtual Meetings in Physical Spaces.

The picture on the right is another team that were doing very well. If my notes serve me well (and please forgive me if they don’t. I came back with my card wallet overflowing!) the photo on the right showsChristian Waechter (on the left) and Peter Keitler (on the right) of the Technische Universitat Munchen.

Germany is certainly leading the way in industrial AR. And I learned how small businesses like Metaio get to work with top research institutions and big companies like VW, thanks to very strong German funding program for AR and VR. The current iteration of a series of funding programs is  called Avilus.  Avilus  is putting 42 million Euros into AR and VR this year alone (click on the slide below to see more about Avilus ).

I wish we had the equivalent of Avilus here in the US.  But there is no equivalent to Arvilus for AR here, and  no AR is  being developed by the US car industry either it seems.  But look at the slide above to get a taste of some of the cool stuff Metaio and other small AR and VR businesses do for VW through the Avilus project.

I also got to meet many people from one of the world’s most important AR hubs -  The Department of Informatics, Technische Universitat Munchen, including Prof. Gudren Klinker on the far right in pic above.  And from left to right, Fabian Doil (VW, co-organizer of contest), Sebastian Lieberknecht , Selim Ben Himane (Metaio), Tobias Eble (Metaio).  Prof. Klinker is the engine behind much of German innovation in AR.

Metaio was one of the few teams to rely mainly on markerless tracking which in this contest was very challenging because of the very different light conditions (see pics below) between the windowless interior and dazzling Florida sunshine outside (pic on the right shows targets under ideal lighting conditions).  Many people in the US may be  familiar with Metaio’s consumer applications, like Junaio,  but thanks to Germany’s efforts to nurture augmented and virtual reality they are also respected software developers in industrial AR.  And I suspect that Metaio will spearhead markeless tracking in consumer AR too.

This post as usual has already expanded to something much longer than I originally attended – pretty typical for me! There is much I have not been able to cover including some of the interesting contributions by augmented reality artists at ISMAR – again I recommend the upcoming videos.

But I cannot end without a hat tip to, Oriel, Nate et al. who won the best student paper award for AR Sketch – again please see Games Alfresco for more on this (pic below from Games Alfresco). AR Sketch, Ori notes, is featured “in our top post and popular video.” And

“Their work is revolutionizing the AR world by avoiding the need to print markers – or any images whatsoever.”