The Missing Manual for The Future (or The Future: The Missing Manual)

O’Reilly Media, is famous for is producing  “missing manuals” for new technologies, but thinking of O’Reilly as just a publisher of books would be like saying Facebook is just a website (this came up in the discussion at Media Round Table at Web 2.0 Expo, NY, 2010).   In recent weeks, I managed to catch Tim O’Reilly at several events, Maker Faire, Web 2.0 Expo, Hadoop World, and the free webcast Tim did with John Battelle on The Battle for the Internet Economy (although Tim spoke several other times during this period!).

It occurred to me, as I immersed myself in the depth and breadth of innovation showcased and discussed at these events that Tim O’Reilly, and the O’Reilly team, are creating, The Missing Manual for the Future.

As Tim puts it, we are “changing the world by spreading the knowledge of innovators.” Tim uses a quote from William Gibson to illuminate what is at the heart of the O’Reilly project:

“The Future is here, it is just not evenly distributed yet.” (William Gibson).

But Tim O’Reilly makes another point about the future when he speaks.  The future unfolds unexpectedly – so we must invent for an unknown future not a known future, or as Alex Steffen put it so well in his post, Why Our Bright Green Futures Will Be Weirder Than We Think, – “The world we need is one we’ve never yet seen.” The magic of attending an O”Reilly event is that it gives you a chance to work on this koan in interesting ways, and to take more responsibility for how things turn out.

Tim O’Reilly also urges that we think more deeply about what we are doing.  His keynote for Hadoop World , NYC, billed as, “The Business of Data” turned towards “The Consequences of Living in a World of Data.” The 900 strong crowd at Hadoop World was probably one of the most savvy crowds in the world about the business of data, so this was a nice turn.

Web 2.0 Expo with the theme, Platforms for Growth, was a deep dive into the business of innovation.  Tim O’Reilly’s keynote at Web 2.0 Expo,  “Thinking Hard About The Future” (or rather “thinking a little bit creatively or differently about the future)  – see video here, developed the call he made at Web 2.0 Expo 2008, to “work on stuff that matters,” into a Four Cylinder Engine for Innovation.   The first of the four cylinders in the firing order is, “Having Fun!” But,  at Maker Faire, Web 2.0 Expo, and Hadoop World I got an inside look at the workings of all four cylinders, and there is more to come, I am sure, as the other O’Reilly events unfold over the coming months including,  Web 2.0 Summit, Strata (a new O’Reilly conference on The Business of Data), and Where 2.0, 2011.

In a free webcast, last week (recording here), previewing Web 2.0 Summit, John Battelle and Tim O’Reilly discussed the Points of Control Map which is developing into a fun and useful tool to examine a very serious topic, “The Battle for the Internet Economy,” and how the “increasingly direct conflicts between its major players” could effect “people, government and the future of technology innovation.”   In my previous post, Platforms for Growth and Points of Control for Augmented Reality, I had a great conversation with Chris Arkenberg using this map as a springboard.  More on Points of Control later in this post.

The Four Cylinders of Innovation

click to enlarge

From Jet Ponies to Jet Packs: The First Cylinder of Innovation – “Have Fun”

The “maker” energy and its spirit of play, and the courage to create, hack, reinvent and re-purpose everything and anything, is a quintessential example of the first cylinder of innovation firing big.  Many “maker” projects also go on to fire on all four cylinders.   But the Maker forte definitely is in the first cylinder zone (and safety third as some of the rides, including Jet Ponies, warned).  The photo opening this post by Marc de Vinck – for more pics see here, is of Tim riding The Jet Ponies at Maker Faire which took  the New York Hall of Science by storm in late September – see The New York Times coverage here.  The ride was “built by the dastardly danger-hackers at the Madagascar Institute.“ See this wonderful interview with Hackett on his work to design “our specific jets from a patent that was filed in 1960s by a Mr. Lockwood, for Valveless Pulse Jets.” Hackett points out:

“Louder than god, glowing white-hot and looking like the trombone of the Apocalypse, pulse jets are also really shitty, inefficient engines,”

But, he adds:

“I have always wanted a jetpack, and one of the reasons I learned to build these things was to further that goal.”

This grand vision behind the Jet Ponies is a key to firing, The Second Cylinder of Innovation,  “Hey, we can change the world!”

But Jet Ponies, as a stepping stone to jet packs, also really struck a chord for me as I have been devoting a lot of time lately to the emerging Augmented Reality industry, a technology which was lumped in the same category of sci fi chimera as jet packs until very recently.

Data is the Gasoline

“The faces are coming from the sky.  The locations are coming from the sky.  All these apps depend on something, somewhere up.  And that, to me, was always the heart of Web 2.0.  And I am so delighted that people are finally getting it.  Because for a long time, people thought, ‘Oh, Web 2.0, it’s about lightweight advertising supported in a web start up.’  So I went, ‘No, no, no. It’s about the fact that we’re building these giant database subsystems in the sky that are going to drive applications.’  And now, of course, the same application is on your PC, it’s on your phone, it’s on you iPad.  And clearly, the applications are just sort of an interface to something that is being driven from the cloud, and that is fabulous.  That’s the difference.  People get it now.” (Tim O’Reilly, said this as part of a response to the first questioner at the Media Round table Web 2.0 Expo)

Answering questions about the importance of “Having Fun” to innovation doesn’t look quite as fun as riding Jet Ponies! Photo above from luca.sartoni’s Flickr stream

“ the data that is generated by the sensors and the applications that use that data is going to be where people are going to be innovative.” (Tim O’Reilly)

During the Media Round Table, I had a chance to ask Tim more about the role of bottom up innovation in a world where big data is the gasoline for increasingly sophisticated engines – platforms integrating machine to machine intelligence and real time analytics.

Tish Shute: You brought up Maker Faire in your keynote, and again now.  I was there, which not many people in the audience were  [not too many hands went up when Tim asked during his keynote].  But I think one of the things that struck me was the jet ponies – they were just earthshaking to stand near.  They made the ground tremble; they made the world shake.  Yet, most of your keynote, and most of what’s on our minds here, at Web 2.0 Expo, is extracting intelligence from the big data [in the sky], and algorithmic intelligences are the jet engines of the internet.  And of course, not to be forgotten, as we are here in New York City, where the trading markets are creating the air we breathe  [although we probably don't realize it until we lose our mortgage or something] and these algorithmic economies or “robot casinos” as Kevin Slavin put it, are all about speed – it’s not just real-time, issues of latency are  so critical that co-location is key to winning the game of the markets.  [Kevin Slavin brilliantly unpacks this in his talk, "Loitering on the Motherboard." For more in this see my conversation with Kevin Slavin below].

So my question is, who’s making the jet ponies for the algorithmic economies in the sky that you just described?   How can we make a play from the bottom up?  I always feel Ushahidi is one of the jet ponies of the data algorithmic space [because of their great work to bring human and machine intelligence together to solve problems in crisis situations].  But who do you think is doing exciting work and how can we ensure that this powerful world of data and algorithmic intelligences does not become hidden in a closed black box [only really accessible to elite players like the NYC trading markets]?

Tim O’Reilly: “Well, I think there’s certainly a lot of interesting things happening in, say, the financial services that a lot of, kind of, the Internet folks are kind of blind to.  I think that there are companies like Next Jump which are really good with data and good with algorithms. But kind of speaking specifically to the maker side of this, that whole sensor enabled world which is going to produce data is in its infancy.  What we have that I think is so powerful right now is we have the first portable sensor platform.  I said in my talk the other day, you know, your phone has ears, it has eyes, it has a sense of where it is.  And these are all available to application developers. You know, you can compare, say, Dodgeball to Foursquare, you can see how different… Dodgeball is Foursquare in the tele-type era.  Foursquare is now possible because there are so many more capabilities on the phone.

And I think that we are going to see a lot of other areas that are revolutionized by the sensors in the device.  It could well be that some of them will come explicitly out of the maker kind of projects, or it could just be that make is sort of a proxy for them.  So yeah, Arduino is this great maker sensor platform, but hey, here’s a consumer sensor platform [holding up phone]. Maybe we vaulted past the maker stage already and we just didn’t know it.

And that’s not entirely true, because Arduino is building a whole economy of special purpose devices.  But it feels a little bit like the days when people rolling their own PCs coexisted with the rise of Dell, who was a kid in his college dorm room who made his own PCs and sold them on the net, but figured out how to scale it pretty quickly and get good at it. But there were still a lot of garage shops, you know, ‘I’ll make a PC and sell it to you’ people for probably a decade before there was really a clue that that was a commodity industry.  In fact, I do think the sensor platforms are going to become a commodity industry.  And the data that is generated by the sensors and the applications that use that data is going to be where people are going to be innovative.”

The internet operating system is a data operating system and it is happening in real time (Tim O’Reilly)

click to enlarge the image above  – a slide from Mike Olsen’s  (CEO of Cloudera) keynote at Hadoop World

Not only do we have a portable sensor platform in our pockets  but developers also have powerful platforms and tools to make sense of data that fuel our apps.   Opensource Hadoop makes available, to anyone with some data munching chops, the power to work with giant unstructured databases and do the kind of real time analytics previously only available to giants like Google.  Big players like Yahoo, Facebook, and Twitter use Hadoop (Jonathon Gray from Facebook noted they add 10TB a day).   But, as this great roundup of Hadoop World points out, while Hadoop gets the press for handling petabytes of data , Mike Olsen (CEO of Cloudera) noted, the fastest growing area of users are working with clusters smaller than 10TB and over half of the Hadoop clusters were under 10TB in size.

Four Square: A Platform for Growth with an ecosytem built on top of data that exists in the real world

As an augmented reality enthusiast it is not hard to guess that one of my favorite platforms for growth is Four Square.  See Dennis Crowley’s keynote at Web 2.0 Expo here.  The Four Square API has been available to developers since November 2009,  and there are already a number of  interesting applications, and there will be many more to come.  The screen shot above is of geopollster – see the gallery of Four Square apps here.

@dens tweeted recently  “Politics + @Foursquare = @GeoPollster” http://geopollster.com <- I love love love that people are using 4SQ to think about election tools

As Kati London pointed out in her keynote, Four Square is the “kind of augmented reality that is aimed at shifting or changing a person’s social reality, e.g. the mayor badges in Four Square that change my relationship to the people and the place I am in, and augment engagement and reputation through socially driven consumer tie ins.” We are already see augmented reality developers beginning to work with the Four Square API – see here, Foursquare + Augmented Reality + Virtual Graffiti = ARstreets.

As augmented reality development tools mature, Four Square will, increasingly, become an important platform for creative AR developers interested in integrating the power of this platform for augmented engagement and reputation with “device aided augmented reality that can shift visual experiences of situated geolocal experiences.” With the Qualcomm vision based augmented reality SDK now available for download, and Wave in a Box soon? to be released, and an ARWave client working on Android (almost!), I have been exploring the Four Square API in my non existent spare time!!

The Four Square API also offers some interesting possibilities for exploring games that take the complex economy of Four Square – not personal data but aggregates of behavior, as their subject matter (for more on this see my conversation with Kevin Slavin later in this post and in an upcoming post).

I took this picture of Dennis at Where 2.0, 2009 at the beginning of Four Square’s phenomenal growth (they are at 4 million plus users now).

Pachube (Patch-Bay): a web service for storing and sharing sensor, energy and environmental data

Eighteen months ago, I interviewed Usman Haque (architect and director, Haque Design + Research) and founder of Pachube – see Pachube, Patching the Planet.   Usman pointed me to this wonderful evocative image from T.R. Oke’s book, “Boundary Layer Climates” (original photo source Prof. L. E. Mount’s The Climatic Physiology of the Pig).  “It’s the same piglets, in the same box, but on the right hand side the temperature has been increased. This small change in how the space is “programmed” has dramatically changed the way the ‘inhabitants’ relate to each other and how they relate to their space.”

The Challenge of Connecting people and environments.

At Web 2.0 Expo, I got the opportunity to talk with Usman Haque again.   Pachube, is becoming an established platform now, Usman explained.  They have a development team of eleven and robust back end.  And, they will now be spending some more time on the front end, including a redesign of the website, making “it a lot easier to widgetize the entire website so that you will be able to take almost any element and embed that into your own website.” And, as Usman mentioned in his presentation, they are working on an augmented reality interface, Porthole, for facilities management and, “as a consumer-oriented application that extends the universe of Pachube data into the context of AR – a ‘porthole’ into Pachube’s data environments..  Usman is also contributing to the AR standards discussion and on the program committee now for the W3C group on augmented reality.

Click to enlarge the image above from Chris Burman’s paper for the W3C, Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual”

Pachube, is sometimes described as the Facebook for Data or an analogy Usman prefers, a Twitter for Sensors.  At Web 2.0 Expo, I had an amazing opportunity to hear from Twitter and Facebook about their strategies as platforms for growth.  This gave me lots of fuel for questions about Pachube’s approach to developing their platform.  Simplicity was a theme that Facebook  and Twitter both affirmed as a key.  One of Pachube’s challenges will be to deliver ease of use, and the equivalent of Facebook’s “like” and  Twitter’s “follow” to gain mass appeal.

Here is a brief excerpt from my upcoming conversation with Usman:

Tish Shute: So as a platform you see Pachube as having more in common with Twitter – a Twitter for Sensors. In what ways is Pachube similar to Twitter?

Usman Haque: Well we are the Twitter of sensors, devices & machines in the sense that, really, the API that enables all this communication is important, much more so than the website itself. It is where, basically, most of the millions of our hits actually go, is to the backend. And we’ve now got dozens of applications built on top of the system, a little bit like Twitter’s applications; you know, all the apps are the important part.

But we are actually going to be doing some quite exciting things with API keys that we haven’t really spoken that much about in public. But we have come up with a pretty innovative solution to make almost every resource have granular privacy options on it, now discussed here.

At Hadoop World, Tim O’Reilly also raised some interesting broader questions that are very relevant to Pachube’s vision to “patch the planet”, e.g, the problem of digital identity in the age of sensors? (Smart phones already know their users by the way they walk!) And, “How should we think about privacy in a world where data can be triangulated?”

Usman talked about Pachube’s approach to both the technical aspects of how to build a massively scalable system, and the conceptual aspects of how people connect to each other, and what they might do with these new opportunities to connect environments and sensor data  (see my earlier talk with Usman, Pachube, Patching the Planet, for a detailed explanation of some of the concepts behind Pachube).

I look forward to posting this conversation. Pachube is growing, and Usman always goes beyond the familiar tropes of connecting human and machine intelligence.

2nd Cylinder of Innovation: “Hey Can We Change the World!”

The possibilities for reimagining of the role of data in healthcare produced some of the most powerful “Hey Can We Change the World” moments for me at both Web 2.0 Expo and Hadoop World.  The slide above is from Esther Dyson’s brilliant Ignite presentation, “What you can and can’t learn from your genes?” are here,   Tim O’Reilly also brought up the powerful role real time data analytics can play in improving healthcare in his Hadoop World Keynote.  Also see Alex Howard’s post, 10 Lessons for Gov 2.0 from Web 2.0 for some more great, “hey we can change the world moments” at Web 2.0 Expo.  The keynote from Lukas Biewald of CrowdFlower and Leila Chirayath Janah of Samasource (screen shot below) in particular, is a provocative exploration of the future of work in the new ecologies of human and machine intelligence.

Changing the World When Our Lives Are Increasingly Shaped by Forces Invisible To Us?

Click to enlarge

Mike Olsen, CEO of Cloudera, noted that “the largest area of data growth does not come from humans interacting with machines; rather, it’s from machines interacting with each other” (see here in Minor Technical Difficulties).   One of the most interesting presentations at Web 2.0 Expo was Kevin Slavin’s, “Loitering on the Motherboard,” which, as Tim O’Reilly pointed out in his keynote at Hadoop World, is a talk that raises all kinds of questions about a system where big players are gaming the data for their own ends.

Kevin Slavin, a founder of Area/Code, notes the operating system of our mortgage, life insurance, the operating system of currencies and gold is now governed by machine to machine intelligence and algorithimic economies outside of human cognitive processes.  The markets are now legible only to bots in an algorithmic arms race with bots surveilling bots, and throwing off false information in a bid for counter-surveillance.  He showed some slides of the eery but beautiful visualizations of traces of the trading bots created from the Nanex API.

The screenshot above is from the Nanex: Crop Circle of the Day – Quote Stuffing and Strange Sequences.  “The common theme with the charts shown on this page is they are all generated in code and are algorithmic. Some demonstrate bizarre price or size cycling, some demonstrate large burst of quotes in extremely short time frames and some will demonstrate both…” This one is a zoom of the NSDQ “Wild Thing.”  Wild price/size repeater from NSDQ running at 1,000 quotes per second, effecting the BBO along the way (I love the great names Nanex gives the different patterns and traces produced by the trading bots).

Nanex supplies a real-time data feed comprising trade and quote data for all US equity, option, and futures exchanges. They have archived this data since 2004 and have created and used numerous tools to “sift through the enormous dataset: approximately 2.5 trillion quotes and trades as of June 2010.” May 6th 2010 (day of the flash crash), had approximately 7.6 billion trade, quote, level 2, and depth records.

Kevin points out that our lives are being shaped by criteria invisible to us and the old hackneyed tropes of machine to machine intelligence such a robots reading HUDs in English are long worn out.  The latter point is, perhaps, something for us augmented reality geeks absorbed in ideas of “making the invisible visible” to chew on.

Changing a world shaped by forces that are, increasingly, invisible to us presents a huge challenge.

But I had the glimmer of a, “Hey Can We Change the World” moment, when I attended Kevin Slavin founder of Area/Code’s presentation and had a conversation with him after his talk.  Could games take these complex economies as their subject matter?  The economies of  Farmville and games like WoW are not opaque at all, and these are environments with complex economic behavior, “where you can actually have enough data to understand what it is” – “it’s not so much about personal data.  It’s more about, like, aggregate behaviors.” “Games that can really model those, and play with those, and take those as the subject the way that Monopoly takes Monopoly as a subject could be really interesting.” Kevin made many fascinating points – more to come on this topic.

Photo by James Duncan Davidson, of Kevin Slavin speaking at Web 2.0 Expo NY, 2010, from the O’Reilly Conferences Flickr stream

Here is the beginning of our conversation:

Talking With Kevin Slavin

Tish Shute: You began your talk today about visibility and where some of the algorithmic masters of disguise went to work, after they had solved the math behind stealth bombers.  I thought perhaps you were leading into ideas about a reverse surveillance society.

But you surprised me, as I felt you made visibility itself kind of a non-issue by the end of your presentation and that counter surveillance became basically a time and speed issue.  Now I am not sure quite how to imagine a counter-surveillance society, something I try to think about…

Kevin Slavin: Well, let’s see.  There’s a couple ways to think about it.  I think one point is just that when we talk about counter-surveillance, we usually locate that as something that comes from  the bottom up, something that comes from the population. Think about the way the plane spotters discovered the CIA black rendition flights.

I think in general, when people talk about counter surveillance, or sousveillance, they imagine it as an inversion of the traditional relationship between the people and the state.

But that’s what’s interesting. What’s happening now, is that there are forms of surveillance and counter-surveillance that are in play beyond any human perceptual horizons. These forms are at their most sophisticated in financial services, in the markets.

If you were a bot, and could read the market legibly (which humans cannot), what you would see, effectively, are bots that are surveilling bots. Then you have bots that are throwing off false information in a bid for counter-surveillance. Many of the bots are, themselves, surveilling other bots; each one of them is trying to figure out what all the other ones are going to do. In essence, it’s an algorithmic arms race, and game theory has become concrete, since the theories are code, the code is action, and the action affects, let’s say: your mortgage.

And so, basically what you have is you have this series of algorithms that are all looking to discern each other, while also trying to prevent themselves from being discerned. I think of the tunnels under the trenches in WWI, tunnels to surveil the trenches, and then, later, tunnels to surveil the tunnels. There’s a few examples of this kind of thing.  But It’s especially strange when it’s computer code, and at the magnitude we’re seeing today.

All of it, as noted in the talk, accounting for 70% of all the trades in the market. 70% of the market trades are never touched by human hands or even seen by human eyes; they don’t move through a conventional cognitive process.  And that’s why you get things like the Credit Suisse algorithm, it was buying, selling 200,000 shares of stocks to itself over and over and over again. It was a bug and it slowed the market to a crawl.

Credit Suisse was fined, in essence, for failing to control an algorithm. Maybe that’s the first time an algorithm was treated like a human, in a way. As if the algorithm broke the law, and Credit Suisse was responsible for letting it do so. For me, that feels like a threshold event.

It’s not that humans never made mistakes when trading on the market. But when algorithms err, they err with magnitude.

The idea that we now have bugs in the United States market economy is really worth looking at.  If Apple can’t keep code bugs from the most simple iPhone apps in a closed and regulated ecosystem, I’m pretty certain we’ll have a lot more Credit Suisse type bugs in the future.

And that will be pretty interesting. There will be viruses, and the operating system they will operate on will be the operating system of the United States. The operating system of your pension, your house, your life insurance. The operating system of currencies and gold.

Tish Shute: I was hard-pressed by the end of your talk to think of like, “Well, what would be the equivalent of, sort of a people’s uprising to create a better fairer society in this kind of world where, really, the things that affect the key aspects of lives most are going on beyond human perception at an algorithmic level?”  But you made a pretty radical suggestion at the end…

Kevin Slavin: Well I think increasingly the markets have become delaminated from anything meaningful. First from goods, then from fundamentals, and now finally from homo sapiens. So that’s hard to fight.

It’s the race towards abstraction that makes it impossible to simply “resist.” The latest version in the long series of fiscal catastrophes was based on Wall Street finding goods that could be rolled up and sold with false valuations, but goods that would take a long time to fail. Mortgages are handy like that. It’s the tradition of extending the abstraction as long as possible, until finally the bill arrives and the banks fail. I don’t know if that’s something to rise up against or not. It’s like a rally against evil.

But really, I think the point is that it won’t be the people that rise up. It will be the financial services themselves that rise up. They’ll just detach completely.

That was harder to do with cotton or with wheat, with simple futures; they keep financial services tied to the ground.  So what we’re doing is creating increasingly complex financial instruments that are further and further removed from anything you can touch.  Like the way a mortgage is abstract. But, of course, the bottom line is that at the end of that mortgage lies someone’s home.

It’s said that Wall Street is now moving onto life insurance, because that’s going to take even longer to fail.  They’re doing the exact same thing. The word is that they are rolling up CDOs made out of crap life insurance policies, same way they rolled them up with crap mortgages a few years ago.

And those will probably take, I don’t know, 15 or 20 years to unwrap and unravel.  But what you see in the meantime, is that they are looking for things that are increasingly abstract, intangible, removed as far as possible from the experience of everyday life.

So maybe this is good. Maybe that’s financial services rising up. Lifting off. I think best case scenario now is that they actually leave humans alone altogether.  That, someday, they are just trading, effectively, completely arbitrary goods, the stocks could be anything at all, maybe for crops that no longer exist, and I’m just saying that then these bots would no longer affect what we do and what we are, it would just be a robot casino, an invisible paradise in the air.

People are the platform: How Games Can Be Engines of Innovation in Our Lives

See the video of Games that Know Where We Live here (screen shot above)

Kati London, Senior Producer, Area/Code, in her keynote showed how games that know where we live can shift players perspectives – from device aided augmented reality that can shift visual experiences of situated geolocal experiences to a kind of augmented reality that is aimed at shifting or changing a person’s social reality, e.g. the mayor badges in Four Square that change my relationship to the people and the place I am in, and augment engagement and reputation through socially driven consumer tie ins.

Area/Code has recently developed two games for the Knight Foundation that take people as the platform.  Macon Money, uses very simple games dynamics (for more see the video of Kati’s keynote) in a game designed to help “Knight’s continuing efforts to support revitalizing Macon and creating a vibrant college town.”

The other game that Area/Code has designed with the support of the Knight Foundation  is for the Biloxi and Gulf Coast community, a game called Battlestorm.  “The game’s purpose is to increase awareness about natural disasters and change the way people prepare for them.”

3rd Cylinder of Innovation: Build products, business models and entire industries.

Glympse – real-time, private location tracking

Julianne Pepitone, Yahoo! Finance, nailed the essence of Web 2.0 Expo, NYC, this year in her post, Web 2.0 Expo startups are big on neighborhoods, storytelling.  She writes:

“At the Web 2.0 Expo in New York City this week, executives from big sites like Facebook, Twitter and Pandora all spoke about industry trends. But the showcase of 27 startup tech companies stole the show.”

Listen carefully to Tim O’Reilly and Fred Wilson, Union Square Ventures, question their picks from the startup showcase at Web 2.0 Expo.  Also see this video of Fred and Tim discussing their conversations with all the start ups.  This  is one of the clearest public windows onto both how to present your company to VC, and how to figure out what are the most important questions for you as an entrepreneur  building a business in a world of data.

Glympse successfully pitches their “jet pony” strategy for a location based business, and is Fred’s pick.  They hold up well under pressure and answer Tim and Fred’s hard questions about how their start up will not get overtaken by an encumbent player with resources and market share before they can gain traction.   food52 responds to Tim’s probing about their strategy for business data analytics that he points out are vital if they want to survive with the small margins of ecommerce.  There is a list of all the participants in the start up showcase in Brady’s post here. hour.ly was the audience pick.

Shazam for Faces!

My favorite start up was a biometric service doing face, iris, and finger print matching, Tactical Information Systems.

Tim and Fred also liked them, and they have an interesting discussion about the merits or not of approaching your platform through a narrow first application as Tactical Information Systems are with WanderID -  an application to help identifying lost Alzheimer patients.  As Fred pointed out, they are potentially the Shazam for faces, so why start so small?

I  had asked TIS the same question when I met them in the “speed dating” session.  This is just their first toe in the water as they are a two person company at the moment. Their vision for their platform is big.  Mary Haskett and Dr Alex Kilpatrick, the founders of this quintessential jet pony for the algorithmic economies in the sky, are not only a partnership with the credentials to do a  Shazam for faces – see their bios here, they are the people I would want to be running a Shazam for faces!  They really get the consequences of living in a world of data – check out Dr Kilpatrick’s absolute killer Ignite talk, “Defeating Big Brother.” (screenshot below)

How Can Augmented Reality Add Value to the Real Time Internet/Data Operating System?

planefinder.net – an augmented reality app that lets you find information about planes by pointing your phone at the sky, “including flight number, aircraft registration, speed, altitude and how far away it is” (via MacLife).

The new opportunities in the algorithmic economies in the sky were center stage at Web 2.0 Expo and there are some interesting AR apps for the real time internet/data operating system emerging, like planefinder.net.  But Augmented Reality was still pretty low profile at Web 2.0 Expo (except that NVidia augmented reality demo attracted a lot of attention at the sponsors expo).  However, everyone working in the emerging industry of AR should recognize that apps big on “neighborhoods and story telling” are heading right up the AR street, and that platforms like Four Square and Pachube present enormous opportunity to explore the possibilities of AR.  And if augmented reality enthusiasts are not already paying attention to real time data analytics, and Hadoop, they should be (see this post for an excellent round up on Hadoop World).

At Hadoop World, Tim O’Reilly referenced the great tagline from the  IBM commercial:

“Would you be willing to cross the street — blindfolded — on data that was five minutes old? Five hours? Five days?”

As I have noted in several earlier posts – see here and here for starters,  we are just seeing the tools  for developing near field, vision based, mobile, social AR become widely available to developers, so there should be a new level of AR apps emerging through 2011.  There is a wonderful discussion in the comments of this post by Mac Slocum, “How Augmented Reality Apps Can Catch On,” between Mac, Raimo one of the founders of Layar, and Chris Arkenberg on what constitutes a platform for growth for augmented reality.

Mac’s post, the comments and Chris Arkenberg’s post on the latest edition of the Gartner Hype Cycle, that rather curiously placed Augmented reality almost at the peak of inflated expectations. really got me excited about exploring an idea I have been thinking about for a while, which is to get the AR community to discuss the Points of Control map.    See my discussion with Chris Arkenberg here, Platforms for Growth and Points of Control for Augmented Reality. The recording of  John Battelle’s and Tim O’Reilly’s webcast on Points of Control is posted here.

The interactive Points of Control map is an amazing tool to think with! Check it out in movements, territory and movements, acquisition mode.  There is a competition for the most interesting comment and most interesting acquisition suggestion.  The prize is a ticket to Web 2.0 Summit!

What is the Future of Social?

The recent “defection” from Google to Facebook – see Lars Rasmussen, Father Of Google Maps And Google Wave, Heads To Facebook,  is as MG Siegler of TechCrunch points out, “the biggest one since Chrome OS lead Matthew Papakipos made the same jump in June” (TechCrunch also notes “current Facebook CTO Bret Taylor was heavily involved in the launch of Google Maps”).

These moves have drawn my particular attention as did Bret Taylor’s response in his conversation with Brady Forrest at Web 2.0 Expo to Brady’s question, “How soon until we get the Facebook firehose?”

If you have been reading Ugotrade you already know how important I think an open, distributed, standard for real-time communications such as the very innovative Wave Federation Protocol could be for AR development  -  see ARWave and my presentation at MoMo13, Amsterdam last year, The Next Wave of AR: Mobile Social Interaction Right Here, Right Now!

The anticipated release of  Wave in a Box, has raised hopes in the developer community that  WFP will soon become easier to work with, and hopefully more widely adopted.  Like many others, I wonder what will happen to Wave in a Box now?

But the innovation of Wave is deep and broad (and as many have pointed out hugely ambitious).  Perhaps the boldest attempt yet to innovate both at the low level of architecture (where Google is so powerful) and at the high level of the Mark Zuckerberg, “big idea,” which as Tim O’Reilly notes is, “What is the future of social?” MG Siegler noted Facebook Groups Is Sort Of Like Google Wave For Human Beings.

But I deeply hope that the open, distributed standard part of the Wave big idea is not lost in the mix here.

Fourth Cylinder of Innovation: Keep the Ecosystem Going, Create More Value than You Capture

The Points of Control map is interactive, so please click here or on the image above for the full experience.

Tim O’Reilly points out that there is a worrisome dark side to the Points of Control Map – see Tim’s keynote here.  To paraphrase some or his points:

There are companies on the map that are forgetting to think about creating a sustainable ecosystem.  Rather than growing the pie, they are trying to divide up the pie and that threatens to cause the fourth cylinder of innovation to misfire.  This fourth cylinder is essential to the ecosystem.

Tim O’Reilly looks back to the lessons of the personal computing industry which was incredibly vital and creative, and lots of people made money until a couple of big players “sucked all the air out of the ecosystem” and innovation had to go elsewhere.

The Power of Platforms is to create value not just for your company but for other people.   Create value for yourself by creating value for other people.  Tim O’Reilly used the wonderful example of  Henry Ford inventing the weekend so that there would be enough people with time and money to buy his mass produced cars.  Think about building the ecosystem that will support the future your are going to build.  Grow the pie rather than cut up the pie.  This will be the vital fourth cylinder of innovation in a Web Squared world.

Tim O’Reilly has long proposed that  Web 2.0 is all about harnessing collective intelligence,  But as Gartner predicts, “By year end 2012, physical sensors will create 20 percent of non-video internet traffic.” Yet another previously unevenly distributed future is going mainstream, and if you haven’t read it already, now is the time to read this paper by Tim O’Reilly and John Batelle, Web Squared: Web 2.0 Five Years On.

The Consequences of Living in a World of Data

To bring this very long post to a close!  Here are just a few of the key questions re The Consequences of Living in a World of Data that Tim O’Reilly raised during his keynote for Hadoop World:

“How would we solve the problem of digital identity in the age of sensors? (Our smart phones are able to know their users by the way they walk – their gait!)

“How will we input data when our devices are smart enough to listen on their own?”

“How should we think about privacy in a world where data can be triangulated?”

“We are moving to a world in which every device generates useful data, in which every action creates information shadows on the net.”

“Shouldn’t we regulate the misuse of data rather than the possession of it?”

“How do we avoid a data arms race?”

“Create more value than you capture.”

The Points of Control map is interactive, so please click here or on the image above for the full experience.

Today at 4pm EST, 1pm PDT John Battelle and Tim O’Reilly will discuss the Points of Control map and The Battle for the Internet Economy in a Free Webcast:

“More than any time in the history of the Web, incumbents in the network economy are consolidating their power and staking new claims to key points of control. It’s clear that the internet industry has moved into a battle to dominate the Internet Economy.

John Battelle and Tim O’Reilly will debate and discuss these shifting points of control as the board becomes increasingly crowded. They’ll map critical inflection points and identify key players who are clashing to control services and infrastructure as they attempt to expand their territories. They’ll also explore the effect these chokepoints could have on people, government, and the future of technology innovation.”

I’ve been wanting to start a discussion on the  Points of Control map in the Augmented Reality community for a while now, and Chris’ recent post on the latest edition of the Gartner Hype Cycle, “Is AR Ready for the Trough of Disillusionment?” and this post by Mac Slocum, “How Augmented Reality Apps Can Catch On,” and the conversation in the comments between Mac, Raimo (one of the founders of Layar), and Chris, all prompted me to get a conversation started…(see below for all that followed!).  Chris put me on the hot seat back in June when he did this very generous interview with me on Boing Boing, so it was time to turn the tables.

Tim O’Reilly, in his  keynote for Web 2.0 Expo, pointed out there is both a fun and a dark side to the Points of Control map.  There are companies on this map, he noted, that rather than “growing the pie,” are trying to divide up the pie, and they are forgetting to think about creating a sustainable ecosystem. I expect the conversation between Tim O’Reilly and John Battelle to dig deep into this Battle for the Internet Economy.  If, like me, you have another engagement at the time of the webcast, you can register on the site to receive the recording.

AR is still too young to figure in the battles of the giants, but there will be a lot to be learned from this conversation.  And, The Points of Control map is good to think with from the POV of AR in many ways.  As Chris Arkenberg observed:

“When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?”   You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development.  And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.”

Chris Arkenberg is a technology professional with a focus on product strategy & development, specializing in 3D, augmented reality, ubicomp and the social web. He uses research, scenario planning, and foresight methodologies to help organizations anticipate change and adopt a resilient and forward-looking posture in the face of unprecedented uncertainty. His personal work is collected at urbeingrecorded, and his professional profile is here.

He is also one of the founder/organizers of AR DevCamp which is currently scheduled for Dec. 4th (somewhere in SF or The Valley!)  Chris said, “No further details atm (still trying to find a venue and get sponsors) but please direct people to http://ardevcamp.org for upcoming information.”

Talking with Chris Arkenberg

Tish Shute: I know some people thought the positioning of AR by Gartner near the peak of the hype cycle was misguided, and based on a very narrow understanding of AR as used in marketing apps. But reading your post I thought you made a lot of good points.

Chris Arkenberg: It’s tracking hype, right? It’s not necessarily tracking the growth of the technologies or their maturation so much as it’s tracking the general attention level. And what’s interesting to me is that tends to affect the amount of money that goes into those technologies.

Tish Shute: I was particularly interested in your post because I have been writing a post about two recent O’Reilly events in NYC, Maker Faire, Web 2.0 Expo, and then Hadoop World, where Tim gave a very interesting 45 minute keynote.   AR was pretty low profile at all three events.   But the NVidia augmented reality demo attracted a lot of attention at the sponsors expo, and Usman Haque, Founder of Pachube announced in his presentation, they are working on an augmented reality interface for Pachube called Porthole, its designed for facilities management and, “as a consumer-oriented application that extends the universe of Pachube data into the context of AR – a ‘porthole’ into Pachube’s data environments.. “  Usman also mentioned, when I talked to him, that he is contributing to the AR standards discussion and on the program committee now for the W3C group on augmented reality.  For more on this standards discussion and the Pachube AR interface, see Chris Burman’s paper for the W3C, Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual.”

I think pioneers in the augmented reality commmunity should pay attention to these wider conversations about the Battle for the Internet Economy, and the exploration of the  “Platforms for Growth” theme at Web 2.0 Expo is very important- this is a course also a nudge to read my upcoming post on these O’Reilly events!

Also I have another project I have been chewing on that I would like to talk to you about.   I want to start an AR conversation about the wonderful Points of Control map produce for Web 2.0 summit by John Battelle. [ Note there will be, "Battle for the Internet Economy" free Web2Summit webcast w/ @johnbattelle & @timoreilly Wed 10/27 at 1pm PT http://bit.ly/b46cmb #w2s]

Up to this point, understandably given the immaturity of the technology, AR has little role in the “Battle for the Internet Economy.”  But this doesn’t mean that the map isn’t good for AR visionaries, enthusiasts, entrepreneurs, and developers to think with.   And both you and Tim have pointed out the potential for AR to leverage the giant data subsystems in the sky.   I have to say the positioning of Cloud Computing on the brink of heading down into the trough of disillusionment in this recent rendition of the Gartner Hype Cycle seems ridiculous!

Cloud Computing is already ubiquitous hardly seems credible that it is headed for a trough of disillusionment!

Chris Arkenberg: Yeah, it’s ubiquitous so why even talk about it when it’s your fundamental infrastructure?

Tish Shute: Yeah and I seriously doubt it is imminently headed for a trough of disillusionment….and this brings me back to the Points of Control Map which as John Batelle points out, “aims to identify key players who are battling to control the services and infrastructure of a websquared world” in which the “Web and the world intertwine through mobile and sensor platforms.”  This instrumented world, of course, creates a great deal of opportunity for augmented reality. Have you seen that, that points of control map?

Chris Arkenberg: I think I have, actually.

Tish Shute: There has been much debate about how this intertwining of the web and the world will play out in augmented reality.   Chris Burman points out in his position paper for W3C,  Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual”, that “trying to draw parallels between a browser based web and the possibilities of AR may solve issues of information distribution in the short-term,”  but it must not have a limiting effect in the long-term.   But now we at least have one web standards-based browser for AR thanks to the work of Blair MacIntyre and the Georgia Tech team.  But I think the discussion in the comments of Mac Slocum’s recent post, “How Augmented Reality Apps Can Catch On” is an interesting starting point from which to think about platforms of growth for AR.  I am not sure if I am stretching his meaning but I think Raimo, Layar, is suggesting that what the Point of Control map call the the Plains of Media content is very important to the growth of the fledgling AR industry right now. And I would agree with this, and add that the neighboring terrain of gaming levels will be pretty key as one of my other favorite AR start ups Ogmento hopes to reveal in the near future! But what do you think was most important in this brief but pithy dialogue between you Raimo and Mac?

[The screenshot above is a teaser video the Gary Hayes from MUVEdesign for his upcoming (2011 release date), game called Time Treasure.  See Gary’s blog for more and Gary’s  post from over a year ago on AR Business models.  Thomas K. Carpenter, on Games Alfresco notes, “I think this is a terrific idea and I find it interesting he’s planning this on a tablet rather than a smartphone.”

Chris Arkenberg: The way I took it…And to give a little bit of context, I came from sort of this apprehension of augmented reality as an expression of the existing Internet. So as sort of a visualization layer that allows you to kind of draw out data, and then, with all the affordances of being able to anchor it to real world things.

And my own sort of path has led me to want to really try to understand that and refine it, particularly with respect to the sort of Internet of things and the smarter planet idea of just having embedded systems everywhere. And specifically, what is the value-add for augmented reality as a visualization layer of an instrumented world?

And so that’s caused me to be a bit biased towards that side of AR. And the way I took Raimo’s comment was that he was saying that, “You know, really what we’re interested in is media.” That he was effectively saying that AR for them is really just about that space between the screen and the the world, or between your eyes and the world, and what you can do there.

Certainly I had considered it in the past, but I hadn’t really focused on it or assumed that it was a priority as a business model. And so he kind of reminded me that, actually, there’s a lot of entertainment applications. There’s a lot of, obviously, advertising and marketing applications.
And so I felt that I was being a little narrow in my focus…

Tish Shute: Yes this comes to the heart of what I am interested in about the role AR can play in opening up new relationships to the world of data that we live, not just making it more accessible and useful to us when and where we need it, but AR as a road to reimagining  it..

Have you seen any interesting work yet to explore these great data economies in the cloud through AR. I mean can you think of any others – there is planefinder.net but others?

Chris Arkenberg: I’ve seen a few just sort of skunk works type applications that people have been playing around with, again, to try and reveal things. One of them was similar to the aircraft, but it was more for military use and being able to identify things of interest in the sky. I’ve seen a couple other for navigation, so being able to identify mountain peaks on a visual plane, for example, but this isn’t so much about revealing an instrumented world.

Tish Shute: Yeah, I think that was from the Imagination right? I know that’s an interesting one. Usman at Web 2.0 Expo, in his presentation, mentioned the work Pachube is doing on an Augmented Reality interface. I interviewed Usman again as my last long interview with him was nearly 18 months ago now and Pachube is well on the way to becoming the Facebook of Data or the analogy that Usman prefers – the Twitter of sensors!

Chris Arkenberg: Hmm, interesting.

Tish Shute: And to go back to your comments on Augmented Reality not getting caught in some of the traps that have made virtual worlds lose relevancy I think that is vital that AR developers understand the strategic possibilities of key points of control in the internet economy because the isolation and Balkanization of virtual worlds were certainly a factor in their rapid slide into the trough of disillusionment – although many would argue that a fundamental flaw in the kind of virtual experience that Second Life and other virtual worlds constructed was really the fatal flaw (see James Turner’s interview with Kevin Slavin Reality has a gaming layer).

But Second Life’s isolation from the other great network economies of the internet was certainly a limiting factor.

Chris Arkenberg: And that’s been exactly my sense, and I’ve, over the years, tried to encourage development in that direction for virtual worlds. I did work, through Adobe, to help develop Atmosphere 3D back in the the early 2000’s. And we did a lot of work to try and understand the marketplace and the specific value-add of doing things in 3D over 2D.

And this is kind of why I keep referring back to VR and VW’s with respect to augmented reality, is that with immersive worlds, there was this idea…there was this big rush. Everybody was so excited about it. It was obviously the next cool thing. And everybody wanted to try to do everything in it. You could do your shopping in virtual worlds. You could have meetings in virtual worlds.

Tish Shute: and shopping, yes ..that didn’t work out so well!

Chris Arkenberg: And everybody was very excited in developing these things. And what it really came down it is, “Yeah, you can, but it’s actually a lot better to do those things on a flat plane or in person.” Meeting Place, WebEx, TelePresence – those tools generally do a much better job at facilitating TelePresence meetings than a virtual world does. The same with TelePresent Education. There are only very specific things that both VR and AR are really good at.

And that’s where I find myself with augmented reality right now, trying to really pick through that and critically look at which uses are really appropriate for an AR overlay. And again, I think that’s why the hype cycle is important, because it reflects back this desire that AR is going to be the next big thing – the be-all, end-all of interacting with data in the cloud – and forces us all to take a critical look at why we should do things in AR instead of on a screen.

AR is not going to work well for most things but it’s going to be very good for certain uses. Right now I’m very keen at trying to understand what those things might be.

Tish Shute: I had this wonderful conversation (more in an upcoming post) with Kevin Slavin one of the founders of Area/Code at Web 2.0 Expo and I think some of what he describes about the data brokerages of High Frequency trading have some interesting implications for AR’s role, say, in ubiquitous computing. The trading markets are now pretty much dominated by machine to machine intelligence; machine to machine brokerages. They are basically game economies on the scale that we can barely wrap our heads around where the speed that bots and algo traders can access the network is the key. We really have no clue what is going on until we lose our house…

Kevin was also interviewed by James Turner on O’Reilly Radar. He talked about how much of the interesting work in location based mobile social apps is defined in opposition to the model of Second Life. He also talked to me about how we are seeing “first life” take on the qualities of “second life.” What goes on the trading floor is largely a performance secondary to a more important world of machine intelligence with giant co-located servers and bots fighting for trading advantages measured in fractions of seconds.

He pointed out how we draw on all these tropes from sci-fi movies, these HUDs based on ideas of machine intelligence where the robot talks to the other robot in English through an English HUD!  Many of our current visual tropes for AR are perhaps just as inadequate for the kind of data driven world we live in.

Of course, when you are thinking of having fun with dinosaurs, or illustrated books, or whatever, this is not, perhaps, an issue.  But if you are thinking of augmented reality interfaces as being important in a battle for network economy, and platforms for growth,  how this new interface helps us live better in a world of data is an important issue.

Chris Arkenberg: Now, does that indicate that the UI just needs more overhaul and innovation, or more that the visual interface for those experiences shouldn’t really leave the screen? It shouldn’t move on to the view plane?

Tish Shute: Yes we have a few concept videos that try and explore this ..

Chris Arkenberg: Well, and I think this will happen at the level of human-computer interface. I mean that’s always been its role, in making coherent the sort of machine mind, for lack of a better term, making it coherent to the human mind. So I mean there is a lot of this sort of machine intelligence, the semantic Web 3.0 revolution, where it really is about enabling machines, and agents, and bots to understand the content that we’re feeding them.

But at the end of the day, they, for now, need to be providing value to us human operators. So there’s always going to be a role for human-computer interface and user experience design to make this stuff meaningful.

I mean, if you look at the revolution in visualization & data viz, this is of incredible value because it takes a tremendous amount of data and collates it into a glanceable graphic that you can look at and immediately comprehend massive amounts of data because it’s delivered in a handy, visual way.

So I see that as a fascinating design challenge, how the user experience of the data world can be translated into meaningful human interaction.

Tish Shute: Yeah. And when we see Stamen Design pursuing a big idea in AR, that’s when we might start to rock and roll, right?

Chris Arkenberg: Yeah. In my article, I sort of jokingly suggested that Apple will create the iShades. But, they’ve got the track record of being way ahead of the curve and delivering the future in very bold forms.

Tish Shute: A key part for the battle for the network economy is to bring the complexity of data into the human realm in a way that increases human agency. Kevin suggests that the giant robot casinos of markets should actually lift off into total abstractions as theses machine-driven trades get back into the human realm in ways that are so damaging to our lives – a lost house or job! The notion of a counterveillence society where people have more agency over the important aspects of their lives, health, housing, job (which I discussed with Kevin – interview upcoming) has gotten pretty tricky!

But I think we will begin to see AR eyewear for specific applications (gaming and industrial) get more common fairly soon – possibly as smart phone accessories.

And it is clear that AR is going to be, increasingly,  a part of our entertainment smorgesborg in coming months. Itouch has a camera (although lower resolution), Nintendo’s are AR-ready and many aspects of the AR vision of hands-free spatial interfaces will go mainstream through Natal.

But we are yet to see an app/platform emerge for mobile. Social AR games that turn every bar and cafe and ultimately the whole city into a gaming venue  -although I think Ogmento and MUVE aim to lead the way here! Will an AR company achieve Zynga level success by using the Foursquare, for example?

My feeling is that the lesson of Zynga is pretty important for mobile social AR games. Could Flash social gaming have taken off without Facebook?

Chris Arkenberg: And that’s the real driver. And again, as you mentioned with Second Life, and this was exactly my own sense, is that they stuck to the closed garden model and didn’t get the power of social and collaboration. They attempted to add some of those affordances within the world, but, you know, ultimately most people aren’t in virtual worlds, and most people aren’t using augmented reality. So leveraging the really predominate platforms like Twitter and Facebook and Foursquare, being able to leverage those affordances, that connectivity, into a platform like augmented reality, I think, is really critical. Because again, you get nothing unless you have the masses, unless you have people present.

Tish Shute: In AR research there is a long history of the notion of powerful AR-dedicated devices, but smart phones and tablets are good enough,  and can launch augmented reality into the heart of the internet economy. I think  the elusive AR eyewear will come to us initially as a smart phone accessory for specific apps.  But, for the moment, most AR apps make little attempt to play in the wider internet economy.

Chris Arkenberg: And I think it’s actually much lower hanging fruit, really, to do gaming, marketing, transmedia. Because then you don’t really care about the cloud, or maybe you only really care about a little part of it that your gaming property is addressing. Then it becomes much more about entertainment, and much more about persuasion, and sensationalism. And if you’ve got dancing dinosaurs on your street, great! It’s entertaining, it’s cool, it’s new. That stuff is fairly straightforward.

I keep coming back to this idea of, you know, the instrumented city. What sort of data trails do you get out of a fully instrumented city? So maybe you get traffic patterns, maybe you get geo-local movements of masses, maybe you get energy usage, that sort of thing, all the, sort of heat maps you can generate from a city. But then what good does it do to be able to have that on an augmented reality layer versus just looking at it on a mobile device or looking at it on your laptop?

Tish Shute: Of course the use cases for “magic lens” AR are different from the kind of hands free, 360 view with tightly registered media, that a full vision of AR has always promised. The 360 view is quite a different metaphor from the web and mobile rectangular screens.

Chris Arkenberg: Yes, yes.

Tish Shute: Did you see that great parody of Michael Jackson’s “Beat It” with the iPads versus the iPhones, right?

Chris Arkenberg: Oh, really?

Tish Shute: I tweeted it cos i thought it was quite funny and a little close to the bone!
[laughter]

“ur wanna an ipatch 2 b the new fad?” #AR gets cameo in Twitter, iPads & iPhone’s Michael Jackson-Inspired Parody via @mashable

It is hard to get away from the importance of eyewear when discussing AR!

Chris Arkenberg: Yes, so the hardware, to me, is a big stumbling point right now, or it’s a large gating factor, I think, for realizing what an augmented reality vision could really be like. That it really does need to be heads up. This holding the phone up in front of you is fun to demonstrate that it’s possible, and it’s valuable in some ways…

Tish Shute: And it’s particularly nice in some applications like the planes app, the Acrossair subway app where you hold the phone down and get the arrow, right?

Chris Arkenberg: Yeah, the way-finding stuff I think is really valuable…

Tish Shute: Sixth Sense really caught people’s imagination because it managed to deliver the gesture interface with cheap hardware, even if projection has limited uses (no brightly lit spaces or privacy for example!).

The other important and as yet unrealized part of the AR dream is real-time communications. Many interesting uses cases would require this. As you know that is my chief excitement, along with federation, in the Google Wave Servers for (which should soon be released at Wave in a Box) for ARWave.

Chris Arkenberg: Well my sense of Wave is that it was a ChromeOS protocol that they instantiated, or that they exhibited in the public deployment of Google Wave. That that was a proof of their sort of low level architectural solution. Because, you know, they’ve been rumored to be working on this cloud OS for some time. And so my sense is that Wave is actually one of their core components of that cloud OS, and that it just happened to incarnate for the public in a test run as Google Wave.

Tish Shute: I do hope that Wave In the Box will lower the barriers to entry to people experimenting with this technology. The FedOne server was just way too hard for most people to take the time to set up. Of course, it is the brilliance of the Wave Operational Transform work that also poses problems in terms of ease of use. But Wave Federation Protocol is pretty innovative. And could even play an important role in a real time communications for AR eyewear connected to smartphones. The challenges that Wave takes on re real-time communications, federation, permissions and filters are pretty important ones for AR…

Chris Arkenberg: Especially when you’re trying to federate a lot of permissions and filter a lot of data, which all of that gets even more important when you have a visual layer between you and the real world.

Tish Shute: You got it. Yeah!

Chris Arkenberg: I think that’s really valuable real estate, both for third parties that want to get access to your eyes, as well as for you, as the user, who still needs to navigate through the phenomenal world and not be occluded by massive amounts of overhead data.

Tish Shute: Yes, I am sure Google has big plans for the next level of cloud computing and Wave looks at some key challenges. I suppose federation poses some key business problems. I think it was Michael Jones who said to me that it was a bit like socialism in that you have to be willing to give something up for the greater good.

Perhaps federation does not present enough appeal because of its challenges re business models?

Chris Arkenberg: Well, I wonder. I mean there’s got to be some value for their ad platform as ads are moving more towards this personalized experience. Advertising is becoming less of a shotgun blast and more of a very precise, surgical strike. So being able to track user data to such a fine degree to mobilize the appropriate ads around them wherever they are, on any platform, is certainly very valuable to Google and their ad ecology.

Tish Shute: Many people have high hopes that HTML 5 by lowering the barrier of entry for  browser style AR could also pave the way for some interesting AR work..

Chris Arkenberg: Well, as much as I would hope that all the different players are going to come together and establish some shared set of standards, really, what’s happening is it’s a rush to the finish line to be the first…to get the most penetration in the marketplace so that Layar, for example, can say, “It’s official. We’re the platform.” And then the consolidation that will follow, where the Googles and the other big players like Qualcomm say, “OK, it’s mature enough. We’ll start buying up all the smaller companies.”

And that’s where the real challenge is right now is that there are no standards. It’s such an immature technology that you have a lot of different players trying to establish the ground rules. And again, this is one of the challenges that faced public virtual worlds, is that you had a lot of different virtual worlds that weren’t talking to each other in any particular way, and that they each had their own development platform. And so you end up with a very fractured ecosystem or set of competing ecosystems, which is kind of what’s happening with AR right now, where a developer has to choose between a number of different new platforms or hedge by deploying across multiple platforms. Basically, the web browser wars are set to be recapitulated by the AR browsers.

Among them, Layar and Metaio seem to be getting the most traction. But there’s still not a really strong case for a unified development ecosystem to emerge.

Tish Shute: So a discussion of ecosystem development brings us back to the Points of Control Map I think. So what do you see as key points of interest for AR developers to watch in the Points of Control Map? And where do you want to sort of put your bets, right? We are still really waiting for mobile social AR to emerge into the mainstream.

Chris Arkenberg: Yes. And that’s primarily the shortcoming of the hardware itself, but also of the accuracy of current GPS technology. That’s another kind of gating factor, because again, AR wants to be able to express the data within a distinct place or object.

So in a lot of ways, other than kind of what we’ve allowed for the broader entertainment purposes, for AR to really work, there needs to be more resolution in GPS location. So for it to be truly locative…because it’s OK to tell Foursquare that you’re in Bar X. But if you want to be able to draw data directly on a wall within that bar, or do advertising over the marquee on the front, you need more factors to accurately register those images on a discreet location. So that’s another, sort of, aspect of the immaturity of AR, is that it’s still very hard to register things on discreet locations without employing a number of diverse triangulation methods.

Tish Shute: Right. The mobile AR games we see at the moment are really just faking a relationship to the physical world unless they rely on markers or some limited form of natural feature recognition which is really just a more sophisticated form of markers. But the Qualcomm SDK does offer some opportunities to tie AR media to the world more tightly as does the Metaio SDK. But in terms of a mobile social AR game that could be like the Cape of Zynga to FourSquare in Location Basin [see the Points of Control map]… We haven’t seen anything close yet.

AR should be able to bring the check-in mode to any object in our environment.

Chris Arkenberg: Yes, yes. And that’s actually one of the early interests I had in the notion of social augmented reality. I wanted a way to tag my community with invisible annotations that only certain people could read, and found pretty quickly that that’s very difficult to do. I mean you can kind of do some regional tagging, like on a beach, for example, but if you wanted to tag the bench that was on the cliff above the beach, it’s very difficult to do that using strictly locative reckoning.

There’s all sorts of really cool social engagement that can be revealed when people are allowed to attach things to the world around them, to the streets they normally pass through, or the points of interest that they normally engage in. To be able to author on the fly on the streets and attach it discreetly to an object effectively.

Tish Shute: And yes we do have all kinds of markers and QR codes. But Erick Schonfeld of Tech Crunch made a good point that QR codes: “Until QR code scanners become a default feature of most smartphones and they start to become actually useful enough for people to go through the trouble to scan them, they will remain a gee-whiz feature nobody uses.”

Chris Arkenberg: So again, this gets back to competing standards and who gets access to the phone stack, the bundle. Who gets the OEM deal…?

Tish Shute: Yes, the battles for the networks on the Handset Plains are pretty important for AR!
[laughter] I think Layar have made some smart moves on The Handset Plains.

And there are a lot of acquisitions of nearfield technology to look at.  If I remember rightly Ebay bought the Red Laser tech from Occipital – now there’s any interesting company. Their panorama stuff rocks!

Chris Arkenberg: Right. There’s a lot of nearfield stuff that’s supposed to hit all of the major mobile platforms in the next year or so.

I mean I think where this is heading, in my mind, is basically smart motes. You know, little nearfield wide-range RFID’s that are the size of a small, tiny square that you could attach to just about anything and then program it to be a representative of your establishment or of an object, that then you can start to tag just about anything. I mean you can’t rely on geo to do it, but if you have a Nearfield chip there that costs maybe like two cents to buy in bulk, and you can flash program it, then you can start to attach data to just about anything.

Tish Shute: Yes ‘cos some things still remain very difficult for near field image recognition technologies like Google Goggles.

Chris Arkenberg: Well, if your phone can interrogate for Nearfield devices, and it detects a chip in its near field, it can then interrogate that chip. The chip may contain flash data on itself, or it may contain the local server in the establishment, or it may go to the cloud and get that data back.

Tish Shute: Yes there is moverment from the top and open source hardware like Arduino has created an opportunity for all sorts of creativity with instrumented environments.  And the handheld sensors in our pockets – our smart phones create a lot of opportunity for bottom up innovation too.

Chris Arkenberg: I mean that’s my guess. If you look at what IBM is doing with their Smarter Planet initiative, they’re partnering with a lot of municipalities, and obviously with a lot of businesses and their global supply chains.

But they’re basically working with municipalities and all these stakeholders to instrument their territory, their business, or their city, as it were. So they’re working to provide embedded sensors and the software necessary to read them out and run reports & viz. And presumably that software can extend to include some sort of mobile device to interrogate the sensors and read the data.

That’s kind of a top-down approach of a very large global company working with top-down governance bodies to do this. Simultaneously you have the maker crowd experimenting with Arduino and such to build from the grassroots, the bottom up approach.

And that’s primarily gated by the amount of learning it takes to be able to program these devices, to be able to hack them. Typically, the grassroots creators who make these devices don’t have the luxury of very large budgets to make things highly usable and Wizywig.

So the bottom up community is a sandbox to create tremendous amounts of innovation, because they are unconstrained by the very real financial needs of the top down innovators. And so you get a lot of fascinating innovation, a very rich ecology from the bottom-up approach, but you don’t get a lot of wide distribution. But that does filter up to and inform the top down approach that has a lot more money to put into this stuff. And it ultimately has to respond to the needs of the marketplace.

I mean if there’s an answer to the question of whether something like AR will succeed through the bottom-up grassroots approach or the top-down industry approach, I would say it would be both. That handsets will be hacked to read the bottom up innovations of the maker community, and handsets will be preprogrammed to read the top down efforts of the IBMs of the world.

Tish Shute: Yes but i have to say it is very time-consuming hacking phones (I have just seen a few days suck up in this myself so that I could upgrade my G1 to try out the new ARWave client!). I mean Android has obviously been the platform of choice because of openness but the business model of iPhone and its market share in the US sure make it important for developers.  It’s like you don’t exist if you don’t have an iphone app for what you are doing.

Chris Arkenberg: Yeah, and that’s the challenge, because at the end of the day developers prefer not to work for free and a solid, reliable mechanism to monetize their efforts becomes very appealing.

When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?” You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development. And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.

Tish Shute: Some of the important platforms for AR don’t appear to have spots on the map like Google Street View and other mapping technologies that hold out so much hope for AR, or am I missing something?

Chris Arkenberg: You mean on the map?

Tish Shute: Yes for the full vision of AR we need sensor integration, computer vision and cool mapping technologies to come together. Do you see where Google Maps and Google Street View… Where would they be?

Chris Arkenberg: Yeah, I mean it’s certainly content, it’s location…

Are you familiar with Earthmine?

Tish Shute: Yes, yes I am, definitely. Earth Mine, Simple Geo, Google Street View, user generated internet photo sets like Flickr all of these could be very important to AR, potentially.

Chris Arkenberg: Well, and the interesting thing about Earthmine is that they’re effectively trying to do an extremely precise pixel to pixel location mapping. So they’re taking pictures of cities just like Street View, except they’re using the Z axis to interrogate depth and then using very precise geolocation to attach a GPS signature to each pixel that they’re registering in their images. Effectively, you get a one-to-one data set between pixels and locations. And so you can look at something like Google Street View, and if you point to the side of a building, in theory, it should know exactly where that is.

They’re rolling this out with the idea of being able to tag augmented reality objects in layers directly to surfaces in the real world. So that’s another approach to trying to get accurate registration and to try and create what are essentially mirror worlds. Then your Google Street View becomes a canvas for authoring the blended world, because if you plop a 3D object into Street View on your desktop, and then you go out to that location with your AR headset, you’ll see that 3D object on the actual street.

Tish Shute: There was some experimental work with Google Earth as a platform for a kind of simulated AR but I suppose Google Earth doesn’t figure in the battle for the network economy as it never got developed as a platform.

Chris Arkenberg: It hasn’t tried to become a platform, to my knowledge. I mean I know some people are doing stuff with it, but as far as I know, Google owns it, they did it the best because they have the best maps, and there’s not a huge ecosystem of development that’s based around it other than content layers.

And my sense of everything else on the Points of Control map is they’re looking more at these sort of platform technologies that…

Tish Shute: Yes, re platforms for growth for AR. Gaming consoles will probably emerge as a significant platform for AR this year.

Chris Arkenberg: There will be much more of a blended reality experience in the living room for sure, and with interactive billboards. Digital mirrors are another area. So I mean if we kind of extend AR to include just blended reality in general, you know, this is moving into our culture through a number of different points. As you mentioned, it will be in the living room, it will be in our department stores where you can preview different outfits in their mirror. We’re already seeing these giant interactive digital billboards in Times Square and other areas.

It’s funny. I mean for me, the sort of blended reality aside, the augmented reality, to me, is actually a very simple proposition in some respects. When I look at this map, augmented reality is just an interface layer to this map in my mind, just as it’s an interface layer to the cloud and it’s an interface layer to the instrumented world. It’s a way to get information out of our devices and onto the world.

Tish Shute: The importance of leveraging existing platforms has become pretty clear but it is interesting Facebook definitely gave Zynga the opportunity but would Facebook be so big without Zinga’s social gaming boost?

Chris Arkenberg: I feel that Zynga has definitely helped its growth…But I think Zynga has benefited a lot more from Facebook than Facebook has from Zynga.

Tish Shute: Zynga certainly proved you could build a profitable business on Facebook’s API!

Chris Arkenberg: They did. And they also really validated the Facebook ecosystem and the platform. They really extended it… Zynga benefited from the massive social affordances that Facebook had already architected and developed. They brought gaming directly into Facebook, and particularly, this emerging brand of lightweight social gaming that when you sit it on top of a massive global social network like Facebook, it suddenly lights up.

Tish Shute: AR pioneers should quite carefully go through this map. There is so much to think about here. I’m a kind of fanatic about Streams of Activity in AR. Real time brokerages and their potential for AR is something I am fascinated by. That is one reason I love the ARWave project.

Anselm Hook, to me, is one of the great thinkers in this area of real time brokerages – with his project Angel, and the work of Ushahidi, which is now the platform for augmented foraging (see here). Anselm is now working on AR at PARC which is exciting.

Chris Arkenberg: Well, there are some challenges working with data streams. Presentation and filtering I think is a big challenge with any sort of stream. Because obviously, you have a lot of potential data to manage, to parse, and to make valuable and comprehensible. So I think this is bound very closely to being able to personalize experiences, or having very discreet valuable experiences. Disaster relief, for example, I think is an interesting idea that ties into the Pachube type of work. Where, if you had the headset and you were a relief worker, and you had immediate lightweight, non-intrusive, heads up alpha channel overlay, waypoint markers showing you all of the disaster locations or points of need, AR becomes extremely valuable, because it’s a primarily hands-free environment. This is why the military stuff is so interesting.

Tish Shute: Ha! We are running into the eye patch/shades/goggles/sexy specs thing again. But filtering and making streams of activity relevant will be very interesting for AR.  Again that it why I love the Wave Federation Protocol work because what they have built into their XMPP extensions. You can have your real-time personal data streams, or community streams, or broadcast publicly – the permissions are built.

And Thomas Wrobel’s original vision of these layers and channels is only fully expressed if you have the eyewear.

Chris Arkenberg: Well, and it becomes redundant if it’s on a mobile. To use a very basic example, Twitter, obviously there’s an app you can view those streams of activity on the camera stream. But you can view that real time data on the screen. Why do you need to see it heads up?

The reason I really pay attention to what the military is investing in, one, because they have a ton of money, but also because they tend to represent the core bio survival needs of the species…So, when I look at computing, I see this very obvious trend of computers getting smaller and smaller and closer and closer to us because they’re so valuable to our success. They give us so much valuable information for engaging our world on a moment by moment basis. So, of course now we have these tiny little handheld devices that give us access to the global knowledge depositories of human history, because it’s so useful to have that stuff right at hand.

The only impediment now is that it takes one of our hands, if not both of them, to access it. So if you are in the natural world, which we are all always in the natural world, ultimately, you want your hands free in order to engage with the world on a physical level.

I see computation, or rather, our access to computation is just going to get thinner and thinner, and we’ll very soon move into eyewear, and inevitably, we’ll move into brain computer interface in some capacity.

So when you’re the disaster worker, or a deployed soldier, or the extreme mountain biker, or the heli-skier, or just an adventurer, there are a lot of very practical reasons to have access to information on a heads-up plane. I see AR as being so profound and so valuable, but we’re getting a glimpse of it in its infancy, and it’s got a ways to go to be able to really contain what it is we’re reaching for.

Tish Shute: I agree.

Chris Arkenberg: And that’s been a big criticism I’ve had with all the existing AR implementations that I’ve seen, is that the UI really needs a revolution. It’s very heavy handed. It is not dynamic, even though it’s supposed to be. It does not take advantage of transparencies. It treats the screen like a screen. It doesn’t treat the screen like a window onto the real world. When you’re looking on the real world, you don’t want a lot of occlusion. You want very soft-touch indicators of a data shadow behind something that you can then address and then have it call out the information that’s important to you.

Tish Shute: Now, that’s a very nice kind of image you’ve conjured for me there. Do you see that more could be done on the smartphone than is being done within that? Or are we like waiting for the old ishades?

Chris Arkenberg: I think there’s definitely a lot of room for improvement on the smartphone UI. Nobody’s really played around with it much. And again, I think that’s in part that there hasn’t been a really established platform with enough money to fund interesting UI work. We see it in some of the concept demos that float around every now and then.

I guess it’s both a blessing and curse that I’m always five steps ahead of where I’m trying to get to.

Tish Shute: Yeah, I am familiar with that feeling!

Chris Arkenberg: So I’m always trying to reach for the vision even though it’s a bit distant. I think there’s going to be a lot of development on the handsets. But again, I think we need a lot of refinement. We need a lot of real critical analysis of why this is a good thing.

To get back to the original point of Raimo’s comment, it struck me. And I knew it, but I just had set it aside as gimmickry. But he’s right. Content is a huge driver for this. Just stuff that’s engaging, and fun, and cool, and shows off the technology so they can get enough money to make it through whatever Trough of Disappointment may be waiting.

Tish Shute: Yeah, don’t underestimate the Planes of Content!  They are a great place to get interest and money to keep AR technology moving on, right?

Chris Arkenberg: Yeah, yeah. Because, you know, there’s a lot of freedom there. And you can piggyback on all the rest of the content that’s out there and jump on memes and marketing objectives, etc…

And there’s a lot of stuff…I’m blanking on some of the names, but some of these historical recreations of city streets. There’s a street in London where they overlaid historical photos in a really compelling experience. [Museum of London - http://www.museumoflondon.org.uk/] Again, I’m completely forgetting the attributions, but hose are the type of things that can really be pursued on the existing platforms. There is stuff that’s really compelling and really cool.

I heard of another interesting use case – and I should say that I can’t find attributions to this anywhere on the web and I may be paraphrasing or mis-representing the actual work, but I think the concept is worth exploring anyway. But the idea was that you could take the locations of border checkpoints and conflict sites in Palestine and Israel and visually overlay them on an AR layer in San Francisco. And it would do some sort of transposition where you could virtually view these things in San Francisco with the same locational mapping superimposed. So you could see where the checkpoints where. You could see where the wall was. You could see where suicide bombings were and where there had been conflicts. [I cannot find any citations for this!]

Tish Shute: But with an AR view? But why would you use an AR view if you are in San Francisco, then?

Chris Arkenberg: Because it superimposes two realities, translating the Gaza conflict into San Francisco as you are walking around. You can interrogate the world. There’s a discoverability aspect where you’re using the headset to reveal things, or the handset rather, to reveal things that you could not see otherwise in your city. It was done as an art piece, but as a provocative, obviously political art piece.

Tish Shute: Very interesting. I’d love to see that. Because that’s interesting to get away from this idea that you actually have to sort of have this one to one relationship between the data and the world is kinda nice, isn’t it? Well, not one to one, but a very literal…getting away from that literalness is kind of good.

Chris Arkenberg: And that’s a possibility of virtual reality and augmented reality merging, that maybe virtual reality is actually going to do best by coming out of the box and writing itself over our reality, so that as you are walking around, you are no longer seeing San Francisco, but you are seeing part of Everquest or World of Warcraft.

Tish Shute: Well this is where Bruce Sterling gets to that point he made in his keynote for are2010, that if we actually have viable AR eyewear, then you get the gothic stepsister of AR, VR rising from the grave!  He asks whether the very charm of augmented reality, is in fact that it adds rather than subtracts from your engagement with the world and that getting get sucked back into the black hole of VR might not be so great.

Chris Arkenberg: And then you get all sorts of interesting challenges to social cohesion if you have a lot of different people experiencing very different worlds, effectively. That if there is no real consensual reality and a majority of your local populous is, in fact, experiencing very different and unique versions of the world, what does that do to social cohesion? How does that reinforce tribalism, for example, when only you and certain others get to opt in to a particular layer view of the world?

Tish Shute: Yes Jamais Cascio wrote an interesting piece on that issue on AR and social cohesion a while back.

An eye patch is a more logical vision than the goggles in many ways but I suppose the loss is stereo vision?

Chris Arkenberg: And actually, there were developments in military helicopter technology many years ago that used a single pane square of glass over the eye mounted to the helmets of pilots. And then they drew various bits of heads-up information on it. So that ensures that you’re having a real strong engagement with the real world, which, obviously, when you’re a helicopter pilot is quite important. But you still have access to the data layer of the invisible world.

Tish Shute: I just went to Hadoop World and I have to say, I was awestruck about how big that’s got. I mean Hadoop has gone from like zero to huge in just a few years. I mean it’s just like now everyone has the power of the Google big table at their fingertips.

What’s the play for AR in the land of search?

I could imagine Hadoop being very powerful tool for AR analytics?

Have you got any thoughts on the land of search and AR? Of course visual search is proceeding at a fast pace and there is a lot of promise for integrations with AR in the future but the latency for visual search is still pretty high?

Chris Arkenberg: In the near term, not a lot. In the medium term, there’s a larger trend towards virtual agents that you can program or teach to keep watch over things for you as an effort to scale down the data overload. So search is something that’s going to become more personalized and more active. There’s a movement to make it so people can essentially deputize these agents to be always searching for them; to be out there looking for the things that they have told these agents are important to them.

So active search for AR I think presents some challenges, obviously because you need to do text input, typically, or voice input. Voice input, I think, is much more achievable than text input for AR. But I can certainly imagine an AR layer that is being serviced by these agents that we have roaming around the web for us reconciling their visual view of the world with our personalizations. AR apps are contextually aware so it knows that if you’re downtown, it’s not going to be giving you a ton of information about Software as a Service infrastructure, or what have you. But that, instead, it’s going to be handing you little tidbits about a particular clothing brand you’ve opted in to follow and information about music venues & schedules, for example. Or perhaps you’ll be on the lookout for other users that have opted in to publicly tag themselves as a member of this or that affinity.

I keep coming back to this idea of AR as really just a simple visualization layer that all of these other technologies can potentially feed into. So in that sense, search becomes a passive thing that AR is just simply presenting to you in a heads-up, hands-free, or potentially hands-free environment.

Tish Shute: Yes, the big challenge is the stepping stones to that point! Small steps that keep interest going into developing the underlying technology (and not just in research labs!) that will bring us that interface.  We have seen some movement already with Qualcomm.

Chris Arkenberg: And there are bandwidth issues as well, as we can see with the Google Goggles, which is a great idea of visual search. But you have to take a picture and send it to the cloud and wait for your results. It’s not a real-time dynamic interrogation of the world.

Tish Shute: Yes we are really only at the very beginning of AR being ready for prime time.. it would be interesting to ask AR developers how many of them use AR on a daily basis.

Chris Arkenberg: I think a lot of us, we’re just informed by the sci-fi myths and fascinated with the potential now that’s it’s starting to become real. But I think we all kinda get that it’s still extraordinarily young. I mean the web is extraordinarily young. And AR is itself far younger in a lot of ways in its implementations.

Everybody has a lot of excitement about all of the great potentials that are being unleashed by this great wave of the Internet and the web and ubiquitous mobile computing. So that’s why, you know, you look at that map and we talk about AR and you can’t talk about any of the stuff without talking about all of it, in a lot of ways, particularly with something like AR where it’s so ultimately agnostic and could be completely pervasive across all of these layers.

So my fascination is with the future, and I measure our progress towards it by the young nascent offerings from the platform players and the developers. And yeah, a lot of it is…it’s akin to getting that first triangle on the screen in 3D. You know, when the renderer finally works and you get a triangle on the screen, and you go, “Oh my God, it renders.” And then you can start to really build polygons and build objects, and start doing boolian operations, and get light and rendering in there, and textures, and on, and on, and on.
So I’m fascinated by the Layars and the Metaio’s…
[laughter]

Tish Shute: Yes and hats off to all the players in the emerging industry, Layar, Metaio, Ogmento, Total Immsersion, and all the others who are finding clever ways to bring fun aspects of AR into the mainstream, and fuel interest to take the technology to the next level.

Chris Arkenberg: Absolutely. And the hype cycle is very valuable. It has really helped launch the AR industry. It’s brought a lot of eyes, and it’s brought a lot of money into the industry. And it’s forcing people like us to have these conversations to understand how to refine its growth and really focus on the potential in all these different venues, whether it’s trying to save lives, or better understand your city, or have really compelling entertainment experiences.

Everybody’s excited, and everybody’s sharing, and everybody’s trying to move it forward in a way that’s the most productive.

Above is an image above  from Total Immersion’s augmented reality experience developed for the “Networked City” exhibition in South Korea, – “a fun scenario created for a u-City’s infrastructure and city management service”

“To the naked eye, the exhibit looks like a bare bones model of a city. But when visitors put on the special AR goggles a whole new world unfolds – as graphics overlaid on the city model.” (Games Alfresco)

“The Networked City,” is a large scale augmented virtuality of a scenario for a networked city. But my guess, reading the     Korea IT Times, is the plan is to move from an augmented virtuality to an augmented reality as Incheon Free Economic Zone  (IFEZ) realizes its vision to become a leading u-City – where reality is turned “inside out” (see Inside Out: Interaction Design for Augmented Reality ).  If you are not familiar with South Korea’s u-Cities, check out this post for a short primer (and note Google Trends search on Augmented Reality shows  South Korea leaving everyone else in the dust).

Ubiquitous computing and augmented reality are like adenine and thymine – a DNA base pair.

A sky view of Incheon Free Economic Zone (from Korean IT Times). For more on the IFEZ vision to become a leading u-City see here.

Korea IT Times writes about the u-city concept:

“Korea began using the term u-City after accepting the concept of ubiquitous computing, a post-desktop model of human-computer interaction created by Mark Weiser, the chief technologist of the Xerox Palo Alto Research Center in California, in 1998. There have been a lot of research in this field since 2002. As a result, many local governments in Korea have applied this concept to various development projects since 2005 based on a practical approach to it.”

The back story to many of my recent posts, including this one, is an understanding of a relationship between ubiquitous computing and augmented reality that emerged, for me, in a February conversation with Adam Greenfield, Towards a Newer Urbanism: Talking Cities, Networks, and Publics with Adam Greenfield.  In cased you missed it, here is the link again because I think it holds up very well considering the rapid developments of recent months.  Also, importantly for this post, it includes a discussion of  moving on from Weiserian visions.

Adam Greenfield’s Speedbird is one of my key sources for understanding “networked urbanism,” and the list he makes of the elements of networked urbanism here (also see the comments) – is my mantra for thinking about the DNA base pair relationship of augmented reality and ubiquitous computing.

Adam Greenfield’s, “summary of what those of us who are thinking, writing and speaking about networked urbanism seem to be seeing” is:

1. From latent to explicit; 2. From browse to search; 3. From held to shared; 4. From expiring to persistent; 5. From deferred to real-time; 6. From passive to interactive; 7. From component to resource; 8. From constant to variable; 9. From wayfinding to wayshowing; 10. From object to service; 11. From vehicle to mobility; 12. From community to social network; 13. From ownership to use; 14. From consumer to constituent.

Augmented Reality – Making Visible the Invisible

The screenshot above is one of  the coolest “making visible the invisible” AR applications. It was developed at Columbia University Graphics and User Interface Lab where Steven Feiner is Director (see the deep list of projects from the lab here).  This app “shows carbon monoxide levels projected over New York City. The height of each ball reflects concentrations of the pollutant.” Credit: Sean White and Steven Feiner  (via Technology Review).

The recent emergence of “magic lens” augmented reality apps for our smart phones – Wikitude, Layar, Acrossair, Sekai Camera, and many others now, have given us a new window into our cities. But we are yet to realize the full potential of the AR/ubicomp base pair that can “make visible the invisible” and give us new opportunities to relate to the invisible data ecosystems of our cities, not merely as a spectator experience,  but as an interactive, in context, real time opportunity to reimagine social relations.

Mark Shepard says in his curatorial statement for, “Toward the Sentient City:” (Much more soon on this very significant exhibit which runs from Sept. 17th to Nov. 7th, 2009.)

“In place of natural weather systems, however, today we find the dataclouds of 21st century urban space increasingly shaping our experience of this city and the choices we make there.”

Augmented Reality, as Joe Lamantia points out, is becoming the great “ambassador of ubiqitous computing.” AR is. “…mak[ing] it possible to experience the new world of ubiquitous computing by reifying the digital layer that permeates our inside-out world,” and we are only just glimpsing the razor thin end of the wedge in this regard.

I am still working on my Gov 2.0 Summit write up  and, amongst other things, I will talk about how an emerging new social contract around open data, here in the US,  will put augmented reality  apps center stage  – “doing stuff that matters.” At Gov 2.0 Expo Showcase Tim O’Reilly tweeted:

Tim O’Reilly Really enjoyed @capttaco (Digital Arch Design) @ #gov20e: “Augmented Reality could be a new public infrastructure” http://bit.ly/18iCx

Also see Tim O’Reilly and Jennifer Pahlka on Forbes.com discuss the The “Web Squared” Era -  “the Web Squared era is an era of augmented reality arriving (like the sensor revolution) stealthily, in more pedestrian clothes than we expected.… …our world will have “information shadows.” Augmented reality amounts to information shadows made visible.”

Again there is back story to how I came to think about Information Shadows in relation to augmented reality.  So in case your missed it the first time, here is the link to a conversation that began in a hallway meeting between Tim O’Reilly, Mike Kuniavsky, ThingM, Usman Haque, Pachube, and Gavin Starks, AMEE, at ETech earlier this year,  “Dematerializing the World, Shadows, Subscriptions and Things as Services: Talking With Mike Kuniavsky at ETech 2009.”

Slide from Augmented City Lab @ Picnic ’09

So What’s Next for Mobile Augmented Reality?

These videos from Daniel Wagner’s team from Graz University of Technology showing Realtime Panorama Mapping and Tracking on Mobile Phones and Creating an Indoor Panorama in Realtime, as Rouli from Games Alfresco points out,  indicate that there is a lot in store for us at ISMAR09.

We may not be so impressed by directory style/”post it” AR anymore, as these applications have become common place so quickly!  But while these early mobile AR apps may be disappointing in relation to some futurist visions of AR – merely AR/ubicomp appetizers,  there are still good implementations of this model coming out (see new comers to the app store Bionic Eye and RobotVision). And Layar, always on the ball, has upped the ante for the new cohort of AR Browsers with Layar 3D.

But as Bruce Sterling notes here:

*In AR, everybody wants to be the platform and the browser, and nobody wants to be the boring old geolocative database. Look how Tim [creator of RobotVision] here, who is like one guy working on his weekends, can boldly fold-in the multi-billion dollar, multi-million user empires of Apple iPhone, Microsoft Bing, Flickr, and Twitter, all under his right thumb

(watch video here)

But if  you looking for something more from AR, you probably won’t have to wait too long.  The two pioneering companies in AR, Total Immersion – founded in 1999, and Metaio – founded in 2003 are both coming out with “mobile augmented reality platforms” in a matter of weeks (see press releases here and here).  And both companies, it seems, will deploy much more sophisticated AR rendering and tracking than we have seen to date.

I approached Bruno Uzzan, founder and CEO of Total Immersion, for an interview as part of my look at the new industry of augmented reality through the eyes of the founding members of the AR Consortium. These consortium members are some of the first commercial augmented reality companies.

The interview below with Bruno began early this summer and then we both went on vacation and it picks up after the announcement of the partnership between Total Immersion and Int13.

The significance of this announcement is that Total Immersion is now positioned to take the augmented reality experiences they have developed for a number of top brands onto multiple mobile platforms with, “Int13′s very clever embedded solution that allows our [Total Immersion's] solutions to work across many [mobile] platforms,” while Int13 gets to extend their reach.

Total Immersion has a 50 person R&D team and their two main focuses have been, firstly getting:

“Augmented Reality to work with as many platforms as possible – PC, Mac, Mobile, Game Consoles, all those are the platforms that we are targeting. We are currently doing lot of work in the R & D team in cross platform compatibility….”

and, secondly:

“Our R&D guys are working on the real world interacting more with the virtual world.  And I have started seeing some results which are pretty much crazy and this will be ready for next year.”

Pandora’s Box – Shared Augmented Realities

Spes or “Hope”; engraving by Sebald Beham, German c1540 (see Wikipedia article on Pandora’s Box)

There are many weaknesses to the mobile smart phone AR experiences we have now, and the lack of near field object recognition (to date), and difficulties with accurate positioning aren’t the only ones.  Note re solving positioning problems in mobile AR, we are yet to see AR  leverage public libraries for analyzing scenes like Flickr’s geo tagged photos, see Aaron Straup Copes’s work on “The Shape of Alpha.” And for more on this my post here.

But, as Joe Lamantia points out:

“One of the weakest aspects of the existing interaction patterns for augmented reality is their reliance on single-person, socially disconnected user experiences.”

In my view, The Pandora’s Box of Augmented Realities is an open, distributed, multiuser augmented reality framework, fully integrated with the internet and world wide web.

As Yochai Benkler has pointed out many times, and argues again in, Capital, Power, and the Next Step in Decentralization, it is “open, collaborative, distributed practices that have been at the core of what made the Internet.”  We have to try to make sure that open, collaborative, distributed practices are at the core of mobile augmented reality.

Can Google Wave be the basis for an Open, Distributed, Multiuser Augmented Reality Framework?

I have been exploring the idea of using Google Wave protocol as the basis for a distributed, multiuser open augmented reality framework with a small group of AR enthusiasts and developers. And I am happy to say the proposal is beginning to get fleshed out a little.  New collaborators are welcome both for “gear heady” input and use case suggestions (but re the latter, you can’t just say everything you see in Denno Coil..!).

This effort started with Thomas Wrobel’s  proposal for an Open AR Framework prototyped on IRC – see here, and click to enlarge the image above of, “Sky Writer: Basic Concept for an Open Multi-source AR Framework.”

But recently we began looking at the Wave Federation Protocol.  And, if you check out this post, and this post, you may get a glimpse of why Google Wave protocol might be a good basis for an open, distributed, AR Framework.  You will notice, if you study what Google Wave has done with the XMPP protocol, that many of  the elements of networked urbanism that Adam Greenfield describes resonate strongly with what is being attempted in Wave.

But enough said for now!  Regardless of the details of implementation,  Google Wave or an AR protocol built from scratch (phew! the latter does seem like a lot of work) -  an open, distributed, multiuser AR framework integrated with the internet and web would explode the potential of AR, creating new possibilities for data flows, mashups ,and shared augmented realities.

And we are excited by Google Wave because, as Thomas puts it:

“The really great thing wave does ….(aside from being an open standard backed by a major player…hopefully leading to thousands of worldwide servers )….is that it allows anyone to create any number of waves, set precisely who can view or edit them, and for them to be able to be updated quickly and continuously (and even simultaneously!) Better yet, changes will (if necessary) propagate to all the other servers sharing that wave. It does all this right now. From my eyes this does a lot of the work of an AR infrastructure already.

I cant see any other protocol actually doing anything like this at the moment, although correct me if I’m wrong, as alternatives are always welcome :)”

Also, Thomas notes, “even the playback system (that is, the ability to playback the changes made to a wave since its creation) …this could give us automatically some of the ideas Jeremy Hight has mentioned in his visionary work here,  and here on “the geo spatial web, interlinked locations and data, immersive augmentation and open source geo augmentation.”

One of the many reasons why an Open, distributed AR Framework would be so cool is it would open up all kinds of possibilities for GeoAR by providing the over-arching standard protocol for communication of updates necessary for the substandards that will facilitate GeoAR.

Also important to note is the  Wave Federation Protocol allows anyone:

“to run wave servers and become wave providers, for themselves, or as services for their users, and to “federate” waves, that is, to share waves with each other and with Google Wave. – “the federation gateway and a federation proxy and is based on open extension to XMPP core [RFC3920] protocol to allow near real-time communication between two wave servers.” See Reuven Cohen’s blog for more here and here, “HTTP is Dead, Long Live the Real Time Cloud.”

Still some people have expressed concern that an AR Framework using Google Wave protocol would give Google disproportionate influence.   Will Google-specific functionality be an issue?  How much stuff is Google specific just because no one else is using it (yet)? And how much is Google specific because it holds no value to anyone else but Google? These are some of the questions that have come up.

You are going to see a variety of suggestions for standards and specs for open AR coming out out in the next few months which as, Robert Rice of the AR Consortium points out is: “a good thing, we need that competition early on to settle down on best case.” Recently,  Mobilizy have offered up an ARML (“an augmented reality mark-up language specification based on the OpenGIS® KML Encoding Standard (OGC KML) with extensions”) for consideration see here.

So it is, perhaps, also important to note, that an Open AR Framework should be neutral/transparent to techniques of  “reality recognition,”  and methodologies of registration/tracking, allowing various ones to work on the system as new techniques evolve, and to support as many evolving standards as possible.

Augmented Reality developers, like Total Immersion and others with powerful rendering/tracking AR software, should be able use an Open AR Framework to exchange the data which their tracking will use. And the tracking/rendering problems they and other researchers have solved are much harder than figuring out data exchange on on a standard infrastructure or protocol!

So I pricked up my ears when I heard Bruno Uzzan, CEO of Total Immersion -  the first and currently the largest augmented reality company, with a 50 person R&D team in France and offices in LA, where Bruno himself is now based, say: “Total Immersion is  only months away from launching shared mobile augmented reality experiences using near field object recognition/tracking across multiple platforms” (for more details read my conversation with Bruno Uzzan below).

I was happy when I asked Bruno about the possibilities for developing an open, distributed, multiuser augmented reality framework fully integrated with the internet and world wide web (possibly using Google Wave protocols), and he replied:

“I think this is feasible. I think that’s doable, that’s just  in my opinion. I mean some people might have another kind of opinion but I think that that’s definitely doable.”

Total Immersion – working with the “symbiosis between augmented reality and brands”

Total Immersion has created many of the best known and most ambitious augmented reality experiences for major brands to date, including Mattel’s new AR toys to be released in conjunction with the James Cameron film Avatar, and AR baseball cards for Topps, video here (or click screenshot above), and the UK’s first augmented reality books.

Bruno founded Total Immersion 10 years ago when he was just 27. And the kind of conviction it took to survive as an augmented reality business in the decade before augmented reality captured the world’s attention is remarkable.

AR’s first steps out into the world after 17 years as predominantly a lab science maybe “wobbly” (what new technology isn’t), and sometimes gloriously kitsch – check out this riotus video of the 3D Interactive Live Show Total Immersion produced in Korea (also see the Total Immersion Augmented Reality Blog for more on the TI’s turn key  Interactive 3D Live Show Solution).

As Lamantia points out here, ” projecting mixed realities into public, common, or social spaces makes them social by default.”

However, the potential for shared location based augmented reality experiences is as yet untapped.  So I see the entry of the most experienced commercial augmented reality company into mobile as pretty interesting.   While  smart phone AR still has significant limitations, and it certainly does differ from some of the futurist dreams of AR (see Mok Oh’s post here on his disappointment in this regard), it is significant that Total Immersion is committing to becoming a leader in mobile AR.

Our smart phones, the powerful networked sensor devices that so many people carry in their pockets, have proved themselves a “good enough for now”  mediating device for early manifestations of the ubiquitous computing and augmented reality base pair.  And now AR and ubicomp is mixed in the  rich, messy soup of everyday life, commerce, business, marketing, art, entertainment, and government, we should get ready to see these technologies grow up fast, and unfold in some surprising ways that lab science didn’t necessarily predict.

And, perhaps, the new dialogue between scientists and entrepreneurs may spur both communities to outdo themselves.

Particularly, as Joe Ludwig notes: “It seems to me that the biggest disconnect between the academics and the entrepreneurs is that they disagree on how far we are from the finish line.”

See the comments’s on Ori Inbar’s post, Augmented Reality Entrepreneurship: Natural Evolution or Intelligent Design?, for  a courteous but spirited discussion on the potential benefits and frictions of the newly expanded AR community of  researchers and  entrepreneurs.

As Blair MacIntyre (see my long conversation with Blair here) notes:

“not all academics and researchers are only interested in the traditional models of impact. Case in point: I wouldn’t be building unpublishable games, nor investing so much time talking to the press, entrepreneurs and VCs if I did not believe strongly in the value of the impact I am having by doing that — and I know others with the same attitude.”

In this vein, check out the Marble Game (video here) developed by Steve Feiner and his team at Columbia U. It’s enabled by Goblin XNA, an open source AR framework built on top of Microsoft’s XNA, which powers XBox live games, Zune games, and some Windows games. For more about Goblin XNA and AR from Columbia U see here.  (Hat tip to Brian Jepson for this link)

While we are still waiting for the kind of sexy AR specs – nothing totally game changing in Gigantico’s AR eyewear rounup (maybe note this Apple patent), that might get wide adoption. But at least researchers are not afraid to explore the possibilities of AR Goggles.

But how far are we now, with or without sexy goggles,  from a fuller expression of the base pair DNA of ubiquitous computing and augmented reality?

We may have a LAN of things before we have an Internet of Things

The picture above is a workshop I attended at Conflux last weekend – Fish ‘n microChips, with Natalie Jeremijenko. We are at the site of the Amphibious Architecture project (a commissioned work for Toward the Sentient City) and “a collaborative project with xClinic, The Living and other intelligent creatures.”

We are probably as far off some grand futurist visions of ubiquitious computing as we are some of the futurist visions of augmented reality. But as it turns out that may not be a bad thing! Recently, @mikekuniavsky noted in a tweet:

“Another argument for the LAN of Things before the Internet of Things: http://tinyurl.com/lgp9uq”

Bert Moore, in the article Mike linked to points out, the grand vision of an “internet of things” with everything connected to everything  can “distract people from thinking about the benefits of RFID in smaller, more easily implemented and cost-justified applications.”  The same argument I think applies to sensor networks and augmented reality.

In New York City, a series of commissioned works for the Architectural League of New York’s exhibit, “Toward the Sentient City” are giving us the opportunity to dip our toes into the ocean of a “networked urbanism.”   For only a small budget, two of the five commissioned works, Amphibeous Architecture and Natural Fuse demonstrate how sensor networks can allow us to explore new kinds of communities – connecting people to environments in interesting ways to create new forms of social agency.

“Amphibeous Architecture” -  from The Living Architecture Lab at Columbia University Graduate School of Architecture, Planning and Preservation (Directors David Benjamin and Soo-in Yang) and Natalie Jeremijenko, Environmental Health Clinic at New York University, uses a skillfully built (electronics and water are notoriously hard to mix) array of partially submerged sensors to pierce the blinding, reflective surfaces of the rivers  surrounding Manhattan and to create a new two way relationship with the ecosystem below – the water, our neighbors the fish and even a beaver that lives in the water surrounding Manhattan.

Image from Toward the Sentient City

In a similar spirit, “Natural Fuse” – Usman Haque, creative director, Nitipak ‘Dot’ Samsen, designer, Ai Hasegawa, designer, Cesar Harada, designer, Barbara Jasinowicz, producer, creates a network of people and electronically assisted plants to explore what it takes to work together on energy consumption and to experience the consequences of “selfish” and “unselfish” behavior interactively before it is too late to modify our actions.

The “Greedy Switch“ from Natural Fuse on the left. On the right “The System” – click to enlarge.

Much more to come in another post on these works, and “Toward the Sentient City.”  Also an update on how Pachube – an important part of both these projects and a very important contribution to ubiquitous computing because it creates the opportunity to connect environments and create mashups from diverse sensor data feeds – has matured since my interview with Pachube founder, Usman Haque, “Pachube, Patching the Planet,” in January this year.

In the picture above Natalie Jeremijenko, and Jonathan Laventhol give the Amphibious Architecture sensor array a last look over, as it will soon be lowered into the East River. Jonathan is on a busman’s holiday to help out at the pre launch of Amphibious Architecture, nr Manhattan Bridge, NYC.

I was very happy to get  a chance to talk to Jonathan Laventhol - more on our conversation in another post. Jonathan Laventhol is CTO of Imagination – one of the world’s leading design, events, and branding agencies. We talked about the importance of  Pachube, which Jonathan called the “The Facebook of Data,”  and  how the symbiosis between brands and augmented reality, and healthcare applications, would  be key to augmented reality emerging into the mainstream.

Natalie Jeremijenko’s workshop at Conflux on the social negotiation of technology and how “everyware” can give us the chance to experience new forms of agency and connection was a totally inspiring.  And I will cover this too in another post.  I have so much awesome stuff  to write about at the moment!

None of the projects in, “Toward the Sentient City,” included a mobile augmented reality, or “magic lens” component, but they all pointed to why “enchanted windows into our newly inside-out reality” are going to be so important. And why the DNA base pair of ubicomp and augmented reality can really do stuff that matters.

Shangri- La – “Transfigured City”

Screenshot from Shangri-La episode Transfigured City

In my AR Consortium founder member interview series, I have found that, understandably, the visionary founders of these first augmented reality companies are a little reticent about sharing their full vision.  They are basically on stealth mode in this regard.  So as you will not, from my interview with Total Immersion founder and CEO, Bruno Uzzan, get a fully drawn scenario of his vision for a next generation of shared augmented reality experiences, here’s a really interesting anime episode from the anime Shangri La called, Transfigured City, to mull over instead.

As you can tell from this rather long and circuitous intro to my my conversation with Bruno Uzzan, I  have been investigating shared augmented realities pretty intensively recently. And Mike Kuniavsky pointed me to Shangri-La, and Transfigured City, in a conversation with Mark Shepard, after Mark’s presentation at Conflux, Sentient City Survival Kit.

Mike Kuniavsky with Tod E. Kurt is founder of ThingM, a ubiquitous computing device studio. Also Mike Kuniavsky researches, designs and writes about people’s experiences at the intersection of technology and everyday life – see Mikes blog Orange Cone.  And I interviewed Mike at Etech- see here.

In Transfigured City, the “Metal Age” group has to figure out how to share and communicate in a city transfigured by augmented realities/virtualities, where no-one sees the same place in the same way.  Only one character can figure out from her previous experience of the city the relationship between the transfigured city and how it used to be.

The conversation I had with Mike Kuniavsky on The Transfigured City continued at a picnic in Washington Square Park the next day with Elizabeth Goodman, who I met at Etech when she gave a brilliant presentation, Designing for Urban Green Space.  We covered so many areas at the picnic related to ubiquitous computing and augmented realities that this conversation probably deserves a post of its own (my writing to do list is growing longer!).

The Plot Synopsis for Shangri La:

“In the mid-21st century, the international committee decided to forcefully reduce CO2 emission levels to mitigate the global warming crisis. As a result, the economic market was transferred mainly into the trade of carbon. A great earthquake destroys much of Japan, yet the carbon tax placed on the country is not lifted, so Tokyo is turned into the world’s largest “jungle-polis” that absorbs carbon dioxide. Project Atlas is commenced to plan the rebuilding of Tokyo and oversee the government organization, which the Metal Age group opposes due to its oppressive nature. However, Atlas is only built with enough room for 3,500,000 people and most people are not allowed to migrate into the city. The disparity between the elite within Atlas and the refugees living in the jungles outside of its walls set up the background of the story.”

Talking With Bruno Uzzan

Tish Shute: We won’t have fully opened the Pandora’s Box of Augmented Realities until we have ubiquitous, shared augmented realities, will we?

Bruno Uzzan: Yes. The most important for augmented reality is the experience we want to share. Now we are working on the cell phone, we can potentially do some marketing components that we already have developed now on cell phone. Done. It’s working.

But the most interesting part of it is how these new components [cell phone AR] will be used for marketing campaigns by brands. And we are also pretty much well positioned to transform some of the AR that we currently have working on Mac and PC and to transform these to applications working on mobile devices.

Tish Shute: We haven’t really experienced yet what it means to actually share mobile AR experiences?

Bruno Uzzan: It’s hard — we did a Facebook app. It’s a first try, it has a way to go.  But to go more and more into social, is the way forward for us – to share and expand AR experiences. But yes, I mean what you’re seeing is how two people on two different applications can share that same expanse.  For sure we are going in that direction. We are currently working on those kind of solutions. How people can share and experience together at the same time. That’s how we start creating excitement in augmented reality, and it’s coming up.

It’s a new market and there’s so much more in store for augmented reality. You know, some people are telling me, don’t you believe that augmented reality is a gimmick? It will be a trend for a few weeks or a few months and then gone? I say, you’re kidding me. This is only the beginning. I mean I can assure you that the applications that are on the market today are one percent of what we will have five years from now.

Tish Shute: I agree.

Bruno Uzzan: And I’m sure that augmented reality will be a part of a lot of components that we are currently using today – GPS, web browser, glasses, I mean there are so many applications that will come up shortly. This is only the beginning. I’m completely convinced that augmented reality will be in three years from now what virtual reality is today, which is a billion dollar market.  I know that it’s not just a gimmick of a few weeks or a few months, because so many brands are jumping into it, spending money, exploring solutions.  I know that it’s not just short term -what they are willing to do and we are willing to do, but also middle and long term. And that’s what makes this adventure pretty much unique and what makes creating a cutting edge technology, very, very much exciting for us.

Tish Shute: First could you explain more to me about your partnership with Int13. I am not sure I understand what is in the arrangement from Total Immersion’s POV. I mean what happens re your own mobile software development? Haven’t you only been licensed the Int13 SDK for a limited period of time and have limited access to all it’s power? Stephane from Int13 said to Ori on Games Alfresco, here, “we have licensed the SDK4 for two years,” and then Ori asks, “but you have basically kept the power to yourselves, right?” So if they are the only ones that can enhance it and develop the software, where will  TI be in two years in mobile if you haven’t really had the chance to develop your own software .

Bruno Uzzan: Actually it’s a real win-win situation. Int13 is a very small company and they have so many requests they can’t possibly fulfill them all. So  this is a way for both of us to be, as quickly as possible, the first mobile provider for all the requests we have. Also they give us exclusivity so nobody else can use INT13 SDK for such applications.  I think that it is a good partnership,

And concerning our own mobile application… First of all we have currently some mobile applications working. But with Int13 we have a mobile solution that can work on many different devices. That’s a fact and that’s working. And, believe me you will hear from us a lot more about this soon. We are fully independent on our mobile development. The reason we closed the partnership with Int 13 is  to be able to deploy mobile in a broad way.

I mean you know that the difficulty with AR mobile is that each separate device needs some customization. Working on the iPhone is different from working on the Nokia, different from working on the Palm; it’s different from working on the Samsung. Each of them have their own operating system inside and so we were interested in Int13′s very clever embedded solution that allows our solutions to work across many platforms.

The reason we are working with Int13 is that we are able to work on so many mobile devices, thanks to Int13. And in the mobile AR race that we are currently in, the next two years will be extremely important to us…

Tish Shute: OK, that definitely clarifies it a lot. So Int13 has done an embedded solution to allow TI developed AR solutions to work easily across many devices?

Bruno Uzzan: Yes  they have kind of an embedded solution, a way to address extremely quickly new cell phone… But, currently on our side, we are in discussions with a mobile company… and that only refers to some very specific mobile devices.  And what they have is also a way to embed deeper our technology into mobile, so that we can have quicker… applications that work on a large number of cell phones.

Tish Shute: So, basically it means you don’t have to go through some complicated negotiations with each of the cell phone companies, is what you are saying?

Bruno Uzzan: Not only negotiations, but also hard development. You know? Working on the Windows mobile is completely different from working on the Palm OS. You know, that’s different! Its a big work, to have a mobile application working on many other devices. So, INt13,  provides us a way for us to save some time and some development cost too.

Tish Shute: And Int13 doesn’t have powerful AR development tools like D’fusion right?

Bruno Uzzan: Right! That’s right. That’s why we say it’s a true win-win solution. They can benefit from our work too. And we can benefit from their work, in order to deploy quicker and faster mobile solutions.

Tish Shute: Now, the second thing is… there is a lot of debate and disagreement about how far mobile augmented reality is from delivering something more that the “post it” approach that has been much publicized in recent months, via all the AR browser apps.

But from my understanding from the conversation we had earlier this summer (see below), Total Immersion is targeting a much higher level of mobile augmented reality than we’ve seen to date?

Bruno: Yes the browser apps we have seen are a kind of augmented reality, but not exactly the way we see it. Let me explain you why. With this kind of application it’s true that you can overlay 3D-information and video. That’s a fact. So, in a sense, that’s augmented reality. But the way that they are working on the position of the 3D on that video is that they are using compass and GPS-information.. so it means that this AR solution will work only on some building and some physical objects that are FIXED. In a fixed and known position.

So you want to go to a theater?

The theater is here, for sure it will not move, so you know the position of the theater, and that’s a fact that you can superimpose an object on the theater. That’s what can be done currently. What we are achieving and what we are doing on mobile is more than that. We want to be able to port our solution with trading cards, with brands, into a smart phone.

I’m assuming that you want a can, a drink can, to be able to trigger an experience. The only way you can do it is to be able to understand what the can, it is. And the current solutions that are out there can’t do that, it’s impossible.

Tish Shute: Right, yes. There’s no near-field object at all in these early browser apps.

Bruno Uzzan: And the solution we have is that we can recognize a can and then — in a very, very precise way and that activates geo-location, so we can superimpose 3D. I mean in that case, it opens up all the applications that we currently have, so they could work on mobile.

Tish Shute: So for example, if you’re working with a soft drink company, people can trigger that experience wherever they see that can?

Bruno Uzzan: Correct.

Tish Shute: Yes. Yes, I assumed that was what you’re doing

Bruno Uzzan: We believe — and maybe that’s not the case, but we believe that our marker-less tracking technology is pretty much unique on the mobile devices.

I haven’t seen yet, from anyone, a full augmented reality mobile solution working.

I really see AR being part of the Web 3.0 next generation. I mean the vision I have is that, you know — today, when you want to have information, you go on a website and then you find your information. AR — and the future is that I think it will be part of the opposite. You want to have information about a product, you just show it to your computer and the information will automatically pop up. I see here a new way to market some key messages, a new way to get information is that some physical product by themselves could be a way to get information, and you don’t have to search anymore for them, it’s coming out to you.

AR is definitely for me, one of these components. Another thing that AR is a solution, another thing that AR itself will create these kind of results in how information is being displayed. But I’m seeing  here a way that could be part of a new way to have access to information. And that’s part of the vision I have. Whatever, if it is through mobile phone or web or PC, Mac, whatever, I really believe that now this kind of new generation of receiving information will come shortly and could be a kind of a new — could be part of the new 3.0 generation of the web.

Tish Shute: My friend Gene Becker did an interesting post recently on some of the current limitations of mobile AR where he pointed out the problem of:

“Simplistic, non-standard data formats – POIs, the geo-annotated data that many of these apps display, are mostly very simple one-dimensional points of lat/long coordinates, plus a few bytes of metadata. Despite their simplicity there has been no real standardization of POI formats; so far, data providers and AR app developers are only giving lip service to open interoperability. Furthermore, they are not looking ahead to future capabilities that will require more sophisticated data representations. At the same time, there is a large community of GIS, mapping and Geoweb experts who have defined open formats such as GeoRSS, GeoJSON and KML that may be suitable for mobile AR use and standardization.”

Bruno Uzzan: That’s interesting. I mean — I know exactly what his is referring to. He is mainly referring to a localization and how you can have a quick, accurate localization.  If you look at current solutions, and you look at this 3-D superimposing on the video, the 3-D is shaking a lot. I don’t know if you see that in some of these early efforts.

It’s hard to use because the 3-D, you know, is  part of the magic of augmented reality, that is when the 3-D is being inserted in a very easy way and smooth way in your solution. Here, when you see this overlay, 2-D or 3-D overlaid on the video, it’s shaking a lot. One reason for this is that the GPS compass is not accurate enough to coordinate the perfect location of the user. And here, what Gene says is interesting. I think we are addressing this localization issue in a pretty smart way.

But to be frank with you, I don’t believe mobile augmented reality in the extremely short term — I’m talking about three weeks, one, two months is mature enough for good AR applications.  It will be shortly.  But for now it is more proof of concept than a true and easy application to use.

But we are starting to see a lot of new application coming out, but I really believe that marketing and entertainment are the two key markets for AR right now.

I’ve been working ten years in augmented reality. And, eight years ago, when I was talking about augmented reality, I was E.T., you know? Nobody understood what I said, and I thought it was crazy. And now, today, yes it’s completely different.

Tish Shute: The Pandora’s Box of Augmented Realities, in my view, is an open, universal and standard, distributed, multiuser, augmented reality framework fully integrated with the internet and world wide web. I have been looking into Google Wave protocols as a basis for this would you be interested in this? Do you think it is feasable?

Bruno Uzzan: I think this is feasible. I think that’s doable, that’s just  in my opinion. I mean some people might have another kind of opinion but I think that that’s definitely doable.

Tish Shute: Yes I suppose an open AR Framework involves cooperation and collaboration, it is more about business and politics than technological problems.

Bruno Uzzan: Yes!  Actually the Web is politics. Business is politics.

Tish Shute: I would be interested if anyone in your R&D team would be interested in looking at some of the ideas that are emerging in our little discussion of Google Wave and an Open AR Framework  to offer feedback. it is an interesting time now to input on the Wave Federation Protocol docs because nothing is set it stone right now.

Bruno Uzzan: Just shoot me an email, I’ll try to put you in touch with the right person and, and a team member that can input on this.

Tish Shute: For mobile augmented reality the best thing we’ve got now is the phone, right?

Bruno Uzzan: Right.

Tish Shute: And the only way we can use the phone is by holding it up, right?  Isn’t this a bit of an an obstacle as you introduce better object recognition and tracking?  People are going to have to stop moving to use their phone. What do you feel about that experience? Isn’t AR eyewear and essential part of a tightly registered AR experience?

Bruno Uzzan: We don’t do hardware and we don’t have the current solution for eyewear that would do all we need for a good mobile AR experience, so I guess we don’t have the current answer for that.  But we are beginning to see the next generation of this — of these glasses.

Tish Shute: But you’re happy enough with the mobile experience of augmented reality on smart phones that you’re investing in this next generation of software for this.

Bruno Uzzan: Yes, I know. We know that some application will not work on the iPhone. And yes, whatever you do, you still need to hold the iPhone, so it means that you can’t play with your hands anymore. So we know that partially, some AR solutions  we have on other platforms will lose the magical effectivities on just the iPhone.

But I’m starting to see on the market some glasses that could perhaps be not too expensive — that’s a challenge!  And easy to use — that’s another big challenge. And, that could fit on anybody’s faces and head — there’s another big challenge. So yes, I’m starting to see that, but so far AR glasses are only applicable for some very, very specific application, like design or theme park or, you know, some specific location where it makes sense to move forward with glasses.

I don’t believe that kids will use glasses for — in our toys and for games in the next months or maybe othe next one or two years. But maybe something will come out shortly and that could be a big breakthrough, and enable us to think another way. But  from what we have seen so far and from what we know in this hardware market, I don’t believe that currently there is a workable solution.

Note: The following section of the interview took place earlier in the Summer.

Tish Shute: You are the first commercial AR company  – you started in 1999 right?

Bruno Uzzan: Yes you are right. We started the extremely early in this augmented reality market. We were the first company worldwide to start doing augmented reality and to start promoting augmented reality. So it’s true, we are pretty old players although the market has been getting bigger and bigger for the last year and a half. So for a long time we were only in the market, and the market was not really there.

But for the past 8 months, the company has been growing really fast.

Tish Shute: Yes I’m sure. Congratulations for hanging in there long enough to get the pay off!

Bruno Uzzan: You know, my background is Financial. So I have been driving the company for many years in a very cash efficient way. So we have been waiting for the markets to reach maturity before starting make some investments. So that’s the reason we are still here, and that’s the reason I think we managed pretty smartly the cash that we raised for the company.

Tish Shute: Yes there is a saying that when a market takes off you can tell a pioneers because they are the ones with the arrows in their backs. But I am glad you are dodging the arrows!

Bruno Uzzan: You know, I’ve always driven the company with revenue. And because revenue was not there at the beginning I was extremely cautious about the cash. So now that the company is getting some revenue, for sure we are making more and more investments, and taking advantage of our situation as a worldwide leader of augmented reality.

This situation is not easy as it appears today but it’s now getting better, as you can see, AR, Augmented Reality, has very good momentum and we are benefiting a lot from all this momentum for augmented reality right now.

Tish Shute: You’ve been very involved in researching developing augmented reality tools. Are you still as active in the research area, or are you too busy keeping up with work for hire now, to be working on research and building new technology for Augmented Reality?

Bruno Uzzan: Both. First of all, we are part of lot of projects either directly with clients like Mattel or with some partners that are using our technology to promote and develop other AR projects. From what we he have seen, many, many, many, projects augmented projects have been done currently with our solutions.

To continue with your previous question. So we are being perceived as this leader in that space, and we  have some pretty heavy demand for our services. But we are coming up with new technology, of course, still connected to Augmented Reality.  But, our R & D is working in two different directions, which of course also bind together.

The first one is platform developments. So we want Augmented Reality to work with as many platforms as possible – PC, Mac, Mobile, Game Consoles, all those are the platforms that we are targeting. We are currently doing lot of work in the R & D team in cross platform compatibility.

Tish Shute: Robert Rice said recently, “markers and webcams equal Photoshop page curls…”

Bruno Uzzan: Yes. There are so many concerns with markers. The quality is extremely bad. As soon as you hide a part of the marker, a slight part of the marker, you’re dead. You can’t track any more of the object. So compared to our solution where I want to say play with cards or where you are going to play with a Mattel toy, even if you hide a part of the toy, it’s still working.

Tish Shute: But you haven’t offered the public an SDK to your engine right? Basically the way people get access to your tools is working in a partnership with Total Immersion right?

Bruno Uzzan: Correct.

Tish Shute: Do you think in the future you might open your SDK? Are you considering that?

Bruno Uzzan: Yes, it would be interesting.

Tish Shute: So that is something we can see coming soon?

Bruno Uzzan: Maybe, because it’s true that Total Immersion is starting to be mature enough for these kind of tools. The only thing is that we have to respect good timing for that.  It’s a big decision. You know what I mean?  It is a big, big decision. We would then compete with others using our technology.

Tish Shute: Oh I know, it is a big decision when you have so much skin in the game! But it would be nice to have your SDK being THE platform for AR, wouldn’t it?

Bruno Uzzan: It is a really big decision that we can’t just take like that, you know.  There are a lot of friends who told me you have to be extremely careful about timing. This timing is pretty much connected to the maturity of the market. For sure, we see the market being more and more mature. But, there are a lot of low hanging fruits we still want to address

To get the best value possible for all the publicity we have and all the clients we have now.

Tish Shute: Yes, I know. You’ve been in this game so long. Now, there is an interesting question here though about tools and platforms because you know, A.R., augmented reality has already expanded  beyond its kind of original purist definition. And when I talk to people  about augmented reality, there are actually lot of different ideas and priorities of where the tools should go right now. You know, obviously we have these kind of browser-like applications, but these browser like applications are not dealing with recognizing near field objects yet.  What are your priorities for tool development and what are your priorities for AR development in the future? What areas are you going to focus on? Oh dear that is a rambling question!

Bruno Uzzan: [laughter]  So, one of our first priorities is we need to create our software with one development, one installer, one software that can be spread on different platforms. The same application, the same software can be used either on a PC, Mac, phone or console. So that’s a lot of work, because that means that our platform has to address many many different devices and that’s a big priority for us because we received this request from our clients. We want to be able to use one application on many different platforms and devices. So, that’s the first one.

And the second one is to add more and more interactivity between the real and the virtual world. So, we are working on some improvements to add some real components that will interact with virtual, and that also part of our big strategy and direction and these two worlds can more and more be bridged together, linked together so they can interact  one with the other.

Our R&D guys are working on the real world interacting more with the virtual world.  And I have started seeing some results which are pretty much crazy and this will be ready for next year.

There are so many different directions for interaction between the real world and virtual world to develop.  I’m sure ten years from now you’re going to have AR applications everywhere.  Its not just temporary fashion stuff or a gimmick for few months. I mean we are getting there, its getting stronger and stronger and we are getting a good adoption rate from our consumers. They like it, they test it, they play with it and brands wants more, people want more and its getting bigger and bigger.

Tish Shute: Yea and I totally agree, its not a gimmick because the interaction between “virtual” and “real” enhances the magic of both. Another question about you RandD operation. Is your R&D still in France or have you moved totally out to LA.

Bruno Uzzan: We are 50 people in France and I started this LA office two years ago and I moved permanently two years to LA. So I’m now permanently located in the US to take care of the US office, knowing that revenues are really getting bigger and bigger in the US. So it means that we are getting a lot of traction, working with large company and now I’m currently located in the US.

Tish Shute: My sister lives in Paris. Could I visit your R&D lab at some point? I’d love to visit!

Bruno Uzzan: Yeah sure sure sure. I mean if you want to go. You won’t have access to all the research. But if you want to go out and meet all the team please do.

Tish Shute: I’d love to.

Bruno Uzzan: No problem. Shoot me an Email you and I will introduce you to Eric Gehl, COO, he is the COO of the French team. And he can definitely take care of that.

Tish Shute: That would be fun. Thank you!

Recently, AR browser applications have really caught the imagination of the web community, eg., Layar and Wikitude?  Where do you think the most important market for AR is at the moment, entertainment,  green tech, business, education?

Bruno Uzzan: I think that all that you mention will be important. The first one that did grab my attention is entertainment particularly dual marketing, because they always searching for new ways to interact with players or the consumers.  But it’s just the tip of the iceberg, you know, I mean medical applications could be huge using augmented reality. Education, and edutainment is definitely using more and more augmented reality components.  And I will just be submitting with big companies – that are considering using augmentation for education. Museums are very important too. Also augmentation as a kind of free sales tool, you know there are so many applications, design, architecture – so many directions that it’s hard to say today which one will take the lead.

But I do believe that on the short term the ones that are really really moving fast are the entertainment business and the digital marketing business.

Tish Shute: What do you think are the biggest shortcomings with current augmented reality and what are the obstacles that no one has solved yet?

Bruno Uzzan: I think the cell phone is not fully ready for augmented reality – a lot of people are working on that but there are still a lot of constraints to get the augmented reality working on a cell phone and I think that from what I heard a lot of manufacturers and a lot of companies are working from direction that are going to help us a lot to develop some great cell phone applications.

And I think that’s one of the biggest part of the game. All the applications that you see on cell phones so far are just gimmicks – the next big key is how to transform some gimmick cell phone application to a real, industrial, robust application that’s going to work on a cell phone. So I think that’s a big challenge for this year.

Most of what we see now is just matching and overlaying some 2d components in a video. This is not what I call AR.  You’re far away – with this kind of application, you are far away from doing the registration that we need to do – you can’t do it. So here’s the challenge: “how can you get a Topps is an application working on cell phone. That’s the big challenge  How we can make that work!” You can’t today get a real AR Topps application working on cell phone because there’s no cell phone  that’s actually ready. But we are working on it and the first one that can make that work, it’s going to be huge.

When you are working with good AR components you need a lot of CPU and GPU programs. So today new cell phone have started to be more and more ready for augmented reality but you need a really good cell phone to make it work. You can’t choose an old cell phone to make it work because you have some recognition, you have some tracking, you have some rendering, so you can’t choose a Nokia cell phone two years old to make that work. For sure the newest iPhone is the one that can make it work, but that’s it for now. There is a lot of research – from large cell phone companies – to get more CPU and GPU into their cell phone.  But so far we are also waiting for these devices to be released to consumers.

Tish Shute: And the current economic climate has put a damper on MIDs hasn’t it. But who can tell? It depends what price points some new MID came out at right?

Bruno Uzzan: Correct.

Tish Shute: Yes,I agree. But basically what’s interesting, the interesting thing is, the iPhone can deliver so much of what is necessary and even if Apple hasn’t given access to the full power of the iphone to AR developers yet, there is really no going back now – the mobile augmented reality cat is out of the bag!

Bruno Uzzan: You’re right, you’re fully right.

Picture on the left Mirrorshades, picture on the right a Metroid Hud.

“Augmented Reality is like a Philip K Dick novel torn off its paperback rack and blasted out of iPhones,” Bruce Sterling in Beyond the Beyond

“a techno visionary dream come true – those are rare, really rare, you have to be patient,  it’s super cyberpunk”… Bruce Sterling, “At the Dawn of the Augmented Reality Industry.”

The Dawn of the Augmented Reality Industry continues to brighten, and now we have two augmented reality companies, Total Immersion and Ogmento, firmly established in Hollywood – the dream mother of so many of our augmented realities.

Ogmento is the most recent of these two pioneering augmented reality companies to set up shop in LA.  Total Immersion’s CEO Bruno Uzzan moved to LA from France two years ago, although he still has a fifty person RandD team in France.   Total Immersion began 10 years ago in the quiet, lonely, hours before the dawn of an AR industry.  But Total Immersion’s AR toys for Mattel, and augmented reality for Topps baseball cards, fired CNet writer Daniel Terdiman up enough to say, “I have seen the future of toys, and it is augmented reality” (see full post here on CNet).

Recently, I talked with  Ori Inbar, one of the founders of Ogmento and  the premier augmented reality blog Games Alfresco about his new venture in Hollywood. Bruce Sterling, @bruces, had some fun with my invention of brand new augmented reality trade jargon here!  Ori pointed out Ogmento brings two important new facets to the rapidly growing augmented reality field: firstly they are bringing leadership from veterans of the entertainment industry into augmented reality development. Brad Foxhoven and Brian Selzer from Blockade have partnered with Ori on Ogmento.  And, in an another important step forward for a young industry, Ogmento announced they will be acting as publishers for a fast growing cohort of augmented reality application developers and helping AR development teams out there bring their concepts to the market.

So I was very happy also to have the opportunity to talk with Brian Selzer.  Bruce Sterling pointed out in his seminal sermon from the flatlands at the Layar Developer Conference, AR is kind of a “Hollywood scene.” We have seen the web early adopter/developer/blogger community  embrace augmented reality browser experiences in recent weeks in an awesome wave of enthusiasm. Are Hollywood creatives equally smitten? For the answers see the full interview with Brian Selzer below.

Brian Selzer (www.brianselzer.com and twitter – brianse7en ) has an extensive involvement with emerging platforms:

“from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC [user generated content] projects (mashade.com, instafilms.com).”

I began my own career in Hollywood doing motion control photography and creating software that bridged the language of robotics and servo motors with the visions of  film directors. Eventually our little company, NPlus1, moved on to 3D vision systems and image recognition stuff.  So yes, I have been really, really patient waiting for this particular techno visionary dream.  And, while I have been waiting for augmented reality to manifest, I have grown to love the internet.  But now, how awesome, It is OMG finally for mobile AR!

Augmented reality is busting out all over – through our laptops, our phones, on the streets, toys, baseball cards, art installations, sticky light calligraphy and more.

Many of my questions to Brian were directed at at how and when we will see augmented realities with near field object recognition, image recognition and tracking and, of course, the illusive eyewear.  As Bruce Sterling points out we are just at the very, very beginning – the dawn of an industry.  I created the photomontage below on the right to compliment Tonchidot’s illustration suggesting the evolutionary inevitability of holding our phones up (below on the left).  The Evolutionary Reality of AR will not end there.  It is just a step into eyewear, hummingbirds or Nano Air Vehicles, and more…….

The Evolutionary Reality of AR

Cartoon on the left by Tonchidot on the right a collage of a stock photo and the Pentagon’s Robot Humming Birds – “Nano Air Vehicles.”

While we finally we have, an affordable mediating device with the horse power, mindshare and business model to bring AR mainstream with the iphone.  The much anticipated Apple 3.1 Beta SDK to be released in September will not, I am sure, open up the Video API at the levels that augmented realities with near field object recognition and tracking require (I would love to be proved wrong though). But the magic wand to deliver even tightly registered AR graphics/media (that require a lot of CPU and GPU) to a wide audience is in our hands, so full access to may not be far off. And others, of course, can/will/might knock the iphone off its current pedestal.  AR made it’s mobile phone debut on the Android after all.

Like everyone else who loves AR, I wish that Apple would open up faster (and I wish Android would manifest on some rocking hardware). But we will see enough of the iphone Video API open for the next generation of mobile augmented reality games and applications to emerge in the coming months.

One of these will be Ogmento’s. Although Ogmento is in stealth mode, they have released a teaser for their first game, “Put A Spell,” developed by ARBalloon – screenshot below.  Ori did reveal to me in this interview that they are doing image recognition and using the Imagination AR engine.

As Brian notes, Hollywood has had the AR bug for a long time. AR has been everywhere in Science Fiction Movies and video games. Nintendo’s SPD3 head Kensuke Tanabe, “effectively the man in charge of overseeing all the Metroid franchise underneath original co-creator Yoshio Sakamoto,” explains the story of Metroid to Brandon Boyer of Offworld here (an image of a Metroid Hud on the right opening this post) :

“the idea of the different visors you use in the Prime games to interact with the world: the scan visor, for instance, set the game apart from other first person shooters in that the player was using it to proactively collect information from the world, rather than having the story come to them passively, in the form of cut-scenes or narration. “Prime could have adventure elements with the introduction of this visor,” says Tanabe, “That’s how we came up with the genre — first person adventure, instead of shooter.”

But as Brian points out:

“the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear).”

Talking with Brian Selzer

Tish Shute: Bruce Sterling’s sermon at the Layar Developer conference, “At the Dawn of the Augmented Reality Industry,” was absolutely awesome. He spread the future feast/orgy of augmented reality before us  – and described many of the dishes we will tasting both delectable and diabolical.  One of the many things he points out is, AR is kind of a “Hollywood scene.” And, as Ogmento is one of only two augmented reality companies in Hollywood, I am interested to hear how it looks from your neck of the woods. We have seen the web early adopter/developer/blogger community  embrace augmented reality browser in recent weeks in an awesome wave of enthusiam – are Hollywood creatives catching the buzz?

Brian Selzer:  It was a thrill to hear Bruce Sterling mention Ogmento. I devoured all of his Cyberpunk books back in the 80′s, along with writers like Gibson, Rucker, Shirley… To me, sci-fi writers are the visionaries who define and influence our technological paths into the future. They make science and tech sexy enough to want to manifest those experiences in the real world. Clearly Bruce sees the AR industry as being sexy. I love that he called it “a techno-visionary dream come true… and super-cyberpunk.”  And yes, kind of a Hollywood scene.

Hollywood creatives caught the AR bug before they knew what AR was.  Look at science fiction movies and video games to see AR everywhere. Terminator, The Matrix, Minority Report, Iron Man.. the list goes on.  Look at any video game with an integrated heads-up display.  It’s clear Hollywood loves AR.  It’s only been in the past few months though that the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear). So yes, the buzz is here and it’s strong.  With that, we all have to be prepared for the good, the bad and the ugly as AR goes mainstream.

It certainly goes to show how young this industry is when Ogmento and Total Immersion are currently the only AR companies based in Los Angeles. It’s very exciting to be the only company right now demonstrating a natural feature tracking (markerless) iPhone experience in Hollywood. We are in talks to bring some very big brand and properties to the mobile AR space. The goal is to deliver experiences that create added engagement and value to the consumer.

Tish Shute: Also in his landmark sermon Bruce Sterling noted that augmented reality has been around for 17 yrs and now at last we are seeing the dawning of  an augmented reality industry. What inspired you to take up the challenge of launching an augmented reality company in Hollywood?  Oh congrats that Bruce Sterling name checked Ogmento in his list of companies that prove that this really is the dawn of an industry!

Brian Selzer: I’ve always been involved in emerging platforms… from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC projects (mashade.com, instafilms.com). Working with all these great CG game assets, I continued to think about what’s next, and that’s when I started to follow AR very closely and started engaging with those who were pioneering in the space.

I remember swapping instant messages with Robert Rice (@robertrice) right after the 2008 Super Bowl.  We were not chatting about the football game, but rather about some of the commercials that aired during the event as a sign that AR was making its way into the mainstream.  A lot of people became aware of AR for the first time when the GE SmartGrid commercial aired.  There were all these YouTube videos popping up of people blowing on holographic wind turbines.

The commercial that really got me excited though was the Coke Avatar commercial.  In that commercial people in the city were sporadically being portrayed as their digital persona’s, avatars, gaming characters, etc..  For me that spot did a great job showing how many of us already have these ‘alter egos” that live in cyberspace, and how the line between these worlds can sometimes be blurred. I remember watching that commercial and thinking that is exactly the type of experience I’d like to create with mobile AR.  I want to overlap the virtual world into our every-day reality. Why cant I bring my World of Warcraft or Second Life persona with me into the real world?

I am big on the notion of “Games and Goals.” I believe that games have the power to motivate people in a very powerful way. By challenging ourselves while playing a game we can climb mountains.  Augmented Reality is the perfect platform to bring gaming into the real world.  By mixing the virtual world with the physical world, this added layer of perception provides a very powerful experience for something like a role-playing game.

One of my earlier social-gaming projects was a website called Superdudes.  This was a “Be Your Own Superhero” concept that celebrated and motivated kids to create superhero avatar/persona’s online, and we gave members all sorts of games, challenges, and rewards, some of which carried into the real world. The site recognized members for teamwork, creativity, volunteer work and things like that. So the Superdudes were often involved in charity events and benefits to help children.  Everybody called each other by their Superhero names, and the line between fantasy and reality were being blurred.  This project really got me thinking about what happens when you take positive role-playing like this and mix it into the real world.  I started to work on a plan for location-based activist missions for points and rewards, but never got to complete that. So I have some unfinished business here.

I think it would be fantastic to be able to show up to some type of fun event with friends, and everybody could see each others alter ego personas standing before them. When you can turn the world into a playground, and use the power of gaming to make a positive impact on the planet… well, I don’t think there is anything better than that.  These are the types of projects that drive me, and I think AR is the best platform to support these types of social gaming experiences.

Tish: Does Ogmento have any RPGs under development?  I noticed in the Google Wave on RPG someone has been working on doing something with the Dungeons&Dragons API.  I am interested in exploring the web of protocols underlying Wave as a transport mechanism for multi-person, mobile, AR experiences (not requiring downloads), on an open global outdoor AR network. If not Wave, what do you see as the potential infrastrucure and protocols we could harness for an open augmented reality network?

Brian:  Ogmento has a deep background in video games and we interact regularly with most of the major game publishers. As a company we are not so much developing our own RPGs right now, but rather exploring what mobile AR extensions make sense for existing brands.  There are many limitations to location-based gaming, but a global AR network is exactly along the lines we are thinking.  Lots of discussions are taking place on protocols, platforms, API’s, and there are numerous ways to approach this.  We need to be able to use what’s available now and continue to refine and customize for AR’s specific needs and issues as we progress.

In general though, Ogmento is focused on what types of experiences can be had today and over the next couple of years. I still think we are several years out from a truly open augmented reality network.  We are certainly looking at launching our own “Ogmented Network” which would support some fun treasure hunt type experiences, or add an entertainment layer on top of traditional outdoor marketing campaigns.

Tish: I don’t know whether you have read Thomas Wrobel’s ideas for an open augmented reality network that I just published here on Ugotrade.  The principals he talks about are very important for augmented reality to become a major part of our lives – .  Considering the difficulty open networks can pose for emerging business models how can we fund the development of an open framework for augmented reality?

“a future AR Network, I mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.
Where people can just point their phone, webcam, or pair of AR glasses anywhere were a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.”

Brian: I think funding for these types of projects will definitely come from Venture Capital groups in the near future.  It’s early in AR, but the VC’s are watching and deciding which horses to bet on.  Until that time, it’s about service work, and developing AR experiences for others with what is possible today. That work will help fund internal development of original AR products, and platform development.

Tish: How did you get started with Ogmento?

Brian: My first conversation with Ori was actually about my interest in Location Based RPG concepts.   We had a long conversation about the possibilities with AR, and it was clear that we shared similar interests, but were coming from different complimentary backgrounds. The idea of collaboration was exciting, so we just kept talking until the timing felt right. Now, with Ogmento we bring a unique blend of AR development experience with a deep backgrounds in AR technology, animation, video games, entertainment, social media, etc.   I think this is a powerful mix that will allow us to do some great things.

It’s still so early, and things are just getting started in AR. There are only so many webcam magic tricks you can enjoy before you are ready for something else.  The location-based apps have the most potential in my opinion, which is why we are really focused on mobile AR.   We have some board-game type projects, which do not instantly scream location-based gaming, but if you look at something like the ARhrrr board game, you can see how much more compelling it can be when the game invites the player to be actively moving around during the experience.

Tish: I am interested in your perspective on how we can create the kind AR experiences that really embody what has always been so exciting about AR – the tight alignment of graphics and media with real world objects and ultimately a rich immersive 3D experience, so I am going to hit you with a bunch of those, “Is this really eyewear or vaporware?” questions.  The real deal eyewear changes everything!

While eyeware is a big challenge technically and aesthetically,  I am pretty sure that there are several outfits out there that can pull off the optics and projection. 
Will the entertainment industry get excited enough to put a major push into delivering the eyewear in short order instead of the 5 to 10 year project that some people still think it is?    The business development challenge is bigger perhaps than the technical obstacles perhaps? What is your view on this?

And, perhaps, the eyewear is a clear example of a need for partnerships. For example, we have seen efforts from companies like Vuzix and Lumus, and recently a Japanese Company, Masunaga.

I have no reports from people who have tried the Maunaga eyewear yet.  But,  limited by small field of view, and tethered, currently eyewear offerings, available at a reasonable price point, are not workable solutions for augmented reality experiences. But the problems are not insurmountable. What will facilitate the real deal?  


It seems that it is critical to start creating hardware relationships now. The industry is costly and slow moving and as Robert Rice put it to me in a recent conversation, “once the software cat is out of the bag, its going to go wild and if the hardware isnt there, its going to stutter.”

As Ori notes some of the hardware companies like Intel and others don’t seem to be paying enough attention to AR.  Ori points out they don’t see the demand yet.  But in order to create an awesome AR experience and demand from a mass audience, don’t we need to work in conjunction with hardware designers?

Brian: It’s fun to think about who will eventually deliver a great hardware solution for AR glasses. It will happen. It would be cool to see somebody like an Oakley or Nike partnered up with a company like Vuzix to deliver something people actually might wear in public.  Perhaps a hardware manufacturer like Apple or Nokia will bring us something like the iSight or the NGaze down the line.  I’d love to see a set of glasses designed by Ideo.  Microsoft or Sony are already playing with technologies like Project Natale and the EyeToy, so I think its only a matter of time before they deliver an eyewear solution. I would even look to the toy companies to eventually make an investment here.

Gamers will be the early adopters, and in a few years we may start to see people running around in the park wearing glasses with headsets, but it will be acceptable because it’s clear they are using them for a game.  It’s going to take a very sexy and stylish piece of hardware for everyday people to be willing to wear AR glasses in public while going about their everyday business.   It’s like the recent cover of Wired magazine where Brad Pitt is wearing a mobile headset in his ear, and the editors point out that even he can’t pull that look off, so why do you think you can.  When AR glasses come in designer frames, and you can’t tell them from non-AR glasses, to me that’s when things get really interesting from a mass-adoption perspective.   Compare how many people were carrying around a mobile phone in the 80s to now.  I think it will be the same thing with glasses.

I was in an AR pitch meeting the other week at a very significant media company, and brought up the point that today’s handheld Smartphones will eventually evolve into tomorrows Smartglasses. My comment was quickly shrugged off as sort of a sci-fi notion that was irrelevant to the business at hand.  Probably true, but I think it is important to understand where digital media and entertainment is going, so you can adapt quickly, and evolve into those spaces more naturally.  The more we see people walking around with their Smartphones in front of their face (like a camera), the sooner it will be that we make the jump to eyeglasses as a key hardware device for AR experiences.

At Ogmento, we definitely are working on AR experiences with the hardware and software available today.  We will get some product out this year, and 2010 will be a banner year for markerless mobile AR in general.  I think the entire AR community is looking forward to bringing this technology to the mainstream in the form of games, marketing campaigns, virtual docent apps, and much more.  It might not be the full experience we are all dreaming about for some time, but we can see the path and the true potential, and it’s pretty spectacular.

You mention the tight alignment of graphics and media with real world objects.  That is really our focus. A lot of well-deserved attention is going to the browser overlay “post-it” approach right now, which uses compass and GPS.  We are focused on markerless natural feature tracking, so once you identify something that is AR enhanced in your environment, you can interact with that integrated experience.  On an iPhone that can be as simple as using your touch screen to interact.  When you are wearing glasses, it becomes more about visual tracking. There are lots of smart people thinking through these issues. Many of which you have interviewed. It is my hope that there are exciting collaborative efforts to be had in the coming months to get us all there together and faster.

Tish: Bruce touched on some of the hard problems that have to be solved for augmented reality – and he noted for instance security needs to be tackled in the early stages. Robert made a nice list, “privacy, media persistence, spam, creating UI conventions, security, tagging and annotation standards, contextual search, intelligent agents, seamless integration and access of external sensors or data sources, telecom fragmentation, privilege and trust systems, and a variety of others.” Will Ogmento be leading the way in solving some of these hard problems?

And, won’t trying to solve these hard problems for networked AR in walled garden scenarios one company at a time lead to a lot of reinventing the wheel wasted energy?

Brian: These are all important issues, and again there are a lot of smart people thinking about solutions to these problems on a daily basis.  Ogmento is interested in partnering with developers and supporting their efforts as a publisher of mobile AR experiences.  While we intend to roll up our sleeves in these areas, we are currently more focused on taking AR mainstream with the hardware and software available today.  As the industry evolves, so will Ogmento. As the opportunities evolve, our ability to make a greater impact tackling these issues will be realized.

Tish: Another area of development that could really kick AR into high gear might be creating augmented reality hotspots  where we use can deliver the kind of location accuracy/instrumentation necessary to create interesting AR experiences (partnership with Starbucks, perhaps ?!).  Augmented reality hots spots, could deliver the kind of high quality AR experience that isn’t possible ubiquitously at the moment, and may be a real way to get people really exploring the potential of AR now, rather than later?

Brian:  Agreed. I see a great opportunity here with this approach.

Tish: Although there are many obstacles to Green AR – the energy hogging servers at the backend for starters! Last week I had a conversation with Gavin Starks, AMEE, and Robert Rice and Jim Purbrick about how to work with AMEE and the technology available and encourage Green Tech AR development (see an early exploration of green tech AR from Pachube here).

We came up with the idea of holding a competition perhaps centered around a targeted instrumented space. But I would really love to hear your thoughts on the topic of Green Tech AR (the energy hogging servers at the back end being the first cloud on the horizon!.)  Cool GreenTech AR imaginings, social gaming ideas, RPGs, not even necessarily even tied to the immediately practical, would be like rain in a drought!

Brian:  I go back to “Games and Goals”… If you make environmental and other activist efforts fun and rewarding, more are likely to be motivated and participate.  Can you imagine having a personal “carbon footprint stat” floating over your self at all times? Or over your home or factory?  How would that change your behavior?  We all love stats. Look at how the Nike+ campaign has used technology and gaming to motivate people to run.  I think there is a lot that can be done to make being green fun. It starts with the individual, and spreads from there.  Keep me posted on that one!

Tish: I would also like to explore further the RRW suggestion that ambient intelligence is both the Holy Grail of AR and possibly snake oil:

“The holy grail of the mobile AR industry is to find a way to deliver the right information to a user before the user needs it, and without the user having to search for it. This holy grail is likely in a ditch somewhere beside a well-traveled road in the district of the semantic Web, ambient intelligence and the Internet of things. Be wary of any hyped-up invitation to invest in a company that claims to have gotten the opportunity right. What we’ve seen in the commercial industry to date is a rather complex version of a keyboard, mouse, and monitor.”

So Holy Grail, Snake Oil, or a ditch somewhere….?

Brian:  I instantly think of Minority Report, where Tom Cruise’s character is being bombarded with holographic ads personalized with his name and to his current situation.  In the future, Spam is a nightmare, especially when it knows who you are.  I think the key thing here is delivering “the right information”, and we still dont have that down. I do see a day where we can truly customize what comes to us, how we want it, when we want it.  My future vision of ambient intelligence is the ability to “turn everything off” if I want to… block out the stimuli and replace it with images of nature, or natural surroundings, etc.  Where I live in Los Angeles, we have those digital billboards everywhere, so it’s like advertising overload wherever you look (hints of Blade Runner).  I personally don’t mind them, but I know there is great debate on there being simply too many billboards everywhere. So AR would only add to the noise of life by adding yet another digital overlay of information, right?

Perhaps the holy grail is to use technology to filter things out. AR might become a solution to leading a simpler life, or a perfectly customized life if you want that. Ultimately the control needs to be with the individual.  I guess I am talking about something like TiVo taken to the extreme.

Tish: And then that other biggy – augmented reality search! I am asking this next question of  Wikitude and Sekai camera too and now I must also ask Acrossair and several others I guess! Obviously a huge area of opportunity in this broader landscape that uses location-awareness, barcode scanners, image recognition and augmented reality is to harness the collective intelligence – a whole new field of search. There is the beginning of a discussion on this in the comments here.

What will it take, in your view, to become a leader in augmented reality search?

Brian:  I’m more of a content guy, so I tend to focus on things like UI, quality of creative, etc..  From that perspective, I am looking forward to evolving beyond the “post-it” text overlay user-experience we see now in AR search. I was impressed with the TAT Augmented ID concept and hope we start seeing more smart design solutions like that emerging in the space.  There are some great new design approaches coming out of the location-aware space that should be applied to AR search. I’ve been studying the heads-up display designs being used in video games, and re-watching movies like Iron Man for ideas. This is another example where Hollywood has painted a polished picture of what AR can and should look like, and the masses have already accepted these design approaches.  So from that perspective, from my view the leaders in search will be delivering sexy, smart and simple solutions. It’s all about the S’s.

I met Stowe Boyd, of Microsyntax.org at Jeff Pulver’s 140 Characters Conference which convened in the middle of a perfect storm for the State of NOW (more mundanely known as the real time web) as thousands of tiny Twitter pipes became a vital conduit for the historic events occurring in Iran (picture on left, Stowe Boyd, from Brian Solis‘ Flickr here, and on the right, Bruce Sterling, presenting at reboot11 from scriptingnews‘ Flickr here).

But, as Clay Shirky pointed out, re Twitter and Iran:

“It’s incredibly messy, and the definitive rules of the game have yet to be written. So yes, we’re seeing the medium invent itself in real time.”

Stowe Boyd is managing director of Microsyntax.org, a non-profit investigating the embedding of structured information within microstreaming applications, particularly Twitter. It is a communitarian project so if you are interested you should get involved – see Stowe’s #140conf. presentation, “The evolution of Microsyntax.” Stowe is an architect of “flow” and a webthropologist of the State of NOW.  I had the opportunity to talk with him at the conference (see the full conversation below). We talked not only about some of the practicalities of implementing microsyntax but about how “the web of flow” produces a fundamental shift in how we communicate, and who we are.  As Stowe Boyd put it:

“You use these tools, and you are changed. And it’s just a question of how long you use them and the longer you use them, the more you use them, the more changed you are. When people shift to a basis of sociality around connection with other people as opposed to mass affiliation, it’s different. It’s completely different. Your whole system of ethics, the way you judge the world and decide what’s important is different. And not only different it’s better. It’s a better way to deal with the world.”

As Wyclef Sean (@wyclef) remarked at #140conf, “Twitter just cuts the middle man in everything.”

At the 140 Characters Conference it was hard not to be captivated by the energy and optimism arising from the successful use of Twitter by Iranians to communicate in the aftermath of the election.  But the subsequent repression in Iran, in which the regime took advantage of central infrastructure controls to silence Iranian twittering (we have similar network technologies in place here in the US), leaves a big question that came to the fore after the conference:

While these real time applications give us the ability to leverage network effects in totally new ways, and they have enormous potential to make our lives better, do we need to give more thought to the infrastructure they rely on?

The videos for the 140Conf are up now. If you haven’t already seen them, after watching Jeff Pulver’s intro to The State of NOW a great place to start is the “Twitter as a News Gathering Tool” (Part 2).  Also see Ann Curry Defends Foreign Correspondents, Twitter; Rick Sanchez Defends CNN and Brian Solis’ post on techcrunch. Christopher R. Weingarten (@1000TimesYes), “Twitter and the End Of Music Criticism,” and Moeed Ahmad’s (@moeed), Gaza in Focus, are two of several must see presentations. The #140Conf was an extraordinary event.  Jeff Pulver orchestrated a brilliant cast of characters and a manifestation of social media “hybrid vigor” that was exhilarating to be part of.

A “Director’s Cut” of #140conf will be re-broadcast (Monday, June 29th and Tuesday, June 30th) at 11AM EST / 8AM PST – http://140conf.com/watchit. Some of the speakers will be tweeting while their session is being re-broadcast (see The Jeff Pulver Blog for more).

(picture above from Brian Solis’ Flickr here)

In a serendipitous convergence of events I found myself in the front row taking photos for Brian Solis (@briansolis) see Brian’s post, “Is Twitter the CNN of the New Media Generation.” I like my photo of Jack Dorsey (@jack) Twitter founder – the lens of my own camera would never have allowed for this one!

I was also sitting close to Stowe Boyd (@stoweboyd), who out of all of attendees at this jam packed event was one of the people I had most hoped to connect with.

Talking with Bruce Sterling about Squelettes, Twitter, Favela Chic, and Gothic High Tech

I have been following the microsyntax.org effort that Stowe has been leading since this post by Bruce Sterling (@bruces) on Pachube Feeds which contained this challenge:

“(((Extra credit for eager ubicomp hackers: combine this [pachube feeds] with Googlewave, then describe it in microsyntax. Hello, 2015!)))”

Stowe pointed out in our conversation at #140conf, that Microsyntax.org is in one sense a very narrow project but on the other hand it’s very broad, because every sort of information that you can imagine is going to be streaming through Twitter and related [real time] applications.

Or as Aaron Straup Cope put it to me: “This is ultimately the “magic word” problem, which is essentially the semweb vs. google-is-smarter-than-you problem.”

There are a bunch of crystal ball posts up at the moment looking into the future of the real time web…. for example, this post on threeminds.organic (via @timoreilly and @buckybit) asking whether we need page rank for people and not just sites…..and this post on readwriteweb that asks is the state of now the harbinger of doom to walled gardens like Facebook. And there seems to be an arms race starting around real time search.

But Bruce Sterling (@bruces) in his cover story for Interactions Magazine examines some of the blinkering on “two inherently forward looking schools of thought and action [design and science fiction].” He writes:

“We have entered an unimagined culture. In this world of search engines and cross-links, of keywords and networks, the solid smokestacks of yesterday’s disciplines have blown out.”

While I was writing up this post, I found myself up at the crack of doom (4 am EST) with insomnia I attribute to a tweet from Mark Vanderbeeken @vanderbeeken which I (@tishshute ) retweeted:

“Internet of Things – An action plan for Europe,” (This EU Doc. cites @agpublic ’s Everyware) http://bit.ly/16uiu3 via @vanderbeeken“

(I wish I had used the new microsyntax in Tweetdeck RE (for more on RE see Stowe Boyd’s post here) then I would have been able to find @vanderbeeken’s original tweet just now.)

So after a quick scan of the EU paper on the internet of things, and in a “here comes everybody” pre-dawn state of mind, craving oracular pronouncement, I impulsively shot an email to Bruce Sterling.

[Note: the following is an asynchronous exchange – not synchronous as a Google Wave would have made possible. Also I have pulled the conversation out of the original email format. Lars and Jens Rasmussen of  Google Wave seem to have hit the nail on the head when they “set out to answer the question: What would email look like if we set out to invent it today?” (see this excellent post by Tim O’Reilly on Google Wave)]

Tish Shute: I shouldn’t be up at 4am EST sending you more questions but I began reading The “Internet of Things – An action plan for Europe,” http://bit.ly/16uiu3 before I went to sleep and woke up thinking: “How can we work on an action plan for everybody?” ((Another highlight of 140Conf. was Kevin Slavin’s talk on “Things that Twitter –  “sensor aesthetics are streamy”)).

Bruce Sterling: *Everybody?  What, all 6,706,993,152 of us?

Tish Shute: How does, “it’s all about the data,” and “google’s smarter than you” thinking versus “bottom up”/”personal informatics”/”sem web” get worked out in the internet of things?

Bruce Sterling: *I’d be guessing via mergers, acquisitions, lawsuits and police crackdowns, but you never know.  You might have a massive financial collapse where innovations like this start coming out of slums and favelas.  I heard such a great term at LIFT last week:  ”Favela Chic.”  That’s when you are totally penniless and without commercial prospects of any kind but still wired to the gills and big on Facebook.

Photo of Bruce Sterling at Lift 2009 by Centralasian

Tish Shute: Could you elaborate on your comment:

“Also, this stuff they’re discussing: this is like all kindsa trouble ten years from now.” (from your post http://www.wired.com/beyond_the_beyond/2009/03/spime-watch-data-shadows/)

Bruce Sterling: *Okay: you know how much trouble SMS messages are in Iran right now, even though ten years ago, cellphones were only for foreigners and rich guys in Iran?  Kinda like that.

Tish Shute: You wrote here: “The idea of living in *abandoned prototypes* or giant failed larval  husks is very contemporary, very New Depression. Very “Favela Chic…”

And:

“Ocasionally squatters move into “squelettes” and bring in some breeze-block, corrugated tin and plastic hoses, transforming squelettes into high-rise favelas. This doesn’t work very well because it’s tough to manage the utilities, especially the water.”

Tish Shute: So what happens when we rely on Google & Twitter repurposed as our main means to access our government?  Not only repressive regimes can cut these utilities off, even though Twitter was asked to delay maintenance so that the Iranian Twitters could keep flowing, Michael Jackson brought Twitter down.

Bruce Sterling: *Google and Twitter aren’t going to last long enough to become main means of an access to government.  It’s not that Google and Twitter go away and we return to a previous status quo, however.  It’s that they are ramshackle digital expedients that get replaced by  even more ramshackle digital expedients.

In the meantime the stuff we used to call “government” gets similarly destabilized.  It’s been privatized, or offshored, or turned into a hollow shell.

Tish: Shute: So is Twitter a squelette (like all our favorite internet platforms, including Google Wave which we haven’t even had a chance to squat yet)? And is microsyntax our breeze-block, plastic hose and corrugated tin-  – very Favela chic but vulnerable to the vagaries of Michael Jackson’s life and death, and deadly shut downs and snooping by repressive regimes that control the underlying utilities? (Squelettes, as Bruce Sterling points out, are:  “one of those coinages like “Prada Goth” that spring out everywhere once they are pointed out.”)

Bruce Sterling: *We can draw a distinction here:  “Gothic High Tech” is the top-end version, while “Favela Chic” is the low-end.  “Gothic High Tech” would be the likes of a “repressive regime” which finds itself forced to conduct cruel, secret, spooky, Guantanamo cyberwars… it’s pretending to transparency, accountability and open elections, while below that surface is a weird, torchlit, Gothic hall of mirrors where invisible hands wreck banks, impoverish the civil population and kidnap people.

It’s “Gothic” because of its magnificent, elaborate appearance — very “Castle of Dracula” — but that no longer maps onto its panicky, extremist, transgressive behavior.

Gothic High Tech doesn’t live in “squelettes.”  Gothic High Tech lives in fancier, more respectable structures called “stuffed animals.”  A stuffed-animal used to be a functional building. From the outside it looks pretty much like it always did, maybe even “conservative.”  Inside it’s half-retrofitted with aging, Frankenstein machineries, already outmoded, rapidly decaying.

A “stuffed animal” might, for instance, be a “savings and loan” where the behavior of the present-day inhabitants involves no actual saving and no actual loaning.  Instead the inhabitants are on television negotiating a position in a crisis narrative and living on bailouts, while, every day, the cobwebs get a little thicker.  “Regulatory capture” is stuffed-animal activity.  “Failed states” and “hollow states” are stuffed animals.

“Favela Chic” is the same basic activity, but with much less money and institutional clout.  In “Favela Chic” nobody bothers to ask for bailouts.  They know the state has failed, or they themselves are engaged in weird activities they prefer to hide from the authorities.   “Favela Chic” lives within openly failed structures, or else in half-structures that are in “permanent beta” and falling down as rapidly as they can be erected.  Favela Chic is bottom-up, open-sourced, heavily networked, subversive and piratical.

There’s a certain amount of class-transition between Gothic High Tech and Favela Chic — like, Twitter was Favela Chic and is heading straight for Gothic High Tech.  But there’s much less transition than there used to be, because of income differentiation — the tiny faction of Gothic moguls “own” what’s left of most of the wealth, which they themselves are rapidly destroying.  The general trend is not toward increasing global prosperity.  The precarity is becoming general.  The Favela beckons for everybody.  That’s where most of the planet’s population lives already, and it’s certainly where most of the young people live.  The idea of a “developing world” needs to be reversed; the end game is in the “developing world” and the rich nations are heading there.

Tish Shute: It seems to me that Twitter and the real time web of flow is a revolution in our means of communication presenting awesome opportunities.  But, are we squatters in an infrastructure that is hard to manage?

Bruce Sterling: *Yes. I’d go farther and say that we are squatters in an infrastructure that methodically destroys previous systems of management.  Especially itself: the closer you are to a revolutionary real-time web flow, the faster you have to reboot.

Tish Shute: And what is the answer to the question at the end of your cover story for Interactions:

“The winds of the Net are full of straws. Who will make the bricks?”

Bruce Sterling: *I frankly have no idea.  The storm-gusts are rising in a hurry and we are in for a whole lot of straws.

*I would point out that, if we could make up out minds about what kind of bricks we wanted, we could make them at tremendous speed.  We’re not helpless: our productive capacity is frankly fantastic.  Clearly we’ve lost the thread and can no longer explain what we’ve done to ourselves or how we get out of our fix.  But we might surprise ourselves.  21st century Favela Chic is no mere favela, and Gothic High Tech isn’t just Gothic, it’s also very high tech.  We’re in a Depression and it’s gonna last, but this is no 1930s Depression.

Talking with Stowe Boyd

Photo from Stowe Boyd’s Flickr stream, “Little” Tower Of Babel, Pieter Bruegel the Younger. It is also a slide from his presentation, “The evolution of Microsyntax.”

[Note: Most of this conversation took part in a busy foyer at #140conf and various people joined in the conversation at different points.  I have cut out these other conversations and tried to maintain the thread of my own questions in the transcription.  But this may have resulted in a sense of choppiness and discontinuity in places.]

Tish Shute: You have been on the front-line of so much web innovation, but, perhaps, you could give me a little back story on how you came to take the lead with microsyntax.org.

Stowe Boyd: Well, I’ve been on twitter 990 days or something. But long before Twitter became a commonplace household word, I’ve been advocating what I’ve been calling flow application, based on the streaming metaphor – the notion that you’d have a stream of updates coming from people that you chose to follow, which is now being called the asymmetric follow model. Years and years ago I postulated that that model was going to come along and completely change all future significant social applications. Back in the late nineties, I introduced a term “Social tools” and said social tools were going to come along and change the way the web worked. So I have a history of being 4 or 5 years ahead of what actually happens.

Microsyntax is sort of an interesting outgrowth of that. In a way it’s a very narrow area, in the sense that it’s focusing on these information patterns, the way that people want to encode information in the twitter stream or in the realtime stream of other apps. So it’s very narrow in the sense that it doesn’t immediately include all sorts of other things like these sports figures talking about how to market their services or whatever. But on the other hand it’s very broad, because every sort of information that you can imagine is going to be streaming through twitter and related applications.

We saw examples today of plants demanding water or DJ’s posting their set lists as they’re playing them, devices or equipment talking about its status, video stream from surveilance cameras. Everything you can possibly imagine will find it’s way in that stream. It’s all going to be encoded in different ways and grappling with that is actually an interesting problem. But more importantly it’s better for us as a community of users if we try to approach it in some systematic fashion. That’s the purpose of Microsyntax.org – this nonprofit. The concept of microsyntax is immediately evident to people who use Twitter, and that is we have a whole bunch of conventions that have emerged, and we have some places where it would be nice if conventions did emerge, but we don’t have them yet. And the idea of creating a nonprofit to do it is a sensible thing to do. So I decided I’ll go along with the request that others have made, because other people asked me to do this. So that’s a little unusual for me.

The Web of Flow

Tish: What first attracted my attention to Microsyntax.org was Bruce Sterling’s post suggesting combining pachube feeds with Googlewave and then describing this in microsyntax. Why do you think Bruce Sterling posed this particular challenge?

Stowe: Well, because he sees that everything is moving into the web of flow. Everything is moving out of the web of pages. In the next ten years we’re going to cease to experience the web as we do now, which is as a bunch of pages and we move around from link to link. And that’s what browsers are about. They help us move from page to page on the web. But Twitter, and before it the minifeed and instant messaging and a handful of other really interesting applications, have suggested a completely different web where information flows from other people to you through streaming mechanisms.

And the really interesting stuff that comes to me now on a daily basis is streaming to me through Twitter, not through my RSS reader, not me wandering around figuring out what to google, news or something. And that’s an indicator of the fact that that’s the hottest, coolest way to do it now, and means that in the future it will be “the way” that it’s done. So there will still be a web of pages out there, but it’ll exist like an archive. And we won’t experience the web that way in general because, “why would I go to the web page and see the guy’s blog post on his page, when it’s been served up to me 16 other ways?” And most importantly I’ve found it initially in some client, because somebody recommended it to me, and I resolved it in a hover window in my Twitter client. I’d never go to the page. I comment on it here…

Tish: I like your framing,  “the web of flow…”

Stowe: Well it’s also that one of the characteristics is the tempo is different. I actually wrote a post about this, that I think it’s fundamentally important. It’s not really gotten much drift yet. I think it’s too hard for people to think this way. They just can’t get it.

The dimension that’s really most interesting is the transition from secret to private to public. The fact that Twitter is inherently public as a default is a breakthrough. I mean there’s nothing else like this. The first time that the idea, except for the blogosphere itself which is the concept it’s built on,  the inherent notion is that you’re publishing stuff and anyone can get access to it. But the tempo thing really matters, the fact that it’s near synchronous so your perception of what you feel like you’re doing is you feel like you’re in a stream of updates from friends. We know that. But the sensation is dramatically different than your close personal relationship with your inbox, which is email. Email is secret, closed, and the sense is the context is that it’s an inbox, like the one on your desk. And you are boxed in by that, and you’re not actually feeling like you’re dealing with people. You feel like you’re dealing with the inbox.

Tish: This was only present in boxes as you say – chat rooms, IM, IRC, MUDs, Virtual Worlds but they all had that realtime experience going on.

Stowe: Yes instant messaging, chat rooms, etc. they were private. You had to invite people. The update paradigm on instant messaging was backwards. It said I want to follow this guys updates, but you had to get his permissin to do it. That seemed like a sensible thing in the mid ’90s when people worried about privacy and so they made it private. And private is not good, actually.

Tish: IRC is exactly like twitter but it’s off in closed worlds…

Stowe: Yes you have to know about them. You can’t just stumble across them, you have to be invited or give the password. It’s another closed model. But instant messaging is the father of all this, or the mother, depending on which way you look at it. But that fundamental last thing, it’s based on a quote by Gabriel García Márquez which is, “All people have three lives. they have a public life, a private life and a secret life.” And we are philosophically moving from a time where things were primarily secret (pre internet) to a time where things were primarily private which is web 1.0 into this new web where things are going to be primarily public and open and immediate. So we are building the scaffolding real fast to allow that to happen. And it’ll take us away from the old web. The old web will go down there. Everything’s built on dirt right? Do you see very much dirt in cities? No. No. The dirt is all concealed. It’s down there. If you want to go find it you can dig underneath the floor, and there’s dirt under there. But most people don’t spend very much time down there we send professionals down there to put plumbing and pipes underneath and we experience the world like this.

Tish: I met Eric Horvitz (Microsoft Researcher) at Where 2.0.  He is interested in community sensing and ideas about how people can share data in a win win way (see here). Do we need to work out ways to make sure people’s relationship to their data is not just to have it harvested by others for profit or repression?

Stowe: I’m interested in this actually. I recently wrote a piece about the governance of Twitter and for the purpose of your question let’s just go along with the premise that Twitter’s going to continue to be benevolent, and everything will be open, and everything will be public and everyone can do whatever they want with it. Well there’s a tremendous amount of things that people will want to do, but most of the things that they will set about doing to begin with will turn out to be irrelevant.

People will want to measure sentiment and all this other stuff, for example. And they’ll do that and they’ll coerce a lot of big brands and so on to pay money for these services. But the thing that’s going on with the now web, my web of flow is that people are disconnecting from self identity based on mass affiliations. So ultimately the more you spend your time doing this, you don’t give a s**t about brands. Nike – I could care less.

So there is defection from the mass media. We heard it today. There’s people here who were like booing these media guys, who think they should be held up as gods because, “Oh I’m one of the first to use Twitter on TV.” Well F*** you, I don’t give a s***. I don’t watch television. Every hour that people spend on the internet is an hour they do not spend watching television. It’s a direct and one to one correlation. Sure people still want to get their fill of whatever, the NBA playoffs, but significantly less than ever before. Which is why they’re increasingly irrelevant.

So the idea that some magicians are going to come along, figure out how to mine this data to find out how I feel about my automobile? I do not have a close personal relationship with an automobile. I don’t. And increasingly people won’t affiliate that way. They won’t bond with their stuff like that. That’s why I say most of this information won’t be helpful. It’ll be interesting sociologically. Webthropologists will be able to make it interesting – and marketing people, who are trying to figure what’s going on, might be able to do the right thing. But if they’re trying to take it and make it do something for them… They’re going to try to take it and use it to change us? To control us? It’s like that line in The Labyrinth,  “you don’t have any power over me anymore.”

Tish: You are actually saying something much more radical than say community sensing or that we need to store our own data. You seem to be saying that in some ways it doesn’t matter whether you store your own data or your data’s in the cloud (although Iran seems to be showing how centralized network control can be a powerful tool of repression).

Stowe: Most of the things that they’re going to try to use it to do won’t work because we’re not the same anymore. It’s inevitable. You use these tools, and you are changed. And it’s just a question of how long you use them and the longer you use them, the more you use them, the more changed you are. When people shift to a basis of sociality around connection with other people as opposed to mass affiliation, it’s different. It’s completely different. Your whole system of ethics, the way you judge the world and decide what’s important, is different. And not only different it’s better. It’s a better way to deal with the world. And these guys are still hoping that the old rules hold, but they don’t. They just won’t.

Tish: This is  rather a broad question. But one of the things that Kevin Slavin brought up in his talk is about things that tweet – your plant is tweeting, your shoes are tweeting, your house is tweeting. Twitter is a natural medium for the internet of things and what Kevin Slavin calls the “streamy aesthetics of sensors.” But with all these things that are tweeting people have had a lot of problems with filtering that kind of flood of tweets.  For example, I may want to listen to a tweet from my plant telling me it needs water when I am actually at home and can do something about it. But I may not want to listen to my plant whining about being thirsty all the time. Can microsyntax help? Or is this a place for those appliances you mentioned earlier?

Stowe:  There’s a whole other category of stuff having to do with priorities – this isn’t really a microsyntax – of different times of day when you’re involved in different activities. You may be more or less interested in different collections of Twitter streams. And the notion of how you go about dealing with that is – it could semi-microsyntactical, but maybe it isn’t at all. Maybe it’s all just having to do with the way that clever client apps work. So maybe if you have a super duper Tweet Deck, and you say it’s evening time and I’m in my evening mode, so a whole bunch get blocked and a different group of people, for example, your Parcheesi evening friends get enabled, and at the weekend when you have time to do house care you listen to your house.

I don’t think this is a microsyntactical issue. I don’t think this is an issue of what’s embedded in the stream except as a notion of priorities. There’s a lot of people who would like to have a mechanism to indicate priority. But I can’t think of any effective way to do it that wouldn’t immediately be abused. Of course anything can be abused. This guy thinks that this is high priority, but maybe once again it’s one of these sort of mutual dimensions where they want to indicate it’s high priority but I say I only believe in priorities from certain people.

But still there might be a case to be made for allowing people to put some kind of indication of priority in a tweet, so that there is a hope that it could rise out of the clutter. I talked about some things that I’m interested in that are just purely operational. One of these things I want to get people to build, in Tweet Deck, but it could be in any kind of a client, I want to be able to say don’t let this tweet go away. So I’m getting them to build the pushpin. So I can put a pushpin in the thing and it’ll stay at the top, or stay at the bottom, wherever I put it. And then I can respond to it later, because if I don’t respond to it right now, in most places it goes bye, and then you’ve got to go search for it – a pain in the ass.

Then I say if I’m going to have pushpins I want to have a record of all the things that I’ve push pinned – a history of pushpins. But it’s all client based. It’s got nothing to do with what’s in the text.

Tish: And knowing how many of your followers had already got a particularly tweet from somewhere else which would be very useful has to be done as an appliance…

Stowe: Yes that’s sort of a downstream metrics kind of thing.

Microsyntax is not the answer to every kind of thing. Like, appropriately dealing with hash tags in a sensible fashion is not purely a function of how we use them. But some of it is the structure itself. That’s why I came up with the subtags model. So everybody at South by South West tagged everything southbysouthwest, so if you searched for it there were 150,000 hits a day. So it was useless. But if people had used the subtags model, or something else like that, you could have searched for the subtag. So you could have searched for south-by-southwest.parties or south-by-southwest.thirtytwo-bit which was a particular party.

And so if you have sensible tools that are doing a better job of aggregating information around more complicated ways of structuring hash information, then we can get past the fact that brute force search just isn’t going to work. It just won’t work. For example somebody going through the stuff from today all the stuff that says #140conf but they want to find just the stuff that had to do with media, they wont be able to do it. They’ll have to do it manually. So some of that is better syntax. But some of it is better tools. I mean somebody should go build a better hashtags.org.

Tish: And in terms of creating a web of flow not all of what we need can been done within the Twitter messages – it has to be done in the client and external applications…

Stowe: Yes, there’s this class of applications that listen very diligently to what you’re doing in Twitter. The primary mechanism of how you influence the app is doing stuff in Twitter. You can always go to the app and look at it and fool with it. But, if in fact, the preponderance of your interaction is, it’s listening or talking to you in Twitter – I call that an appliance, to distinguish it from these other apps. Any external application might provide you with the mechanism to dump information into Twitter, but you have to go to the app to do the primary kinds of interaction. In fact major functionality may not be available at all in Twitter or maybe no functionality, except for like Brightkite allows you to dump stuff into Twitter. But the idea is that primarily you do it there. Or there’s a very limited thing like you get with Brightkite, you can send a message saying, “I’m somewhere.”

Tish: Should location be put into tags?

Stowe: I don’t think that location should be put into tags. In other words, if I talk about Paris, then using hashtags is sensible. Or I’m talking about Sherlock Holmes and his relationship to London. It’s a conceptual thing – like talking about Heaven. It doesn’t actually have to exist on the planet somewhere. But it’s really different if you say I am in New York City right now or the more interesting case I think really is, “I am going to be in Boston colon next week” or June 15 dash 17. And I want that information to be available to everybody or a select group of my friends, or just to myself and have it find it’s way into my calendar. But that’s really different than saying “I’ve always enjoyed it when I visit HASH New York.”

Tish: I liked Kevin Slavin’s  phrase “the streamy aesthetics of sensors.” I guess streamy aesthetics is something you have given a lot of thought to?

Stowe: First of all I read a lot of poetry, so I believe in poetics in reading and writing. But I don’t think punctuation marks really degrade that dramatically. I mean it’s OK to have periods and exclamation marks and commas, and things can still be poetic. I think it’s important to try to dream up microsyntax that doesn’t take your eyes off the content, the stuff that people are really trying to say. So that’s why for example I hate L: as a location queue because anything that has letters in it, if you’re not supposed to say them, – if you’re not mentally supposed to say them, or if you’re not supposed to say them if you read it aloud, causes you to do a stutter step when you’re reading the tweet.

But if you use punctuation marks, special characters at various points or placement conventions, like where do things appear in order in a tweet, those things don’t have the same toe stub, that I think really ugly syntactic conventions would. So it’s possible to make these things pretty. For example I’m testing out trying on various conventions for what do you do with a re-tweet. If you want to re-tweet it, if you actually want to have people see it, and then you want to make your own comment. So the question is how do you separate the two? So, RT – guy’s name and then text. Well then how do you know where his text ends and my text begins. So certain things don’t work for me. I mean like a comma is not enough because there might be a comma in the text. And a period doesn’t work because there might be multiple sentences. So it has to be something else.

Tish: And aren’t there confusions that arise because there are already conventions of usage…

Stowe: Yes, I have problems with angle brackets, for example. Sometimes when the tweets wind up in not particularly smart rendering systems, it gets confused because it thinks they’re html. For example, somebody was using the open angle bracket, and even though it’s just text, and it’s not html, when I took that tweet and put it in a blog post, it thought it was the start of an html tag, and so it disappeared. You could use an html escape character but that’s the kind of thing that causes problems. The other problem is there are other ways that it’s been used a lot. People have used this as the thing to introduce the comment that they’re making after a re-tweet.

Tish: There must be very few characters not being used for other things?

Stowe:  Yes but for example, when we use geoslashes there’s a blank in front of it, or it’s the first character in the tweet – so in that particular example  it is similar because slash is used for other things. But, in all the places where it is used, generally there’s a character that precedes it – like “w/o” for without or a fraction or a long list list of  these options. [Geoslash is microsyntax for user location using slash (‘/’) — as in ‘just arrived /SFO’ or ‘heading to /New York: tomorrow/’ for more see Stowe’s post here.]

When I was rooting around for a character I looked for a long time.  And also I wanted to make sure that the slash was easily reachable on cell phones, which, for example, angle bracket isn’t. So if you’re on a phone and you want to say I’m here – I don’t know how far you have to go on your phone, but it isn’t in the first eight characters of Symbian. I looked carefully to make sure it wasn’t a common character that people use widely in everyday speech like commas and semicolons and exclamation marks, but was still easily used. There are still other alternatives. It’s not the only one. There are cases to be made for all of these things – pros and cons for all of them.

Anyway I was making the case of experimenting with different things for this re-tweet, “Here’s my comment.” And I was trying all sorts of stuff like double colon, I tried all kinds of things I wanted to see what it looked like. So starting this week I used the solid bar, the upright bar. It sets it off. It really feels like there’s a divide. There’s a cleavage point, and that’s that guy and this is this guy. So I’m going to write it up as one of the candidates. Some people use square brackets and many other things. There are many personal conventions but nothing has become a real convention, accepted as the norm.

[ Note: Our conversation ended here as the presentations had resumed at 140 Characters Conference ]

The biggest news at Where 2.0, 2009 came from the Yahoo! Geo Technologies Group. Tyler Bell, announced Yahoo! Placemaker and the opening up of the GeoPlanet data set, “all of the WOEIDs [Where On Earth (WOE) IDs] available as a free download under Creative Commons in June” (see Brady Forrest’s post for more details).

WhereCamp 2009 was held immediately after Where 2.0 and was a great place to chew on the events and ideas of Where 2.0.  In the picture above Anselm Hook addresses the WhereCamp morning circle in the courtyard outside the Social Text offices in Palo Alto. Anselm pointed out to me:

“there are interesting implications of placemaker in combination with other yahoo assets – in particular YQL – placemaker by itself is neat – but placemaker combined with everything else is a natural missing piece that is a big enabler.  Yahoo has been impressive.”

With all the Geo platform power available to us now, also (also see New Geo for Devs from Google I/O), there isn’t a shadow of a doubt in my mind Brady is right when he said, just before the Where 2009 conference: “Location is no longer a differentiator it’s going to become oxygen” (quote from WebMonkey).

The Yahoo! GeoPlanet team at WhereCamp – Tyler Bell, (talking to Brady Forrest in picture on the left) is sporting his spatial junkies T-Shirt. Photo on right, Aaron Cope, Tyler Bell, Martin Barnes, Gary Gale.

WhereCamp was alive with key figures from the social geography movement who knew the power of these new tools (see some of my photos of WhereCamp on Flickr here).

The importance of the Yahoo! announcement really became clear to me at WhereCamp where I attended sessions all day Saturday including the Curating Big Data Session led by Tom Carden, Stamen Design and Aaron Straup Cope, Flickr, (see Aaron’s slides from his Where 2.0 presentation on “The Shape of Alpha” here and video here).

Anselm Hook, a prime mover for WhereCamp, is a leading philosopher of place making and veteran software developer who led Platial engineering and is now at web consultancy http://makerlab.com. If you missed Anselm at WhereCamp he will be presenting on, Ubiquitous Angels at The OpenSource Bridge, Portland, Oregon, June 17th -19th, 2009.

Anselm describes where he thinks the challenges are:

“We should be mapping information that in some ways has been historically unmappable because it is 1) not valued or is 2) actively seen as threatening or is 3) simply too hard to map using traditional tools.”

The WhereCamp Schedule

The Shape of Alpha

Screen capture from Aaron’s Where 2.0 presentation on “The Shape of Alpha. Original photo from Flickr user “NickIsConfused”.

Aaron Straup Copes’s work on “The Shape of Alpha” puts key questions about curating big data center stage.

Firstly, the exploration of what it means to curate/collaborate over meaning from “the abundance of data produced in the precise but distant language of machines” (also see The Interpretation of Bias (and the bias of interpretation). The Shape of Alpha uses a process of reverse-geocoding to translate machine-generated geographic data into place names that people can understand and relate to.

The shapefiles are built with nothing but geotagged photos and some code called clustr (written by the brilliant  Schuyler Erie – co-author of Mapping Hacks). Anyone can make these shapefiles. You can get the shapefiles out of the  Flickr API. Aaron has been keying off WOEIDs (Where On Earth (WOE) IDs) but as Aaron noted you can key off anything you like – tags are an obvious choice.

Wow! You can reinvent mapping with this stuff.

Very importantly, “The Shape of Alpha,” tells us something about how we relate to place versus location. The emotions, disputes and behavior related to place also emerge through crowd sourced corrections.  For more see this very evocative post by Aaron about corrections and treating airports as cities.  There is a glorious thread/riff and ode to the genius of  J. G. Ballard pursued by Aaron and Dan Catt in their posts (also see Dan Catt’s, J.G. Ballard, Flickr, naked singularities and 3-letter airport codes, and Aaron pointed me to this brilliant “geo-detective work” on Concrete Island, by Mike Bonsall .

Dan Catt created geobloggers and “seeded the geotagging community around the Web.” I met Reverend Dan Catt (Twitter @revdancatt ) at Where 2.0 when he was kind enough to share part of his seat so I could join a very interesting discussion with Aaron on The Shape of Alpha.

As Aaron points out they decided to treat “the airport itself as the town…”  not (only) because they admired the work of J.G. Ballard,  “but because it is the right thing to do.”

Dan Catt has excellent blog posts “describing the nuts and bolts of how ‘corrections’ works.”  Aaron points out,  “in the nerdier of the two Dan sums it up nicely by saying”:

“On a slightly more philosophical level, it’s a never ending process. We’ll never reach a point where we can say “Right that’s in, all borders between places have been decided.” But what we should end up with are boundaries as defined by Flickr users.

…

For us, it’s a first small step into an experiment, and actually a pretty big experiment as we’re potentially accepting “corrections” from our millions and millions of users. We’re not quite sure how it’ll all turn out, but we’re armed with Maths, Algorithms and kitten photos.”

Psychosynthography – “Wearing Geography as a Perfume”

Psychosynthography screen capture from Aaron Straup Cope’s Where 2.0 presentation . Original photo from Flickr user,  “NiteLynx.”

As I mentioned before, many of the ideas raised at Where 2.0 were unpacked and worked through at WhereCamp. For example, Aaron introduced a word psychosynthography in the last 24 seconds of his talk at Where 2.0.

So I spent as much time as I could listening to Aaron at WhereCamp, and asking him about psychosynthography and more (post of this interview upcoming).

Aaron urged the Where 2.0 audience to pay attention to the Psychogeography movement seeded by Guy Debord, and “to wear geography like a perfume.”

Joseph Hart writes in a “New Way of Walking” psychogeography is:

“a whole toy box full of playful, inventive strategies for exploring cities…just about anything that takes pedestrians off their predictable paths and jolts them into a new awareness of the urban landscape.”

Curating Big Data

Tom Carden, Stamen, (picture above) paired with Aaron for the Curating Big Data session. Tom noted:

“The Curating Big Data session for me was an attempt to learn from other attendees (as opposed to teach/lead, as with the Stamen session, “Real Time Web-Based Visualization and Mapping”).  Also, it was an excuse to get Aaron to recap parts of the Flickr Shapefile story for WhereCamp folks, and to get *input* on how to do more things like it. I was a bit disappointed that nobody had really good examples for us, but I was happy with Brad Stenger’s suggestion to look into the upcoming census data as a relevant area.”

Aaron’s work on the The Shape of Alpha and The Corrections project shows, as Tom noted:

“what you can do once you have 150 million geotagged photos, and millions of users who are willing to say I took this thing here and my name for that place is …..”

And part of the significance of opening up the GeoPlanet data set is that now:

“we can try and start talking about the same places, as far as, [for example], these shape files go. So if you are interested in what comes out of the Flickr shape files project and but you also have your own opinion about what shape those places are so the IDs have be open you have to be sure that you are talking about the same thing in the first place.”

And, as Tom pointed out, collaborating over geo data informs us about curating any big dataset:

“it should lead to an overarching discussion about any kind of dataset geo or otherwise and ways in which we can talk about it, and think about patterns for improving that data, for collaborating, even on things like cleanup.”

Warp speed geo-genius Andrew Turner, Fortius One, took these excellent notes for the “Real Time Web-Based Visualization and Mapping” (on left) and “Curating Big Data” (on the right).

On my way to Where 2.0 I took the train from SFO to San Jose which was a delight but a little slower than I imagined. So, unfortunately, I arrived on Tuesday just after Michal Migurski (Stamen Design), Shawn Allen (Stamen Design) presented  Maps from Scratch: Online Maps from the Ground Up. This was on my MUST attend list and it was a wonderful opportunity to get into, “Real Time Web-Based Visualization and Mapping.”   I did get a chance to talk to Michal and Shawn a bit later in the conference but I will try to catch up with them soon for an in depth story.  Below is  Shawn Allen’s map of overlapping data sets from, “Trees, cabs and crime in San Francisco:”

Another follow up I am really looking forward to making is with Liz Barry and her work on S+EM, “an environmental mapping and social networking design project that links New York City trees with the people who care for them” (also see, Creating a Greener San Francisco Tree by Tree).  Also I got a chance to talk to another fellow New Yorker (we have to travel to the West Coast to find time to chat!), John Geraci of DIY City who presented DIY City:  An Operating System for Cities.

Machine Intelligence and Human Intelligence

Aaron Cope, Flickr, on the left is talking to Andrew Turner on the right the CTO of FortiusOne (see Andrew’s presentation at Where 2.0, “Your Own Private Geo Cloud”)

Many of the most interesting conversations happened in between sessions at WhereCamp and Where 2.0.

I caught this one in which Aaron Cope and Andrew Turner where discussing some of ideas Aaron raised in his presentation, “Capacity planning for meaning in the age of personal informatics” (see Aaron’s blog post, Tree planting and tree hugging in the age of personal informatics). The core question they were discussing was what happens when you wire the world at the scale people are talking about and it breaks… Aaron argues that you already have a whole class of people in systems operations that can tell us a lot about how to answer this question.

Ryan and Anselm shared the pulpit for the morning circle pulpit with Ross Mayfield of Social Text who was the generous host to WhereCamp.

Social Reality Mining

“As it stands today, we have no idea what terms and limits of a cloud based citizenship of the Google Caliphate will entail and curtail. Some amalgam of post-secular cosmopolitanism, agonistic radical democracy, and post-rational actor microecomics, largely driven by intersecting petabyte at-hand datasets and mutant strains of Abrahamaic monotheism. But specifically, what is governance (let alone government) within this?” from Benjamin Bratton’s talk at ETech 2009 (picture above), Undesigning the Emergency: Against Prophylactic Urban Membranes.

The other big take away from WhereWeek – Where 2.0 and WhereCamp, was not so much news, but a confirmation of something that has been pretty clear for a while now. (Check out Brady’s posts on reality mining at Where 2.0 last year).

We are moving headlong into the era of reality mining with all its myriad possibilities from: “hedonistic optimization” (this term came from Stig Hackvan when I asked him about some of the ideas central to the HeadMap Manifesto -more about HeadMap later in this post); to new forms of marketing (social reality mining the inside to predict if someone is going to trade business cards in the next 120 seconds – Alex “Sandy” Pentland, MIT, Where 2.0);  to stuff that matters to save us from mass extinction like distributed sustainability – greening production and consumption and our cities; to open government;  empowering indigenous communities (also see Rebecca Moore’s Indigenous Mapping: Emerging Cultures on the Geoweb Presentation); and not to be forgotten, the troubling possibility of new forms of social control.

Smart phones are powerful networked sensor devices in the palm of our hand

As Sandy Pentland MIT pointed out in his Where 2.0 keynote, “Reality Mining for Companies, or, How Social Networks Network Best,” mobile phones have created an ubiquitous instrumented reality that goes way deeper than location awareness. Smart phones are powerful networked sensor devices in the palm of our hand that know a lot more about us than location. With proximity, motion, (accelerometers), voice, images, call logs, email – what is enabled is not just knowing where people are but knowing more about them.

Many of the issues raised by Adam Greenfield in Everyware and in my interview with Adam were on my mind during WhereWeek, also questions that were distilled and explored in this presentation by Matt Jones last year, Polite, Pertinent, and… Pretty: Designing for the New-wave of Personal Informatics and Timo Arnall’s presentation, The Web in the World.

Google Wave, Pachube  Feeds, Sensor Networks and Microsyntax!

Visualizing 24 hours of Pachube logs, feeds all around the world -  built with Processing.

I found myself really wishing Pachube founder Usman Haque had been able to come to Where 2.0 this year – Usman was originally on the Where 2.0 schedule but had to drop out. My small contribution to WhereCamp was to discuss Pachube, Natural Fuse and OpenShaspa in the, Urban Eco-Managment session (see my interview with Pachube Founder, Usman Haque here).

Pachube announced – mapping mobile feeds in realtime, with 3d datastream value time & location based graphing just before Where 2.0.

And, as I was writing up this post, I was delighted to see this post by Bruce Sterling on Pachube Feeds and his challenge, offering:

“(((Extra credit for eager ubicomp hackers: combine this [pachube feeds] with Googlewave, then describe it in microsyntax. Hello, 2015!)))”

Also Anselm Hook, who has an extensive background in video game development, made an interesting point about Google Wave to me:

“btw – there is a preexisting metaphor for the wave – the wave is notable in that it is making the web like a videogame – its bringing real time many participant shared interaction to the web”

“Jens, Lars, and team re-imagined email and instant-messaging in a connected world, a world in which messages no longer need to be sent from one place to another, but could become a conversation in the cloud. Effectively, a message (a wave) is a shared communications space with elements drawn from email, instant messaging, social networking, and even wikis.”

For more on microsyntax see microsyntax.org

Aaron pointed out to me re microsyntax:

“This is ultimately the “magic word” problem, which is essentially the semweb vs. google-is-smarter-than-you problem.”

I will have some more questions for Aaron on the the “magic word” problem in my upcoming interview post.  At the moment I am busy studying some of the thoughts in these links.

http://delicious.com/straup/magicwords

http://www.slideshare.net/straup/the-papernet/22

http://www.xml.com/pub/a/2005/02/16/edfg.html

http://xtech06.usefulinc.com/schedule/paper/135

Privacy: Towards a Win Win and Community Sensing

While a key element of  Yahoo! Geo Technologies portfolio of platforms, FireEagle, not only gives an important set of tools to allow people to “share their location with sites and services through the Web or a mobile device” but also offers up some vital privacy tools, the community sensing work of Eric Horvitz takes privacy and data sharing into new terrain.

Eric didn’t have time to discuss his privacy work in his Where 2.0 presentation, Where, When, Why, and How: Directions in Machine Learning and Reasoning about Location, – it came up in his very last slide. But I ran up after his talk with my trusty old ipod recorder in hand, and got the part we missed! Fascinating stuff that will be the subject of an upcoming interview post. Here’s a little taste of what is to come. Eric describes one of the directions his team will be exploring.

“One thing I want to do, on our research team, I’d like to develop something very simple for people to use. A challenging problem with privacy is usability and controls. Aunt Polly and Uncle Herbie just don’t get all these authentication controls and sliders, nor do they want to invest in figuring them out. They also don’t get why they’re being asked with pop up windows to yes or no to various questions and so on. One Idea is having a useable privacy lens, that you can hold up anywhere and it tells you what you’re showing anybody or any organization, what does the world know about you. And you would like to have buttons to turn sharing off for some items. You’d also like to have a way to go back in time and view prior sharing and logging over periods of time, and to have buttons to push to say erase that segment of your logs.”

Understanding the social implications of what it means to live in an instrumented world is a topic that we cannot afford not think about. But luckily there are lot of people who have been thinking pretty deeply about this for a while now.

And I did my best at both Where 2.0 and WhereCamp to seek out as many of geothinkers as I could, and do interviews wherever possible (I have not had time to mention everyone I talked to in this post but hopefully all the interviews will get on Ugotrade soon!)

HeadMap Manifesto

In the bar of The Fairmont on the last night of Where 2.0, I heard some of the history of Where 2.0, GeoWanking, and The HeadMap Manifesto from Sophia Parafina, Director of Operations for OpenGeo and Rich Gibson, programmer, GeoWanker,  Gigapanner and co-author of Mapping Hacks with Schuyler Erie and Jo Walsh (Jo did a lot of key early work on bottom up urban informatics but unfortunately couldn’t make it to WhereWeek this year).

Check Gigapan.org out! “The GigaPanSM process allows users to upload, share, and explore brilliant gigapixel+ panoramas from around the globe.”

Also I interviewed Paul Ramsey, Senior Consultant, OpenGeo, so more on OpenGeo is upcoming (see Paul’s Where ReCap). Justin Deoliveira (OpenGeo) and  Sophia Parafina did a session, GeoServer, GeoWebCache + OpenLayers: The OpenGeo Stack, which unfortunately I missed as it was before I arrived Tuesday.

I met Rich Gibson at Etech 2009 playing Werewolf and Rich introduced me to his co-author on Mapping Hacks and alpha geek supreme, Schuyler Erie, who also wrote the clustr code that The Shape of Alpha uses.

Joshua Schachter founder of Delicious and the GeoWanking mailing list, [and GEOURL - and MemePool!] now at Google came to WhereCamp and was mobbed by a small crowd eager to get their hands on one of the developer G Phones he was handing out from a large box.

GeoWanking, which is now run by O’Reilly Media, has been the incubator for all things location aware and “neogeography” discussions since 2003 – check out ‘sproke for a Paleogeography vs Neogeography (which, as Sophia notes, was a common topic of discussion at Where 2.0) smack down in which geowanking rules in the form of a list traffic comparison.

Sophia and Rich shared some of their perspective on the early days of GeoWanking and the creation of the HeadMap Manifesto with me and pointed me to many other people to talk to. The prime mover of the Headmap manifesto, Ben Russell, has retired from the scene – perhaps bored by seeing a radical vision gone thoroughly mainstream, or exhausted by the rigors of carrying an idea through the early blue sky years, or just s simply doing something else? I don’t know.

The HeadMap Manifesto is still vibrant today even as much of what it envisaged has already been realized. HeadMap assembled the future in a poetry of fragments:

“you can search for sadness in new york people within a mile of each other who have never met stop what they are doing and organize spontaneously to help with some task or other.”

Anselm explained to me what powered all this social cartography revolution, from his POV, was actually IRC.

“We had a channel on IRC called “#geo”.  And many of us met there.  I met Ben Russell at MathEngine in the UK. Ben and I were fascinated by the future of maps.  Ben, Jo and I met Schuyler, Dav, Dan Brickley (who worked for Tim Berner’s Lee who invented the Web), Rich Gibson, Joshua Schachter (who was just a geek at Morgan Stanley at the time ) … and the snowball took off….  many others.

We stormed ETECH ( Schuyler met Jo there).  We got invited to FooCamp. Schuyler was married to Jo by Marc Powell (Food Genome) and lived at his house.  We pushed so hard on the social cartography revolution.

I did a spinny globe for geourl – a project by some hacker named Joshua Schachter…  we were all friends for years and we had never even met.”

“Can AR researchers harness these new approaches to index reality?”

Radiohead’s laser (as opposed to video) clip made using Lidar

If you have read my interview with Ori Inbar, you will know how excited I was to attend The Mobile Reality panel.  The video is up and it is really awesome to hear Raven Zachary (on twitter @ravenme) get into the fray with augmented reality.

The main take away for me from the Mobile Reality panel was that we shouldn’t get too hung up on the difficulties of achieving fully immersive visual augmented reality and twiddle our thumbs waiting for the long anticipated sexy lightweight eyeware – which is still in a coming soon phase (for more on immersive augmented reality see my upcoming interview with Blair MacIntyre). Because, in the meantime, there are plenty of delightful and useful ways to augment our experience of the world – and not all of these augmented realities rely soley on smart phones as John S. Zeleck showed in his presentation on “Wearable Sensory Substitution Device for Navigation.” Also I had an interesting discussion at lunch with Ori Inbar about the use of audio for augmented reality projects.

Where 2.0 clearly demonstrated that we have an unprecedented amount of information from mapping our world, Ori Inbar noted in his conference roundup. Ori writes:

“My point is not a shocker: all we need is to tap into this information and bring it, in context, into people’s field of view.”

As Ori noted Earthmine and Velodyne’s Lidar showed off two new approaches to mapping the world that have potential to create new opportunities for augmented reality:

“Earthmine uses its own camera-based device to index reality, at the street level, one pixel at a time. They have just announced Wild Style City an application that allows anyone to create virtual graffitis on top of designated public spaces. However, at this point, you can only experience it on a pc!”

Lidar, Ori notes, has also embarked on a mission to map the outdoors. But, the question Ori highlights is:

“Can AR researchers harness these new approaches to index reality?”

Brady Forrest inspects John S. Zelek’s “Wearable Sensory Substitution Device for Navigation” at Where Fair before putting it on and being guided by sensory nudges at the cardinal points in the belt.

Coolest Mobile Locative Media App. at Where Fair

Atsushi Shionozaki of Place Engine – “a core technology that enables a device equipped with Wi-Fi such as a laptop PC or smart phone to determine its current location,” demoed the coolest location aware mobile app in Where Fair – Oedo Yokai. Working with ethnologist, Dr. Hiro Kubota and artist Atsushi Morioka, “Oedo Yokai” is Koozyt’s “first attempt to cross IT (Location Information) and Folkloristics.”

“The Japanese “Yokai” are known to dwell and appear at specific locations. They can frequently be seen within the grounds of shrines and temples, believed to be the border between this world and the afterlife, or in more common places like on a hill or at a crossroads. If the “Yokai” symbolize the mystery, legend, and lore associated with places, as our interests fade from actual locations, the rol, es they play in modern day society will diminish, and the “Yokai” might then cease to appear at all.”

I love this idea of bringing the ancient spirits of place back into our lives with our new tools of location awareness.

Odeo Yokai also reminds me of Aaron Straup Cope’s work on “the idea of every spot being a “history box” which he explained is “one of the threads behind the ‘nearby’ project at Flickr.”

The Food Genome

I cannot end this roundup of WhereWeek without a mention of The Food Genome.

“Food Genome is a big hungry brain that scours the internet, trying to learn everything there is to know about food.”

Watch out for the upcoming launch of this project, it stole the show with an exciting presentation at WhereCamp. You can follow @foodgenome on Twitter now.

To get one of the gorgeous Food Genome brochures you had to ask Mark Powell a good question. Notice an eager hand reaching out in the picture below. I asked, “how would the basic building blocks of the food genome be licensed?” I got my brochure and a rain check on an answer to my question.

The Ubiquitous Media Studio

Another highlight of WhereCamp was hearing from Gene Becker about his new project, Ubiquitous Media Studio which will be located in Palo Alto. The project is still in the early stages of devlopment but it sounds really exciting. I am looking forward to being involved from East Coast.  If you’re curious where this is going, follow @ubistudio on Twitter to stay updated.

HomeCamp is a home hacking, automation and green technology community that will be gathering in London tomorrow, Saturday 25th April 2009, 10am until 6pm BST (GMT + 1), and in an OpenSim event running alongside for virtual participation, to brainstorm new possibilities for distributed sustainability, creative smart meters, monitoring, graphing and visulaizing energy usage.

More details and videos on the blog. The wiki, which includes signup, is the main portal to all the online activity.

As James Governor notes here:

there has been a huge amount of code and applications released focused purely on using technology for home energy monitoring and automation.  We have an active google group and quite a few videos and content showcasing the various applications and hardware currently being used by geeks to save money and live greener.

Now the challenge is to see how this seedling home energy management movement can really grow into widely adopted distributed sustainability solutions that everyone can use, and participate in.

Both Chris Dalby (see here), Andy Piper, James Governor of Monkchips (see here),  and Tom Raftery of GreenMonk (see here), have posted on tomorrow’s HomeCamp event. So I am just going to add some quick notes, especially to highlight some of what will be going on virtually for those of you, like me, who can’t make it to London.

You can tune in either on the live video ustream, or sign up on ReactionGrid and join the OpenSim event. Also, you can keep up on what is happening on Twitter #homecamp. I highly recommend that you catch Tom Raftery’s talk which will be streamed from Spain live into the London meeting, the OpenSim event on ReactionGrid, and Ustream. Tom Raftery, a leading Green technology analyst at RedMonk (see also GreenMonk), will be picking up, in depth, on some themes raised in his brilliant ETech 2009 presentation, “Electricity 2.0: Applying the Lessons of the Web to Our Energy Networks.”

There will be homecampers dropping in to virtual homecamp in ReactionGrid throughout the day, including Chris Hart (the awesome “girl-geek”@dstrawberrygirl), MiketheBee, and Cminion, who has a number of cool projects to demo, including his energy turbines.  Dave Pentecost (pictured above with his Tweetawatt, Pachube Orb) and I (see our presentation for EarthWeek SL here) plan to be at Virtual Homecamp on ReactionGrid between 9am and 10.30am EST. Dave has done a number of cool energy monitoring hacks including a Pachube link to and from OpenSim.

Also keep your eye on Dave’s blog, The Daily Glyph, for what’s new in distributed sustainability. Dave just posted some great links on Sustainable Interaction, design and work by ITP researchers and others in sustainable use of technology.

Sustainable Interaction | Main / Papers

Sustainable interaction design | Sustainable Minds

Design For the Other 90% | Cooper-Hewitt, National Design Museum

If you are in London, look out for Oliver Goh of Shaspa as Oliver will be at Homecamp in London. As I mentioned in my previous post, Oliver will soon be launching both Shaspa commmunity and enterprise hardware and software packages for “Intelligent Energy Management.”

For a bit of homecamp history, James Governor (picture below from Chinposin), recaps  some of the successes of  the first HomeCamp here.

And last but not least, a big thanks to sponsors, CurrentCost, Greenmonk, Pachube, Onzo, and ReactionGrid,  and media partner theattick.tv who are making the London and virtual homecamp events possible.

]]> https://www.ugotrade.com/2009/04/24/homecamp-2-home-energy-management-and-distributed-sustainability/feed/ 2 https://www.ugotrade.com/2009/04/19/sensor-networks-and-sustainability-connecting-real-virtual-mobile-and-augmented-reality/ https://www.ugotrade.com/2009/04/19/sensor-networks-and-sustainability-connecting-real-virtual-mobile-and-augmented-reality/#comments Sun, 19 Apr 2009 06:32:59 +0000 http://www.ugotrade.com/?p=3381

Today, I did a presentation, on connecting real, virtual, mobile, and augmented spaces to support sustainability, for Earth Week SL, with Dave Pentecost and Jim Purbrick, who presented on Carbon Goggles.

Dave and I focused on sensor networks, open data, Pachube, OpenSim, and sustainability from perspective of, “hack local, think global.”  Dave and I will be picking up on some of these themes of sensor networks and sustainability next week in our presentation with Dimitri Darras at ITP,  NYU, Aprl 24th, 6.30 pm to 8 pm – details here.  If you are in New York City, I hope to see you there.

We got some interesting insights into augmented reality from Jim Purbrick whose Carbon Goggles project prototypes how we can use augmented reality to read carbon identity and to combine well organized, verified data from AMEE – a neutral aggregation platform to measure the “carbon footprint” of everything on earth, with crowd sourced tagging and linking.

Shaspa – “the sensor network system that has it all”

We also discussed, recently launched, Shaspa. Shaspa’s energy management packages connect spaces – real, virtual, mobile and augmented.  Shaspa has been blogged  by Maxping and Virtual World News, so you can read all about it, but the Shaspa device kit won’t be available until next week. Some key features of the Home Energy  package are listed on the slide above.  However, this evening, Dave Pentecost and I got a sneak preview of both the Shaspa commmunity and enterprise hardware and software packages from Shaspa founder Oliver Goh. We were pretty impressed.

Dave: “It’s the ultimate hackable device for energy management!”

Oliver: “Bring us any sensor device – with documentation, and within three days we will put a driver into Shaspa.”

Oliver is on the right and Dave on the left in the picture above. The picture below shows Shaspa in OpenSim. Oliver and I will be attending the 3D Training, Learning and Collaboration Conference in Washington, DC, next week.

Links

Here are some of the links that came up in the presentation as many people asked for them to be published. Dave also has them on his blog.

SLIDES on GOOGLE DOCS:
Earth Week SL Presentation, April 18th, 2009 – Google Docs

Pachube, sensor networks

Dave’s blog covering Maya archaeology, jungle ecology, and technology

Maya Frontier, Usumacinta River videos

Collapse

Arduino

Pachube – tutorials

Pachube Apps -

Arduino-SL-Pachube data site

SL to Pachube site

Dave’s Danger Shield – Pachube tutorial

TweetaWatt site (LadyAda)

Dave’s post on TweetaWatt to Opensim/SL

Peter Quirk’s post on Importing Sketchup into RealXtend

Opensim

RealXtend

ReactionGrid

homecamp

cminion -wind turbines in OpenSim

MiketheBee

Is it “OMG finally” for Augmented Reality?

Smart Planet: Interview with Andy Stanford-Clark

Orange Cone – Information Shadows and Things as Services

ETech 2009 was all about making interesting and deeply socially effective technological interventions in the world. And dematerializing products into services seemed to be one of the most powerful concepts elaborated there to accomplish this.  Mike Kuniavsky in his presentation, “The dotted-line world, shadows, services, subscriptions,” noted:

“There’s great opportunity here to create an ecology of services embodied as robust, valuable, exciting new tools with focused, limited functionality, tied together with item-level identification and wireless networks. Whole classes of things that can enrich our lives and bank accounts are now possible thanks to the way ubiquitous computing interweaves services and devices at an intimate, everyday level….

We now have the technology to create whole new classes of tools for living in a way that is more useful and fun for individuals, more sustainable for society, and more profitable for companies. That way is to recognize the connectedness of all everyday things, and to build on it, rather than ignoring it.”

The picture opening this post is from Mike’s presentation (see Mike’s blog for  a PDF with all of the images and notes (884 PDF), and the original presentation description).

An ecosystem using  item-level identification, wireless networking, and data visualization is evolving that links everyday objects to information about those objects – what Kuniavsky calls their “information shadow.”  Because every object can be uniquely identified and that identification can be associated with a cluster of metadata, it “exists simultaneously in the physical world and in the world of data.”

Mike mentioned Tom Coates’ “Age of Point-At Things” blog post to say that although Tom was talking about TV listings data, the same ideas can be applied to anything that’s uniquely identified. Also, Mike noted, he often references Ulla-Maaria Mutanen’s Thinglink project and her observation about Amazon ASINs to explain this concept which is, of course, closely related to the internet of things.

Until recently, Mike explained, accessing the information shadow was difficult. The world of objects and the world of  information shadows were separated by the difficulty of getting at the information. But now, increasingly:

“we can instantaneously see the world of information shadows as we’re interacting with the world of objects.”

Mike’s is not only conceptualizing these ideas, his company with partner Tod E. Kurt, ThingM, is producing hardware that will enable this vision.

“We’re a ubiquitous computing consumer electronics company, which sounds fancy, but we’re pretty small. We design, manufacture and sell ubicomp hardware.”

ThingM may be small now but they are at the leading edge of huge transformation.  When asked, “How do you see the near-future city working with ubiquitous computing…” Adam Greenfield put it succinctly to Lalie Nicolas for Le Hub’s Ludigo project:

“I would go so far as to say that there will be no area or domain of urban activity that is not somehow disassembled and recomposed as a digital, networked, interactive process over the next few years. Objects, buildings and spaces will be reconceived as network resources; cars, subways and bicycles will be reimagined as on-demand mobility services; human communities are already well on the way to becoming self-conscious ‘social networks.’”

For the rest of this short interview see Adam’s post, and for my recent long interview with Adam see here.

“‘Almost everything in this room is in a landfill, but just doesn’t know it yet.’  This needs to change”

(Tim O’Reilly responding on Twitter to a quote from @AlexSteffen‘s talk)

Chart above from Jeremy Faludi’s presentation Priorities for a Greener World: If You Could Design Anything, What Should You Do? Presentation [PDF]

Interconnecting themes at ETech,  Inhabitat noted, “formed bridges between luminary speakers from a variety of backgrounds, as Alex Steffen, Mary Lou Jepsen, Jeremy Faludi, and others reinforced the need to create repairable, open-source, long lasting products, reveal energy usage, and pursue forward-thinking strategies for a greener tomorrow.” But Jeremy Faludi, a sustainable design strategist and researcher, put the design challenge most directly:

“If you really care you need to dematerialize, turn products into services…”

The idea of data shadows has been a part of the conversation in ubiquitous computing for a long time (since Marshall McLuhan perhaps?).  But, at ETech 2009, it seemed to have come of age.

It came up again and again, in the need to dematerialize stuff that seemed to be part of every conversation, from Faludi’s comments on the amount of toxic mining waste created in the manufacture of one laptop, to Raffi Krikorian’s presentation of Wattzon’s Embodied Energy Database (see slides here), and AMEE founder, Gavin Stark’s presentation, (also see Gavin’s blog on Energy Identity here).

The path to dematerializing the burdensome stuff that spells doom for our environment was not only presented conceptually and in creative solutions to specific problems (e.g. ThingM) at ETech. There were also hands on workshops (see my post on the two I attended) from Maker gurus, who were also often to be found in the Makershed, providing opportunities to experiment with and prototype your own solutions (my hat is off to Brady Forrest and the ETech committee for pulling all this together).

Connecting the dots…

In the wake of an “econolypse,” (neologism pulled from Bruce Sterling’s twitter feed -  @bruces) and on the eve of environmental catastrophe, we may well have, as Adam Greenfield said to me here, “seriously screwed the pooch.”

But that does not mean we should not do everything we can to try to save the day.

And in the serendipity peculiar to a conference, I was talking in the corridor to Gavin Starks of AMEE who is working to create “the world’s energy meter” (on the right in the picture below), and Tony Mak from O’Reilly AlphaTech Ventures (to Gavin’s right), and Usman Haque of Pachube (on Tony’s right) - see my earlier interview with Usman here), when Tim O’Reilly (far left) came by with Steven Levy of Wired  (to Tim’s left).  More on Pachube, WattzOn, AMEE and Path Intelligence and how these projects may connect in an upcoming post.  Path Intelligence like AMEE is funded by the O’Reilly Venture group.

And no sooner had I snapped the photo below, Mike Kuniavsky arrived.

It seemed such an historic meeting, I asked everyone if I could switch my recorder on.

Tim had just been explaining how the concept of “data shadows” fit with something he’d learned from Gavin in a breakfast conversation.  Gavin was talking about what AMEE is learning from smart meter data collected from 1.2 million homes in the UK.  The energy signature from each device is so unique that you can tell not only the make and model of major appliances in each home, but its age.   Gavin is worried about the privacy implications (as we all should be), but nonetheless, you can see the implications for business. Tim framed a vital question: What new businesses are growing in the data shadows?

Tim O’Reilly: Here’s the other member of this conversation I was trying to broker. This is Mike Kuniavsky, Gavin Starks. I was talking in your session about the point he made in his session…Steve Levy from Wired…

Tish Shute: sorry, could you recap the point?

Tim O’Reilly: …just the idea about data shadows, I just think it’s just such a powerful metaphor that every .. and you went on to explain that potential for subscriptions and so on…

Mike Kuniavsky: Yes well what I was saying was that essentially every object that has an identifier associated with it, and there are a number of different kinds of identifiers out there, simultaneously lives in kind of the world of physical objects, and of the world of data. And the identifier links those two.

Steven Levy: Just like Sterling’s Spimes?

Mike Kuniavsky: A spime, it’s related obviously because we’re talking about RFIDs, but I’m really specifically talking about the fact that there is this information shadow that exists out there.

Tim O’Reilly: I think we’ll find it lots of different ways, that was my excitement in connecting these points.

Gavin Starks: My take on it is energy identity – that everything and everybody ends up with an energy identity that is the embodiment of their physical consumption.

Mike Kuniavsky: And I would say, not to argue, I would say that energy comes as part of my information shadow. Like I carry this baggage of data along with me. And whatever data is potentially appropriate can be glommed on to that. And then that can then be carried to something else that can manipulate it. And also that’s true about every object. And now that we have RFID tracking of individual objects, it’s true about literally every object, not just every class of objects.

Usman Haque: There’s a really beautiful story by Julio Cortazar where he uses the phrase “dressing the shadows” and it’s about the idea the shadow is not this sort of flat black thing but we can sort of put things onto it and slowly sort of grow it into something. It’s actually sort of more of a love story. But it’s a really interesting idea that the shadow’s not just the absence of but that it’s kind of the important part of it [for more see Usman’s paper, Dressing the shadows of architecture – which is also available in spanish here.]

Mike Kuniavsky: It’s the Peter Pan Barrie [JM Barrie, the author] thing. When Peter Pan’s shadow gets cut off and Wendy has to resew it back on. Potentially what all of these item level identification technologies are doing is they’re sewing the shadow back to the objects that they came from. And so you’re getting the information.

Gavin Starks: It’s like the two and a half kilo Macbook which has a 460 kilo carbon shadow.

Tim O’Reilly: It’s just a very powerful concept. That’s all I’m saying. I think it’s a metaphor that as soon as you have it, it makes it very easy to understand and to see a whole lot of things. So I’m very fond of it. Already it’s my new favorite toy. And it is great running into you all in the same place in the hall so I could introduce you all.

Image from Mike’s ETech presentation

“To create these new experiences we need to think about the design of both digital devices and infrastructures differently. We need step back from standalone tools and think about what service those tools deliver, then construct new avatars that fit better into people’s everyday experiences. We also need to step back from our infrastructural products and think about what services they enable. The electrical grid did not first start out as an abstract electrical grid in South Manhattan; it started as a way to deliver electric light. The electric bulb was not a standalone device, it was an avatar of Edison’s light delivery service and it was, first and foremost, designed to solve a specific problem for a large consumer market. Only then did the infrastructure it created expand to solve other kinds of problems.” Mike Kuniavsky’s ETech presentation, 2009

Talking With Mike Kuniavsky

Mike Kuniavsky and Elizabeth Goodman playing Bocci after ETech

The conversation with Mike began with a discussion about how to encourage participation. Usman Haque was present but he was called to lunch shortly.  The question of encouraging participation in deep social change was another recurring theme at ETech.  And, as Mike noted in his presentation:

“The design of these avatars [Kuniavsky's term for objects that are closely tied to services] is quite challenging. They can’t really be as personalized. You just can’t pimp your City Carshare car. You only get one kind of bike in the Call a Bike program. That’s an important problem to solve. We love to have our stuff be ours. However, the same technologies can bring that, too. Our key fob can bring our whole world with us, and whether sit down in a minivan, on a chair or in a plane we can bring our world with us. The thing can become our preferred colors, with our favorite music, and a picture of our loved ones on the dahboard, desk, or wall. Is it the same thing as owning it and   leaving your stuff in it? No, but it’s closer.”

Moreover:

.. objects have to change at a fundamental level. They have to be designed differently and they have to be described and discussed differently. The “owner’s” relationship to the object changes. The very idea of ownership changes. The solid object grows a dotted line that is filled-in as-needed, when-needed, and with the features that are needed. This is not the same thing as renting or co-ownership, its anytime/anywhere nature-enabled by the underlying technology makes these new service objects fundamentally new (Kuniavsky’ presentation at ETech).

Elizabeth Goodman’s brilliant presentation at ETech, Designing for Urban Green Space, discussed a study of urban green space volunteership as a way “to rethink urban green space as a spectrum of places with varying types of ownership and management.”  Mike began the conversation by citing Elizabeth’s work.

Picture from Elizabeth Goodman’s presentation.

Mike Kuniavsky: Well what I was saying [re participation], citing my wife Elizabeth Goodman’s work …She did all this work at Intel on people’s health practices and the issues [around] instrumenting people’s lives in order to produce behavioral change and the problems with that.

The question is how do you, sense to encourage, rather than sense to punish, when all the indicators are going down, like economic indicators, ecological indicators. They’re just not going to be going up perceptibly in a very long time. You don’t want to discourage people. The way to create behavioral change is not to essentially keep punishing people for the past. And so I don’t know if I have a good answer for this, but there is this entire kind of thinking about how do you encourage people to keep doing things even when the actual easy-to-measure indicators like the first order indicators are all pointing down. It’s the classic thing about how do you get people to stay fit even as they’re aging. They are never going to be as healthy as they were when they were 50 again.

Usman Haque: I think you really hit on it when you said it’s not about the first order but about the second order measurements because that is exactly the kind of thing you want to change. It’s not that you want to stop it from falling because sometimes it’s impossible, you want to slow it’s rate.

Mike Kuniavsky: Exactly. You want to slow the rate because at the bottom maybe you can start looking at the first order indicator. But you can’t look at the first order indicator while things are going to hell. And so you can just say it’s less bad than it would have been. And figuring out how to take the first order sensory data and turn it into this kind of second order data that might be helpful for actually creating behavioral change, because ultimately that’s what all of this is talking about.

Tish Shute: This discussion about behavioral change wasn’t elaborated in your presentation was it?

MK: I presented on essentially  the combination of being able to identify individual objects and the idea of providing services as a way of creating things… the servicization of things …turning  things into services is greatly accelerated by network technologies and the ability to track things and what leads this to the potential of having fundamentally different relationships to the devices in our lives and to things like ownership.

Like we now have the technology to create objects that are essentially representatives of services – things like City Car Share.  What you own is not a thing but a possibility space of a thing.  This fundamentally changes the design challenges.  I am pretty convinced that this is how we should be using a lot of these technologies is to be shifting objects from ownership models to service models.  We can do that but there are significant challenges with it. What is happening is that we have had the technology to do this for a while, but we haven’t be thinking about how to design these services.  We haven’t been thinking about how to design what I call the avatars of these services – the physical objects that are the manifestation of them, like an ATM is the avatar of a banking service.  It is useless without the banking service it is a representative of, essentially.

If you imagine a this as an abstract idea, the ATM pokes out of [the service and into] a specific thing, but so does the bank tellers and so does the web site.

TS: It seems like this is a major shift in how we conceptualize our economy, culture and even government – what are the avatars of government?

MK: I think change in government is very hard. The example I have been using is the light bulb.   Start by solving a problem. The interesting thing about lightbulbs is that it was not the invention of an incandescent filament that glowed in a vacuum–that had been invented long before–it was the system that it was part of.  And that is was part of a much larger design project that was created specifically for delivering the service of light to lower Manhattan in 1884.

TS: The grid hasn’t changed since Edison right – one of the earlier speakers mentioned this, that if Edison came back now he would say, “the grid is where I left it.”

MK: My point is that he wasn’t creating an abstract electrical grid, he was solving a problem by creating a system that had as its avatar – as its end point this bulb. But the bulb is actually not the system, it is merely the end point.

As we are thinking about the capabilities of these technologies my argument is we have to be designing service systems along that model.

TS: Web services?

MK: Not just designing Web services.  I am a big fan of thinking about digital tools outside the context of general purpose computing devices. I consider laptops general purpose and I consider phones general purpose.  Yes originally the handset started out just as a phone but now it is essentially a computer terminal and now you have netbooks and netbooks are essentially this halfway point between a phone and a laptop because now you are going to get net books with G3 cards.  Essentially it is already a big phone.  Those are general purpose computing platform, and I am not very interested in those.

TS: What motivated you to make that move in your thinking?

MK: I thought it was very narrow kind of thinking.  I thought that the costs of computing represented by the technologies in the middle of the Moore’s Law curve – rather than on the right – that the cost of that had dropped so far that it seemed we could be making all kinds of devices that had information processing as part of what it is without being general purpose computing platforms.

The ipod is a good example.  The ipod is a computer and you can run linux on it. It has more computing power than an computer did in the seventies. But who cares? The point of it is that you are using that power to solve a problem. You are applying the capabilities of information processing to solve specific problems. I have actually worked on infrastructural stuff. Twenty years ago I was associated with some early distributed computing stuff, then I did ten tears of web site design stuff, but i am essentially done with that. Because what I am really interested in is  creating new kinds of tool, new classes of tools that use information processing as the core of what makes them interesting and valuable.

TS: Do these tools have to leverage networks to be useful?

MK: No I think it is possible to use information processing in a small scale without having to be online all the time.  That is another one of the big toolboxes.  It creates a deep shift in the capabilities of what you can do if you have a network.  But the network can be really, really low bandwidth and simple for it still to be useful. You get these things that wake up once a month and spit out a packet with their telemetry.  And they are incredibly valuable but they are not what you would normally consider to be an always-on device.  It changes what they can do very fundamentally.  But it is not this thing that requires there to be blanket wifi.

You can have devices out there and this is the sort of a cliched example but the guy riding a bicycle around with a wifi access point in rural area where you have no infrastructure to do it otherwise.  But you have a little computer in every area and as he rides by they will exchange some data.

You don’t have to have fibre at the curb to really, really make interesting deeply socially effective technological interventions in the world.

Aaron Straup Cope, Flickr, Tod E. Kurt, and Mike Kuniavsky – discussing The Shape of Alpha (more on this upcoming!)

MK: What we are trying to do is to do that.  We make a BlinkM – we make hardware – you saw my business partner Tod E. Kurt, he does all the heavy engineering and I am the guy who waves his hands around a lot and sends faxes.  We came out with our first product a year ago was a smart LED.  It is very simple RGB LED, it has a microcontroller and the microcontroller has firmware on it that kind of abstracts out the complexity of incorporating LEDs into a hobbyist product.  So you can do arbitrary colors, so it can do smooth fades between any two points in RGB space, you don’t need to know anything about Pulse Width Modulation or even microcontrollers.  You don’t have to know anything about anything except a little bit about electricity to use the thing. [In addition to BlinkM, BlinkM MaxM – the smart LED, Thingm has developed prototypes for other products such as the WineM RFID wine rack and LoveM LCD chocolate box.]

TS: I made a LilyPad enabled Tshirt yesterday, if I used your LED what difference would that make to my Tshirt?

MK: You could have the LED without changing the circuit at all, you could have it blink in any pattern, be any color, fade between colors. With our new one which is bigger than the old one, we actually have inputs. You could stick a wire on it or weave it into your shirt, and when you touch the wire it would change the behaviour of the LED.

TS: Nice, you are giving me even more incentive to finish my T-Shirt. I noticed that Tim O’Reilly was connecting you to Gavin Starks, CEO of AMEE just now, and Usman Haque of Pachube.  What is the connection between you work on Thingm and these projects?

MK: I think what Gavin’s doing, as I understand it from Tim, he is essentially creating this new kind of sensor network that monitors electrical usage and allows you to feed it back. What that does is that creates a new kind of data in the data shadow of your house, you refrigerator or whatever. It suddenly grows this extra lobe out in the data world that then has these new capabilities that can be attached to.

TS: In terms of what you do with ThingM how are these ideas expressed through BlinkM?

MK: We’re still building stuff that’s on a slightly lower level, components. Our corporate goal this year is to make our first product, a stand alone solution to something. One of the easiest things you can do with our technology right now is you can replicate an Ambient Orb in about ten minutes. You could tie into their work. But you could also tie into it in a more subtle way where you could make lights smart so that when the net electricity cost goes above a certain threshold the lights know to dim or to turn off. And that can be dependant on how people use them. So rather than having a light you essentially associate a function or purpose with a light. Then the light knows based on electricity usage when it’s purpose has high priority enough to be on.

Not all of these ideas pour into our products, we can only afford to make LEDs.

TS: Still it is amazing how ThingM really is a flagship for what is big and important shift in the way we can relate to stuff. And what about Usman’s Pachube. Where does ThingM fit with that?

MK: I see Pachube less as a monolithic service than as a standard for device communication. Essentially it’s a proposal for interdevice communication, and potentially an easy way for people to define the way devices behave within their own personal ecology of smart devices. It’s something that’s in the early stages, and I think the barriers are not technological, the barriers are social. The barriers are understanding what this is for and why to use it. It’s not about will it work. It’ll work.

Image from Mike’s ETech presentation – original image source: Yottamark

“You can, hypothetically, look at any object and know where it was made, what it is made of, what your friends think of it, how much it sells for on Ebay, how to cook it, how to fix it, how to recycle it, whatever. Any information that’s available about an object can now be available immediately and associated with that object.”

“Connect it with location information and you have Location Based Services for anything. This is Cabspotting by Stamen. As Tom Coates says, once we have a handle, you can throw the data around.” (Kuniavsky)

More to come on Stamen Design later! Tom Carden (Stamen Design) ran a workshop at ETech 2008, “Live, Vast and Deep: Web-native Information Visualization,” outlining the process of taking a real data set from an online API (such as Flickr or Dopplr) and shaping it into an informative, beautiful, and useful interactive graphic presentation and this year, Michal Migurski (Stamen Design), Shawn Allen (Stamen Design) gave a workshop on “Maps from Scratch: Online Maps from the Ground Up.” Eric Rodenbeck, founder and creative director of Stamen Design, presented on, “New Data Visualization: Reaching Through Maps.”

The picture above is of Eric Rodenbeck and Shawn Allen playing Bocci.

“Come to ETech; Experiment with Physical Computing and RFIDs” said Brady Forrest. The ETech RFID tag that I activated at registration is a gateway to several internetworked worlds.  It allows you to check into pulse stations to tell you about people with similar interests to you based on your traffic movements around the conference.  There is a photo booth that allows you to upload photos to Flickr. And even a Fortune Teller from Josh and Tarikh of Uncommon Projects (makers of the awesome Yahoo! geo-bike) that will be arriving tomorrow, and more.

But the first day of ETech 2009 was packed with hands-on workshops.  And I actually managed to make, in Tom Igoe’s, Hands-On RFID for Makers workshop, my first RFID to web interface that could read Etech’s elegant RFID tags (also see my photo set on Flickr to get a glimpse of the action in the workshop). Amazingly it worked perfectly first time (I did have help from the very patient executive editor of Maker Media Books, Brian Jepson. And Tom Igoe’s step by step instructions on his website are invaluable (picture of Tom Igoe below).

It was very exciting to actually get hands-on with the Arduino open source electronics prototyping platform, and Processing – a very accessible language to do dynamic and interactive graphics for screen-based media, . You’ll know how much I love to write about these things if you have checked out some of my previous posts.

My “build” is sitting on the right of my workshop neighbor Ahmed Riaz (ebay) in the photo opening this post. We shared power supplies and a great discussion on interaction and user experience design (see my previous post, Towards a Newer Urbanism: Talking Cities, Networks, and Publics with Adam Greenfield, for an idea of some of the topics that we touched on). We also discovered a shared interest in User Experience Design sketches – see Ahmed’s blog here and his flickr stream for his project on UX sketches. I have reposted here one of my favorite UX sketches done by an eight year old, especially for Ahmed.

If you look closely at the picture below you will see that Ahmed’s RFID to web interface has read my Etech RFID tag and pulled up my Etech Conflink profile and picture.

In the evening, Tom Igoe announced during his Ignite presentation that an Arduino MEGA will be available next week – more pins, more ports, more memory.  I can’t wait to see what people come up with for the MEGA, especially as Pachube (another favorite project of mine – see my interview with founder Usman Haque here) is designed to work with Arduino and Processing.

I think I’m hooked on Maker culture. I can’t wait to check out the Etech Maker Shed that opens today. I got a feel for the excitement of rapid prototyping in the morning doing the LilyPad Electronic Fashion workshop with Leah Buechley, a brilliant and patient teacher. Leah is checking out Raffi  Krikorian and Tom Igoe’s progress in the photo below.

There was some big talent in the Lilypad workshop. The Wattzon team, Raffi  Krikorian and Jeremy Cloud, and Wattzon-phile Tom Igoe stitched and ironed (see my Flickr stream here), and helped out noobs like me. Possibly we will see some programmable T-Shirts displaying carbon footprint data. But certainly you can use Wattzon to compute the embodied energy data of all the Lilypad components.

I was a little hampered by my appalling needlework skills. But Maker culture came to the rescue when I twittered about needlework phobia and LilyPad love. @dpentecost replied in seconds inviting me to “sew and tell” at a NYC Lilypad meetup when I return to NYC. Below is a picture of Jeremy Cloud’s excellent stitching with the challenging silver plated thread.