I am super excited to be speaking at SXSW Interactive 2012, as part of Will Wright’s Stupid Fun Club, on “A Lifestyle with a Gaming Sense.” Michael Trice just did a post on our session for SXSW.com, Screen Burn Panels at the Palmer Presents Will Wright’s Stupid Fun Club. The photos of Will Wright, Tish Shute (me!) and Peter Swearengen are by Anya Zavarzina. Thank you Anya for such great photos!

I have been too busy to blog much lately, but there is a lot to unpack in future posts in my quote in Michael’s SXSW post!

“Really we’ve entered a new era where the world has become a platform for storytelling and the goal is to turn everyday life into an opportunity for play, relatedness, and new forms of autonomy and fun. We’ve now come to a point where software has moved out of the computer and into the world. Rather than viewing this process in terms we’ve already grown out of, like gamification, we view this as an opportunity to explore everyday activities as possibility spaces.”

For the complete post, including Peter Swearengen of Stupid Fun Club on StoryMaker, see here http://sxsw.com/node/9969

I’m really looking forward to the O’Reilly Strata events that are coming to NYC in a couple of weeks. I’m fascinated to see where the art, science, and business of data has gone since February, when I attended the first Strata Conference in Santa Clara – a sold out event imbued with an awareness that this was an important gathering of cognoscenti working on the next big thing.

Strata in New York City is a sequence of events,  Strata JumpStart, Sept. 19th, and then The Strata Summit, “The Business of Data,” Sept. 20th & 21st, and followed by the Strata Conference, “Making Data Work,” Sept. 22nd, 23rd.

“The future belongs to those who understand how to collect and use their data successfully.”

Below is a transcript of a conversation I had last Friday with Strata Program Chair, Edd Dumbill about some of the highlights of the schedule from my perspective.  However, I highly recommend taking a good look at all that is planned through the three events because there is a depth and breadth that could not be covered in one conversation.

The video opening this post is from visual.ly.com – a start-up making it easier for people to create, explore, share, and promote data visualizations and infographics.

Talking with Edd Dumbill

Tish Shute: It seems a dialogue between the art of data and the science of data is going to be center stage at Strata NYC, and there will be much discussion about story telling with data.

Is that observation correct or is there something else going on there?

Edd Dumbill: No, I think that’s a great characterization. For the Summit, the core realization for me has been that when you have these tools for getting value from data and when you can drive what you’re doing by data, then actually, the biggest consequences are human ones, and they are organizational ones, and they are strategic ones once you have the technology in place.

So what the summit is doing is really looking at how, in a variety of industries, governments, and within disciplines within those, how the amount of data, the ease of which it can be communicated and mined is changing the way industry is shaped.

Tish Shute: Also, I noticed  that the Strata Summit Schedule (Sept 20th & 21st), and even through to the Strata Conference (Sept 22nd & 23rd), has more of an emphasis on pop culture; sports – baseball, dating – OKCupid, and Narrative Science, all have a place on the schedule, for example?

Is this the culture of New York City being reflected – interests in media and marketing, or is there something else going on?  Has the data tool stack matured since the Strata Conference in Silicon Valley at beginning of the year?

Edd Dumbill: Yes, there’s certainly a different flavor to the event because we’re in New York. And, yes, the tool stack has matured, but it is, by no means mature, and the maturity’s only coming at the lowest level.

I think there’s many years left in maturing the tool stack. But one of the beauties of big data is that once you have the data together, the algorithms to get value from it initially are pretty simple.

So, focusing on the stories of success of being data driven, particularly in the Summit, is important to us because the two questions people are asking are, “One, I’ve got data. Two, What do I do with it?”  We don’t need to make the argument that data is important anymore. But we do need to demonstrate what you can do with it.

The data isn’t necessarily big; it’s just there. It’s about having an analytical approach to your business that compliments your intuition, and compliments your vision.

“One of the most powerful ways of presenting data to people is in a story,” Edd Dumbill

Tish Shute: Yes I can see the emphasis in the schedule on how to tell meaningful stories with data. Narrative Science seem to be doing something very interesting re turning data into stories?

Edd Dumbill: Yes. They absolutely fascinate me with what they do. There’s this kind of hierarchy and sort of chain of needs right now where business is going, “We need data scientists. Find me data scientists. Train me data scientists. Hire me data scientists.” And the data scientists are all going, “I need visualization. I’ve got this data, I now need to turn it back into a story that’s going to be useful to people or provide interfaces that are going to help people understand and explore this,” because it doesn’t scale to have to have an interpreter all the time between the data and the results.

You need to be able to present it in a way that means something to people.

People can look at a graph and get many things out of it, maybe not even get anything at all out of it if they are not used to it. But particularly for digesting certain kinds of high-level summaries and results, if you can put the data back into prose, it makes it very accessible to people.

Tish Shute: Natural Language Generation from data really opens up so many possibilities..

Edd Dumbill: Yes, it’s interesting. I think it’s a very novel use. A lot of people would consider that the end result of their data was a spreadsheet or a graph that they are processing.

But if you turn that back into a story, I think there’s a lot of potential of helping executives understand what’s going on. It makes it possible to use language to understand the results.

Tish Shute: I am really excited to see the emphasis on stories, data design and visualization, and the way we experience data is as much part of The Strata Summit and The Strata Conference as some of the more hardcore big data challenges and analytics stuff.

Edd Dumbill: Yes. We are definitely ramping up on visualization. And I think that’s going to become more important. Having a fundamental grasp of how to use graphics and charts is still incredibly core to what we’re saying. But I’m also interested in ways that go beyond, because at least 50% of the point of visualization is to help people understand the dynamics of the data, to really augment their senses with the results of the computation.

You know, the people who are some of our best leaders, the ones who know how to ask the right questions of the data, have a sort of indefinable fingertip feel that you get for numbers when you live around them for a while. And anything we can do with interfaces to accelerate this is going to be very beneficial, whether it comes to being visual and flying through the data or hearing it in natural language.

Tish Shute: Have I missed anything in that in terms of what you’ve got on the schedule re visualization? VisualizingData.com published an ideal schedule from the visualizing data perspective. But have you added anything recently?

Edd Dumbill: Well, there’s one event which isn’t actually listed on the schedule yet, which is on Tuesday night. There’s a venue called EyeBeam in New York; we’re having a visualization showcase that evening. So there will be stuff to walk around and then a few talks, really from some of the most interesting companies doing viz and viz approaches. So that’s not up on the schedule yet, but that will be in addition. It gives a nice focus on Tuesday night.

Tish Shute: Oh, that’s super awesome. I’ll definitely go to that.

Tish Shute: I am very interested in mobile social communications and augmented reality – especially augmented reality that feels different, not just looks different, as Kevin Slavin puts it.

I am excited to see people thinking about data not just in terms of visualization, but in other ways too that we can feel it through our secondary senses as well (see Mike Kuniavsky’s talk at ARE2011, “Somatic Data Perception”).

Edd Dumbill: Yes, absolutely. That is where we view this as going. I will be incredibly depressed if I’m still looking at the world through a glowing rectangle in 10 years time.

Tish Shute: Yes, it would be! I am looking forward to see the new data start ups too.

Edd Dumbill: Yes, there are a variety of interesting startups, that I feel are particularly important in the data space. Media Sift and Data Sift, for example, Data Sift is doing a lot of real time processing on the Twitter fire hose. They provide real time analytics on Twitter, which I think is very important.

Tish Shute: In terms of using data to provision mobile experiences, real time is massively important, isn’t it?

Edd Dumbill: Absolutely. Yes.

Tish Shute: But real time data is still a big challenge, isn’t it?

Edd Dumbill: Yes. I mean right now, our focus on real time is probably at the technology level. Looking at real time, people are kind of building out the frameworks, companies like Media Sift and Data Sift creating parts of the experience.

And yes, our Where 2.0 conference will be focused more on the mobile experience.

Tish Shute: Re mobile experiences, I am very excited about Infochimps and their new geo APIs, and sensor data is becoming such a big part of the picture now too. But the Kinect has also opened up a whole set of possibilities for the future of sensor data!

Edd Dumbill: Yeah. I still think Kinect is probably one of the most exciting things going down because of the democratization of that kind of capability. Interesting things happen when the sensors become cheap, right?

When alongside a little camera in your iPad you have a Kinect sensor equivalent. That’s become extremely interesting because everybody has it with them and can do things based off it.

So the things that always fascinate me are when it becomes cheap and hackable.

Tish Shute: And if Kinect went mobile, that would be exciting?

Edd Dumbill: I think it’s entirely likely in the next couple years, yes.

The more sensors we can start instrumenting our mobile and personal devices with, I think it’s going to always result in some much more novel uses that we ever dreamed of.

Tish Shute: There was a lot of hoo-ha about Color when they launched this year. They were unable to capture a user base, but if they had issues of privacy might have come to the fore because they were really collecting more sensor data than any other app, right?

We are still waiting to see a breakthrough app in that area in terms of using all the phone sensors in ways that will really enhance a user experience rather than just the aims of data mining, aren’t we?

Edd Dumbill: Yes. I think this is one of the things where, in parallel, we’re really learning out the social and privacy implications of this kind of technology. It seems to me the focus has shifted from the tech in the second half of the year too. Frankly, everybody getting kind of freaked out about the amount of data that’s being mined and, you know, what’s acceptable use for that.

But on a slightly more prosaic level, there are some rather fabulous things being done. If you look at the Google Maps navigation experience on an Android phone. For instance, there’s some very practical applications of sensors collecting data with traffic and a variety of other augmentations going in that to actually do something useful.

So maybe we’d like to think we carry our sixth sense around with us in our pocket, and maybe we will. But we certainly can in our car right now with all the automatic rerouting and so on. That’s slightly more prosaic, but I think a lot more significant in terms of a pattern of how that can be applied.


Tish Shute: One of the Startups that really excited me in February at Strata, Santa Clara was Singly and The Locker project. They are really thinking innovately in the area of putting people at the center of their data.

I am looking forward to seeing the fruition of that work. And, while I’m enjoying Google +, it seems, we are just sort of holding up our hands and saying, “Well, there’s only one business model for data, and that is a centralized Fort Knox,” isn’t it? Or is there something that I’m missing?

Edd Dumbill: You’re right. I mean I think Google +, for instance, is rather the walled garden is a hedged garden. You know, there is a certain barrier there that I think is more about the fact that you need to put certain barriers up to actually create a decent user experience in the first place. I think user experience is one of the BIG problems with open data, and private data, to be honest.

There’s a reason we are not all writing PGP encrypted emails to each other, right? Because it’s so hard to make a UI for encryption that’s safe. Most people don’t use passwords properly. And I think a lot of the same user experience considerations come into this whole data thing.

Facebook can get away with anything they want to because have you ever tried using their privacy settings? Google, I think, more than anybody has tried to address this issue using sensible defaults, making the explanations clear. And they probably succeeded for a geek tech audience.

So I honestly think, probably, Locker’s biggest challenge, in that kind of approach, is definitely UI and giving the concept to the users so they can understand it.

But there’s certainly a very useful contribution to this conversation.

I think there are parallels in blogging, actually. There is a case where people have information they want to disseminate. And do you choose to do in on your own website, set everything up, publish for yourself, host for yourself, so you have complete control, or do you cede, for convenience, control to Blogger or Tumbler, knowing that you are being monetized somehow and that you’re playing in somebody else’s walled garden and don’t have that control?

So I haven’t really expanded that thought too much, but I think there’s something there in following that along and seeing where that actually leads.

But, you know, there is a whole technical challenge as well.

I really like the idea of being able to give permission to people. Being able to say, well, “I’m engaging you to do X,Y,Z in return for such and such. That seems like a good bargain to me. Giving up my data is a decent bargain for the services I’m getting back.” I mean that’s generally the contract we make in real life with people anyway.

That’s another thing re Google+, –why it’s a promising approach. At least in their rhetoric, they’re trying to say, well, “We’re trying to model this on the real life economy, the economy of real life interactions.”

Tish Shute: Yes. Any movement towards saying, well, “I’m not just collecting your data randomly, I’m collecting this data because I want to give something back to you that will enhance your interactions,” definitely feels like an improvement, doesn’t it?

Edd Dumbill: Yes. I think that bargain is clear. I’m just fascinated by who could be trusted and… I do actually wonder if there will be some kind of, rather than necessarily everything being decentralized like Lockers suggests, there might be an idea of a variety of inter-operating, trusted identity brokers. People who we would actually trust. Banks, right? We do that right now. Banks are pretty much our identity brokers. Who knows?

Tish Shute: I think, that is where the Locker project’s going with Singly, isn’t? Isn’t Singly the trusted broker for the Lockers, right?

Edd Dumbill: Yes. Now the question is whether you trust a startup with that or whether you’re going to trust… I mean, who knows? Trust levels are at such all-time lows with everybody right now. People in America won’t trust the government. I think Google are probably one of the most trusted brokers out there online.

Tish Shute: Perhaps, that’s interesting, isn’t it?

Edd Dumbill: I did write a piece, which kind of speculated that Google may become some sort of center brokering of social information and kind of a platform.

Tish Shute: Oh, yes, “Google+ is the social backbone” – a very thought provoking piece! It deserves an interview on it’s own!

But back to the Strata schedule! I notice you have DePodesta doing the Moneyball talk, right? What’s the 2011 twist on Moneyball?

Edd Dumbill: I think the twist on that is that the’re a lot more people can play now, really, which is why we’re having Strata in the first place. That 10 years ago the people doing this kind of stuff are McDonalds and Walmart and sports teams. Everybody, where there was large money, they could afford to gather the data. Maybe they could try this service out in making decisions based on it.

Well, we’re now in a very instrumented society where every business, every person has instrumented data about their interactions. I think the kind of resistance and dynamics and opinions that Moneyball brought up are the ones that people are going to be facing again right now as they seek to be more data-driven in what they’re doing.

It’s also very interesting to know 10 years on, what do you think? You’ve had 10 years of this, of sort of sabermetrics and so on. Have you matured in your view, have you softened?

What I’m endlessly and ultimately fascinated by is, where does this fit in the decision process and in the organization tree? Where does it mesh with vision?

Steve Jobs achieved it perfectly. He had vision and all kinds of things for his products. But Apple succeeded through a relentless operational efficiency. Absolutely relentless in their suppliers, their supply train, their manufacturing lines down to their detail. They are an utterly data-driven, process-driven organization at the same time as melding that with vision, design values and good quality. That’s a case where it worked together.

I’m eager to try and tease it out, figure out how that really works and how those things come together.

Tish Shute: And that’s another thread I see being explored at Strata, NYC. It’s not human versus machine or machine trumps human, but it’s human with machine. This is another theme, isn’t it?

Edd Dumbill: Exactly. We all operate by feedback loops. Really, what machines are doing enables us to get better quality data and in a tighter feedback loop.

Tish Shute: One feedback loop that we’re finding machines very useful for is understanding how we feel. I think that’s really interesting.

Edd Dumbill: Yes. I’m very fascinated by all the quantified-self stuff and where that can take us. At the end of the day, we have a very personal little organization to deal with, which is ourselves.


Quid: Building Software and Mathematical Solutions 
to Simplify Complex Decisions

Tish Shute: Yes! But the thing is we don’t understand ourselves in isolation, do we? I am definitely going to attend the session by Sean Gourley, CTO of Quid, on semantic clustering analysis. It seems like sentiment analysis is going big-time now, isn’t it?

Edd Dumbill: Yes. I mean, sentiment analysis is actually becoming a checkbox feature in databases now. The latest release of Greenplum has it built it. It’s that kind of level of feature that people want as social data is so important. Of course a lot of this is being driven by marketing and advertising.

Tish Shute: Yes but even re marketing data story telling has been taking some interesting and quirky turns hasn’t it?

Edd Dumbill: Yes, absolutely. I think there’s a lot of interesting research ahead of us there as well.

OKCupid Trends

Tish Shute: OkCupid is a very interesting example of data story telling that leverages our desire to know ourselves, and ourselves in relation to others.
.
Edd Dumbill: Yes. I mean they’re an example of a shift that’s happening in the PR industry, actually, which is companies understanding that telling marketing stories with data is very, very compelling. OkCupid really used that to hit well above their weight. Of course they got acquired as a direct result of that and their profile.

Tish Shute: I know OKCupid got acquired by Match.com, but you were saying they hit above their weight by using this analysis? How did that work?

Edd Dumbill: I think a lot of it’s down to their blog. That they analyze these things, publish them on their blog. It got a lot of attention, generated a lot of media stories, which brought them to Match.com’s attention. There’re millions of – well a large number of dating sites. But they differentiated themselves through the smart use of their data.

Tish Shute: Data and Games is an area I am very interested in. Zynga changed the game with game analytics and social games. And now we are seeing Rovio partner with Medio for analytics, (see Green pigs and data). But I noticed that you don’t have games as a strong theme on the schedule?

Edd Dumbill: I think you’ll see more of that on the West Coast to be honest. It’s not that we’re not interested. I just feel that the center of gravity to that topic is probably back on the West at the moment.

Tish Shute: So what’s after Zynga in terms of game analytics? A nice easy question!

Edd Dumbill: Sure. Let me predict the future for you.

Tish Shute: Yes please do!

Edd Dumbill: I don’t know, to be honest. One of the very interesting things about games is that it helps us understand the real world by modeling and playing around. I’m highly fascinated to see some more of those things played out through real life actors. There’s been some examples right out of Scavngr and whatnot. But if any of those techniques can really start to make a way into mobile technology, that’s one interesting thing.

What lessons can we take from what we’ve actually learned in game analytics that are reproducible and useful elsewhere?

Gamification is a bit of a trend right now. I am slightly skeptical… But I am fascinated by a lot of systems that are having these game elements added to them. And so the second question is, if you’re having games added to things, like losing weight or saving money or writing a book, I’ve seen that too, what can you apply from the analytics world on top of that, and learn about systems and tweak them?

I don’t have that good of an answer for you. How my game is, is not steeped in that. But I am aware that there’s probably a lot of progress in games that has yet to be applied anywhere else.

Zynga and whatnot, is kind of a space race, isn’t it, to monetize that. Space races generate technologies that can be applied in a variety of places.

What are the spinouts of game analytics that we can actually use elsewhere?

“These Bloom Instruments aren’t merely games or graphics. They’re new ways of seeing what’s important.”

Cartagr.am by Bloom

Tish Shute: Last February, at Strata, I was very struck by the new work by Ben Cerveny and Bloom on “pop cultural instruments for data expression” (also see Ben Cerveny’s talk at ARE2011).

Edd Dumbill: Yeah. I love every time the visualization comes onto a tablet….there’s an interesting back channel there.

And Google has done this in extreme to add to their great advantage. There’s a potential when you read an E-book, or you interact with the visualization of a tablet, that it can learn from your interactions.

If you read an E-book, and the book is instrumented and sends stuff back, then the book can read you at the same time that you’re reading it. That kind of collective intelligence can then be harnessed.

So what if Bloom’s pop culture visualizations are instrumented so that they know how people are using it? Well what can they learn about that? About either the quality of the visualization, about what’s interesting to data and back at the same time?

This is what the fundamental principles I think even of Web 2.0 and definitely in this era of big data that we’re in, is that the secondary signals, the exhaust from any electronic product, can be incredibly valuable.

We know that every time you run Google you are probably a part of at least one experiment that they are running to determine an optimal, and optimize their product through that. And how can you turn this up to generalize that out?

Tish Shute: I agree.  This is at the core of the art, science and business of data.  I hear your phone ringing, but do I have time for one more quick question?

Edd Dumbill: Oh yes.

Tish Shute: So it sort of follows on from my previous question.  The relationship between the crowd sourced intelligence and machine intelligence has played a huge role in making data work and  solve real world problems – Crowd Flower, for example.

Where are we at now with this relationship between crowdsourcing power of, for example, Crowd Flower and Mechanical Turk when combined with machine intelligence. Is there anything new going on here?

Edd Dumbill: What we’re actually starting to do is learn where to apply these tools. We’re reaching a point of understanding what crowd-sourcing is for, how to better design crowd-source tasks and so on in innovative uses.

One of the things I am particularly excited about is Natala Menezes who was at Amazon working on Mechanical Turk, she’s now moved to a company called GigWalk, which is a Turk platform that’s mobile.

So if you want to assign tasks that depend on people being in particular places and being able to do particular things, this is a platform for turking using that, which I think is fascinating. That’s definitely a new approach.

Tish Shute: Yes GigWalk is awesome – I saw that Photosynth is partnering with GigWalk. That is interesting – perhaps a step towards strong AR! ( see Read Write World and Blaise Aguera Y Arcas’s work on Photosynth was big news at ARE2011).

Edd Dumbill: Natala will be talking about GigWalk. I think the session is called quirky crowdsourcing. I want to call it Quirky Turks.

Tish Shute: [laughs] I like that.

Singly’s appearance at the startup showcase at Strata 2011 this week has excited thought leaders across the web since the story got out. Singly is a new startup that exists to provide oxygen and commercial support to the open source Locker Project, and new protocol TeleHash.

With some wonderful serendipity I met Singly on my first night at Strata.  The next day, I talked in depth to Jeremie Miller and Simon Murtha-Smith, two of the three Singly co-founders (see later in this post). I also had the opportunity to ask Tim O’Reilly, Alistair Croll and Roger Magoulas for some of their thoughts on the significance of this project (see below for their comments).

It was a real – pinch myself in case I need to wake up from a dream experience – for me, to stumble across Jeremie Miller with Simon Murtha-Smith sitting behind a hand written sign demoing Singly at Strata (see my  pic opening this post). As Marshall Kirkpatrick notes:

“Jeremie Miller is a revered figure among developers, best known for building XMPP, the open source protocol that powers most of the Instant Messaging apps in the world. Now Miller has raised funds and is building a team that will develop software aimed directly at the future of the web.”

Singly’s appearance at Strata began auspiciously when they won the judges choice award in the startup showcase. And following Marshall Kirkpatrick’s post, Creator of Instant Messaging Protocol to Launch App Platform for Your Life , and The Locker Project: Why Leave Data Tracking to Others? Do It Yourself, Singly have been burning up Twitter.

Singly, by giving people the ability to do things with their own data, has the potential to change our world.  And, as Marshall Kirkpatrick notes, this won’t be the first time Jeremie has done that.

“Pop-cultural instruments for data expression and exploration,” by Bloom

I was drawn over to the Singly table when an awesome app they were demonstrating caught my eye. Fizz, which is running on a locker with data aggregated from three different places is a first glimpse of one of Bloom’s, “pop-cultural instruments for data expression and exploration.”

Fizz is an intriguing early manifestation of capabilities never seen before on the web – the ability for us to control, aggregate, share and play with our own data streams, and bring together the bits and pieces of our digital selves scattered about the web (for more about Bloom and Singly, see Tim O’Reilly’s comments below). The picture below is my Fizz. In Fizz, large circles represent people and small circles represent their status updates. Bloom says:

“Clicking a circle will reveal its contents. Typing in the search box will highlight matching statuses.
This is an early preview of our work and we’ll be adding more features in the next few weeks. We’d love to hear your feedback and suggestions.”

If you are not already familiar with The Bloom team, Ben Cerveny, Tom Carden, and Jesper Sparre Andersen – go directly to their about page and you will understand why the match of Bloom and The Locker Project is a cause for great delight.

The Locker Project – a whole new way to connect from the protocol up

As Jeremie began explaining the depth and breadth of what The Locker Project is facilitating, I was utterly gob smacked. And when the penny dropped and I realized this is the whole 9 yards, bringing awesomeness to people with a whole new way to connect, from the protocol up, all I could think was, OMG finally!

Luckily I have had time to catch up with the whole team since then, and recovered my composure enough to ask some coherent questions. But I can still barely contain my enthusiasm for this project.

Singly, The Locker Project and TeleHash take on, and deliver a simple, elegant, and open solution to some of the holy grails of the next generation of networked communications. I have written on, and been nibbling at the edges of some of these grails in various projects myself for quite a while now. Even if you haven’t been reading Ugotrade, just a glance at the monster mash of my pre Strata post will give you an idea of how important I think Singly is.

My previous post raised the question of how to invert the search pyramid and to transform search into a social, democratic act. But if you are really interested in social search, I suggest staying keyed into what Singly is doing with The Locker Project!

One of Singly’s three founders, Simon Murtha-Smith, was building a company called Introspectr, a social aggregator and search product. Singly’s other founder Jason Cavnar was working on another similar project. And they came together as Singly because social aggregation and search is a very hard problem for one company to solve, and they realized that the basic infrastructure needs to be open source and built on an open protocol.

As Jeremie puts it, “We shouldn’t…(every startup that wants to do something interesting) have to spend this much time aggregating the data, building robust aggregators.”

To me what is so important about the Locker Project is that it is built on a new open protocol, TeleHash. And having the Singly team focused on supplying tools and the trust/security layer for the Locker Project will mean that developers have the whole stack they need to do some interesting stuff very soon.

I asked Jeremy to explain the relationship between TeleHash, The Locker Project and Singly.

Tish Shute: So TeleHash…

Jeremie Miller: It’s a peer-to-peer protocol to move bits of data for applications around. Not file sharing, but it’s for actual applications to find each other and connect. So if you had an app and I had an app, whenever we’re running that app on our devices, we can actually find those other devices from each other and then connect. Our applications can connect and do something.

For the entire edge of the network, basically, out there in the wild, and let those things mesh together.

And TeleHash is actually what has led to the Locker project itself.

Tish Shute: So TeleHash led to the The Locker Project and the Locker Project led to Singly?

Jeremie Miller: Singly is a company who is sponsoring the open source Locker Project…the three of us as founders, [left to right in pic below - Jeremie Miller, Jason Cavnar, Simon Murtha-Smith, ]

I took the pic above of all three founders being interviewed by Marshall Kirkpatrick of Read Write Web for his post, “Creator of Instant Messaging Protocol to Launch App Platform for Your Life.“  I think we will look back on this moment and say it was an inflection point for the web. At least I tweeted that!

Jeremie Miller: TeleHash is a protocol that lets the lockers connect with each other and share things. The locker is like all of your data. So it’s sort of like a digital person…

Tish Shute: A locker for bits and pieces your digital self?

Jeremie Miller: Yes. So TeleHash lets the lockers connect and directly peer-to-peer connect with each other and share things. Singly, as a company, is going to be hosting lockers first and foremost. But the Locker Project is an open source project. You can have a locker in your machine or you can install it wherever you want…

Tish Shute: Yes it’s often too difficult for a lot of people to set up something locally…so Singly makes it easy to have a locker right?

Jeremie Miller: A lot of people see this cool app or this cool thing they want to do, it’s something that you run in your locker that they need to be able to turn on a locker somewhere very easily.

Tish Shute: So Singly will provide the trust layer and hosting?

Jeremie Miller: Yeah, Singly is a company that will host lockers, as well as…when people build applications that run inside your lockers or use your data, you need to be able to trust them. Maybe it’s like social data and you don’t care that much. But especially once you start to get any of your transactions in there, your browsing history, your health data, like your running logs or sleeping…fit-bit stuff…then it’s much more important to be careful about what you’re running inside your locker and sharing.

So Singly will also look at the applications that are available that you can install and actually run them and look at what data they access, and look at who created them, and be able to come back and either certify or vouch for them.

And I hope in the long-run, as this grows and builds, that power users may actually be able to buy a little device that they can plug into their home network that is their locker. Wouldn’t that be cool? This little hard drive or whatever that you plug in.

Tish Shute: Wow – that would be very cool! Architecturally is TeleHash and the Locker Project related to your work on XMPP?

Jeremie Miller: Architecturally, some of the stuff I’ve learned, XMPP, in Jabber it was designed for the specific purpose of instant messaging, but it was still a federated model, in that you still had to go through sort of a central point so you could…a server that lived somewhere. So it was really optimized for like businesses and small groups, teams, as well as big companies out there; ISPs can use it.

So it was designed with that in mind—for the communication path to be routed through somewhere. And where I’ve sort of evolved over the years since then is really fascinated with truly distributed protocols that are completely sort of decentralized so that things are going peer-to-peer instead of actually through any server.

The last 10 years, peer-to-peer has gotten a pretty bad rap with file sharing.

Tish Shute: A really bad rap, yes.

Jeremie Miller: Yeah. And almost because of that, and because it’s really hard to do, that it hasn’t gotten…the potential for it’s awesome. There’s so many really good things that can be done peer-to-peer. And it hasn’t gotten used very much.

But the other side of the peer-to-peer thing that I think is critically important, look at the explosion of the computing devices around a person anymore, both in the home and on our person. We have one, two, three, four even. And the number of devices that are online for you that are yours…I look at my home network router and I’ve got 30 devices in my house on Wi-Fi. What the heck? That’s a lot of devices.

But right now, all of those devices, for me to work with them, I’m almost always going through a server somewhere, through a data center somewhere, which is ridiculous at face value. You go five, 10 years out from now, there’s probably going to be 300 devices on me in some form.

Tish Shute: So we need a peer-to-peer network just to manage our own devices?

Jeremie Miller: A peer-to-peer, yes. You know, my phone should be talking straight to my computer, or to the iPad, or to the washing machine, or refrigerator. The applications in my TV, or whatever, they should all be talking peer-to-peer. And it should be easy to do that. It shouldn’t be that the only way you can do that is to go through a data center somewhere.

[Our conversation continued, but to sum things up, for now, here is the final question I asked Jeremie which pretty much packs in everything I would like to do with TeleHash, the Locker project, and Singly tools/trust layer all in one!]

Tish Shute: How can TeleHash, the Locker, and Singly help people combine personal data from different sources – web and mobile for example, so the data locked up in our social graph on the web can be integrated with, for example, the location data and “the data wake” from our cell phone sensors, to know not only where we have been but to give us more ways to know where we are going?

Jeremie Miller: That’s a pretty packed question, but here’s my simple answer, hopefully just seeds the right discussion

Telehash is the protocol that lets the apps (mobile, sensor, or anywhere) talk to a locker as well as lockers talk to each other, it’s the chatter, moving the bits around the network. The locker is the storage for a person’s data and the crunching ability to analyze it or trigger actions from it. Singly is the company sponsoring the project(s) and helping anyone dev apps atop it. We’re going to build the platform and looking to the world to create some amazing things on top of it (we have lots of our own personal ideas we already want to create, hah!).

The Locker Project is not just “one more rebel army trying to undo these big data aggregations,” Tim O’Reilly

@lockerproject: “We’ll be posting updates on the Locker Project (https://github.com/quartzjer/Locker) here as we make progress, very awesome stuff “

During the Strata Media Conversation I asked Roger Magoulas about Singly and The Locker Project because Roger played Yentl and brought Singly and Bloom together!  Although there was not much time to discuss it, the relationship of TeleHash, The Locker project and Singly to the social network encumbents, came up, and Roger Magoulis and Tim O’Reilly gave some very insightful comments on this when I talked to them afterwards (see below).

Roger Magoulas pointed out:

“I think Singly has Facebook like aspects, but I think a better description is an app platform that integrates your personal and social network data – including data from Facebook. Sing.ly is likely to have challenges with some of their data sources, particularly if Sing.ly gains traction with users.

I like the app platform business model, although they face risks getting critical mass and app developer attention, and I like how they plan on using open source connectors to keep up with changing social network platforms. Jeremie has credibility with the open source community and is likely to find cooperating developers. The team seems to bring complementary strengths to the project and you can tell they all work well together. ”

And Tim O’Reilly went on to elaborate the awesome potential of this platform to bring something new to the ecosystem, and to comment on just how interesting Bloom’s insight into, “data visualization as a means of input and control” is.

Talking with Tim O’Reilly

Tish Shute: So will the Locker Project be able to break the lock of Facebook’s and other big sites’ control of everyone’s data. Sometimes I feel we are stuck in the era of Zyngification, where you have to do what Zynga did and leverage the system in order to gain traction or do anything with social data?

Tim O’Reilly: I don’t think that is the objective of the Locker Project —to break the Facebook lock, because I tend to agree, the value of Facebook is having your data there with other people’s data. What Singly may be able to accomplish is to give people better tools for managing their data. Because if you can actually start to abstract the data from various sites and you can set it and manage it yourself, then you can potentially make better decisions about what you’re going to allow and not allow. Because right now, the interfaces on a lot of these sites make it very, very difficult to understand exactly what the implications are.

And I think all this done right will create a marketplace where people will build better interfaces that will give people more control over their data. They’ll still want to put it on those sites, because why do you put your money in the bank? You know, because it’s more valuable being with other people’s money.

And I think that to conceive of it as one more rebel army trying to undo these big data aggregations is just the wrong way to frame it.

Tish Shute: Yes and framing the question the way you just did – that this is not just one more rebel army, might mean that the stage at Strata will be filled with new startups next year! That’s what I thought when I found out what The Locker Project and Singly was about – that we are about to see an explosion of creativity with personal and social data.

Tim O’Reilly: Yeah, sure. I mean, because at the end of the day, if you can start to extract your personal data in ways that make it more useful, you can potentially create the ability for people to build better interfaces. It’s not just Facebook.

You know, you think, “Oh, wow, I’d really like to have a management console for all my contacts.” And you go, “Well, I’m stuck with, I can use Facebook, I can use LinkedIn, I can use my address book in Outlook or Gmail or whatever, or on my local machine.” The tools are pretty primitive. And if we get a better set of tools, I think we’ll see a lot of innovation.

Now, some of those startups might well be acquired by a Facebook or a Google. But it if moves the ball forward in giving people better visibility and control over their data, that’s a good thing.

Bloom’s insight, “data visualization will become a means of input and control.”

Tish Shute: I loved the marriage with Bloom, which is interesting, because Ben and the Bloom team haven’t really talked a lot about Bloom yet, but I gather Bloom is moving to consumer facing work with data?

Tim O’Reilly: What’s really interesting about Bloom is the notion…You know people think of data visualization as output. And the insight that I think Ben has had with Bloom is that data visualization will become a means of input and control.

Tish Shute: Right, very cool.

Tim O’Reilly: So I’ve started to feel like data visualization as a way of making sense of complex data is kind of a dead-end. Because what you really want to do is to build these feedback loops where you actually figure something out, some particular atomic action well enough that you can create an application that let’s somebody actually do something with it. But the idea of visualization as a way of manipulating the data in real-time, data visualization as interface rather than as a report, it’s a small but subtle shift that I think becomes kinda cool.

Talking with Alistair Croll

Sing.ly – Join or Die from Singly Inc on Vimeo.

Alistair Croll: So I’m a big fan of Singly. They were my choice for the Startup Showcase. I think it’s certainly the right time – the team can execute on it. But the thing I like the most is I thought back to the early days of Photoshop.

So, Photoshop was a neat application that could take data in the form of an image and manipulate it. But the real value from Photoshop came from these amazing plugins. Like, there’s a company called Kai’s Power Tools that made these things that would allow you to do manipulations. Today, commonplace things that are built in. But at the time, they were things like building bubbles, and spheres, and drop shadows and stuff like that, cutouts, in amazing ways.

Another company, I think, called Alien Skin that made these things. There’s whole ecosystems of plugins. So you could go and get a plugin and transform that original data in ways you hadn’t thought of. And eventually, there was a macro language for scripting how you could do those things, and that found its way into the Photoshop environment.

But you think about the transformation of digital design from Photoshop, I think if you can take that same pattern of you create the basic ecosystem of a few tools, and then you allow people an open system on top of that, that’s unprecedented. I think it really does allow you to take ownership of that.

And then when you allow people the proper tools to federate that information. I was actually thinking of starting a company a couple of years ago based on data federation like that. But what you really want to say is I’ve got a pattern…It’s almost like a multi-channel mixer. You’ve got a band that is your health, your weight, your blood pressure, family photos, words you’ve used. You know, the more data I record when I carry my phone around with a headset of whatever, all of that stuff goes in, all my searches, everything.

And then I say, “Ah, I want to federate height, weight, and blood pressure with my doctor. I want to federate sleep cycles and nutrition with my child’s teacher,” and so on and so on. And you start to create these federated sources of data where now you have a teacher data mining, in a safe manner, the sleep and health habits of all the students along with report card information. And you suddenly realize that Johnny is borderline diabetic and falls asleep at recess.

That’s something that never would have happened. And that happens when you have tools to federate data and then compute on top of them. So this idea of, like, lifestreaming or life logging, this is a logical consequence of the whole lifestreaming movement; that whole recorded future stuff.

Tish Shute: Yes it really is a wonderful fruition to the visions of the lifestreaming movement [see this interview with David Galernter]. And best of all it sits on a new open protocol, TeleHash and the open source Locker Project that will give tools to everyone to work with these data streams.

Alistair Croll: Exactly. This is the toolset that sits on top of that stuff. Because once I’ve life-streamed everything, great, I have this bucket of stuff that I did that I never look at again. But if I can suddenly unlock that with data mining tools and analyze patterns, all of a sudden that life logging has a reason to have existed.

The biggest problem we have with data right now is we don’t have apriori knowledge of what will be useful. We could have been recording crime reports in the city of Chicago, and a year later it turns out that data is really useful for predicting diabetes in the city, but we didn’t know it was related.

So the problem, and one of the things I think that distinguishes big data from traditional data, traditional data is collected to some apriori knowledge of how it will be used. Big data tends to be collected for the sake of…it’s almost collected on faith that later on it will be useful for something.

Tish Shute: I am very interested in this idea of federation, I actually went as far as to deep dive into Wave servers because of this….

Alistair Croll: Yeah, Wave was a great example of federation, just too complicated. When it was canceled, both users [and developers] were furious.

Tish Shute: Yeah, I suppose you could see Google Wave as a bit of an Icarus project, right? I am so excited by Singly because it is coming sort of bottom-up – a very different approach…

Alistair Croll: And remember, Facebook didn’t work before Friendster. The only difference between being wrong and being too early is that too early costs a lot of money. So it may be that this is an idea that works now, but a couple years ago didn’t work.

Alistair Croll, co-chair of Strata 2011, in his post, reframed the question, “Who Owns Your Data?” as, “It’s not who owns the data, it’s about who can put the data to work.”

And I am sure there will be many more people able to put data to work, and into play, in a multitude of  interesting ways, now we have TeleHash, the Locker Project, and Singly.

Photo by Duncan Davidson.
Strata 2011 is presented by O’Reilly Media. Produced by Good Company Communications.

I think the photo above gives a good idea of how I felt on the last day at the Strata conference. Yup – like the cat who got the cream!

And in case you are wondering where AR is in this story – it is everywhere!  Below is a pic of the AR concept designs that were omnipresent in the media communications at Strata.  The one below I snapped off the job board.  But as Sophia Parafina noted,  “AR is maturing from displaying last year’s text bubbles and dinosaurs to big data overlaid on the world.”

We are in the age of unearthing and uncovering data, and only just at the beginning of the age of processing data and dealing with it (see my interview with Anselm Hook, Part 2 upcoming).  O’Reilly’s Strata Confernence 2011, will explore, “the change brought to technology and business by data science, pervasive computing, and new interfaces.” It is, perhaps, one of the most important events of 2011.

Data is driving a revolution much as coal, oil, and steel powered the industrial revolution.  And the world changing insight from Karl Marx that “the industrial revolution polarized the world into two groups: those who own the means of production and those who work on them,” is taking on on new life, as  Alistair Croll, co-chair of Strata 2011, points out in his post,  “Who Owns Your Data?”

“The important question isn’t who owns the data. Ultimately, we all do. A better question is, who owns the means of analysis? Because that’s how, as Brand suggests, you get the right information in the right place. The digital divide isn’t about who owns data — it’s about who can put that data to work.”

Strata is where a vanguard will be meet, not only to discuss this revolution’s futures, but to define how to create, handle, and build the platforms and experiences that will harness the data. My flight is booked!  (Also check out BigDataCamp which takes place the night before Strata.)

The picture opening this post is from Michael EdgeCumbe’s  Fall 2010: ITP Winter Show Project.  A project exploring ways to intuitively get the feel of what it going on with big data sets using “the gestural manipulation and stereoscopic visualization of complex data to create a meditative state for data analysis.”  Michael project will be part of the Science Fair at Strata.  For more on Michael’s work see Noise Derived. I also have a number of the  interesting new interface sessions at Strata in my schedule.

The daily Strata Gems on O’Reilly Radar are great place to get a gestalt of some of the Strata themes, and this post by Edd Dumbill, program chair for Strata, Three key data trends for 2011, looks at the year ahead.  This week, I got the chance to ask Edd a few of the questions that I will have on mind at Strata – see his responses below.

If you have been reading Ugotrade, you will know I am interested in our mobile social augmented futures and there is no question in my mind that these will be unleashed by our new capacities to work with data (see my post here).

Data is the how.

The pic above is from “Secrets of BackType’s Data Engineers.” This post on ReadWriteHack by Pete Warden, an ex-Apple engineer, and founder of OpenHeatMap, really lives up to its title.  Check it out if you want to know how  “three guys (the BackType team ) with only seed funding process a hundred million messages a day?”

I asked on Quora, “What would be the most important developments for Augmented Reality in 2011,” Michal Avny, Strategist & Real Time search expert, wrote:

“AR strongly relies on localized personalized real time information.

Having a stream of tweets based on keyword search, location or circle of friends doesn’t really make the AR experience; it is the processed real time relevant information that will make AR useful and intensify the experience.”

In 2011 Real Time search and Social Search will drastically change to provide the infrastructure required.”

I followed up on Michal’s Quora answer with some more questions – see below in this post.

Also note the response from David Molnar, here is an excerpt:

“2. A wave of actionable, important data APIs opened up, enabling useful non-gimmicky AR apps for the first time. Think geoloqi.com , or the work Max Ogden has done with Portland civic data. Plus of course face.com , email providers and calendar providers, etc.”

Amber Case, one of the founders of Geoloqi, is on the programming committee of Strata and will be speaking. Be sure to catch her session! Posthumans, Big Data and New Interfaces, and if you haven’t already seen it, Amber’s TED talk is a must see.

Geographic proximity is a powerful filter, as is route, and time. But clearly social proximity, social relevance, and shared tastes are also key dimensions for location based experiences, (see my convo with Schuyler of Simple Geo, upcoming).

While the whole business of location based search and curation of augmented mobile social experiences is still, for the most part, uncharted terrain, the danger of key points of control being only really accessible to elite players looms large. I asked Sophia Parafina, a pioneer in the open geo space for some thoughts on real-time local /geosearch and geomessaging, and the future of openess & big data (see Sophia’s response below).

Is the market ready yet for P2P cloud computing?

This is another question I’m following,  Is the market ready yet for P2P cloud computing? It is one of those questions that we seem to have been asking in various forms for a very long while now, but without a  major shift in sight.  The pic above is from, The Cloud Made Open Source “Invisible” This Year.  But, perhaps, we are at the point when open p2p clouds will find a place in the market because of their potential importance in real time social search and discovery. Borislav Agapiev, Search Entrepreneur and founder of Vast.com, writes on Quora:

“I believe a P2P cloud is ideally suited for social & real-time search and discovery.

Consider MapReduce, a very interesting and popular paradigm for distributed computing. MapReduce is very much about bringing computation to data i.e. doing computation at nodes (map) and then aggregating results through network (reduce).

It is very clear now that user attention data (what they click on) is very valuable for search and discovery, yet a centralized model relies upon uploading all that to a single location and then doing a supposed local MapReduce. Clearly, MapReduce could be done across the network, without any centralized uploads.

In addition to the efficiency argument raised here, it is even more important to consider privacy issues. Uploading massive amounts of user attention data to a centralized location is not something that is going to make users warm and fuzzy as we are increasingly seeing.

In a P2P cloud, there is no big brother watching over anyone, all computation and data storage is done in the cloud, fragmented in many, many small encrypted pieces ala BitTorrent.”

Picture above from Brynn Marie Evans, “It takes two to tango: review of my social search panel“

The Delta of Now – Transforming Search into a Social Democratic Act

Picture of Maneki Neko “beckoning” cats from Journeyetc

New ecologies of human and machine intelligence are beginning to change basic social structures – see the Future of Work (Biewald and Chirayath Janah 2010). And projects like Swift River, using search and machine mining to filter out streams on topics of interest that can then be subsequently curated by human beings. This may be extended to the curation of real-time data streams and employment of machine learning algorithms based upon the explicit relationships.

Augmented mobile social experiences are a new frontier in which ideas and practices from a number of fields collide, including: ambient findability (Morville 2005), urban psychogeography, narrative structures, ambient games and devices, 4d (time-space), explorations of place and memory, enchanted objects and people (Kuniavsky 2010), and designed animism (Laurel 2010), to mention just a few.

Mobile local interaction presents an opportunity to invert the search pyramid and to transform search into a social, democratic act (see my interview with Anselm Hook upcoming).  Up until now search has been predicated around a very narrow revenue model. Google has an implicit model of a B2C – business to consumer brokerage. We are only just beginning to get a glimpse of the disruptive potential of C2C – consumer to consumer brokerages. Mobile local C2C brokerages that allow us to transact in a trustworthy way over our local geography in close to real time (Hook 2010) have the potential to enable new forms of social organization. Bruce Sterling’s short story about a networked gift economy, Maneko Neki, is a brilliant glimpse at the disruptive potential of such re-imaginings.

Augmented experiences that shift or change a person’s situated geolocal experience of social reality, and change our relationship to the people and the place by augmenting engagement in, and reputation through, socially driven consumer tie ins and game dynamics, like Four Square, & Gowalla are beginning to emerge, as Kati London pointed out in her excellent keynote at Web 2.0 Expo. And, while the integration of mobile local interaction and an augmented view that shifts our geolocal experience visually will involve creative solutions to some well churned mobile, tracking, mapping and registration challenges, the exploration and development of new dimensions through which we can filter and create trusted and meaningful augmented mobile social experiences is vital, whether you are considering a mobile screen, map, camera view, or futuristic HUDs and gestural interfaces.

Talking with Edd Dumbill

Picture from O’Reilly Community.

Tish Shute: First congratulations on Strata! On the Strata homepage there is a quote from Jason Hoffman:

“My gut feeling is that we’re going to look back at the upcoming Strata Conference like we do at the Web 2.0 Conference in 2004/2005.”
—Jason Hoffman, CTO/Founder, Joyent, Inc.

Why do you think Jason’s comparison might be prescient?

Edd Dumbill: Web 2.0 is a development that ran through every brand that has a web presence and radically changed the way business is done for many companies and brands.

Strata will have a similar impact: every business has data, every business collects an increasing amount of data. This data is the new oil – a valuable raw material that when refined or combined creates value and opportunity.

Tish Shute: The rise of real time was one of your three key data trends for 2011. Hadoop is bringing the capacity to work with big data to more than just a few elite players. But the challenge is still real time. You mention we will be seeing a hybrid approach to real time and batch MapReduce processing. Will we hear more about these approaches to real time at Strata? And, what do you see as the most important conversations on real time data analytics emerging at Strata?

You point out “open source projects and cloud infrastructure means developers can evaluate and learn to love technologies without requiring support or approval from above.” What are the most exciting developments on the horizon for open source tools?

Edd Dumbill: Here are some projects worth watching, in the key areas of real time, cluster management and Hadoop.

* Cassandra and MongoDB — NoSQL databases that will prove vital for anybody with real time big data needs

* Mesos — a compute cluster management tool, modeled after that which powers Google

* Hadoop ecosystem’s continuing maturation, especially HBase and Hive.

Tish Shute: Do you think the market is ready for p2p cloud computing?

Edd Dumbill: The market is emerging for decentralized and distributed cloud computing, and P2P technologies are one way of achieving that. They key trends will be moving computation nearer the data sets or nearer the point of user consumption of the result.

P2P is a difficult model for anybody wanting to commercialize a service, so I think it will tend to form part of a hybrid solution.

Tish Shute: We have seen enormous strides in our ability to work with giant unstructured databases recently. Do you think, perhaps, that the dream of a web of linked data – “a web of data that can be processed directly and indirectly by machines,” will be attained through brute force – i.e. through our ability to harness the power of massively parallel processing, as much as by Semantic Web approaches focused on machine readable metadata? [Also see my question on Quora, "Is this a good approach (www.dist-systems.bbn.com/people/...) to use Hadoop to build a scalable, distributed triple store?"]

Edd Dumbill: I’ve been an observer of the SW for over a decade and I tend to believe that on the web, data means to you whatever meaning you give it as the consumer. With that model, the links are made by the consumer rather than sitting out there explicitly. Some links become de facto standards, and some very few become web standards.

I think the actuality will be a mix of both explicitly stated metadata and that which is inferred. The Semantic Web is a great framework for certain operations, especially interoperable exchange of metadata. A great many more private meanings, never intended to be shared, will be created by consuming software.

There’s no question that machines will learn how to process most of the Web. Furthermore, machines will learn how to process most of the physical world we’re in. And that by the end of this decade.

Talking with Sophia Parafina

Picture of Sophia at Where 2.0

Tish Shute: Sophia you have worked in the trenches for a long time now to support the growth of open geo data. What do you hope to see emerge in 2011 in the field of geo-data?

Sophia Parafina: Better support for displaying and handling location data across multiple apps. Fred Wilson recently blogged about content-shifting, he talks about overcoming content silos across devices. We’ve worked very hard to reduce data silos via formats, but devices are creating their own silos. I would like to see a standard method for sending geo data and geo information to mobile devices.

Producing content for mobile is different from producing content for a computer browser. Web 2.0 produced a lot of infrastructure for browser based interfaces, but in mobile devices that gap has been filled with apps which is fragmenting how data is handled by various devices. What is even more interesting in the mobile space is that devices can push data back that contains location, user updates, photos and even sensor data.  If mobile data standardizes, it could lead to browser based applications and stem the continued fragmentation of the mobile application market.

Tish Shute: Simple Geo and Factual are startups emerging in the geodata space. What do you see on the horizon in terms of both the growth of business opportunities and an open geo data community?

Sophia Parafina: In the near future think we’ll see startups providing curated data + API and in response we will also see companies that provide a single interface across multiple data providers. We saw this when everyone released a mapping API and companies such as Mapufacture provided a single interface across multiple APIs.

We will see a resurgence in data providers repackaging the the 2010 US Census data in different ways to respond to market segments, some of this will be open data but all of it will be provided through an API instead of file. Additionally, we’ll see more data from outside the US.

Tish Shute: What are the biggest obstacles to having the open geodata sets available that we need to enable mobile local interactions and social augmented experiences?

Sophia Parafina: Licensing for both crowd sourced data and private curated open data will become an issue. We recently seen VLC, the open source video player, pulled from the Apple app store because of licensing issues. Also, licensing of content by geography will be problematic, limiting searches by geographical location. In addition, how will licensing of data that is updated by crowd sourcing work?

Multiple APIs for accessing data sources. The current trend for each provider to create an API for their data sets will result in data silos – there needs to be a single sign-on equivalent for requesting data.

Size of data on the wire, the current models for delivering data is based on broadband connections. However, as mobiles increasingly become the way people use the web, the data needs to be sized accordingly. This also goes for mobile interfaces. Have you tried to shop on a mobile device, or buy a train or plane ticket? It’s frustrating and error prone. There is a large untapped market of people who only use the Internet on mobile devices.

Tish Shute: You pointed me to this link in Strata Gems re “an interesting and pertinent (also a competitor to GeoLoqi),” – the Android Tasker app. What do these emerging services bring to the table in terms of the next generation of location based services?

Sophia Parafina: This app let’s your device interact with the environment. I think that this is a great way of using the sensors on existing platforms to increase interaction and to implement ambient findability. The basic premise of Tasker is that some action happens in response to an event in an application, time, date, location, event, or gesture. Tasker has defined 180 actions that can occur based and number or combination of events. This can provide a basic vocabulary for interaction between the user and the device and more importantly between users. Tasker also can use Android script plugins, which lowers the bar to creating your own ambient application.

Programs such as Tasker can provide a way for people to interact with social networks beyond sending messages. People can use their mobile devices to interact with their surroundings with out having to interact with the device.

Tish Shute: We have had many conversations about emerging ideas of geo-search, geo-messaging and geo-fencing. What are the most interesting developments in these areas and what do you see on the horizon for 2011?

Sophia Parafina: The map will fade into the background and become less important. Display of information will be context aware, that includes location. For example, let’s say I make a grocery list, when I’m at the grocery story, the list will just pop-up without the need for me to find the app that has the list. Or reminders or offers pop-up when you are near a place at a certain time, let’s say you need to buy a present for a birthday party for a child, you could send out a request that you are looking for an item and retailers could offer “on the spot” discounts if you are in the area.

Geo-search, geo-messaging, and geo-fencing are geared to towards mobile devices, so I expect to see them soon as part of apps. Building generic applications that implement geo* will fail because that sort of information is useful only within a context. Geo* apps are solutions looking for an problem. The killer mobile app will use these functions transparently to reduce the cognitive load of the user who is busy moving around in the world.

User data gathered from multiple web applications will become consolidated profiles that will used for context aware applications. For example, there could be a service which matches prices of items that you have shopped for on the web, so for example the service would have access to your cookies, know your favorite retailers, things you have shopped for, your location and activity patters (when you are at home, work, restaurant). When you are in the vicinity of a brick and mortar retailer with the same or similar items, the service can send you alert to match the price of the item you found on line. So your digital life will become more closely linked with your day to day activities.

Talking with Michal Avny

Tish Shute: At Web 2.0 Summit, one of the highlights for me was the, Q&A:The New Search Insurgents lunch where Charlie Cheever of Quora, IMO, stole the show. I tweeted:

“One of my takeaways from #w2s is that #quora points to future of augmented mobile social experiences – a search filter for experience! #AR”

In your view what are the biggest challenges for location Q&A to emerge as a search filter for location based experiences?

Michal Avny: The biggest location Q&A challenges yet to be conquered are immediacy (real time dynamic data), relevancy (strong personalized filters) and user experience (simplified interface).

Location Q&A enables different use cases. The most prominent are Follow (follow places, topics and friends to learn about a location), Interact (meet new people based on common interests), Plan ahead (plan a trip, night out or a shopping day by asking and searching for local information) and On-site (check for recommendations, friends, deals, events and traffic nearby).

Unlike Follow, Interact and Plan ahead that can be added to existing Q&A platforms (such as Quora) by attending location specifics as they share similar characteristics, the on-site mode introduces a completely different experience, first and foremost it requires immediate attention. It is real time based and the nature of the data is dynamic. Traffic updates, current events, nearby friends, all that changes constantly. Posting a location question on-site implies the response should be in real time (e.g. best kid friendly restaurant), the normal Q&A response latency wouldn’t work.

Strong relevancy filters are required to accommodate for the overwhelming flood of information. Moreover, some of the data should be filtered by user behavior and preferences, check in notifications (type of relation), restaurant recommendations (type of food, price level, etc), shopping deals (commercial categories) and more.

Mobile experience requires ease of use and simplicity. A new Q&A interface and query language that allows for posting questions should be defined as well as coherent summarized response interface. User on the go should not have to post lengthy questions, browse through tens of results or search for the right service, but instead use a simple intuitive tool.

Tish Shute: Real- time location based search is in its infancy. Real time questions can be answered using different services such as Yelp, TripAdvisor, Waze, Foursquare, IMDb and more. But what are the challenges to moving forward with aggregating these sources and then into “locals” that are able to process and deal with vast amounts of information?

Michal Avny: Using some of the leading location services to answer question is sufficient to start with.

In order to provide broad coverage (worldwide) and reliable information, aggregation of the different services is required for instance to normalize product and service rank, aggregate classified, and more. This is quite challenging as there is no one standard available.

When location Q&A user base is big enough, I foresee a tendency to rely more on ‘locals’ input as the base of information. As the platform grows, communities will be formed with different cultures, relationships and trust levels, making the information more valuable and customizable. Some of the challenges I already mentioned are implementing filters, query language and interfaces to enable using the vast amounts of real time data in a mobile environment. More of the challenges lying ahead are integrating the ‘locals’ data with location based services as they are integral components of the Q&A ecosystem. Merging trust levels and relationships while adhering to different privacy guidelines is a challenge yet to be explored. (This should be discussed in more detail under the protocols topic).

It is quite evident that Quora is now facing growing pains and is struggling to maintain its character. Same as with Quora, it will also be a challenge to support and maintain the ecosystem while allowing for massive scale-up.

Tish Shute: I have been very interested in exploring protocols that will be enablers to micro local interaction and mobile social interaction for AR – particularly the XMPP extensions and operational transform work of Google Wave (now Apache Wave), and PubSub protocols like PubHubSubbub and Erlang based RabbitMQ. We are beginning to see protocols emerging that could enable new real time local services. What do you think are some of the most valuable use cases for “locals” that this new generation of real time protocols can enable?

Michal Avny: AR is about interacting with digital information; the AR ecosystem is composed of layers and components such as devices, platforms, browsers, applications and content. For the different components to interact new protocols, security guidelines, and privacy policies must be in place. A standard will enable local vendors and service providers to publish specials, deals, updates and events for any application to broadcast, identify people and places by proximity (without having to use the same application or device), local recommendations will be shared by services, devices will be able to interact, location based platforms, such as Q&A, will have access to vast breadth of information, geo aware devices will provide consistent experience globally, and much more.

Tish Shute: What do you think are the biggest challenges to going mainstream for this emerging field of real time social discovery?

Michal Avny: The biggest challenge is building towards real time, geo-aware, localized, personalized ambient data. Discovery is in its infancy, location social based Best, Top, and Trending lists with some basic filtering options are available, and this is great as people are getting accustomed to information surrounding them. To some degree it can intensify the AR experience, for instance suggest the most popular dish in a restaurant, or map the best coffee shops nearby, but it is customized at best by friend recommendations and depends on the coverage and broadness of the specific discovery service.

There is a need for the next generation of discovery, customized geo social aware discovery that filters the vast amount of real time data by learning user preferences and behavior (built on top of the much needed local social real time open protocol)

Tish Shute: Who are your favorite startups/upstarts in the the field of real time search and why?

Micha Avny: My6Sense - My6sense provides a sharper and better way to experience your information from feeds you subscribe to (Social Networks, News, RSS feeds, etc.). It’s personal – Content is ranked based on what’s relevant to you. It learns what’s valuable to you by translating your consumption behavior into a personalized ranking function.
My6Sense – because it is a personalized prediction filter, a critical foundation for AR

Topsy – Topsy is realtime search powered by the social web that finds the most relevant conversations happening online. The site’s underlying technology examines popular links as well as the influence of each person citing a link. Topsy augments traditional search engines by finding information that people are talking about.
Topsy – because its ranking is based on retweets and influencers, a great social experience

Collecta – Collecta is a real-time search engine for the social web. It monitors the update streams of popular realtime blogs and sites like Twitter, WordPress, and Flickr, and shows results as they happen. Results can be filtered by status updates, comments, stories, or photos. The entire engine is built around the XMPP standard, which pushes out data on a continual basis, so that for every search you end up watching a stream that keeps updating itself.
Collecta – because it is built around XMPP, a real time experience

The Missing Manual for The Future (or The Future: The Missing Manual)

O’Reilly Media, is famous for is producing  “missing manuals” for new technologies, but thinking of O’Reilly as just a publisher of books would be like saying Facebook is just a website (this came up in the discussion at Media Round Table at Web 2.0 Expo, NY, 2010).   In recent weeks, I managed to catch Tim O’Reilly at several events, Maker Faire, Web 2.0 Expo, Hadoop World, and the free webcast Tim did with John Battelle on The Battle for the Internet Economy (although Tim spoke several other times during this period!).

It occurred to me, as I immersed myself in the depth and breadth of innovation showcased and discussed at these events that Tim O’Reilly, and the O’Reilly team, are creating, The Missing Manual for the Future.

As Tim puts it, we are “changing the world by spreading the knowledge of innovators.” Tim uses a quote from William Gibson to illuminate what is at the heart of the O’Reilly project:

“The Future is here, it is just not evenly distributed yet.” (William Gibson).

But Tim O’Reilly makes another point about the future when he speaks.  The future unfolds unexpectedly – so we must invent for an unknown future not a known future, or as Alex Steffen put it so well in his post, Why Our Bright Green Futures Will Be Weirder Than We Think, – “The world we need is one we’ve never yet seen.” The magic of attending an O”Reilly event is that it gives you a chance to work on this koan in interesting ways, and to take more responsibility for how things turn out.

Tim O’Reilly also urges that we think more deeply about what we are doing.  His keynote for Hadoop World , NYC, billed as, “The Business of Data” turned towards “The Consequences of Living in a World of Data.” The 900 strong crowd at Hadoop World was probably one of the most savvy crowds in the world about the business of data, so this was a nice turn.

Web 2.0 Expo with the theme, Platforms for Growth, was a deep dive into the business of innovation.  Tim O’Reilly’s keynote at Web 2.0 Expo,  “Thinking Hard About The Future” (or rather “thinking a little bit creatively or differently about the future)  – see video here, developed the call he made at Web 2.0 Expo 2008, to “work on stuff that matters,” into a Four Cylinder Engine for Innovation.   The first of the four cylinders in the firing order is, “Having Fun!” But,  at Maker Faire, Web 2.0 Expo, and Hadoop World I got an inside look at the workings of all four cylinders, and there is more to come, I am sure, as the other O’Reilly events unfold over the coming months including,  Web 2.0 Summit, Strata (a new O’Reilly conference on The Business of Data), and Where 2.0, 2011.

In a free webcast, last week (recording here), previewing Web 2.0 Summit, John Battelle and Tim O’Reilly discussed the Points of Control Map which is developing into a fun and useful tool to examine a very serious topic, “The Battle for the Internet Economy,” and how the “increasingly direct conflicts between its major players” could effect “people, government and the future of technology innovation.”   In my previous post, Platforms for Growth and Points of Control for Augmented Reality, I had a great conversation with Chris Arkenberg using this map as a springboard.  More on Points of Control later in this post.

The Four Cylinders of Innovation

click to enlarge

From Jet Ponies to Jet Packs: The First Cylinder of Innovation – “Have Fun”

The “maker” energy and its spirit of play, and the courage to create, hack, reinvent and re-purpose everything and anything, is a quintessential example of the first cylinder of innovation firing big.  Many “maker” projects also go on to fire on all four cylinders.   But the Maker forte definitely is in the first cylinder zone (and safety third as some of the rides, including Jet Ponies, warned).  The photo opening this post by Marc de Vinck – for more pics see here, is of Tim riding The Jet Ponies at Maker Faire which took  the New York Hall of Science by storm in late September – see The New York Times coverage here.  The ride was “built by the dastardly danger-hackers at the Madagascar Institute.“ See this wonderful interview with Hackett on his work to design “our specific jets from a patent that was filed in 1960s by a Mr. Lockwood, for Valveless Pulse Jets.” Hackett points out:

“Louder than god, glowing white-hot and looking like the trombone of the Apocalypse, pulse jets are also really shitty, inefficient engines,”

But, he adds:

“I have always wanted a jetpack, and one of the reasons I learned to build these things was to further that goal.”

This grand vision behind the Jet Ponies is a key to firing, The Second Cylinder of Innovation,  “Hey, we can change the world!”

But Jet Ponies, as a stepping stone to jet packs, also really struck a chord for me as I have been devoting a lot of time lately to the emerging Augmented Reality industry, a technology which was lumped in the same category of sci fi chimera as jet packs until very recently.

Data is the Gasoline

“The faces are coming from the sky.  The locations are coming from the sky.  All these apps depend on something, somewhere up.  And that, to me, was always the heart of Web 2.0.  And I am so delighted that people are finally getting it.  Because for a long time, people thought, ‘Oh, Web 2.0, it’s about lightweight advertising supported in a web start up.’  So I went, ‘No, no, no. It’s about the fact that we’re building these giant database subsystems in the sky that are going to drive applications.’  And now, of course, the same application is on your PC, it’s on your phone, it’s on you iPad.  And clearly, the applications are just sort of an interface to something that is being driven from the cloud, and that is fabulous.  That’s the difference.  People get it now.” (Tim O’Reilly, said this as part of a response to the first questioner at the Media Round table Web 2.0 Expo)

Answering questions about the importance of “Having Fun” to innovation doesn’t look quite as fun as riding Jet Ponies! Photo above from luca.sartoni’s Flickr stream

“ the data that is generated by the sensors and the applications that use that data is going to be where people are going to be innovative.” (Tim O’Reilly)

During the Media Round Table, I had a chance to ask Tim more about the role of bottom up innovation in a world where big data is the gasoline for increasingly sophisticated engines – platforms integrating machine to machine intelligence and real time analytics.

Tish Shute: You brought up Maker Faire in your keynote, and again now.  I was there, which not many people in the audience were  [not too many hands went up when Tim asked during his keynote].  But I think one of the things that struck me was the jet ponies – they were just earthshaking to stand near.  They made the ground tremble; they made the world shake.  Yet, most of your keynote, and most of what’s on our minds here, at Web 2.0 Expo, is extracting intelligence from the big data [in the sky], and algorithmic intelligences are the jet engines of the internet.  And of course, not to be forgotten, as we are here in New York City, where the trading markets are creating the air we breathe  [although we probably don't realize it until we lose our mortgage or something] and these algorithmic economies or “robot casinos” as Kevin Slavin put it, are all about speed – it’s not just real-time, issues of latency are  so critical that co-location is key to winning the game of the markets.  [Kevin Slavin brilliantly unpacks this in his talk, "Loitering on the Motherboard." For more in this see my conversation with Kevin Slavin below].

So my question is, who’s making the jet ponies for the algorithmic economies in the sky that you just described?   How can we make a play from the bottom up?  I always feel Ushahidi is one of the jet ponies of the data algorithmic space [because of their great work to bring human and machine intelligence together to solve problems in crisis situations].  But who do you think is doing exciting work and how can we ensure that this powerful world of data and algorithmic intelligences does not become hidden in a closed black box [only really accessible to elite players like the NYC trading markets]?

Tim O’Reilly: “Well, I think there’s certainly a lot of interesting things happening in, say, the financial services that a lot of, kind of, the Internet folks are kind of blind to.  I think that there are companies like Next Jump which are really good with data and good with algorithms. But kind of speaking specifically to the maker side of this, that whole sensor enabled world which is going to produce data is in its infancy.  What we have that I think is so powerful right now is we have the first portable sensor platform.  I said in my talk the other day, you know, your phone has ears, it has eyes, it has a sense of where it is.  And these are all available to application developers. You know, you can compare, say, Dodgeball to Foursquare, you can see how different… Dodgeball is Foursquare in the tele-type era.  Foursquare is now possible because there are so many more capabilities on the phone.

And I think that we are going to see a lot of other areas that are revolutionized by the sensors in the device.  It could well be that some of them will come explicitly out of the maker kind of projects, or it could just be that make is sort of a proxy for them.  So yeah, Arduino is this great maker sensor platform, but hey, here’s a consumer sensor platform [holding up phone]. Maybe we vaulted past the maker stage already and we just didn’t know it.

And that’s not entirely true, because Arduino is building a whole economy of special purpose devices.  But it feels a little bit like the days when people rolling their own PCs coexisted with the rise of Dell, who was a kid in his college dorm room who made his own PCs and sold them on the net, but figured out how to scale it pretty quickly and get good at it. But there were still a lot of garage shops, you know, ‘I’ll make a PC and sell it to you’ people for probably a decade before there was really a clue that that was a commodity industry.  In fact, I do think the sensor platforms are going to become a commodity industry.  And the data that is generated by the sensors and the applications that use that data is going to be where people are going to be innovative.”

The internet operating system is a data operating system and it is happening in real time (Tim O’Reilly)

click to enlarge the image above  – a slide from Mike Olsen’s  (CEO of Cloudera) keynote at Hadoop World

Not only do we have a portable sensor platform in our pockets  but developers also have powerful platforms and tools to make sense of data that fuel our apps.   Opensource Hadoop makes available, to anyone with some data munching chops, the power to work with giant unstructured databases and do the kind of real time analytics previously only available to giants like Google.  Big players like Yahoo, Facebook, and Twitter use Hadoop (Jonathon Gray from Facebook noted they add 10TB a day).   But, as this great roundup of Hadoop World points out, while Hadoop gets the press for handling petabytes of data , Mike Olsen (CEO of Cloudera) noted, the fastest growing area of users are working with clusters smaller than 10TB and over half of the Hadoop clusters were under 10TB in size.

Four Square: A Platform for Growth with an ecosytem built on top of data that exists in the real world

As an augmented reality enthusiast it is not hard to guess that one of my favorite platforms for growth is Four Square.  See Dennis Crowley’s keynote at Web 2.0 Expo here.  The Four Square API has been available to developers since November 2009,  and there are already a number of  interesting applications, and there will be many more to come.  The screen shot above is of geopollster – see the gallery of Four Square apps here.

@dens tweeted recently  “Politics + @Foursquare = @GeoPollster” http://geopollster.com <- I love love love that people are using 4SQ to think about election tools

As Kati London pointed out in her keynote, Four Square is the “kind of augmented reality that is aimed at shifting or changing a person’s social reality, e.g. the mayor badges in Four Square that change my relationship to the people and the place I am in, and augment engagement and reputation through socially driven consumer tie ins.” We are already see augmented reality developers beginning to work with the Four Square API – see here, Foursquare + Augmented Reality + Virtual Graffiti = ARstreets.

As augmented reality development tools mature, Four Square will, increasingly, become an important platform for creative AR developers interested in integrating the power of this platform for augmented engagement and reputation with “device aided augmented reality that can shift visual experiences of situated geolocal experiences.” With the Qualcomm vision based augmented reality SDK now available for download, and Wave in a Box soon? to be released, and an ARWave client working on Android (almost!), I have been exploring the Four Square API in my non existent spare time!!

The Four Square API also offers some interesting possibilities for exploring games that take the complex economy of Four Square – not personal data but aggregates of behavior, as their subject matter (for more on this see my conversation with Kevin Slavin later in this post and in an upcoming post).

I took this picture of Dennis at Where 2.0, 2009 at the beginning of Four Square’s phenomenal growth (they are at 4 million plus users now).

Pachube (Patch-Bay): a web service for storing and sharing sensor, energy and environmental data

Eighteen months ago, I interviewed Usman Haque (architect and director, Haque Design + Research) and founder of Pachube – see Pachube, Patching the Planet.   Usman pointed me to this wonderful evocative image from T.R. Oke’s book, “Boundary Layer Climates” (original photo source Prof. L. E. Mount’s The Climatic Physiology of the Pig).  “It’s the same piglets, in the same box, but on the right hand side the temperature has been increased. This small change in how the space is “programmed” has dramatically changed the way the ‘inhabitants’ relate to each other and how they relate to their space.”

The Challenge of Connecting people and environments.

At Web 2.0 Expo, I got the opportunity to talk with Usman Haque again.   Pachube, is becoming an established platform now, Usman explained.  They have a development team of eleven and robust back end.  And, they will now be spending some more time on the front end, including a redesign of the website, making “it a lot easier to widgetize the entire website so that you will be able to take almost any element and embed that into your own website.” And, as Usman mentioned in his presentation, they are working on an augmented reality interface, Porthole, for facilities management and, “as a consumer-oriented application that extends the universe of Pachube data into the context of AR – a ‘porthole’ into Pachube’s data environments..  Usman is also contributing to the AR standards discussion and on the program committee now for the W3C group on augmented reality.

Click to enlarge the image above from Chris Burman’s paper for the W3C, Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual”

Pachube, is sometimes described as the Facebook for Data or an analogy Usman prefers, a Twitter for Sensors.  At Web 2.0 Expo, I had an amazing opportunity to hear from Twitter and Facebook about their strategies as platforms for growth.  This gave me lots of fuel for questions about Pachube’s approach to developing their platform.  Simplicity was a theme that Facebook  and Twitter both affirmed as a key.  One of Pachube’s challenges will be to deliver ease of use, and the equivalent of Facebook’s “like” and  Twitter’s “follow” to gain mass appeal.

Here is a brief excerpt from my upcoming conversation with Usman:

Tish Shute: So as a platform you see Pachube as having more in common with Twitter – a Twitter for Sensors. In what ways is Pachube similar to Twitter?

Usman Haque: Well we are the Twitter of sensors, devices & machines in the sense that, really, the API that enables all this communication is important, much more so than the website itself. It is where, basically, most of the millions of our hits actually go, is to the backend. And we’ve now got dozens of applications built on top of the system, a little bit like Twitter’s applications; you know, all the apps are the important part.

But we are actually going to be doing some quite exciting things with API keys that we haven’t really spoken that much about in public. But we have come up with a pretty innovative solution to make almost every resource have granular privacy options on it, now discussed here.

At Hadoop World, Tim O’Reilly also raised some interesting broader questions that are very relevant to Pachube’s vision to “patch the planet”, e.g, the problem of digital identity in the age of sensors? (Smart phones already know their users by the way they walk!) And, “How should we think about privacy in a world where data can be triangulated?”

Usman talked about Pachube’s approach to both the technical aspects of how to build a massively scalable system, and the conceptual aspects of how people connect to each other, and what they might do with these new opportunities to connect environments and sensor data  (see my earlier talk with Usman, Pachube, Patching the Planet, for a detailed explanation of some of the concepts behind Pachube).

I look forward to posting this conversation. Pachube is growing, and Usman always goes beyond the familiar tropes of connecting human and machine intelligence.

2nd Cylinder of Innovation: “Hey Can We Change the World!”

The possibilities for reimagining of the role of data in healthcare produced some of the most powerful “Hey Can We Change the World” moments for me at both Web 2.0 Expo and Hadoop World.  The slide above is from Esther Dyson’s brilliant Ignite presentation, “What you can and can’t learn from your genes?” are here,   Tim O’Reilly also brought up the powerful role real time data analytics can play in improving healthcare in his Hadoop World Keynote.  Also see Alex Howard’s post, 10 Lessons for Gov 2.0 from Web 2.0 for some more great, “hey we can change the world moments” at Web 2.0 Expo.  The keynote from Lukas Biewald of CrowdFlower and Leila Chirayath Janah of Samasource (screen shot below) in particular, is a provocative exploration of the future of work in the new ecologies of human and machine intelligence.

Changing the World When Our Lives Are Increasingly Shaped by Forces Invisible To Us?

Click to enlarge

Mike Olsen, CEO of Cloudera, noted that “the largest area of data growth does not come from humans interacting with machines; rather, it’s from machines interacting with each other” (see here in Minor Technical Difficulties).   One of the most interesting presentations at Web 2.0 Expo was Kevin Slavin’s, “Loitering on the Motherboard,” which, as Tim O’Reilly pointed out in his keynote at Hadoop World, is a talk that raises all kinds of questions about a system where big players are gaming the data for their own ends.

Kevin Slavin, a founder of Area/Code, notes the operating system of our mortgage, life insurance, the operating system of currencies and gold is now governed by machine to machine intelligence and algorithimic economies outside of human cognitive processes.  The markets are now legible only to bots in an algorithmic arms race with bots surveilling bots, and throwing off false information in a bid for counter-surveillance.  He showed some slides of the eery but beautiful visualizations of traces of the trading bots created from the Nanex API.

The screenshot above is from the Nanex: Crop Circle of the Day – Quote Stuffing and Strange Sequences.  “The common theme with the charts shown on this page is they are all generated in code and are algorithmic. Some demonstrate bizarre price or size cycling, some demonstrate large burst of quotes in extremely short time frames and some will demonstrate both…” This one is a zoom of the NSDQ “Wild Thing.”  Wild price/size repeater from NSDQ running at 1,000 quotes per second, effecting the BBO along the way (I love the great names Nanex gives the different patterns and traces produced by the trading bots).

Nanex supplies a real-time data feed comprising trade and quote data for all US equity, option, and futures exchanges. They have archived this data since 2004 and have created and used numerous tools to “sift through the enormous dataset: approximately 2.5 trillion quotes and trades as of June 2010.” May 6th 2010 (day of the flash crash), had approximately 7.6 billion trade, quote, level 2, and depth records.

Kevin points out that our lives are being shaped by criteria invisible to us and the old hackneyed tropes of machine to machine intelligence such a robots reading HUDs in English are long worn out.  The latter point is, perhaps, something for us augmented reality geeks absorbed in ideas of “making the invisible visible” to chew on.

Changing a world shaped by forces that are, increasingly, invisible to us presents a huge challenge.

But I had the glimmer of a, “Hey Can We Change the World” moment, when I attended Kevin Slavin founder of Area/Code’s presentation and had a conversation with him after his talk.  Could games take these complex economies as their subject matter?  The economies of  Farmville and games like WoW are not opaque at all, and these are environments with complex economic behavior, “where you can actually have enough data to understand what it is” – “it’s not so much about personal data.  It’s more about, like, aggregate behaviors.” “Games that can really model those, and play with those, and take those as the subject the way that Monopoly takes Monopoly as a subject could be really interesting.” Kevin made many fascinating points – more to come on this topic.

Photo by James Duncan Davidson, of Kevin Slavin speaking at Web 2.0 Expo NY, 2010, from the O’Reilly Conferences Flickr stream

Here is the beginning of our conversation:

Talking With Kevin Slavin

Tish Shute: You began your talk today about visibility and where some of the algorithmic masters of disguise went to work, after they had solved the math behind stealth bombers.  I thought perhaps you were leading into ideas about a reverse surveillance society.

But you surprised me, as I felt you made visibility itself kind of a non-issue by the end of your presentation and that counter surveillance became basically a time and speed issue.  Now I am not sure quite how to imagine a counter-surveillance society, something I try to think about…

Kevin Slavin: Well, let’s see.  There’s a couple ways to think about it.  I think one point is just that when we talk about counter-surveillance, we usually locate that as something that comes from  the bottom up, something that comes from the population. Think about the way the plane spotters discovered the CIA black rendition flights.

I think in general, when people talk about counter surveillance, or sousveillance, they imagine it as an inversion of the traditional relationship between the people and the state.

But that’s what’s interesting. What’s happening now, is that there are forms of surveillance and counter-surveillance that are in play beyond any human perceptual horizons. These forms are at their most sophisticated in financial services, in the markets.

If you were a bot, and could read the market legibly (which humans cannot), what you would see, effectively, are bots that are surveilling bots. Then you have bots that are throwing off false information in a bid for counter-surveillance. Many of the bots are, themselves, surveilling other bots; each one of them is trying to figure out what all the other ones are going to do. In essence, it’s an algorithmic arms race, and game theory has become concrete, since the theories are code, the code is action, and the action affects, let’s say: your mortgage.

And so, basically what you have is you have this series of algorithms that are all looking to discern each other, while also trying to prevent themselves from being discerned. I think of the tunnels under the trenches in WWI, tunnels to surveil the trenches, and then, later, tunnels to surveil the tunnels. There’s a few examples of this kind of thing.  But It’s especially strange when it’s computer code, and at the magnitude we’re seeing today.

All of it, as noted in the talk, accounting for 70% of all the trades in the market. 70% of the market trades are never touched by human hands or even seen by human eyes; they don’t move through a conventional cognitive process.  And that’s why you get things like the Credit Suisse algorithm, it was buying, selling 200,000 shares of stocks to itself over and over and over again. It was a bug and it slowed the market to a crawl.

Credit Suisse was fined, in essence, for failing to control an algorithm. Maybe that’s the first time an algorithm was treated like a human, in a way. As if the algorithm broke the law, and Credit Suisse was responsible for letting it do so. For me, that feels like a threshold event.

It’s not that humans never made mistakes when trading on the market. But when algorithms err, they err with magnitude.

The idea that we now have bugs in the United States market economy is really worth looking at.  If Apple can’t keep code bugs from the most simple iPhone apps in a closed and regulated ecosystem, I’m pretty certain we’ll have a lot more Credit Suisse type bugs in the future.

And that will be pretty interesting. There will be viruses, and the operating system they will operate on will be the operating system of the United States. The operating system of your pension, your house, your life insurance. The operating system of currencies and gold.

Tish Shute: I was hard-pressed by the end of your talk to think of like, “Well, what would be the equivalent of, sort of a people’s uprising to create a better fairer society in this kind of world where, really, the things that affect the key aspects of lives most are going on beyond human perception at an algorithmic level?”  But you made a pretty radical suggestion at the end…

Kevin Slavin: Well I think increasingly the markets have become delaminated from anything meaningful. First from goods, then from fundamentals, and now finally from homo sapiens. So that’s hard to fight.

It’s the race towards abstraction that makes it impossible to simply “resist.” The latest version in the long series of fiscal catastrophes was based on Wall Street finding goods that could be rolled up and sold with false valuations, but goods that would take a long time to fail. Mortgages are handy like that. It’s the tradition of extending the abstraction as long as possible, until finally the bill arrives and the banks fail. I don’t know if that’s something to rise up against or not. It’s like a rally against evil.

But really, I think the point is that it won’t be the people that rise up. It will be the financial services themselves that rise up. They’ll just detach completely.

That was harder to do with cotton or with wheat, with simple futures; they keep financial services tied to the ground.  So what we’re doing is creating increasingly complex financial instruments that are further and further removed from anything you can touch.  Like the way a mortgage is abstract. But, of course, the bottom line is that at the end of that mortgage lies someone’s home.

It’s said that Wall Street is now moving onto life insurance, because that’s going to take even longer to fail.  They’re doing the exact same thing. The word is that they are rolling up CDOs made out of crap life insurance policies, same way they rolled them up with crap mortgages a few years ago.

And those will probably take, I don’t know, 15 or 20 years to unwrap and unravel.  But what you see in the meantime, is that they are looking for things that are increasingly abstract, intangible, removed as far as possible from the experience of everyday life.

So maybe this is good. Maybe that’s financial services rising up. Lifting off. I think best case scenario now is that they actually leave humans alone altogether.  That, someday, they are just trading, effectively, completely arbitrary goods, the stocks could be anything at all, maybe for crops that no longer exist, and I’m just saying that then these bots would no longer affect what we do and what we are, it would just be a robot casino, an invisible paradise in the air.

People are the platform: How Games Can Be Engines of Innovation in Our Lives

See the video of Games that Know Where We Live here (screen shot above)

Kati London, Senior Producer, Area/Code, in her keynote showed how games that know where we live can shift players perspectives – from device aided augmented reality that can shift visual experiences of situated geolocal experiences to a kind of augmented reality that is aimed at shifting or changing a person’s social reality, e.g. the mayor badges in Four Square that change my relationship to the people and the place I am in, and augment engagement and reputation through socially driven consumer tie ins.

Area/Code has recently developed two games for the Knight Foundation that take people as the platform.  Macon Money, uses very simple games dynamics (for more see the video of Kati’s keynote) in a game designed to help “Knight’s continuing efforts to support revitalizing Macon and creating a vibrant college town.”

The other game that Area/Code has designed with the support of the Knight Foundation  is for the Biloxi and Gulf Coast community, a game called Battlestorm.  “The game’s purpose is to increase awareness about natural disasters and change the way people prepare for them.”

3rd Cylinder of Innovation: Build products, business models and entire industries.

Glympse – real-time, private location tracking

Julianne Pepitone, Yahoo! Finance, nailed the essence of Web 2.0 Expo, NYC, this year in her post, Web 2.0 Expo startups are big on neighborhoods, storytelling.  She writes:

“At the Web 2.0 Expo in New York City this week, executives from big sites like Facebook, Twitter and Pandora all spoke about industry trends. But the showcase of 27 startup tech companies stole the show.”

Listen carefully to Tim O’Reilly and Fred Wilson, Union Square Ventures, question their picks from the startup showcase at Web 2.0 Expo.  Also see this video of Fred and Tim discussing their conversations with all the start ups.  This  is one of the clearest public windows onto both how to present your company to VC, and how to figure out what are the most important questions for you as an entrepreneur  building a business in a world of data.

Glympse successfully pitches their “jet pony” strategy for a location based business, and is Fred’s pick.  They hold up well under pressure and answer Tim and Fred’s hard questions about how their start up will not get overtaken by an encumbent player with resources and market share before they can gain traction.   food52 responds to Tim’s probing about their strategy for business data analytics that he points out are vital if they want to survive with the small margins of ecommerce.  There is a list of all the participants in the start up showcase in Brady’s post here. hour.ly was the audience pick.

Shazam for Faces!

My favorite start up was a biometric service doing face, iris, and finger print matching, Tactical Information Systems.

Tim and Fred also liked them, and they have an interesting discussion about the merits or not of approaching your platform through a narrow first application as Tactical Information Systems are with WanderID -  an application to help identifying lost Alzheimer patients.  As Fred pointed out, they are potentially the Shazam for faces, so why start so small?

I  had asked TIS the same question when I met them in the “speed dating” session.  This is just their first toe in the water as they are a two person company at the moment. Their vision for their platform is big.  Mary Haskett and Dr Alex Kilpatrick, the founders of this quintessential jet pony for the algorithmic economies in the sky, are not only a partnership with the credentials to do a  Shazam for faces – see their bios here, they are the people I would want to be running a Shazam for faces!  They really get the consequences of living in a world of data – check out Dr Kilpatrick’s absolute killer Ignite talk, “Defeating Big Brother.” (screenshot below)

How Can Augmented Reality Add Value to the Real Time Internet/Data Operating System?

planefinder.net – an augmented reality app that lets you find information about planes by pointing your phone at the sky, “including flight number, aircraft registration, speed, altitude and how far away it is” (via MacLife).

The new opportunities in the algorithmic economies in the sky were center stage at Web 2.0 Expo and there are some interesting AR apps for the real time internet/data operating system emerging, like planefinder.net.  But Augmented Reality was still pretty low profile at Web 2.0 Expo (except that NVidia augmented reality demo attracted a lot of attention at the sponsors expo).  However, everyone working in the emerging industry of AR should recognize that apps big on “neighborhoods and story telling” are heading right up the AR street, and that platforms like Four Square and Pachube present enormous opportunity to explore the possibilities of AR.  And if augmented reality enthusiasts are not already paying attention to real time data analytics, and Hadoop, they should be (see this post for an excellent round up on Hadoop World).

At Hadoop World, Tim O’Reilly referenced the great tagline from the  IBM commercial:

“Would you be willing to cross the street — blindfolded — on data that was five minutes old? Five hours? Five days?”

As I have noted in several earlier posts – see here and here for starters,  we are just seeing the tools  for developing near field, vision based, mobile, social AR become widely available to developers, so there should be a new level of AR apps emerging through 2011.  There is a wonderful discussion in the comments of this post by Mac Slocum, “How Augmented Reality Apps Can Catch On,” between Mac, Raimo one of the founders of Layar, and Chris Arkenberg on what constitutes a platform for growth for augmented reality.

Mac’s post, the comments and Chris Arkenberg’s post on the latest edition of the Gartner Hype Cycle, that rather curiously placed Augmented reality almost at the peak of inflated expectations. really got me excited about exploring an idea I have been thinking about for a while, which is to get the AR community to discuss the Points of Control map.    See my discussion with Chris Arkenberg here, Platforms for Growth and Points of Control for Augmented Reality. The recording of  John Battelle’s and Tim O’Reilly’s webcast on Points of Control is posted here.

The interactive Points of Control map is an amazing tool to think with! Check it out in movements, territory and movements, acquisition mode.  There is a competition for the most interesting comment and most interesting acquisition suggestion.  The prize is a ticket to Web 2.0 Summit!

What is the Future of Social?

The recent “defection” from Google to Facebook – see Lars Rasmussen, Father Of Google Maps And Google Wave, Heads To Facebook,  is as MG Siegler of TechCrunch points out, “the biggest one since Chrome OS lead Matthew Papakipos made the same jump in June” (TechCrunch also notes “current Facebook CTO Bret Taylor was heavily involved in the launch of Google Maps”).

These moves have drawn my particular attention as did Bret Taylor’s response in his conversation with Brady Forrest at Web 2.0 Expo to Brady’s question, “How soon until we get the Facebook firehose?”

If you have been reading Ugotrade you already know how important I think an open, distributed, standard for real-time communications such as the very innovative Wave Federation Protocol could be for AR development  -  see ARWave and my presentation at MoMo13, Amsterdam last year, The Next Wave of AR: Mobile Social Interaction Right Here, Right Now!

The anticipated release of  Wave in a Box, has raised hopes in the developer community that  WFP will soon become easier to work with, and hopefully more widely adopted.  Like many others, I wonder what will happen to Wave in a Box now?

But the innovation of Wave is deep and broad (and as many have pointed out hugely ambitious).  Perhaps the boldest attempt yet to innovate both at the low level of architecture (where Google is so powerful) and at the high level of the Mark Zuckerberg, “big idea,” which as Tim O’Reilly notes is, “What is the future of social?” MG Siegler noted Facebook Groups Is Sort Of Like Google Wave For Human Beings.

But I deeply hope that the open, distributed standard part of the Wave big idea is not lost in the mix here.

Fourth Cylinder of Innovation: Keep the Ecosystem Going, Create More Value than You Capture

The Points of Control map is interactive, so please click here or on the image above for the full experience.

Tim O’Reilly points out that there is a worrisome dark side to the Points of Control Map – see Tim’s keynote here.  To paraphrase some or his points:

There are companies on the map that are forgetting to think about creating a sustainable ecosystem.  Rather than growing the pie, they are trying to divide up the pie and that threatens to cause the fourth cylinder of innovation to misfire.  This fourth cylinder is essential to the ecosystem.

Tim O’Reilly looks back to the lessons of the personal computing industry which was incredibly vital and creative, and lots of people made money until a couple of big players “sucked all the air out of the ecosystem” and innovation had to go elsewhere.

The Power of Platforms is to create value not just for your company but for other people.   Create value for yourself by creating value for other people.  Tim O’Reilly used the wonderful example of  Henry Ford inventing the weekend so that there would be enough people with time and money to buy his mass produced cars.  Think about building the ecosystem that will support the future your are going to build.  Grow the pie rather than cut up the pie.  This will be the vital fourth cylinder of innovation in a Web Squared world.

Tim O’Reilly has long proposed that  Web 2.0 is all about harnessing collective intelligence,  But as Gartner predicts, “By year end 2012, physical sensors will create 20 percent of non-video internet traffic.” Yet another previously unevenly distributed future is going mainstream, and if you haven’t read it already, now is the time to read this paper by Tim O’Reilly and John Batelle, Web Squared: Web 2.0 Five Years On.

The Consequences of Living in a World of Data

To bring this very long post to a close!  Here are just a few of the key questions re The Consequences of Living in a World of Data that Tim O’Reilly raised during his keynote for Hadoop World:

“How would we solve the problem of digital identity in the age of sensors? (Our smart phones are able to know their users by the way they walk – their gait!)

“How will we input data when our devices are smart enough to listen on their own?”

“How should we think about privacy in a world where data can be triangulated?”

“We are moving to a world in which every device generates useful data, in which every action creates information shadows on the net.”

“Shouldn’t we regulate the misuse of data rather than the possession of it?”

“How do we avoid a data arms race?”

“Create more value than you capture.”

The Points of Control map is interactive, so please click here or on the image above for the full experience.

Today at 4pm EST, 1pm PDT John Battelle and Tim O’Reilly will discuss the Points of Control map and The Battle for the Internet Economy in a Free Webcast:

“More than any time in the history of the Web, incumbents in the network economy are consolidating their power and staking new claims to key points of control. It’s clear that the internet industry has moved into a battle to dominate the Internet Economy.

John Battelle and Tim O’Reilly will debate and discuss these shifting points of control as the board becomes increasingly crowded. They’ll map critical inflection points and identify key players who are clashing to control services and infrastructure as they attempt to expand their territories. They’ll also explore the effect these chokepoints could have on people, government, and the future of technology innovation.”

I’ve been wanting to start a discussion on the  Points of Control map in the Augmented Reality community for a while now, and Chris’ recent post on the latest edition of the Gartner Hype Cycle, “Is AR Ready for the Trough of Disillusionment?” and this post by Mac Slocum, “How Augmented Reality Apps Can Catch On,” and the conversation in the comments between Mac, Raimo (one of the founders of Layar), and Chris, all prompted me to get a conversation started…(see below for all that followed!).  Chris put me on the hot seat back in June when he did this very generous interview with me on Boing Boing, so it was time to turn the tables.

Tim O’Reilly, in his  keynote for Web 2.0 Expo, pointed out there is both a fun and a dark side to the Points of Control map.  There are companies on this map, he noted, that rather than “growing the pie,” are trying to divide up the pie, and they are forgetting to think about creating a sustainable ecosystem. I expect the conversation between Tim O’Reilly and John Battelle to dig deep into this Battle for the Internet Economy.  If, like me, you have another engagement at the time of the webcast, you can register on the site to receive the recording.

AR is still too young to figure in the battles of the giants, but there will be a lot to be learned from this conversation.  And, The Points of Control map is good to think with from the POV of AR in many ways.  As Chris Arkenberg observed:

“When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?”   You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development.  And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.”

Chris Arkenberg is a technology professional with a focus on product strategy & development, specializing in 3D, augmented reality, ubicomp and the social web. He uses research, scenario planning, and foresight methodologies to help organizations anticipate change and adopt a resilient and forward-looking posture in the face of unprecedented uncertainty. His personal work is collected at urbeingrecorded, and his professional profile is here.

He is also one of the founder/organizers of AR DevCamp which is currently scheduled for Dec. 4th (somewhere in SF or The Valley!)  Chris said, “No further details atm (still trying to find a venue and get sponsors) but please direct people to http://ardevcamp.org for upcoming information.”

Talking with Chris Arkenberg

Tish Shute: I know some people thought the positioning of AR by Gartner near the peak of the hype cycle was misguided, and based on a very narrow understanding of AR as used in marketing apps. But reading your post I thought you made a lot of good points.

Chris Arkenberg: It’s tracking hype, right? It’s not necessarily tracking the growth of the technologies or their maturation so much as it’s tracking the general attention level. And what’s interesting to me is that tends to affect the amount of money that goes into those technologies.

Tish Shute: I was particularly interested in your post because I have been writing a post about two recent O’Reilly events in NYC, Maker Faire, Web 2.0 Expo, and then Hadoop World, where Tim gave a very interesting 45 minute keynote.   AR was pretty low profile at all three events.   But the NVidia augmented reality demo attracted a lot of attention at the sponsors expo, and Usman Haque, Founder of Pachube announced in his presentation, they are working on an augmented reality interface for Pachube called Porthole, its designed for facilities management and, “as a consumer-oriented application that extends the universe of Pachube data into the context of AR – a ‘porthole’ into Pachube’s data environments.. “  Usman also mentioned, when I talked to him, that he is contributing to the AR standards discussion and on the program committee now for the W3C group on augmented reality.  For more on this standards discussion and the Pachube AR interface, see Chris Burman’s paper for the W3C, Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual.”

I think pioneers in the augmented reality commmunity should pay attention to these wider conversations about the Battle for the Internet Economy, and the exploration of the  “Platforms for Growth” theme at Web 2.0 Expo is very important- this is a course also a nudge to read my upcoming post on these O’Reilly events!

Also I have another project I have been chewing on that I would like to talk to you about.   I want to start an AR conversation about the wonderful Points of Control map produce for Web 2.0 summit by John Battelle. [ Note there will be, "Battle for the Internet Economy" free Web2Summit webcast w/ @johnbattelle & @timoreilly Wed 10/27 at 1pm PT http://bit.ly/b46cmb #w2s]

Up to this point, understandably given the immaturity of the technology, AR has little role in the “Battle for the Internet Economy.”  But this doesn’t mean that the map isn’t good for AR visionaries, enthusiasts, entrepreneurs, and developers to think with.   And both you and Tim have pointed out the potential for AR to leverage the giant data subsystems in the sky.   I have to say the positioning of Cloud Computing on the brink of heading down into the trough of disillusionment in this recent rendition of the Gartner Hype Cycle seems ridiculous!

Cloud Computing is already ubiquitous hardly seems credible that it is headed for a trough of disillusionment!

Chris Arkenberg: Yeah, it’s ubiquitous so why even talk about it when it’s your fundamental infrastructure?

Tish Shute: Yeah and I seriously doubt it is imminently headed for a trough of disillusionment….and this brings me back to the Points of Control Map which as John Batelle points out, “aims to identify key players who are battling to control the services and infrastructure of a websquared world” in which the “Web and the world intertwine through mobile and sensor platforms.”  This instrumented world, of course, creates a great deal of opportunity for augmented reality. Have you seen that, that points of control map?

Chris Arkenberg: I think I have, actually.

Tish Shute: There has been much debate about how this intertwining of the web and the world will play out in augmented reality.   Chris Burman points out in his position paper for W3C,  Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual”, that “trying to draw parallels between a browser based web and the possibilities of AR may solve issues of information distribution in the short-term,”  but it must not have a limiting effect in the long-term.   But now we at least have one web standards-based browser for AR thanks to the work of Blair MacIntyre and the Georgia Tech team.  But I think the discussion in the comments of Mac Slocum’s recent post, “How Augmented Reality Apps Can Catch On” is an interesting starting point from which to think about platforms of growth for AR.  I am not sure if I am stretching his meaning but I think Raimo, Layar, is suggesting that what the Point of Control map call the the Plains of Media content is very important to the growth of the fledgling AR industry right now. And I would agree with this, and add that the neighboring terrain of gaming levels will be pretty key as one of my other favorite AR start ups Ogmento hopes to reveal in the near future! But what do you think was most important in this brief but pithy dialogue between you Raimo and Mac?

[The screenshot above is a teaser video the Gary Hayes from MUVEdesign for his upcoming (2011 release date), game called Time Treasure.  See Gary's blog for more and Gary's  post from over a year ago on AR Business models.  Thomas K. Carpenter, on Games Alfresco notes, "I think this is a terrific idea and I find it interesting he’s planning this on a tablet rather than a smartphone."

Chris Arkenberg: The way I took it…And to give a little bit of context, I came from sort of this apprehension of augmented reality as an expression of the existing Internet. So as sort of a visualization layer that allows you to kind of draw out data, and then, with all the affordances of being able to anchor it to real world things.

And my own sort of path has led me to want to really try to understand that and refine it, particularly with respect to the sort of Internet of things and the smarter planet idea of just having embedded systems everywhere. And specifically, what is the value-add for augmented reality as a visualization layer of an instrumented world?

And so that’s caused me to be a bit biased towards that side of AR. And the way I took Raimo’s comment was that he was saying that, “You know, really what we’re interested in is media.” That he was effectively saying that AR for them is really just about that space between the screen and the the world, or between your eyes and the world, and what you can do there.

Certainly I had considered it in the past, but I hadn’t really focused on it or assumed that it was a priority as a business model. And so he kind of reminded me that, actually, there’s a lot of entertainment applications. There’s a lot of, obviously, advertising and marketing applications.
And so I felt that I was being a little narrow in my focus…

Tish Shute: Yes this comes to the heart of what I am interested in about the role AR can play in opening up new relationships to the world of data that we live, not just making it more accessible and useful to us when and where we need it, but AR as a road to reimagining  it..

Have you seen any interesting work yet to explore these great data economies in the cloud through AR. I mean can you think of any others - there is planefinder.net but others?

Chris Arkenberg: I’ve seen a few just sort of skunk works type applications that people have been playing around with, again, to try and reveal things. One of them was similar to the aircraft, but it was more for military use and being able to identify things of interest in the sky. I’ve seen a couple other for navigation, so being able to identify mountain peaks on a visual plane, for example, but this isn’t so much about revealing an instrumented world.

Tish Shute: Yeah, I think that was from the Imagination right? I know that’s an interesting one. Usman at Web 2.0 Expo, in his presentation, mentioned the work Pachube is doing on an Augmented Reality interface. I interviewed Usman again as my last long interview with him was nearly 18 months ago now and Pachube is well on the way to becoming the Facebook of Data or the analogy that Usman prefers - the Twitter of sensors!

Chris Arkenberg: Hmm, interesting.

Tish Shute: And to go back to your comments on Augmented Reality not getting caught in some of the traps that have made virtual worlds lose relevancy I think that is vital that AR developers understand the strategic possibilities of key points of control in the internet economy because the isolation and Balkanization of virtual worlds were certainly a factor in their rapid slide into the trough of disillusionment - although many would argue that a fundamental flaw in the kind of virtual experience that Second Life and other virtual worlds constructed was really the fatal flaw (see James Turner's interview with Kevin Slavin Reality has a gaming layer).

But Second Life’s isolation from the other great network economies of the internet was certainly a limiting factor.

Chris Arkenberg: And that’s been exactly my sense, and I’ve, over the years, tried to encourage development in that direction for virtual worlds. I did work, through Adobe, to help develop Atmosphere 3D back in the the early 2000’s. And we did a lot of work to try and understand the marketplace and the specific value-add of doing things in 3D over 2D.

And this is kind of why I keep referring back to VR and VW’s with respect to augmented reality, is that with immersive worlds, there was this idea…there was this big rush. Everybody was so excited about it. It was obviously the next cool thing. And everybody wanted to try to do everything in it. You could do your shopping in virtual worlds. You could have meetings in virtual worlds.

Tish Shute: and shopping, yes ..that didn't work out so well!

Chris Arkenberg: And everybody was very excited in developing these things. And what it really came down it is, “Yeah, you can, but it’s actually a lot better to do those things on a flat plane or in person.” Meeting Place, WebEx, TelePresence - those tools generally do a much better job at facilitating TelePresence meetings than a virtual world does. The same with TelePresent Education. There are only very specific things that both VR and AR are really good at.

And that’s where I find myself with augmented reality right now, trying to really pick through that and critically look at which uses are really appropriate for an AR overlay. And again, I think that’s why the hype cycle is important, because it reflects back this desire that AR is going to be the next big thing - the be-all, end-all of interacting with data in the cloud - and forces us all to take a critical look at why we should do things in AR instead of on a screen.

AR is not going to work well for most things but it’s going to be very good for certain uses. Right now I’m very keen at trying to understand what those things might be.

Tish Shute: I had this wonderful conversation (more in an upcoming post) with Kevin Slavin one of the founders of Area/Code at Web 2.0 Expo and I think some of what he describes about the data brokerages of High Frequency trading have some interesting implications for AR’s role, say, in ubiquitous computing. The trading markets are now pretty much dominated by machine to machine intelligence; machine to machine brokerages. They are basically game economies on the scale that we can barely wrap our heads around where the speed that bots and algo traders can access the network is the key. We really have no clue what is going on until we lose our house…

Kevin was also interviewed by James Turner on O’Reilly Radar. He talked about how much of the interesting work in location based mobile social apps is defined in opposition to the model of Second Life. He also talked to me about how we are seeing “first life” take on the qualities of “second life.” What goes on the trading floor is largely a performance secondary to a more important world of machine intelligence with giant co-located servers and bots fighting for trading advantages measured in fractions of seconds.

He pointed out how we draw on all these tropes from sci-fi movies, these HUDs based on ideas of machine intelligence where the robot talks to the other robot in English through an English HUD!  Many of our current visual tropes for AR are perhaps just as inadequate for the kind of data driven world we live in.

Of course, when you are thinking of having fun with dinosaurs, or illustrated books, or whatever, this is not, perhaps, an issue.  But if you are thinking of augmented reality interfaces as being important in a battle for network economy, and platforms for growth,  how this new interface helps us live better in a world of data is an important issue.

Chris Arkenberg: Now, does that indicate that the UI just needs more overhaul and innovation, or more that the visual interface for those experiences shouldn’t really leave the screen? It shouldn’t move on to the view plane?

Tish Shute: Yes we have a few concept videos that try and explore this ..

Chris Arkenberg: Well, and I think this will happen at the level of human-computer interface. I mean that’s always been its role, in making coherent the sort of machine mind, for lack of a better term, making it coherent to the human mind. So I mean there is a lot of this sort of machine intelligence, the semantic Web 3.0 revolution, where it really is about enabling machines, and agents, and bots to understand the content that we’re feeding them.

But at the end of the day, they, for now, need to be providing value to us human operators. So there’s always going to be a role for human-computer interface and user experience design to make this stuff meaningful.

I mean, if you look at the revolution in visualization & data viz, this is of incredible value because it takes a tremendous amount of data and collates it into a glanceable graphic that you can look at and immediately comprehend massive amounts of data because it’s delivered in a handy, visual way.

So I see that as a fascinating design challenge, how the user experience of the data world can be translated into meaningful human interaction.

Tish Shute: Yeah. And when we see Stamen Design pursuing a big idea in AR, that’s when we might start to rock and roll, right?

Chris Arkenberg: Yeah. In my article, I sort of jokingly suggested that Apple will create the iShades. But, they’ve got the track record of being way ahead of the curve and delivering the future in very bold forms.

Tish Shute: A key part for the battle for the network economy is to bring the complexity of data into the human realm in a way that increases human agency. Kevin suggests that the giant robot casinos of markets should actually lift off into total abstractions as theses machine-driven trades get back into the human realm in ways that are so damaging to our lives - a lost house or job! The notion of a counterveillence society where people have more agency over the important aspects of their lives, health, housing, job (which I discussed with Kevin - interview upcoming) has gotten pretty tricky!

But I think we will begin to see AR eyewear for specific applications (gaming and industrial) get more common fairly soon - possibly as smart phone accessories.

And it is clear that AR is going to be, increasingly,  a part of our entertainment smorgesborg in coming months. Itouch has a camera (although lower resolution), Nintendo’s are AR-ready and many aspects of the AR vision of hands-free spatial interfaces will go mainstream through Natal.

But we are yet to see an app/platform emerge for mobile. Social AR games that turn every bar and cafe and ultimately the whole city into a gaming venue  -although I think Ogmento and MUVE aim to lead the way here! Will an AR company achieve Zynga level success by using the Foursquare, for example?

My feeling is that the lesson of Zynga is pretty important for mobile social AR games. Could Flash social gaming have taken off without Facebook?

Chris Arkenberg: And that’s the real driver. And again, as you mentioned with Second Life, and this was exactly my own sense, is that they stuck to the closed garden model and didn’t get the power of social and collaboration. They attempted to add some of those affordances within the world, but, you know, ultimately most people aren’t in virtual worlds, and most people aren’t using augmented reality. So leveraging the really predominate platforms like Twitter and Facebook and Foursquare, being able to leverage those affordances, that connectivity, into a platform like augmented reality, I think, is really critical. Because again, you get nothing unless you have the masses, unless you have people present.

Tish Shute: In AR research there is a long history of the notion of powerful AR-dedicated devices, but smart phones and tablets are good enough,  and can launch augmented reality into the heart of the internet economy. I think  the elusive AR eyewear will come to us initially as a smart phone accessory for specific apps.  But, for the moment, most AR apps make little attempt to play in the wider internet economy.

Chris Arkenberg: And I think it’s actually much lower hanging fruit, really, to do gaming, marketing, transmedia. Because then you don’t really care about the cloud, or maybe you only really care about a little part of it that your gaming property is addressing. Then it becomes much more about entertainment, and much more about persuasion, and sensationalism. And if you’ve got dancing dinosaurs on your street, great! It’s entertaining, it’s cool, it’s new. That stuff is fairly straightforward.

I keep coming back to this idea of, you know, the instrumented city. What sort of data trails do you get out of a fully instrumented city? So maybe you get traffic patterns, maybe you get geo-local movements of masses, maybe you get energy usage, that sort of thing, all the, sort of heat maps you can generate from a city. But then what good does it do to be able to have that on an augmented reality layer versus just looking at it on a mobile device or looking at it on your laptop?

Tish Shute: Of course the use cases for “magic lens” AR are different from the kind of hands free, 360 view with tightly registered media, that a full vision of AR has always promised. The 360 view is quite a different metaphor from the web and mobile rectangular screens.

Chris Arkenberg: Yes, yes.

Tish Shute: Did you see that great parody of Michael Jackson's “Beat It” with the iPads versus the iPhones, right?

Chris Arkenberg: Oh, really?

Tish Shute: I tweeted it cos i thought it was quite funny and a little close to the bone!
[laughter]

“ur wanna an ipatch 2 b the new fad?” #AR gets cameo in Twitter, iPads & iPhone’s Michael Jackson-Inspired Parody via @mashable

It is hard to get away from the importance of eyewear when discussing AR!

Chris Arkenberg: Yes, so the hardware, to me, is a big stumbling point right now, or it’s a large gating factor, I think, for realizing what an augmented reality vision could really be like. That it really does need to be heads up. This holding the phone up in front of you is fun to demonstrate that it’s possible, and it’s valuable in some ways…

Tish Shute: And it’s particularly nice in some applications like the planes app, the Acrossair subway app where you hold the phone down and get the arrow, right?

Chris Arkenberg: Yeah, the way-finding stuff I think is really valuable…

Tish Shute: Sixth Sense really caught people’s imagination because it managed to deliver the gesture interface with cheap hardware, even if projection has limited uses (no brightly lit spaces or privacy for example!).

The other important and as yet unrealized part of the AR dream is real-time communications. Many interesting uses cases would require this. As you know that is my chief excitement, along with federation, in the Google Wave Servers for (which should soon be released at Wave in a Box) for ARWave.

Chris Arkenberg: Well my sense of Wave is that it was a ChromeOS protocol that they instantiated, or that they exhibited in the public deployment of Google Wave. That that was a proof of their sort of low level architectural solution. Because, you know, they’ve been rumored to be working on this cloud OS for some time. And so my sense is that Wave is actually one of their core components of that cloud OS, and that it just happened to incarnate for the public in a test run as Google Wave.

Tish Shute: I do hope that Wave In the Box will lower the barriers to entry to people experimenting with this technology. The FedOne server was just way too hard for most people to take the time to set up. Of course, it is the brilliance of the Wave Operational Transform work that also poses problems in terms of ease of use. But Wave Federation Protocol is pretty innovative. And could even play an important role in a real time communications for AR eyewear connected to smartphones. The challenges that Wave takes on re real-time communications, federation, permissions and filters are pretty important ones for AR…

Chris Arkenberg: Especially when you’re trying to federate a lot of permissions and filter a lot of data, which all of that gets even more important when you have a visual layer between you and the real world.

Tish Shute: You got it. Yeah!

Chris Arkenberg: I think that’s really valuable real estate, both for third parties that want to get access to your eyes, as well as for you, as the user, who still needs to navigate through the phenomenal world and not be occluded by massive amounts of overhead data.

Tish Shute: Yes, I am sure Google has big plans for the next level of cloud computing and Wave looks at some key challenges. I suppose federation poses some key business problems. I think it was Michael Jones who said to me that it was a bit like socialism in that you have to be willing to give something up for the greater good.

Perhaps federation does not present enough appeal because of its challenges re business models?

Chris Arkenberg: Well, I wonder. I mean there’s got to be some value for their ad platform as ads are moving more towards this personalized experience. Advertising is becoming less of a shotgun blast and more of a very precise, surgical strike. So being able to track user data to such a fine degree to mobilize the appropriate ads around them wherever they are, on any platform, is certainly very valuable to Google and their ad ecology.

Tish Shute: Many people have high hopes that HTML 5 by lowering the barrier of entry for  browser style AR could also pave the way for some interesting AR work..

Chris Arkenberg: Well, as much as I would hope that all the different players are going to come together and establish some shared set of standards, really, what’s happening is it’s a rush to the finish line to be the first…to get the most penetration in the marketplace so that Layar, for example, can say, “It’s official. We’re the platform.” And then the consolidation that will follow, where the Googles and the other big players like Qualcomm say, “OK, it’s mature enough. We’ll start buying up all the smaller companies.”

And that’s where the real challenge is right now is that there are no standards. It’s such an immature technology that you have a lot of different players trying to establish the ground rules. And again, this is one of the challenges that faced public virtual worlds, is that you had a lot of different virtual worlds that weren’t talking to each other in any particular way, and that they each had their own development platform. And so you end up with a very fractured ecosystem or set of competing ecosystems, which is kind of what’s happening with AR right now, where a developer has to choose between a number of different new platforms or hedge by deploying across multiple platforms. Basically, the web browser wars are set to be recapitulated by the AR browsers.

Among them, Layar and Metaio seem to be getting the most traction. But there’s still not a really strong case for a unified development ecosystem to emerge.

Tish Shute: So a discussion of ecosystem development brings us back to the Points of Control Map I think. So what do you see as key points of interest for AR developers to watch in the Points of Control Map? And where do you want to sort of put your bets, right? We are still really waiting for mobile social AR to emerge into the mainstream.

Chris Arkenberg: Yes. And that’s primarily the shortcoming of the hardware itself, but also of the accuracy of current GPS technology. That’s another kind of gating factor, because again, AR wants to be able to express the data within a distinct place or object.

So in a lot of ways, other than kind of what we’ve allowed for the broader entertainment purposes, for AR to really work, there needs to be more resolution in GPS location. So for it to be truly locative…because it’s OK to tell Foursquare that you’re in Bar X. But if you want to be able to draw data directly on a wall within that bar, or do advertising over the marquee on the front, you need more factors to accurately register those images on a discreet location. So that’s another, sort of, aspect of the immaturity of AR, is that it’s still very hard to register things on discreet locations without employing a number of diverse triangulation methods.

Tish Shute: Right. The mobile AR games we see at the moment are really just faking a relationship to the physical world unless they rely on markers or some limited form of natural feature recognition which is really just a more sophisticated form of markers. But the Qualcomm SDK does offer some opportunities to tie AR media to the world more tightly as does the Metaio SDK. But in terms of a mobile social AR game that could be like the Cape of Zynga to FourSquare in Location Basin [see the Points of Control map]… We haven’t seen anything close yet.

AR should be able to bring the check-in mode to any object in our environment.

Chris Arkenberg: Yes, yes. And that’s actually one of the early interests I had in the notion of social augmented reality. I wanted a way to tag my community with invisible annotations that only certain people could read, and found pretty quickly that that’s very difficult to do. I mean you can kind of do some regional tagging, like on a beach, for example, but if you wanted to tag the bench that was on the cliff above the beach, it’s very difficult to do that using strictly locative reckoning.

There’s all sorts of really cool social engagement that can be revealed when people are allowed to attach things to the world around them, to the streets they normally pass through, or the points of interest that they normally engage in. To be able to author on the fly on the streets and attach it discreetly to an object effectively.

Tish Shute: And yes we do have all kinds of markers and QR codes. But Erick Schonfeld of Tech Crunch made a good point that QR codes: “Until QR code scanners become a default feature of most smartphones and they start to become actually useful enough for people to go through the trouble to scan them, they will remain a gee-whiz feature nobody uses.”

Chris Arkenberg: So again, this gets back to competing standards and who gets access to the phone stack, the bundle. Who gets the OEM deal…?

Tish Shute: Yes, the battles for the networks on the Handset Plains are pretty important for AR!
[laughter] I think Layar have made some smart moves on The Handset Plains.

And there are a lot of acquisitions of nearfield technology to look at.  If I remember rightly Ebay bought the Red Laser tech from Occipital – now there’s any interesting company. Their panorama stuff rocks!

Chris Arkenberg: Right. There’s a lot of nearfield stuff that’s supposed to hit all of the major mobile platforms in the next year or so.

I mean I think where this is heading, in my mind, is basically smart motes. You know, little nearfield wide-range RFID’s that are the size of a small, tiny square that you could attach to just about anything and then program it to be a representative of your establishment or of an object, that then you can start to tag just about anything. I mean you can’t rely on geo to do it, but if you have a Nearfield chip there that costs maybe like two cents to buy in bulk, and you can flash program it, then you can start to attach data to just about anything.

Tish Shute: Yes ‘cos some things still remain very difficult for near field image recognition technologies like Google Goggles.

Chris Arkenberg: Well, if your phone can interrogate for Nearfield devices, and it detects a chip in its near field, it can then interrogate that chip. The chip may contain flash data on itself, or it may contain the local server in the establishment, or it may go to the cloud and get that data back.

Tish Shute: Yes there is moverment from the top and open source hardware like Arduino has created an opportunity for all sorts of creativity with instrumented environments.  And the handheld sensors in our pockets – our smart phones create a lot of opportunity for bottom up innovation too.

Chris Arkenberg: I mean that’s my guess. If you look at what IBM is doing with their Smarter Planet initiative, they’re partnering with a lot of municipalities, and obviously with a lot of businesses and their global supply chains.

But they’re basically working with municipalities and all these stakeholders to instrument their territory, their business, or their city, as it were. So they’re working to provide embedded sensors and the software necessary to read them out and run reports & viz. And presumably that software can extend to include some sort of mobile device to interrogate the sensors and read the data.

That’s kind of a top-down approach of a very large global company working with top-down governance bodies to do this. Simultaneously you have the maker crowd experimenting with Arduino and such to build from the grassroots, the bottom up approach.

And that’s primarily gated by the amount of learning it takes to be able to program these devices, to be able to hack them. Typically, the grassroots creators who make these devices don’t have the luxury of very large budgets to make things highly usable and Wizywig.

So the bottom up community is a sandbox to create tremendous amounts of innovation, because they are unconstrained by the very real financial needs of the top down innovators. And so you get a lot of fascinating innovation, a very rich ecology from the bottom-up approach, but you don’t get a lot of wide distribution. But that does filter up to and inform the top down approach that has a lot more money to put into this stuff. And it ultimately has to respond to the needs of the marketplace.

I mean if there’s an answer to the question of whether something like AR will succeed through the bottom-up grassroots approach or the top-down industry approach, I would say it would be both. That handsets will be hacked to read the bottom up innovations of the maker community, and handsets will be preprogrammed to read the top down efforts of the IBMs of the world.

Tish Shute: Yes but i have to say it is very time-consuming hacking phones (I have just seen a few days suck up in this myself so that I could upgrade my G1 to try out the new ARWave client!). I mean Android has obviously been the platform of choice because of openness but the business model of iPhone and its market share in the US sure make it important for developers.  It’s like you don’t exist if you don’t have an iphone app for what you are doing.

Chris Arkenberg: Yeah, and that’s the challenge, because at the end of the day developers prefer not to work for free and a solid, reliable mechanism to monetize their efforts becomes very appealing.

When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?” You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development. And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.

Tish Shute: Some of the important platforms for AR don’t appear to have spots on the map like Google Street View and other mapping technologies that hold out so much hope for AR, or am I missing something?

Chris Arkenberg: You mean on the map?

Tish Shute: Yes for the full vision of AR we need sensor integration, computer vision and cool mapping technologies to come together. Do you see where Google Maps and Google Street View… Where would they be?

Chris Arkenberg: Yeah, I mean it’s certainly content, it’s location…

Are you familiar with Earthmine?

Tish Shute: Yes, yes I am, definitely. Earth Mine, Simple Geo, Google Street View, user generated internet photo sets like Flickr all of these could be very important to AR, potentially.

Chris Arkenberg: Well, and the interesting thing about Earthmine is that they’re effectively trying to do an extremely precise pixel to pixel location mapping. So they’re taking pictures of cities just like Street View, except they’re using the Z axis to interrogate depth and then using very precise geolocation to attach a GPS signature to each pixel that they’re registering in their images. Effectively, you get a one-to-one data set between pixels and locations. And so you can look at something like Google Street View, and if you point to the side of a building, in theory, it should know exactly where that is.

They’re rolling this out with the idea of being able to tag augmented reality objects in layers directly to surfaces in the real world. So that’s another approach to trying to get accurate registration and to try and create what are essentially mirror worlds. Then your Google Street View becomes a canvas for authoring the blended world, because if you plop a 3D object into Street View on your desktop, and then you go out to that location with your AR headset, you’ll see that 3D object on the actual street.

Tish Shute: There was some experimental work with Google Earth as a platform for a kind of simulated AR but I suppose Google Earth doesn’t figure in the battle for the network economy as it never got developed as a platform.

Chris Arkenberg: It hasn’t tried to become a platform, to my knowledge. I mean I know some people are doing stuff with it, but as far as I know, Google owns it, they did it the best because they have the best maps, and there’s not a huge ecosystem of development that’s based around it other than content layers.

And my sense of everything else on the Points of Control map is they’re looking more at these sort of platform technologies that…

Tish Shute: Yes, re platforms for growth for AR. Gaming consoles will probably emerge as a significant platform for AR this year.

Chris Arkenberg: There will be much more of a blended reality experience in the living room for sure, and with interactive billboards. Digital mirrors are another area. So I mean if we kind of extend AR to include just blended reality in general, you know, this is moving into our culture through a number of different points. As you mentioned, it will be in the living room, it will be in our department stores where you can preview different outfits in their mirror. We’re already seeing these giant interactive digital billboards in Times Square and other areas.

It’s funny. I mean for me, the sort of blended reality aside, the augmented reality, to me, is actually a very simple proposition in some respects. When I look at this map, augmented reality is just an interface layer to this map in my mind, just as it’s an interface layer to the cloud and it’s an interface layer to the instrumented world. It’s a way to get information out of our devices and onto the world.

Tish Shute: The importance of leveraging existing platforms has become pretty clear but it is interesting Facebook definitely gave Zynga the opportunity but would Facebook be so big without Zinga’s social gaming boost?

Chris Arkenberg: I feel that Zynga has definitely helped its growth…But I think Zynga has benefited a lot more from Facebook than Facebook has from Zynga.

Tish Shute: Zynga certainly proved you could build a profitable business on Facebook’s API!

Chris Arkenberg: They did. And they also really validated the Facebook ecosystem and the platform. They really extended it… Zynga benefited from the massive social affordances that Facebook had already architected and developed. They brought gaming directly into Facebook, and particularly, this emerging brand of lightweight social gaming that when you sit it on top of a massive global social network like Facebook, it suddenly lights up.

Tish Shute: AR pioneers should quite carefully go through this map. There is so much to think about here. I’m a kind of fanatic about Streams of Activity in AR. Real time brokerages and their potential for AR is something I am fascinated by. That is one reason I love the ARWave project.

Anselm Hook, to me, is one of the great thinkers in this area of real time brokerages – with his project Angel, and the work of Ushahidi, which is now the platform for augmented foraging (see here). Anselm is now working on AR at PARC which is exciting.

Chris Arkenberg: Well, there are some challenges working with data streams. Presentation and filtering I think is a big challenge with any sort of stream. Because obviously, you have a lot of potential data to manage, to parse, and to make valuable and comprehensible. So I think this is bound very closely to being able to personalize experiences, or having very discreet valuable experiences. Disaster relief, for example, I think is an interesting idea that ties into the Pachube type of work. Where, if you had the headset and you were a relief worker, and you had immediate lightweight, non-intrusive, heads up alpha channel overlay, waypoint markers showing you all of the disaster locations or points of need, AR becomes extremely valuable, because it’s a primarily hands-free environment. This is why the military stuff is so interesting.

Tish Shute: Ha! We are running into the eye patch/shades/goggles/sexy specs thing again. But filtering and making streams of activity relevant will be very interesting for AR.  Again that it why I love the Wave Federation Protocol work because what they have built into their XMPP extensions. You can have your real-time personal data streams, or community streams, or broadcast publicly – the permissions are built.

And Thomas Wrobel’s original vision of these layers and channels is only fully expressed if you have the eyewear.

Chris Arkenberg: Well, and it becomes redundant if it’s on a mobile. To use a very basic example, Twitter, obviously there’s an app you can view those streams of activity on the camera stream. But you can view that real time data on the screen. Why do you need to see it heads up?

The reason I really pay attention to what the military is investing in, one, because they have a ton of money, but also because they tend to represent the core bio survival needs of the species…So, when I look at computing, I see this very obvious trend of computers getting smaller and smaller and closer and closer to us because they’re so valuable to our success. They give us so much valuable information for engaging our world on a moment by moment basis. So, of course now we have these tiny little handheld devices that give us access to the global knowledge depositories of human history, because it’s so useful to have that stuff right at hand.

The only impediment now is that it takes one of our hands, if not both of them, to access it. So if you are in the natural world, which we are all always in the natural world, ultimately, you want your hands free in order to engage with the world on a physical level.

I see computation, or rather, our access to computation is just going to get thinner and thinner, and we’ll very soon move into eyewear, and inevitably, we’ll move into brain computer interface in some capacity.

So when you’re the disaster worker, or a deployed soldier, or the extreme mountain biker, or the heli-skier, or just an adventurer, there are a lot of very practical reasons to have access to information on a heads-up plane. I see AR as being so profound and so valuable, but we’re getting a glimpse of it in its infancy, and it’s got a ways to go to be able to really contain what it is we’re reaching for.

Tish Shute: I agree.

Chris Arkenberg: And that’s been a big criticism I’ve had with all the existing AR implementations that I’ve seen, is that the UI really needs a revolution. It’s very heavy handed. It is not dynamic, even though it’s supposed to be. It does not take advantage of transparencies. It treats the screen like a screen. It doesn’t treat the screen like a window onto the real world. When you’re looking on the real world, you don’t want a lot of occlusion. You want very soft-touch indicators of a data shadow behind something that you can then address and then have it call out the information that’s important to you.

Tish Shute: Now, that’s a very nice kind of image you’ve conjured for me there. Do you see that more could be done on the smartphone than is being done within that? Or are we like waiting for the old ishades?

Chris Arkenberg: I think there’s definitely a lot of room for improvement on the smartphone UI. Nobody’s really played around with it much. And again, I think that’s in part that there hasn’t been a really established platform with enough money to fund interesting UI work. We see it in some of the concept demos that float around every now and then.

I guess it’s both a blessing and curse that I’m always five steps ahead of where I’m trying to get to.

Tish Shute: Yeah, I am familiar with that feeling!

Chris Arkenberg: So I’m always trying to reach for the vision even though it’s a bit distant. I think there’s going to be a lot of development on the handsets. But again, I think we need a lot of refinement. We need a lot of real critical analysis of why this is a good thing.

To get back to the original point of Raimo’s comment, it struck me. And I knew it, but I just had set it aside as gimmickry. But he’s right. Content is a huge driver for this. Just stuff that’s engaging, and fun, and cool, and shows off the technology so they can get enough money to make it through whatever Trough of Disappointment may be waiting.

Tish Shute: Yeah, don’t underestimate the Planes of Content!  They are a great place to get interest and money to keep AR technology moving on, right?

Chris Arkenberg: Yeah, yeah. Because, you know, there’s a lot of freedom there. And you can piggyback on all the rest of the content that’s out there and jump on memes and marketing objectives, etc…

And there’s a lot of stuff…I’m blanking on some of the names, but some of these historical recreations of city streets. There’s a street in London where they overlaid historical photos in a really compelling experience. [Museum of London - http://www.museumoflondon.org.uk/] Again, I’m completely forgetting the attributions, but hose are the type of things that can really be pursued on the existing platforms. There is stuff that’s really compelling and really cool.

I heard of another interesting use case – and I should say that I can’t find attributions to this anywhere on the web and I may be paraphrasing or mis-representing the actual work, but I think the concept is worth exploring anyway. But the idea was that you could take the locations of border checkpoints and conflict sites in Palestine and Israel and visually overlay them on an AR layer in San Francisco. And it would do some sort of transposition where you could virtually view these things in San Francisco with the same locational mapping superimposed. So you could see where the checkpoints where. You could see where the wall was. You could see where suicide bombings were and where there had been conflicts. [I cannot find any citations for this!]

Tish Shute: But with an AR view? But why would you use an AR view if you are in San Francisco, then?

Chris Arkenberg: Because it superimposes two realities, translating the Gaza conflict into San Francisco as you are walking around. You can interrogate the world. There’s a discoverability aspect where you’re using the headset to reveal things, or the handset rather, to reveal things that you could not see otherwise in your city. It was done as an art piece, but as a provocative, obviously political art piece.

Tish Shute: Very interesting. I’d love to see that. Because that’s interesting to get away from this idea that you actually have to sort of have this one to one relationship between the data and the world is kinda nice, isn’t it? Well, not one to one, but a very literal…getting away from that literalness is kind of good.

Chris Arkenberg: And that’s a possibility of virtual reality and augmented reality merging, that maybe virtual reality is actually going to do best by coming out of the box and writing itself over our reality, so that as you are walking around, you are no longer seeing San Francisco, but you are seeing part of Everquest or World of Warcraft.

Tish Shute: Well this is where Bruce Sterling gets to that point he made in his keynote for are2010, that if we actually have viable AR eyewear, then you get the gothic stepsister of AR, VR rising from the grave!  He asks whether the very charm of augmented reality, is in fact that it adds rather than subtracts from your engagement with the world and that getting get sucked back into the black hole of VR might not be so great.

Chris Arkenberg: And then you get all sorts of interesting challenges to social cohesion if you have a lot of different people experiencing very different worlds, effectively. That if there is no real consensual reality and a majority of your local populous is, in fact, experiencing very different and unique versions of the world, what does that do to social cohesion? How does that reinforce tribalism, for example, when only you and certain others get to opt in to a particular layer view of the world?

Tish Shute: Yes Jamais Cascio wrote an interesting piece on that issue on AR and social cohesion a while back.

An eye patch is a more logical vision than the goggles in many ways but I suppose the loss is stereo vision?

Chris Arkenberg: And actually, there were developments in military helicopter technology many years ago that used a single pane square of glass over the eye mounted to the helmets of pilots. And then they drew various bits of heads-up information on it. So that ensures that you’re having a real strong engagement with the real world, which, obviously, when you’re a helicopter pilot is quite important. But you still have access to the data layer of the invisible world.

Tish Shute: I just went to Hadoop World and I have to say, I was awestruck about how big that’s got. I mean Hadoop has gone from like zero to huge in just a few years. I mean it’s just like now everyone has the power of the Google big table at their fingertips.

What’s the play for AR in the land of search?

I could imagine Hadoop being very powerful tool for AR analytics?

Have you got any thoughts on the land of search and AR? Of course visual search is proceeding at a fast pace and there is a lot of promise for integrations with AR in the future but the latency for visual search is still pretty high?

Chris Arkenberg: In the near term, not a lot. In the medium term, there’s a larger trend towards virtual agents that you can program or teach to keep watch over things for you as an effort to scale down the data overload. So search is something that’s going to become more personalized and more active. There’s a movement to make it so people can essentially deputize these agents to be always searching for them; to be out there looking for the things that they have told these agents are important to them.

So active search for AR I think presents some challenges, obviously because you need to do text input, typically, or voice input. Voice input, I think, is much more achievable than text input for AR. But I can certainly imagine an AR layer that is being serviced by these agents that we have roaming around the web for us reconciling their visual view of the world with our personalizations. AR apps are contextually aware so it knows that if you’re downtown, it’s not going to be giving you a ton of information about Software as a Service infrastructure, or what have you. But that, instead, it’s going to be handing you little tidbits about a particular clothing brand you’ve opted in to follow and information about music venues & schedules, for example. Or perhaps you’ll be on the lookout for other users that have opted in to publicly tag themselves as a member of this or that affinity.

I keep coming back to this idea of AR as really just a simple visualization layer that all of these other technologies can potentially feed into. So in that sense, search becomes a passive thing that AR is just simply presenting to you in a heads-up, hands-free, or potentially hands-free environment.

Tish Shute: Yes, the big challenge is the stepping stones to that point! Small steps that keep interest going into developing the underlying technology (and not just in research labs!) that will bring us that interface.  We have seen some movement already with Qualcomm.

Chris Arkenberg: And there are bandwidth issues as well, as we can see with the Google Goggles, which is a great idea of visual search. But you have to take a picture and send it to the cloud and wait for your results. It’s not a real-time dynamic interrogation of the world.

Tish Shute: Yes we are really only at the very beginning of AR being ready for prime time.. it would be interesting to ask AR developers how many of them use AR on a daily basis.

Chris Arkenberg: I think a lot of us, we’re just informed by the sci-fi myths and fascinated with the potential now that’s it’s starting to become real. But I think we all kinda get that it’s still extraordinarily young. I mean the web is extraordinarily young. And AR is itself far younger in a lot of ways in its implementations.

Everybody has a lot of excitement about all of the great potentials that are being unleashed by this great wave of the Internet and the web and ubiquitous mobile computing. So that’s why, you know, you look at that map and we talk about AR and you can’t talk about any of the stuff without talking about all of it, in a lot of ways, particularly with something like AR where it’s so ultimately agnostic and could be completely pervasive across all of these layers.

So my fascination is with the future, and I measure our progress towards it by the young nascent offerings from the platform players and the developers. And yeah, a lot of it is…it’s akin to getting that first triangle on the screen in 3D. You know, when the renderer finally works and you get a triangle on the screen, and you go, “Oh my God, it renders.” And then you can start to really build polygons and build objects, and start doing boolian operations, and get light and rendering in there, and textures, and on, and on, and on.
So I’m fascinated by the Layars and the Metaio’s…
[laughter]

Tish Shute: Yes and hats off to all the players in the emerging industry, Layar, Metaio, Ogmento, Total Immsersion, and all the others who are finding clever ways to bring fun aspects of AR into the mainstream, and fuel interest to take the technology to the next level.

Chris Arkenberg: Absolutely. And the hype cycle is very valuable. It has really helped launch the AR industry. It’s brought a lot of eyes, and it’s brought a lot of money into the industry. And it’s forcing people like us to have these conversations to understand how to refine its growth and really focus on the potential in all these different venues, whether it’s trying to save lives, or better understand your city, or have really compelling entertainment experiences.

Everybody’s excited, and everybody’s sharing, and everybody’s trying to move it forward in a way that’s the most productive.

Today Metaio is holding Inside AR in Munich, Germany.   Metaio (the picture above shows Metaio co-founders Thomas Alt and Peter Meier), is behind some of the best known commercial and industrial AR experiences of recent years.  But as important as the many AR projects they have executed are the AR tools that Metaio has made available to developers.  Metaio’s AR products and tools have played an important role in bringing AR to a wider public, and given many developers the opportunity to explore AR.

Inside AR is a great opportunity to see what these AR pioneers  will be up to in the coming months.   I could not make it to Munich this year.  But, fortunately, I had the opportunity to talk with Thomas Alt, recently.   In this conversation – see below, I got a chance to discuss what was going on inside AR with Metaio.

The Fall season is always jam packed with great events, and I wish I could be in two places at once.  But this week, I will be in my home town, NYC, attending Web 2.0 Expo which, reflecting the heat in the NYC tech community, is a sold out event with a very exciting schedule this year (more on some of the presentations that I will be attending later in this post).  If you missed out on tickets to Web 2.0 Expo, all Keynotes will be Streamed Live: TUES 9/28 to THURS 9/30, and keep your eye on @w2e and #w2e on twitter.

Meanwhile, I am missing Inside AR, which had some great speakers lined up, including fellow New Yorker, John Swords, partner and Ringleader at Circ.us.  Hopefully, Swords will share his experiences at next month’s ARNY Meetup which will be “joining forces with another vibrant community – NY Gaming – for an unforgettable night of Augmented Reality Games” on Tuesday, Oct 19th, 6:30 PM at AOL Ventures in New York, NY.

At the most recent ARNY @swords gave a brilliant talk on the possibilities for AR Game development on the newly available opensource Parrot ARDrone platform.  It was great to hear from social game guru @swords on his plans for Parrot ARDrone games, and more.  The picture below of an ARDrone camera view is from John Swords Flickr set.  Swords was flying it inside his garage because the winds outside were too strong.

Also, I kicked off what will hopefully be an ongoing discussion on, “Story Telling with AR and the Big ARNY a collaborative AR Game for NY,” with a few slides.  I have opened up the presentation document for collaboration, so please ping me if you would like to be added as a contributor/editor, and are interested in getting involved.

Ori Inbar, CEO and co-founder of Ogmento, Games Alfresco, ARNY and my co-chair on Augmented Reality Event 2010, suggested The Big ARNY – A Collaborative AR Game Development Project modelled after A Swarm of Angels last year at the First ARNY Meetup – so let’s make it happen!  I will be catching up with Ori in October about what Ogmento has been up to since they became the first VC backed AR Game company!

“Games allow us to see each other, for a moment, in a way that living in a city prevents” – Kevin Slavin

I believe that, AR, to get beyond the stage of “interface du jour” needs to offer us new ways to relate to each other and the world around us so that we can actually improve  and deepen our engagement with reality not just create experiences that are primarily optical  (see James Turner’s interview with Kevin Slavin “Reality has a gaming layer” on not letting “the pleasure of a game and the meaning of a game and the experience of a game rest primarily in the optics.“  And see my recent post, Urban Augmented Realities and Social Augmentations that Matter: Talking with Bruce Sterling, Part 2).

Two of the most inspired creators of urban games,  Kevin Slavin and Kati London of Area/Code will be speaking at Web 2.0. Expo tomorrow.   And you can be sure I will be at both these sessions. Loitering on the Motherboard, Kevin Slavin, is 2:35pm Tuesday, 09/28/2010, and Games that Know Where you Live, Kati London – is a keynote that will also be live streamed – 4:55pm Tuesday, 09/28/2010

Recently Kevin Slavin was interviewed by James Turner, on O’Reilly Radar.   This, Reality has a gaming layar, is a must read piece about a “world where games shape life and life shapes games”  (see @timoreilly). 

Interview with Thomas Alt

Tish Shute: Perhaps you could just start with your background Thomas because I think there’s a lot of newcomers to AR but you are really one of the first movers in commercial AR. How long you’ve been involved in this?

Thomas Alt: Actually I’m an ex-researcher in augmented reality. I started with me actually after getting my master’s work in engineering from the Technical University of Munich working for a big company called Volkswagen. And at that time,1999, we got a research grant for researching how augmented reality could change manufacturing processes in the automobile industry.

And from the research work there, I basically went back to school, did my PhD about augmented reality. And while speaking at a conference, I met Peter Meier who is the co-founder of the company who was also a master’s student writing his thesis about augmented reality. That was in 2002.

And so it really was in the very early days of augmented reality. And both Peter and myself we got really excited about the technology; we saw endless possibilities. We said, ‘OK. Let’s just found a company. We actually founded the company in early 2003 with virtually no money. As a matter of fact the founding capital of the company was 25,000 Euros and this 25,000 Euros were won in a case competition in Germany – a business plan competition.

Tish Shute: So you won 25,000 Euros on this case competition and that’s where Metaio started….

Thomas Alt: Exactly. And to legally found a company in Germany it takes exactly 25,000 Euros so that was the founding capital. We started pretty much like good old SAP started.  It wasn’t in a garage though it was a very small office and we basically built up the business through work, so we don’t have any investors or whatever. Right now we are 66 people located in Munich where our headquarters have been for five years. We have some presence in the US, and we have a venture company in Seoul, South Korea.

Tish Shute: Awesome. I just noticed how fast you’ve been growing. So right now, I’m going to ask a couple of questions about where you see the technology and the emerging industry going.

First, what are the platform of choice for Mobile Augmented Reality at the moment?

Thomas Alt: Obviously in the cellphone hardware space there’s a fierce competition going on. It’s yet to be defined what will be the prevailing platform right now, obviously it’s the iPhone is big now, right? But Android is catching on very, very fast.

Tish Shute: You have pioneered bringing a cross platform SDK for vision assisted AR to a wide community of developers with Junaio and with your partnership with Kooaba – a visual search company from Europe.

Thomas Alt: Yes, yes, and this is how we would, also in the future like to position ourselves with Junaio.  Junaio will be a platform, a technology platform, which will allow users to do whatever they want to do in augmented reality. The API of Junaio is huge in the sense you can do anything from outdoor gaming, to visual search, to normal, uh, lay out style, you know, find the next burger king a mile away kind of super impositions.

Tish Shute: The only licensing you pay is for unifeye right? When you want to use your tool kit right?

Thomas Alt: Yes and this is how we’re distinguishing it. Junaio is our consumer brand targeting newbie AR developers, with limited programming skills, while the Unifeye platform is really our B to B platform where B to B customers can create their individual AR experiences.

Tish Shute: Yes which is what my friend Patrick O’Shaughnessey, Patched Reality, did for the Ben and Jerry’s app he created using Unifeye.

Thomas Alt: exactly…

Tish Shute: It is a lot of work developing for so different mobile platforms isn’t it. Junaio is on Android and iphone but you havent moved Junaio to Symbian?

Thomas Alt: To be honest with you right now its a matter of priorities we have other things we want to do first. And from analyzing the user base, iphone was a big step Android was a big step and now we are pretty much seeing what is happening next. As you know Nokia going into different directions as far as their smart phone operating system goes, and so on and so forth.  There are also capacity constraints. And right now obviously the most – potentially not the most possible users, but the users most inclined to do AR on a day to day basis are the ones using the iphone and android devices. But obviously there are a lot bigger cellphone manufacturers out there.  But just you know even the mobile web users aren’t as strong as the users on the iphone and android devices?

Tish Shute: So what do you think the iphone 4 has that brought to the AR picture?

Thomas Alt: Very fast camera access, very good for marker recognition. If you go to the Metaio site you’ll find a movie where we show on the iphone 4 app for a real augmented reality Leggo peice – this is something which is very nice

Tish Shute: Yes I see that, yes that is nice, yes, yes very nice. The Unifeye SDK is really putting markerless AR into the mainstream.

Thomas Alt: Yesterday we launched the first, a err very nice shopping… shopping solution for , but that’s completely external.

Tish Shute: Oh yes – the augmented reality shopping for seventeen.com, i was going to ask you about that, because it is the first augmented reality online shopping using natural feature tracking.

Also I am very excited to see the gestural interface, awesome!

The seventeen augmented reality shopping app is a PC experience but are you working on developing gestural interfaces for mobile AR?

Thomas Alt: We are continually pushing the envelope of what’s possible with AR. Gestural interfaces for mobile AR is certainly the next step in taking what we’ve done on the PC and making it more portable by using the mobile platform. One thing to keep in mind here is the limitations of mobile platforms and size of the screen needs to fit and make sense for the user experience.

Tish Shute: I know you started off as an AR researcher (although as you mentioned earlier you have been working in commercial AR and building Metaio for a long while now.

So in addition to how we are progressing towards gestural interfaces for augmented reality, I would like to ask some questions about AR eyewear. We won’t really have hands free AR without eyewear. What is your projection on when we will see consumer AR eyewear? And, Do you have a any comments on those speculating that we will not see AR eyewear go mainstream for 20years?! What is Metaio doing to move eyewear technology along?

Thomas Alt: Well as you know, it’s always, you know on the technological roadmap, and we’re still doing research projects, in our AR research department. We have worked on things like calibrating eyewear for augmented reality. There is some nice development there.

But really, commercially, the whole thing with eyewear has never caught on on a level which would make it a valuable avenue, business avenue, at least for Metaio. So, I guess as an ex-researcher, it’s still a very interesting, a very good technology. And it would definitely change the marketplace radically when available. But as per right now, there are very few commercial applications.

Tish Shute: Are the obstacles to AR eyewear technological obstacles or is it just a question of a a business model. I mean is it realistic to see eyewear in the next 3 to 5 years at a price point affordable to consumers, where you really, truly can have eye tracking? You know, the whole problem there was with virtual reality and eyewear giving people a headache. How far have we come in terms of the technology?

Thomas Alt: Well, it’s not so much technological factors ’cause all fundamental problems are solved. It’s more a rather large corporation, I guess, would have to step up to the plate and say okay, do, let’s get all the state of the art in electronics and develop just a perfect HMD.

Tish Shute: Something Yohan Baillot’s company Simulation 3D is doing at is looking at is hooking up eyewear to smart phones.

Thomas Alt: Yes exactly that would be even better. Metaio has done a strategic move into this HMD space for augmented reality about a year ago by acquiring a bankrupt company. I mean, we had considerable IP around it from our research base but in the long term we still believe in it and we did a move about a year ago in buying what was left over from a bankrupt company including a lot of IP, which basically goes into the direction of mobile augmented reality but also mobile augmented reality in connection with head mounted displays.

There’s actually a press release about this but that’s about a year ago…

I know that the whole HMD thing… I mean, I’ve seen companies come and go. Metaio has worked previously, very closely, with Microvision of Seattle. We have worked with a German company, doing HMDs and we have worked with Vuzix.  We are still working with Vuzix, so we’re still consider it very valuable.  But right now, I mean, it’s just not a big part of our commercial pipeline, to put it that way.

Tish Shute: It was interesting what Bruce Sterling said in his keynote at ARE 2010.  He actually made a strong case for why smart phone augmented reality may be more interesting because it’s less immersive. I mean, he raised the question of the fact that if you really truly had AR eyewear and HMDs you’d re-enter the world of virtual reality or as he called it AR’s Gothic step sister VR would rise from the grave….

Thomas Alt: Yeah, well, that’s a cultural or even a philosophical question and we have discussed it a lot, especially in the industrial domain. Also will the deployment of HMDs come about from end consumers using it in their spare time, or from professional users using the idea in their work time?

Tish Shute: Do you think it surprised people who have been working in augmented reality research how much people have engaged with the idea of smart phones as the mediating device for AR, and that rather than having the always on experience that eyewear would give us, we use smart phones as a magic lens of a smart phone when we need to or want to. Some people were skeptical that anyone would want to hold up a little window to look at augmentations of the world – a magic lens. I mean, it wasn’t self-evident that that would be an experience people enjoyed, and it turns out that it was.

Thomas Alt: That’s actually a very good analogy. And also in my view, I mean, certain behaviors just change also, right? I mean, this is exactly what Apple’s trying with the iPad, right? You’re taking the iPad, and all of the sudden you’re not constrained to a laptop or whatever. And it’s truly a companion of the couch, in-bed Web, in the kitchen, and so on and so forth. So digital usage with the iPad, which is a different market, and I’m aware of that but as an example, the iPad has changed our behavior. And obviously, the augmented reality guys are hoping that something similar happens with AR.

Tish Shute: Which of course brings up the question, I’m assuming that some of the next generation of slates/ipads are going to have front and back cameras, GPS, and compass, right?

Thomas Alt: Actually we know that.

Tish Shute: You know that, yes. I assume that you know that, because are you working on some prototypes, and have you got some plans?

Thomas Alt: You have to understand that I cannot talk as I’ve talked as a researcher. It’s the rules, so I have to be a little bit careful about what I say. We very much think that a webpad, or whatever pad, you would want to call it is on some occasions very good device for AR.

Tish Shute: Yeah. But it is an interesting point with holding up a larger device, because you hands aren’t free, but the neat thing about the phone for augmented reality is that you really can do a lot with your thumb, as we’ve found and just the position of the phone. But, how will this work it with the two hands on the larger device?

Thomas Alt: Keep in mind, everyone’s talking about mobile augmented reality, but really where the case for augmented reality, at this point, is the strongest is in the installation business, it’s in the web business… Not necessarily only commercially, but also use case-wise. There are tons of museums out there which are using our augmented reality system in an installation fashion, and to communicate products better, and more efficiently, and so on, and so forth.

So, I know that the hype is clearly on the mobile augmented reality side, but there are many examples augmented reality experiences where holding up a larger device is not a big obstacle.

Tish Shute: Well this brings me to some questions about the future of mobile AR. My interview with Jay Wright focused on how we are now in a new period for AR bringing together computer vision, visual search into a mobile stack that is really optimized for AR. What do you see emerging in mobile AR as we move beyond compass, GPS, camera, accelerometer based AR into markless image-based AR. What will the new use cases and where will we see mainstream users getting in AR. Will AR games be the first mainstream AR experiences?

Thomas Alt: My partner is actually, first of all, one of my best friends, second of all, very emotional, third of all, very intelligent, and he said the other day something I think very valuable in this area. He said, basically think about Mobile Augmented Reality, Thomas. There’s really a very limited number of use cases which you can do if you look at these classical Point and Find applications, ok? But there are almost unlimited number of use cases when Augmented Reality becomes a day to day companion, ok? So what he meant is, ok, I’m looking at my normal day… I’m looking at the city, I’m walking throughout this, I’m coming home, I’m having dinner basically. I can deploy Augmented Reality in a pure POI search fashion perhaps not even once. Ok when I’m travelling it’s a different story, but in an ordinary day I might use a POI search not even once.

But where this ultimately leads, you know, is even in the 15 minutes I’m having breakfast, I’m using AR – looking at the cereal box with my cell phone, I’m taking part in a sweepstakes or whatever. So from that we draw the conclusion that as a general strategy for Junaio, we should basically throw as much technology as possible into Junaio, make it halfway self-explanatory, and just give people the possibility to come up with ideas on how to deploy augmented reality continuously.

We have actually got a creative team from an Art School working on that, and just, you know, with very little programming skills coming up with things you can do with Augmented Reality on a day-to-day level. And it could be a scavenger hunt game, in the city, with monsters flying around, it could be the normal POI routine, it could be marketing purposes, and so on and so forth. And I think that’s really the roadmap, and this is a little bit similar on a more technical level, to what Qualcomm is doing, ’cause they’re floating out possibilities or capabilities I want to call them, and Metaio is doing that, but on a higher level [re the tools] meaning on a Junaio level.

Junaio is a capability platform. It is also a way for Metaio to demonstrate the capabilities of our technology. We will offer all the possibilities for AR and more that we have already demonstrated on PC augmented reality.

Tish Shute: What is the business model for Junaio? Are you encouraging developers to develop business on your platform ?

Thomas Alt: Junaio is our end consumer platform and our business model is similar to the way Google structures its business model. We work with OEMs, content partners, brands, and developers to offer free content to our end users. Where we do charge is on the advertising side, more specifically contextual and location based advertising. At the current stage, we are focused on building the content base, fostering our developer community, but in the near future, we will be introducing advertising channels.

First of all you have to have very good use cases in the platform basically. And then to put a business model on top of that from a technology stand point is not hard – its a pay channel. Its all prepared for this.

Tish Shute: You have quite a broad base as a company don’t you – you do everything from industrial AR, marketing to technology licensing and more?

Thomas Alt: Basically, there’s a lot of things people don’t see us. There is also the Unifeye PC SDK and we have a client base and partners who are sourcing software from us, and we are doing great pieces. I mean the hype of augmented reality is really coming to a peak. There are lots of pieces that are not even talked about any more. China’s GQ magazine launched with AR from Metaio, the biggest AR campaign anywhere – there are a lot of potential readers in China. And um, so that’s our business model… we have our IP, our patents and so on. And on this we can move on onto the mobile platform whenever it’s advisable or feasible.

Tish Shute: Right. Yeah. I mean uh, you’re very fortunate to have this base built on uh, years of developing IP. What are the most important areas of AR that Metaio holds IP and patents in, in your view?

Thomas Alt: there’s sleepless nights in that too…

So far we’re extremely excited about what’s going on with Junaio, it’s one of our big, big success stories. But we are sensible and trying to experiment because, you know, analogies from the past won’t really work in my view for Augmented Reality in a sense, that, you know, you better bring for a new system to fly, for a new technology to fly, you better bring a very concrete use case to the table, ok? And a well-defined use case. And we are, right now, with Junaio, in a state where we are checking out what could be such a very defined use case.

Tish Shute: So how many users does Junaio have now?

Thomas Alt: Let’s put it that way, we are, especially in the last 2 months, we are very satisfied. But we are not disclosing that, because users, and we’re seeing that from the competitive landscape, always needs 1 page of description what exactly a user is.

You understand what I’m saying? So this is why, ’cause we don’t want to up or downplay things, we are very careful saying, with users. Because I mean we have people who are actually also commercially very interested in Junaio…  We go through with them and discuss what exactly a user is. Cause there’s more then…a download is not a user. An app or something on your phone is not a user, basically, in my opinion.

Tish Shute: I am still waiting to see someone do something with AR and the Four Square API, or now the Facebook Places API. Do you see an interesting potential in the marriage of the rapidly emerging location based social networking space and augmented reality?

Thomas Alt: Definitely. Augmented reality offers a way for users to find information around them easily. Adding in the social networking component such as geo-tagging, rating, commenting can enhance the user experience and create engagement beyond just viewing the information. For example, within junaio, an average user can create their own personal channel and geo-tag photos or leave text messages at different locations. They can create a virtual tour of San Francisco and share it with their friends. By connecting the social side with good content, the augmented reality experience becomes more fun and interactive.

Tish Shute: And, of course, there’s the Junaio API. Are you beginning to see developers use that?

Thomas Alt: Yeah exactly, I mean if you go to Junaio.com, you can get a login, you have an API description. And the way it works is, that you bascially set up a call, which contains the information you would like to have in your individual channel. You submit it to us, it will get checked for profanity and other things, basically. And then we admit it to Junaio basically. The API is huge.   You can also use Junaio indoors.

This is very relevant. And there’s a tool chain for that, and so on and so forth. You can do mission-based search with Junaio. It’s in there, it’s called Junaio Glue. And there will be another very interesting feature coming up in a couple of weeks. And you can just do it, you can do a scavenger hunt, a game, normal POI search, and so on and so forth. And it’s all active. And that’s, what’s sometimes difficult for us to communicate, is it’s really a capabilities platform, but on the other hand it’s obviously very good to developers. And I mean on the developers side there’s huge interest.

Social Augmented Experiences leveraging geoawareness and human and machine intelligence to create real time information brokerages, combined with an augmented reality view, can create a new opportunities to reimagine our relationships with each other and our environment.

This Summer, I have been on a blogging hiatus, which has meant I haven’t been sharing as frequently and, unfortunately, the second half of two conversations I had earlier this year, both of which have much influence my thinking on social augmented reality, have languished in private mode -  part 2 of my talk with Bruce Sterling (see Interview with Bruce Sterling, Part I: At the 9am of the Augmented Reality Industry, are2010, and part 2 of my conversation with Anselm Hook - Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, Part 1. Time to get caught up on some blogging!  The lightly edited transcript of Part 2 of my conversation with Bruce Sterling is posted in full below.

Bruce Sterling has been blogging all the key developments in augmented reality (amongst other topics of interest!) on his Wired Blog, and he brought my attention to Boskoi the Ushahidi based app for Android phones, augmented foraging pictured in use above – for more pics see fightthegooglejugend.

Augmented Reality and Real Time Information Brokerages

Picture above is the path the “nomads” took through the Westhaven cryptoforest with Pieter Bol,co-auteur of the book Biological Globalisation and Theun Karelse of Urban Edibles Amsterdam “who presented his ‘augmented foraging’ app Boskoi.“  For more see, The Cryptoforests of Utrecht and, Westraven Psychogeography, 6 June 2010. Note: Cryptoforests: 1) Urban forests hidden from view 2) Urban fallows that might or might not be considered as forests 3) Gardens gone wild)

My interest in the Ushahidi family of ideas was already fired up by a conversation with Anselm Hook early this year.  We discussed a number of Ushahidi related projects, Swift, Crisis Filter and Anselm’s project Angel, Augmented Reality, and my own keen interest in an open, real time, distributed platform for augmented reality – ARWave.

The Ushahidi platform and the related project Swift has pioneered the real time brokerage of information with people acting in curatorial roles or matchmaking roles coevolving with machine assisted matching to connect wants to haves.  Ushahidi uses multiple gateways including SMS, and Twitter.  But the Ushahidi family of ideas is extremely interesting when combined with augmented reality and suggests many new possibilities for social augmented experiences, as Anselm pointed out, for human to human communications, human to civilization communication, and human to environment communications (e.g., perhaps, how machine intelligence can help bridge the difference in time scale that Kate Hartman explores in her, Research for Glacier-Human Communication Techniques).

Ushahidi, which means “testimony” in Swahili, is a website that was initially developed to map reports of violence in Kenya after the post-election fallout at the beginning of 2008. It is now an open platform with a wide range of applications and growing developer community.  See What is the Ushahidi Platform? from Ushahidi on Vimeo.

Swift - a project that emerged from the Ushahidi dev community, is a human sensor/real-time brokerage for dealing with emergencies, enabling the filtering and verification of real-time data from channels such as Twitter, SMS, Email and RSS feeds.

Boskoi – augmented foraging is the first app,  I have seen, to begin linking Ushahidi with augmented reality – although I don’t think there is a full augmented view for Boskoi developed yet?

“The whole point of AR is to see things from a different point of view…”

Click to enlarge poster from upcoming ARWave demo at Software Freedom Day – for more see below

I am often asked what augmented reality brings to the table with respect to location based social networking, which is on the verge of going mainstream in smart phone apps like Four Square. While the first part to my answer is usually to explain what is unique to augmented reality.

As Bo Begole notes, the full vision of AR requires machine perception technologies to detect the identity and physical configuration of objects relative to each other to accurately project information alongside/overlaid with a physical object (see this post on the PARC Blog by Bo Begole on the difference between AR and ubiquitous computing – thank you Rouli for bringing my attention to this).

But it is only in recent months that we have begun to see the kind of tools that make this possible become freely available to developers – see my interview with Jay Wright of Qualcomm here.   Also see this post on Bundler: Structure from Motion for Unordered Image Collections an open source system that allows the creation of 3D point clouds from unordered image collections, e.g. internet image collections.  We now have many tools available to move mobile augmented reality beyond the recent crop of apps relying on GPS and compass alone for positioning into a new era of vision assisted AR apps that will increasingly bring the full vision of AR into our daily lives.

Further, the integration of visual search applications like Google Goggles and Kooaba which can detect the identity of particular objects will add another vital tool to machine perception technologies enabling AR “checkins” on potentially anything in the physical world around us, and more fuel to the Gamepocalypse (e.g. it would be easy to turn every trash can in the city into a basketball hoop as we discussed at the ARNY meetup last month).  And soon, the Pandora’s Box of  facial recognition (Google Goggles have the capability though it is not released to the public yet) will open up.

Jesse Schell described the importance of AR in a nutshell in his keynote for are2010:

“The whole point of AR is to see things from a different point of view…How can there be a more powerful art form than one that actually changes what you see?”

But how AR matures as a social experience will be the key to Jesse’s suggestion that:

“Augmented Reality will be one of the things that fundamentally define the 21st century”

There are many interesting forms of AR that are not reliant on a tight registration between media and physical objects – several are put forward by Bruce in the convo below.  And, it is likely we will see AR eyewear as an occasional useful accessory to a smart phone long before we have the sexy, affordable augmented reality eyewear worn that we wear throughout the day.   These speech to text glasses would be a very useful and viable accessory to a smart phone right now for the hearing impaired.

For the moment, as Bruce notes, some of the most interesting and useful augmented experiences to date have not been in the cell phone space:

“There are other aspects of AR besides the cell phone space. There’s Total Immersion’s big display screens. There’s the web-based fiduciary stuff. And there’s projection mapping. And then there’s experience design just for people who need their reality augmented for whatever personal or social reason.”

On of my favorite social AR experiences is this SMS Slingshot.

But I have been excited for a long while about the intersection of mobile social augmented reality, real time communications, and ubiquitous computing see Total Immersion and the “Transfigured City:” Shared Augmented Realities, the “Web Squared Era,” and Google Wave.  And I have described in many places why I think real time, open, distributed communications for AR are so important to developing social augmented experiences – see the slides for my talk at Augmented Reality Event here, here and here for starters.

ARWave at Software Freedom Day 2010, September 18th 2010

Thomas Wrobel and Bertine van Hovell will demo the first ARWave Android client at Software Freedom Day this weekend!

A number of people have asked me, (including Bruce), What will be the future of ARWave now that Google Wave is no longer a stand alone application?  Yes, the recently announced release of Wave in a Box (see here and here) is very exciting for the ARWave team.

The ARWave Android client is the first open AR client built on an open, real time, distributed platform -  based on a server that anyone can download and set up, currently the “FedOne” server but Wave in a Box, hopefully, will be even easier to deploy.  Wave in a Box seems perfect for ARWave’s needs – for more see the WiaB Google Group here.  And for more information on the ARWave client -  click to enlarge the poster below, see the ARWave concept video here, and for more, and how to get involved see arwave.org.   Props to Thomas Wrobel and Bertine van Hovell (posters below from demo for Software Freedom Day), Mark Evin, Davide Carnovale, and Markus Strickler, for all their hard and brilliant work on ARWave.  Also to JCPT the open Android 3D engine that has saved a lot of work!

click to enlarge slide

Social Augmented Experiences that Matter

My ideas on the future of social augmented experience have been deeply informed by the the conversations I had with Bruce Sterling and Anselm Hook this year.

Bruce Sterling notes in the conversation below, location based social apps like, Four Square, are interesting because they are not “urban geography like Google’s satellite stare from above,” but rather “groups of citizens are doing portraits of their own region.” Augmented Reality, with its of lauded power to make the invisible visible is, of course, is the ideal tool for “citizen portraits”  to the next level.  Cory Doctorow described to me three years ago (see here) an “inverse surveillance society,” enabled by an augmented view  – “where all the data from the positional and temporal characteristics of all the objects that we own were in aggregate visible and available so that we can mix and match them remix them understand them and have more agency in the world.”

It is very cool to go back to reread this conversation now that it is becoming possible to build the kinds of apps Cory described, and Bruce Sterling envisioned in Shaping Things (see Amazon.org  page 111).

click to enlarge

My  conversation with Bruce earlier this summer (see below) took place on the heels of are2010 – Augmented Reality Event.   See the video of Bruce’s keynote, “Bake a BigPie,” here,  and the final keynote, “Seeing,” by Jesse Schell (see video here) in which Jesse riffed on AR and the man with the X-ray eyes.  Both these awesome talks are still fresh in my mind.  Bruce noted how we should pay attention to augmentations for people and situations that could really use some augmentation… and not get too fixated on the coming of AR Goggles.  He elaborated on this in our conversation (again full transcript below):

“Well, it’s a matter of deciding whose reality it is that you’re trying to augment. I’m not trying to be a bleeding heart about it, but obviously there are people in our society right now with reality that could really use some augmentation. They are mostly disadvantaged people. They are vision impaired, or maybe they have autism. They might be senile and just can’t remember where they put their shoes. These are people who could really use some help, right?”

So, start with people who really need sensory or cognitive help. Before you turn our geeks into Superman, why don’t you try turning some people who are harmed into more functional individuals? Then you’ll be able to learn how to do that. Then maybe you can ramp it up to these Nietzschian heights of the superb Man With the X-ray Eyes. Whatever.”

What will make AR interesting and useful long before and long after we see the full vision of AR eyewear manifest is its social aspects.  Bruce points out:

“My argument would be that if you want people to be more sensitive toward certain, say, issues and problems, it’s better to find the people who are already sensitive to those issues and problems, and give them a bigger stake in your augmentation system.”

“Say that I am really worried about public health. Well, if I have a lot of nurses that are using my system, people who are aware of my issues, then I could be walking around and I’ll see a lot more tags saying, “This is where he got food poisoning!” “In this shooting gallery, many people have caught AIDS!” Or, you know, “Tuberculosis has been spotted over here in this building.”

At that point, I could simply share their knowledge and get some social intelligence. As opposed to trying to amp the basements of my little hacker-mind and drag stuff up that’s escaped my conscious attention.”

Finding new ways to broker information – bring together needs with haves and different participants, empowered and disempowered  is., as Anselm discussed with me, one way to change our view of human to human, human to environment and human to civilization communication (particularly in light of this  “sobering account of how open data is used against the poor in Bangalore” that as @timoreilly noted recently OpenData Empowering the Empowered).

The key idea in a crisis filter, Anselm noted,  was to break up the participants into different kinds, to connects wants with haves:

“There are people who are in  situation.  We call them citizens.  And then there are reporters, people who report situations back to Twitter.  And then there are curators, people that canvas Twitter looking for important Tweets.  And then there are first responders, people who take the curating collection of responses and then act on them.”

This kind of brokerage between people acting in a curatorial role or matchmaking role with each other can be extended into and coevolve with machine assisted matching as Anselm explains.

It is also a vital part of creating social augmented experiences that matter.

One of Anselm Hook’s projects, which is called Angel is the the most radical expression of connecting wants with haves in that the idea is that “you have a situation, you broadcast that situation, and help magically appears.  You don’t even sign up for a service.  You just get help …

As Anselm explains this is the same idea of a brokerage for dealing with emergencies, but applied to the long tail of crisis response.  As Anselm describes it:

“I am interested in personal crisis.  ‘I lost my cat.  Help.  I can’t find where my kid is.  I am out of gas.  I have a flat tire.  My house is on fire.  My aunt is trapped in the bedroom.’  The kind of personal crisis that is just as important, but is not enough to get a national movement to help you…

I will publish this conversation with Anselm in full in an upcoming post.

Zorop – an ARG for World Peace

If you want to be part of a really exciting experiment to reimagine our relationships with each other and can be in San Jose this weekend, I highly recommend exploring this “rabbit hole”.

Thank you Gene Becker, Lightning Laboratories and Ubistudio for sending me this invite:

“Ken Eklund (@writerguygames) is developing a wonderful game for the 01SJ Biennial called ZOROP, aimed at creating World Peace(!). Some of you might know Ken from his work on the amazing ARGs EVOKE and World Without Oil. Anyway Ken, along with his collaborator Annette Mees, are furiously working to get ZOROP ready to go for the Sept 17th premiere at 01SJ.

Are you intrigued? I thought so, and here are your next steps down the rabbit hole:

> Check out http://zorop.org to learn about the game

> Follow @ZoropPrime to watch it unfold: http://twitter.com/zoropprime

> ‘Like’ ZOROP on FB for a different view: http://www.facebook.com/pages/Zorop/141140772593618

> Become one with the game; consider volunteering as a Zoropathian: curious@zorop.org

> Head down to San Jose on the 17th, play the game, and ride the ZOROP Mexican Party Bus. Seriously.”

Interview with Bruce Sterling

Click on image above to see video clip from from brucesflickr

[Note the first part of this interview is here and I broke in anticipation of Part 2 just as I started experimenting with an idea Joshua Kauffman - an advisor and entrepreneur working on design in the public sphere gave me for an interview technique - the All Souls College one-word question interview.  Although apparently they recently scrapped it and I am not very good to sticking to a single word!]

Tish Shute: We were talking about these proximity-based social work networks like Foursquare and Gowalla and how they may influence the emergence of social augmented experiences.

So Joshua’s suggestion for the first word was “territorialization” e.g. how do these new mobile social experiences like Foursquare, and the observation that actually rather than breaking down territorialization – which would be a good thing, tend to support territorialization…

Bruce Sterling: Yeah, they’re re-intensifying it in a very odd, electronic fashion.

Tish Shute: Yes.

Bruce Sterling: It’s not true of projection mapping or the webcam fiduciary display stuff. But with the handheld stuff, and especially the urban informatic stuff, it really can’t help but take on a local flavor. Layar is like “Augmented Dutch Reality.”

And TonchiDot is “Augmented Japanese Reality.” It’s hard to imagine a Layar interface going gangbusters at Tokyo. Whereas the TonchiDot interface, which is so clearly influenced by Anime and cartoon graphics…. Maybe it could find some niche of hipsters in Amsterdam hash bars…

Stuff that’s socially generated by people on the ground, as with Foursquare and Gowalla, is bound to take on a regional influence. Right? It’s like the New York hipsters who were early adopters of Foursquare. They’re not mapping New York! They’re mapping Hipster New York.

It’s all about Williamsburg and places where 24-year-olds go to drink… They found a demographic niche there. These guys are building the service for them. They’re people who are willing to work for Foursquare for free, because they want to wear the little king hat.

Tish Shute: I got the far far away badge ‘cos I live on the Upper West Side!

Bruce Sterling: But that’s not urban geography, right? I mean, that’s not like Google’s satellite stare from above. That’s a group of citizens doing a portrait of their own region. You’re going to see interesting things happen because, of course, people who use Foursquare elsewhere are going to check into New York, and they’re going to look at the “New York Foursquare.” They’re going to be aliens who interact with Foursquare people in New York and annotate what they’re seeing.

Tish Shute: Oh! Yes. Good point.

Bruce Sterling: That Foursquare community has a certain émigré soul. It’s different from the normal émigré soul of simple tourists on New York. So you’re friend there is right about the territorialization.

Tish Shute: Yes, Joshua Kauffman is a smart guy! Yes I am interested to see what interesting kinds of deterritorializations proximity based social networks and the hyperlocal view of augmented reality might bring, not just the new territorializations.

Bruce Sterling: It’s not the intense kind of territorialization, like gangs putting down graffiti markers and beating people up. It’s an inherent regional character that comes with using peer production to build your database.

Tish Shute: We were discussing whether AR could break down the walls between people – people who share the same physical space but actually inhabit different territories even if they are sitting on the table next to you.

Bruce Sterling: You know, I just wrote an article for my Italian magazine column. I think I mentioned this to you – a report about ARE 2010. I titled it, “Chicks Dig Augmented Reality.”

Tish Shute: [laughs]

Bruce Sterling: There is a very heavy social element to AR, and a phone based element. So the question is: Why would a woman wear a fiducial marker? Like our Metaio speaker at ARE2010 who had a fiducial marker on her lapel pin.

Tish Shute: Right. Lisa!

Bruce Sterling: Why would a woman go out in public with her Facebook profile on her body?

Tish Shute: Well I can think of some reasons…

Bruce Sterling: So that men will approach her, of course.

Tish Shute: Yes the core of all successful social networks is always a form of dating app.

Bruce Sterling: You do it a social icebreaker. It’s like: I’m a woman, I’m sitting here alone, and you can sort of glide by and, you know, take a snap of me. Then you retreat and have a beer with your friends and you work up the courage, and then you come and say, “So! Susan! I understand you like bicycling! And, boy, me too!” Right?

Tish Shute: There are all kinds of social barriers between people in cities that AR might be helpful in breaking down. An extreme example is the dilemma you actually quite often face as a New Yorker as you walk around a city. There are people asleep on the pavement and you don’t know if they’re dead or alive.

Bruce Sterling: Right.

Tish Shute: And you sort of like have this awful ethical dilemma of like, “Am I walking by someone I should be shaking by the shoulder, right, to wake them up so they don’t die, right?”

Bruce Sterling: Yes.

Tish Shute: You said in your keynote that we should pay attention to augmentations for people and situations that could really use some augmentation..

Bruce Sterling: Right. There actually is such an app in Britain right now. I posted about it: two Augmented Reality schemes for rubbish and hobos.

Tish Shute: Right. Yes I saw that!

Bruce Sterling: “Any sufficiently advanced technology is indistinguishable from garbage and hobos.” You don’t need to personally find out whether this hobo is worth your help. What you need is a good way to report the hobo to a hobo check-up service. They come in, and they look on their own database or supply a database to you, or a facial recognition unit, whatever. The service says: “Oh, well. That’s Fred. He’s a paranoid schizophrenic. He always sleeps in that alley. Let him be.”

The same goes for the rubbish — although I don’t want to compare rubbish to hobos. In fact, people do go out with their AR kits and take pictures of abandoned garbage bags and broken glass. They upload them with geolocated tags for the local garbage guys. Guys who are sitting around doing pretty much nothing because they don’t know where the rubbish is.

And they will come out and get the rubbish! I mean, they just deputize guys to go out and follow these alerts. Right?

But nobody predicted — least of all me — that you were going to have a high-tech Augmented Reality system that consisted of removing rubbish and derelicts. Right? But rubbish and derelicts always go profoundly under-reported. It’s just hard to get people’s attention.

But it’s very easy to set up a system so that, if you get ten reports on the same piece of rubbish, that’s going to work its way to the top of the stack. That’s why I was trying to get AR people away from the romance of the hottest app for the shiniest machine. More toward a design stance that’s more user-centric.

Where are the actual problems about stuff that we perceive? Stuff we can’t do anything about? Or people whose mechanisms of perceptions are harmed. They could be doing good work, being more participative, if they didn’t, basically, walk around without their glasses on.

Tish Shute: Well this leads well into the second word, Joshua suggested was interesting spring board – sensitivity.

On the one hand we can do these things for people who maybe need the augmentation because they have difficulty with one or another sense, e.g., their eyes are not functioning, or their ears are not functioning. But on the other hand, we can’t cross the social bridge to communicate with people who are temporarily disempowered in relation to the rest of society e.g. hobos and people who sleep on the streets of New York City.  And even though Augmented Reality could potentially be helpful it can even be more disempowering to the already disempowered.

Bruce Sterling: Right.

Tish Shute: But re “sensitivity” – does augmentation increase or decrease our sensitivity? This is a problem that Will Wright brought up [see video of Will Wright's keynote at are2010], e.g, the problem of parking HUDs getting in the way of your intuitive parallel parking skills. The Lexus that takes driving control from you when you look back, ‘cos it knows that you’re looking at the road, and it starts to brake. Right?

Bruce Sterling: Right.

Tish Shute: The fact that the problem with technology is that it makes us less sensitive, right, augmentations sometimes get in our way?

Bruce Sterling: I suppose that’s true. But I’ve heard that said about practically every medium. Especially television.

Everybody wants to blame machinery for their lack of morality. It’s hard to top something like the Kitty Genovese killing in New York. This sort of legendary New York horror story from the 1960s. A woman is stabbed to death in public, no one does anything.

Tish Shute: Right.

Bruce Sterling: I don’t think that our media is making us any less humane or more callous.

Tish Shute: All right. Oh no! I see what you’re saying. Perhaps I misrepresented what Will was suggesting by putting it that way. The question is perhaps more how do we get the sensitivity into the technology. Human bodies are fantastically sensitive and sensory.

Bruce Sterling: Right.

Tish Shute: And we have these like sensitivities. For instance, How could augmentations of reality be like a blush ? You definitely want an interaction that’s not just this data being pushed at you. But what is the data that counts, right? Will shows a slide often of an iceberg with the tip of the iceberg which is the conscious mind.

Bruce Sterling: Oh, I see. Yeah.

Tish Shute: And underneath it is all the preconscious stuff that really counts, right? Any thoughts on that?

Bruce Sterling: I did take interest in that. Will has obviously been spending a lot of time studying cognition.

Tish Shute: Yes.

Bruce Sterling: I’m not convinced that AR has got a lot to do with that. There is certainly a trend there. There are a lot of people who want to do body hacks and brain hacks. I can imagine AR being used for that purpose, but it seems like a niche application. What is the point of our accessing even more stuff that’s outside of our consciousness?

Tish Shute: One of the things he is talking about is game dynamics, is it? The role of the imagination in play. For example, he shows the high dynamic range photos that make the world magical. Something you want to engage with playfully. This he points out increases a sense of agency because you are encouraged to engage and to play with the world.

Bruce Sterling: Well, I’m a literary guy. Italo Calvino did a lot of writing about this. He talked about the classics of literature. Why do we read the classics? Calvino said we do not read, but reread the classics. And the reason we do that is that, at first, we read a classic book and we think, “Boy, this book is really good.” Then, five years later, we read it again and we think, “Boy, this is a really good book, and it’s got so much more in it than I thought it had when I was 18.” Then we read it again at 28, and it’s like, “OK, now I really seem to understand this book, and it means something to me now that I didn’t know when I was 18 and 25.”

What you are doing through that access is learning something about yourself. So Will is arguing is what I really need is like a better augmentation. So that I can go in there and sop up the book all at once. I can grab every cultural nuance in it, instead of the stuff that’s sliding past me because I’m 18 and kind of young and hasty. Maybe I could have certain words and phrases helpfully underlined, that are like, “OK, well, this part is problematic for you.” In some sense, that’s not allowing me to be 18.

I’m never going to have the experience of my own maturation against this text, because I’ve devoured it all in one gulp.

My argument would be that if you want people to be more sensitive toward certain, say, issues and problems, it’s better to find the people who are already sensitive to those issues and problems, and give them a bigger stake in your augmentation system.

Tish Shute: Yes the social augmented experiences are going to be the most valuable.

Bruce Sterling: Say that I am really worried about public health. Well, if I have a lot of nurses that are using my system, people who are aware of my issues, then I could be walking around and I’ll see a lot more tags saying, “This is where he got food poisoning!” “In this shooting gallery, many people have caught AIDS!” Or, you know, “Tuberculosis has been spotted over here in this building.”

At that point, I could simply share their knowledge and get some social intelligence. As opposed to trying to amp the basements of my little hacker-mind and drag stuff up that’s escaped my conscious attention.

Tish Shute: Interesting that seems to bring us to another kind of repetitive theme in AR, the people tend to pigeon hole it as “merely” a visual interface. But actually, it’s the intersection, isn’t it, of social intelligence and augmentation.

Bruce Sterling: Well, it depends entirely on how you design the system. If I’ve got a military augmented reality, I would expect that to be mostly about urban fighting. It’s going to be about kicking in a door and shooting terrorists. If I pull that helmet off my head and put that on the head of an emergency worker or a cop, I’m going to get a militarized cop or a militarized emergency worker.

Tish Shute: Well the histories of the two great mass media of the twentieth century – TV and the atomic bomb were intertwined, and I suppose the evolution of ubiquitous media, augmented reality and urban warfare is already intertwined too.  So how can we encourage augmented realities to move beyond military roots that is common to much technology and into more peaceful urban realities?

Bruce Sterling: Well, it’s a matter of deciding whose reality it is that you’re trying to augment. I’m not trying to be a bleeding heart about it, but obviously there are people in our society right now with reality that could really use some augmentation. They are mostly disadvantaged people. They are vision impaired, or maybe they have autism. They might be senile and just can’t remember where they put their shoes. These are people who could really use some help, right?

So, start with people who really need sensory or cognitive help. Before you turn our geeks into Superman, why don’t you try turning some people who are harmed into more functional individuals? Then you’ll be able to learn how to do that. Then maybe you can ramp it up to these Nietzschian heights of the superb Man With the X-ray Eyes. Whatever.

Tish Shute: Did you notice that a couple of apps actually like TagWhat have apps geared towards people with disabilities – I haven’t had a chance to check it out.

Bruce Sterling: I’m sorry, I wasn’t looking at their tags.

Tish Shute: I was discussing this with Joshua who mentioned Zachary Lieberman’s Eye Writer, which is for people with locked-in syndrome. Do you know that?

Bruce Sterling: Sure. And people appreciate that because the poor guy, he’s laid up with Lou Gehrig’s Disease. Now they’ve given him a way out. AR is like a spark of new hope that gives his life meaning. What’s wrong with that?

Tish Shute: Yeah. And Tim Byrne using Sixth Sense for Autism is interesting.

Bruce Sterling: Let’s consider it the other way. Let’s say this graffiti writer there, instead of him being sick and weak, let’s say he’s an athlete. So I want to make him into a super-human graffiti writer. I want him to run around graffiti-tagging the entire town before dawn. Is that a good idea? Do we need that? Super human, super taggers? What if he’s going to spray up stencils of Nietszche? I kinda wonder whether the game is worth the candle.

Tish Shute: Yes I suppose it is not a great social scenario to be always augmenting the lives of the elites! Hmm, the third single word interview question is “homophily,” and earlier were you’re saying that we’ve kinda got to accept this is very much part of AR – as how it works, because hyperlocal experiences gets created by local communities – that up to know have tended to be homophilies.

Bruce Sterling: Well, I think that’s easily handled with some design thinking. You’ve got to do some user observation and show some sympathy with the user, and to be aware that you’re designing for the user and you’re not designing for yourself.

In a field as young as this, it’s mostly geeks building cool stuff for geeks. In a lot of ways, it’s a “can you top this” contest. That’s OK, but it’s not good design to be your own client all the time. It’s like writing novels to amuse yourself, or sitting on the porch singing the blues on your own guitar with only yourself to hear.

Tish Shute: What will it take for AR mature out of this “geeks building cool stuff for geeks” phase do you think?

Bruce Sterling: It’s necessary to master some of the tools first. I think of the way the web has developed over the years. When the World Wide Web first appeared, it was just for physicists, and was all line commands and quite unstable and difficult. Then you got usability studies, and things like Ajax and so forth. It’s a very painstaking thing.

We’re not best at building interfaces for the best computer scientists. Web 2.0 was built from things like watching people cry while they were trying to fill out insurance forms. “Well, why are you so upset?”

“Well, I got to the end of the webpage, and then it said I took too long, and it cut me off and now I have to start all over!” Jesse James Garrett, right? Benefactor of mankind.

If you’re experienced, you think: “Why don’t I build a little module here, and kind of move the form over here, then I’ll periodically update it with some asynchronous Java and XTML.” And people are like, “Gee, how odd.” But that really works for real people. It comes from studying what people want to do. Whereas, the current AR approach to a problem like the insurance form would be like, “I will give you the ability to record the entire insurance form, and it will flash before your eyes!” OK great, that’s a cool hack, but I don’t really need X-Ray Eyes to fill out my insurance form. What I need is a more user friendly interface.

Tish Shute: Well it seems like we are moving into the terrain of Joshua’s fifth word “ventilation,” – if I understand it rightly – it is at least partially the antidote to territorialization because it’s this idea that a place needs air so we come out of our hermetically sealed boxes of the way we relate to a place and what kind of augmentation would bring more oxygen to that space.

There was an interesting moment in the Auggies because when Maarten Lens-FitzGerald presented the guerrilla shopping Layar and basically Mark Billinghurst and Jessie Schell who spoke first didn’t seem too impressed. They didn’t want to walk to shopping – that was what web shopping did, it saved us from walking to shop… but I felt, to me you picked up on something which might have some bearing on “ventilation” in that this AR shopping Layar was kind of squatting Prada – a favela chic AR shopping thing?

Bruce Sterling: I wasn’t sure if I was interpreting what Maarten had in mind by that. But I think Maarten sees his structure accurately as an experience thing rather than a mapping thing. I think he’s proudest of things like the Berlin Wall app on Layar, as opposed to Layars that help you go get a hamburger. It’s like…so when Layar inserts parasitic augmented shopping over other people’s real shopping? That was rather a subversive thing.

I think the key there is that his client is called “Hostage T-shirts,” right? I mean it’s actually kind of a transgressive little hippy T-shirt store that Layar can dump anywhere in the world. Layered right over, say, Versace and Prada. I don’t know what becomes of that effort. And I’m not sure about the term “ventilation,” because that’s a term of art I haven’t heard much.

Tish Shute: Maybe it’s like in a cafe. Ventilation would mean we were able to communicate with all these different categories of people that we normally would be unable to connect to, even though we might be sitting only a few feet apart.

Bruce Sterling: So it means ventilation in the bottles of our homophilies.

That’s not a personal problem for me. I commonly live in foreign cities and, you know, and spend a helluva lot of time talking to strangers at conferences. So I don’t think I’d have that particular tight little social island problem.

Tish Shute: Of the three judges at the Auggies, you seemed most enthusiastic about the Layar entry.

Bruce Sterling: It may be they’re not as familiar with the business models of locative AR as I am, and as Maarten is. It was kind of a subtle in-joke he was making about Layar’s own business model there.

Tish Shute: How do you explain that?

Bruce Sterling: Well, you know, Layar’s in the business of selling software to make mapping and urban structures into ecommerce.

The ideal way to do that obviously would be to move the richest customers into the most expensive shops in the most rapid way possible. Or at least distribute them in the directions they want to go, a la Google. Whereas this app that Maarten was talking about puts big barnacles in the way that are selling punk t-shirts.

Tish Shute: Right! Right!

Bruce Sterling: The Dutch are a bit subtle in their humor. I rather imagine there’s a lot of discussion in Layar’s inner circle about exactly what they want developers to do with their platform. They’re going to have considerable political difficulty deciding who can have a Layar key and how you discipline people when they start doing weird stuff. “The Oakland Medical Marijuana layar.”

Tish Shute: Well, finding nudists is one of the top layars at the moment.

Bruce Sterling: You know, obviously so. And finding narcotics in Amsterdam, or a prostitution layer. I warned them nine months ago this was bound to happen. I’m sure they’re aware of it. I don’t think Layar wants Google’s style of cool, technocratic detachment.

Tish Shute: But that’s pretty difficult to do in current augmented reality because we don’t have all the mathematical voodoo for full on AR search yet, do we?

Bruce Sterling: Well, you can hire it out. Somebody’s going to do it, if they get interested enough. There’s Nokia-Yahoo. Nokia-Yahoo! just did a big corporate deal…involving Nokia’s mapping system and Yahoo’s localization. So the Nokia-Yahoo! mash-up is called Nooo! Or could be called Yahno. Yakia! Unfortunately ridiculous names.

Tish Shute: It’s interesting because you mentioned the spiders’ mating problem at Google. They’ve got all the pieces to make this kind of level of AR obviously right now. But they actually haven’t done it yet.

Bruce Sterling: There must be at least some discussion in Google, but the same goes for Microsoft. I’m frankly baffled by Microsoft, because it’s just full of insanely brilliant people. What the hell are they doing in there? Name one serious innovation that’s come out of their labs in five years. They make Integral Research look dynamic. It’s really kind of sad.

Tish Shute: It’s a very curious situation with AR though, because AR more than any new technology relies on these big hordes of data particularly for the mapping, right? And only the big four have the data – although we are beginning to see upstarts, Earth Mine, Simple Geo… Did you get a chance to meet Di-Ann Eisnor from Waze – real-time maps and traffic information based on the wisdom of the crowd.  Waze is a very interesting project that is a potential giant killer.

Bruce Sterling: No, I didn’t talk to them. I’ve seen people speculate that Earthmine and Apple are going to make an allegiance. I guess if you’re thinking that urban informatic mapping is a super big thing for AR, that must be true. But I’m not convinced that’s necessarily the case. People have pointed out that you can just use Google Maps, and you don’t have to walk around with a little visor. There are other aspects of AR besides the cell phone space. There’s Total Immersion’s big display screens. There’s the web-based fiduciary stuff. And there’s projection mapping. And then there’s experience design just for people who need their reality augmented for whatever personal or social reason. [dog barking]

Tish Shute: Right. Oh, I’m in the middle… My son’s come. What a good hair cut!

Bruce Sterling: Hi, there.

Tish’s Son: Hi.

Bruce Sterling: How’s it going, sir? Good to see you…

Tish’s Son: Good.

Tish: [laughs]

Bruce Sterling: Yeah. Nice looking shirt. I like the back of it.

Tish Shute: That’s from the American Shaolin Temple. [laughs]

Bruce Sterling: All right. Awesome. Kung Fu geek shirt.

Tish Shute: Yup he is a bit of Kung Fu Geek. He and his dad did an iPhone app on it for Yu-Gi-Oh, for Yu-Gi-Oh scoring.

Bruce Sterling: Awesome. Plenty of Pokémon-style combat in Yu-Gi-Oh.

Tish Shute: Yeah. Well, it’s interesting because you’ve talked about this aspect. That all of this, the Pokémon aspect of AR hasn’t kicked in yet. But it’s obviously a match made in heaven to some degree, isn’t it?

Bruce Sterling: One would think so, yeah. The whole little kid gaming thing. What does that have to do with Google or Bing? You don’t need a massive database for stuff like that.

Tish Shute: Yeah, you’re right. But good tracking, mapping and registration requires a lot of mapping…

Bruce Sterling: Well, our current tracking, mapping and registration requires that. Maybe there’s some other way to hack it that we don’t know about yet.

Tish Shute: That’s a very interesting point. We always have to stretch the way we think about mapping… perhaps its a real-time understanding of the location you’re in…perhaps the map is being negotiated through several social processes?

Bruce Sterling: There are maps, and then there are maps. There’s a kind of artillery map where you need to know the precise location of target spaces. And then there’s the kind of social map where I’m really looking for the IN-N-OUT Burger where my sister went last Tuesday. That’s a different system.

Tish Shute: And I think AR, at the moment, we’re getting the most out of the social maps certainly. And the other [machine perception technologies to detect the identity and physical configuration of objects relative to each other to accurately project information alongside/overlaid with a physical object] is still kind of the big dream, isn’t it?

Bruce Sterling: They say that men never ask for directions and women never read maps. Clearly, the genders have different ways of navigating the world. Who’s to say what manner of augmenting our experiences is hottest? I’m not convinced that today’s rather rigid geolocativity is really what our society wants from that particular service. Maybe what we want is something more folksy. Some useful nudge in the right direction as opposed to grids with 200 meters here and instructions to turn such-and-such.

Besides, there’s other hacks we haven’t considered. We’re very dependent on GPS, but just suppose all those satellites are blown out of the sky in a solar storm. Would we really want to give up mapping? Wouldn’t we just come up with some other nifty hack? Radio beacons, let’s just say. Atomic clock timers in towns. Or maybe just little QR codes on lampposts that give you the exact location of that lamppost, and just click the thing and have it calculate where you are.

Tish Shute: Yes the NextHope OpenAMD project had a clever way of triangulating location indoors.

Bruce Sterling: Well, GPS is there and people all want to use it. It’s got good API, so of course you want to. And the guys who are good at doing it are real geolocative freaks. But the mere fact that we can do it this way, and that you can make it pay, doesn’t mean that it’s the ultimate way to provide that service to people. It’s like saying that Egyptian hieroglyphics must be the greatest way to write, because we’ve got a lot of them and they’re hard to learn. What if somebody comes along with an alphabet? It’s going to be a little embarrassing.

Tish Shute: Yeah, that’s a very good point. Now, this is a more simple ordinary question about the event. YDreams went off the map in the Auggie voting, and walked away with The Auggies. No one doubted that that was the most…

Bruce Sterling: I don’t know. I thought those Occipital guys with the panoramic painting…. That was hairy. I would have been tempted to give them the prize myself, actually.

Tish Shute: And what did you like best about that? Because I agree. I love Occipital.

click to enlarge

Bruce Sterling: I thought it was a more technically difficult stunt than the hand registration thing. Using a hand as a 3-D cursor is hot, but not like painting a panorama in 3-D in real time. That was an impressive technical feat.

Tish Shute: And they hinted at the 2.1.1 AR, more AR version of that. What do you see coming out of that as possibilities?

Bruce Sterling: Well, I’d heard of YDreams, so I wasn’t stunned. But I’d never heard of those guys. I wonder what else the heck they’ve got in the attic.

Tish Shute: very cool stuff…

Bruce Sterling: Well, more power to them. But clearly YDreams was the popular favorite. And who couldn’t like it? It was just so AR.

Tish Shute: It was so AR and so gorgeous.

Bruce Sterling: It was pretty, actually.  Except for their ugly menu button and poor font choice.

Tish Shute: Oh, yes. You didn’t like that, did you? [laughs] But with the Occipital panorama, what do you see the next stage of that?

Bruce Sterling: Well, obviously quicker and faster. Quicker and faster and more accurate in a network. Let’s just say I’m in New York and you’re in New York and I’m calling you for help. And you say where are you? I just whirl around like this and I mail it to you on a Google Wave. And you whirl around like that, and then we compare the two panoramas and do an instant triangulation. And you say: I’m over here on this red dot of your screen.

Tish Shute: Yeah, exactly.

Bruce Sterling: We’re navigating with panoramas by having two connected panoramas and considering the difference.

Tish Shute: Yeah, very interesting…

Bruce Sterling: Not shabby, right?

Tish Shute: Not shabby at all.

Bruce Sterling: If you could do it in real time.

Tish Shute: Then the other thing I missed because I was going to meet Will was I missed the Launch Pad competition. Did you catch that?

Bruce Sterling: I didn’t see it either. I thought of another app though.

Tish Shute: Oh!

Bruce Sterling: You’ve got a panorama maker in your home office, and it just scans the office 24 hours 365 and tags anything that moves, right? OK, where’s the clipboard?  At 8:15 it was over here. Now it’s vanished. Now another object is viewed over here. So, logically, ping, you hit it with a sticky light and there it is, right?

Tish Shute: Oh, that’s cool also knowing what has changed in any environment would be a big enabler for a lot of AR visions.

Bruce Sterling: I’m sure there are many other things you could do with panoramas.

Tish Shute: My jet lag is beginning to kick in big time – so many ideas to pursue from are2010 – those panoramas are very exciting though.

Bruce Sterling: Oh, well, it’s all right. We can augment reality! I’ve got three heads and six hands!

Recently, Qualcomm announced an SDK for vision based augmented reality – currently in private beta and open to the public this fall. The Qualcomm augmented reality (AR) bonanza will launch with a $200,000 developer challenge and a SDK that will put vision based augmented reality into the hands of developers without licensing fees.

This is a big step forward for augmented reality and a very important move made by an industry giant to support the rapidly evolving AR industry. Innovation at all levels of the AR stack, particularly at the hardware level (CPU/GPU optimization) is vital for the full vision of augmented reality – media tightly registered to physical space, to take center stage. Vision based AR takes mobile AR beyond compass/GPS based AR post-its, which are only loosely connected to the world (but the staple of most current AR apps), towards the holy grail of AR – markerless tracking with the whole world as the platform.

Click on the image above or see here for a video demo of an AR version of Rock’em Sock’em Robots game.  Mattel, one of the first companies working with the SDK demoed AR Rock’em Sock’em, at the Uplinq 2010 conference (see Chris Cameron’s ReadWriteWeb write-up on Uplinq 2010).

The Qualcomm AR stack, which reaches from the metal to developer APIs, will give Android developers an important edge in AR development. And, when vision based AR starts getting integrated with visual search capabilities, and combined with cool tools like Unity, we will start to see the augmented world get really interesting.

Visual search is already an area of AR getting a lot of attention, with Google Goggles, Point and Find, Japan’s NTT DoCoMo set to launch “chokkan nabi,” or “intuitive navigation,” in September, and the recent partnership between Layar and Kooaba. Metaio’s mobile augmented reality platform Junaio is already integrated with Kooaba’s computer vision capabilities.

And, of course, I am particularly excited about including open distributed real time communications for AR in this stack, which is why I asked a group of developers who have been inputting into the ARWave project if they had questions for Jay Wright, Qualcomm.  Thank you Yohan Baillot, Gene Becker, Anselm Hook, Patrick O’Shaughnessey, Thomas Wrobel, Markus Strickler, and Davide Carnovale for your input.  [Note: see my upcoming post, about the future of ARWave and real time distributed communications for AR following this Google announcement.]

Jay Wright, “is responsible for developing and driving Qualcomm’s augmented reality commercialization strategy.” He “handles partnerships with leading innovators in industry and academia and leads Qualcomm’s efforts in enabling augmented reality within the mobile ecosystem.” In the interview below, Jay very generously answers our questions in detail.

A key contributor of questions for this interview is Yohan Baillot. Yohan is working on a full vision of AR – integrating computer vision, visual search, open distributed real time communications and AR eyewear. Yohan Baillot is founder of Simulation3D, a consulting and system integration company specializing in interactive visualization systems and eyewear-based AR systems. (I hope to bring you an interview with Yohan soon!).

Qualcomm was the title sponsor for are2010, Augmented Reality Event, and played a vital role in making this event an historic gathering of the talent and creative minds at the heart of the emerging AR industry. Watch out for the videos of the are2010 sessions to be posted at the end of August. My are2010 co-chair, Ori Inbar, is preparing them to go online while kicking his newly funded start up, Ogmento, into high gear! Ogmento is also one of the start ups pioneering vision based AR.

Metaio, (with Total Immersion, they are one of the first augmented reality companies), has played a key role in bringing a vision component to smart phone augmented reality apps with their Unifeye mobile SDK.  Junaio, Metaio’s own mobile augmented reality platform has gone beyond location based AR with “junaio glue” – “the camera’s eye is now able to identify objects and “glue” object specific real-time, dynamic, social and 3D information onto the object itself,” (see my upcoming interview with Metaio founder, Thomas Alt).  Also, recently, Layar – who continue to innovate at a breathtaking pace, announced a partnership with the computer vision company Kooaba.

Both Maarten Lens-FitzGerald, Layar, and Thomas Alt, Metaio, when I spoke to them recently, saw the Qualcomm SDK as a very positive development for AR, and they look forward to exploring its capabilities and integrating it where appropriate with their AR tools.  See more about Layar’s upcoming visit, to the US here – August 10th NYC, and August 12th SF.  Also save the date, Sept 27th, Munich, for InsideAR, Metaio’s upcoming conference.

It is clear that vision based AR will be driving the next wave of AR apps. And, as Maarten and Thomas both pointed out, it will be interesting to see which use cases capture the imagination of users the most. Having more tools freely available to AR developers will certainly be a boost to creativity. And, Qualcomm’s SDK is going to give Android developers, in particular, a big opportunity to take the lead.

Interview with Jay Wright, Director, Business Development, Qualcomm

Tish Shute: Before I start with questions on the new Qualcomm vision based augmented reality SDK, I want to briefly look ahead to what many people feel is vital for the full realization of augmented reality – head mounted displays, or more specifically, comfortable, sexy AR eye wear. Is Qualcomm going to be involved in the development of augmented eye wear and wearable displays?

Jay Wright: I think there’s some core technology that needs to come together so we can have what we think needs to be a see-through head mounted display with a decent field of view. And that looks like something that is quite possibly further than a three to five year horizon.

Tish Shute: Gene Becker asked some interesting general questions about the Qualcomm AR initiatives. He said, “I’m unclear exactly what Qualcomm’s goal is.” It would be interesting to hear from you the Qualcomm view, from the top down.

Jay Wright: Our largest revenue stream comes from sales of chipsets. And we see augmented reality as a technology that drives demand for increasing amounts of processing power. So we want to create demand for chips, higher-end chips, and augmented reality does that. Specifically vision based augmented reality because it is so computationally intensive.

Tish Shute: Yes. And I think that is why people are very excited by the Qualcomm SDK. It is not only the first free toolkit for developers to build vision apps from, isn’t it? There’s been nothing freely available before this, has there? But also Qualcomm is paying attention to the complete AR stack to support vision based AR development, from the chips to game/app development tools like Unity.

Jay Wright: That’s really the goal. We’re not here to be in the augmented reality applications business. Qualcomm’s role in the ecosystem has been to serve as an enabler. And that’s what we want to do with augmented reality: provide the enabling technology that allows the entire ecosystem to flourish.

“Augmented Reality has a number of attributes that make it a great fit for Qualcomm’s core competencies”

Augmented Reality has a number of attributes that make it a great fit for Qualcomm’s core competencies. It’s very computationally intensive, algorithmically complex, requires tight integration of hardware and software, and benefits from tight integration of multiple hardware components. And that’s the kind of problem we like here, where we can apply our core competence of really optimizing complex systems for performance, while at the same time minimizing power consumption.

And as you know Tish, mobile AR is really extremely power sensitive. We sometimes talk about it as a battery’s worst nightmare. It’s roughly equivalent to playing a 3D game and recording a video all at the same time.

Whenever there is something that takes a lot of power, that’s a definite opportunity for us to optimize it.

Tish Shute: Right. One of the core business is chips right, but for Qualcomm there’s basically a lot of profit in licensing. When I talked to the developer community about the Qualcomm SDK developers first question was, “What’s the licensing? What’s this going to cost us in the long run to develop on this SDK re licensing?” And they had all different takes on this. So everyone had different ideas about what your approach to licensing might or might not be. Could you clarify the approach to licensing, as I think this is a core concern for developers.

Jay Wright: Anytime you see something for free, you kind of say, “Hey, what’s the hook?” So yes, it’s definitely a logical question. Our intent is not to generate licensing revenue from application developers using the SDK. So the SDK will be made available free of charge for development, and it will also be free of charge for developers to deploy applications.

Tish Shute: Now, this is another question. You also include not just image recognition capabilities but Unity in the package you are offering developers. Unity products usually involve a license. They do have some free products too, I think. But how does this work? And how do you separate your part from their part, or don’t you?

Jay Wright: That’s a good question. What we’re trying to do with the platform is incorporate it into tools that people already know how to use. So we’re actually going to have the SDK support two different tool chains. One of them is the Android SDK and NDK. And then the other one, is Unity.

We’re working with Unity to create an extension to the Unity environment that will be available as part of the Unity installer when you install Unity from the Unity website. Developers will still be paying whatever license fees are associated with Unity’s products on their existing pricing schedule.

Tish Shute: One of Thomas Wrobel’s question is whether developers can just use the image recognition without Unity? Your answer is yes, you can work with the computer vision component of the SDK separate from Unity?

Jay Wright: Yes, you can.

Tish Shute: Good because we would like to build a completely open Android client for ARWave, and not tie it to Unity unless people choose to. He’s using the open Android JPCT 3D engine, which he’s adapting for AR. So he could actually use the part of the SDK that does image recognition and association with that, right?

Jay Wright: That’s correct. You are not required to use Unity. Unity is just one option for building the application.

Tish Shute: Great! That’s very good. But I’m sure many developers are going to jump on the chance to use Unity. But I mean it’s nice to be flexible because it’s so early for AR that people have different ideas and new use cases coming up all the time. I think it’s excellent you’ve divided that.

Another of Thomas’s questions was, “Can developers use their own positioning data sharing solution?” He’s really talking about AR blips.

Jay Wright: With data sharing solutions, I am assuming that by data he means referring to augmentation data or graphics?

Tish Shute: Yes, and I’ll ask him to elaborate. But, at the moment, everyone is using different ideas for POI, aren’t they?

“The goal with our platform is to make it just as easy for a developer to create 3D content for the real world as it is for a game world or a virtual world.”

Jay Wright: Yes. So let me answer it this way, Tish. The goal with our platform is to make it just as easy for a developer to create 3D content for the real world as it is for a game world or a virtual world. So all we’re really trying to do is provide the computer vision piece that makes the real world look like a bunch of geometric surfaces and potentially some meta data that is associated with this so you know what you are looking at.

So that means from a developer’s perspective, you are still doing all of the 3D content, all of the animations, all of the game logic, all of the rendering. You are still doing that all yourself. So if you think about doing an AR game, you are doing everything you used to do, except you are not creating a virtual terrain. You are just going to map it in the real world.

So if you want to do a browser that is doing POI’s, your POI data, or augmentation, or meta data, or whatever it is, that can be in your application, it can be in the cloud, it can be wherever you want to put it. We’re not putting any constraints on what that content is or where it’s stored.

Tish Shute: Right, and that’s what I hoped for. And I think that does answer the question. People are interested to know how far Qualcomm is going with this. For instance, Gene Becker asked: “do they see a business at a certain level in the AR stack?” As you said AR development basically feeds into the core business of chip development, right? But does Qualcomm also see some new business models developing?

Jay Wright: I think it’s foreseeable that Qualcomm could identify other business opportunities down the line. But we’re certainly not there today. Today, our motivation for the investment in AR is to create technology that is going to advance the chipset business.

Tish Shute: When the news came out about Qualcomm’s support of a game development studio at Georgia Tech at the same time as the SDK I think I wondered what was the scope of Qualcomm’s interest [for more on using Unity for AR development see Vision-Based Augmented Reality Technical Super Session video from Uplinq 2010].  For example, I am interested to know how the Qualcomm initiative in developing an AR stack connects to the effort to introduce an AR browser based on web standards, i.e., the Kharma/Kamra KML/HTML Augmented Reality Mobile Architecture from Blair MacIntyre and the Georgia Tech team (image below)? Are you supporting the open standards based browser development too?

Jay Wright: Blair is going to continue to work on the browser effort. And it’s our expectation that he will use our SDK and technologies for vision pieces of the browser effort where appropriate. So they are certainly not mutually exclusive. I would just think about our technology as one element of what may be used in that browser, as I expect it would be an element of what any other app developer would put in their application, whether it be browser, or game, or whatever.

Tish Shute: Yes Now, this is an interesting question, which is sort of connected…I’m trying to keep some form of narrative for this! It follows from the question about Blair’s web-based standards browser. A few people have asked me why we haven’t heard more from Qualcomm in all these various standard discussions that are starting to come up. I mean is it just too early, or are you too busy, or what?

Jay Wright: No, let me explain. The type of standards that have come up so far have been around how HTML should be extended for geo-browser type applications. And while that’s interesting, I think the standards efforts that Qualcomm would be more likely to be associated with in the near term are those related to API’s that are hardware accelerated.

So one of the things that we are in the process of doing right now, Tish – because as you know, Qualcomm is a company that adheres to standards and strives to produce a leading implementation of those standards on our hardware and software – is we are in the process of determining what API set within the existing SDK should be standardized.

Tish Shute: Right.

Now, my next question is, “Who are the other players at this level of the AR stack in the standards conversation? Who else is working at that level?” Obviously, the AR Lab in Graz was, but now they are Qualcomm, right?

Jay Wright: They are still independent. Qualcomm is the exclusive industrial partner of the Christian Doppler Handheld AR LAB in Graz.

Tish Shute: Does this compete with, say, the work that other AR start ups are doing?

Jay Wright: Our intent is not to compete with companies that have done augmented reality technology. Our intent is to enable the entire ecosystem. So we would like to work with both Metaio and Total Immersion to find ways that they can benefit from our technology. That would be the hope – that our technology can kind of lift and float all boats in the ecosystem.

Tish Shute: There are not many implementations of vision based AR right now? I mean obviously Microsoft is doing stuff because they have Georg Klein now, right, and there is Google Goggles, Total Immersion, Metaio, and it will be interesting to see where Layar’s partnership with Kooaba will lead?

Jay Wright: Yes. I think there are relatively few commercial implementations of vision based AR stacks.

Tish Shute: One of Patrick O’Shaughnessey’s question is he wants to understand what features are going to be in the vision component, very specifically. Patrick O’Shaughnessy, Patched Reality, working with Circ.us, Edelman, and Metaio used the Unifeye SDK to do a vision based AR app for Ben and Jerry’s that’s been getting all the attention lately. He was a speaker at are2010.

He very specifically wants to know what features will be included in the computer vision component. He says, “I’m most interested in understanding what features are going to be in the vision component. Is it marker based?” Well I know it’s more than marker based. I saw some of it in Chris Cameron’s ReadWriteWeb write-up on Uplinq 2010. Is it “NFT? PTAM? other? Also, are you are integrating any backend services.” That is an interesting question!

Jay Wright: So let’s get to the features on the client side, the vision based features. There’s support for, what AR aficionados would know as natural feature targets, or image based targets. And we use those to represent, obviously, 2D planar surfaces.

The other thing that we are trying to do to set expectations, Tish, about where these can be used is to let people know that they work best in what we’re calling near-field environments. So the idea isn’t that you use the system to create a large scale AR system that can recognize buildings indoors and outdoors. It’s the idea where I can recreate 3D experiences that take place on surfaces that are in my immediate field of view, whether that be on the table in front of me, or on the floor, or on the wall, or on the shelf.

Also, when you talk about near field experiences, there are some other constraints that are implied. Like, if it’s in front of me and my immediate field of view is probably going to be pretty well lit. And lighting, of course, is an important requirement.

So we’ll support these natural feature targets, or image targets. And we also have support for sort of a hybrid marker image type. It’s something called a frame marker, which has kind of a black border with some dots on it.

Click on the image above or here to view Vision-Based Augmented Reality Technical Super Session video from Uplinq 2010

Jay Wright: So there’s this additional type. And the reason for this additional hybrid marker type is it has a lower computational requirement than a natural feature target. So the idea is these things can be used as game pieces or elements of play where I want to have a large number of them detected and tracked simultaneously.

So you can have, for example, one big natural feature target that serves as a game board or game surface, and you can use these other things as smaller game pieces. And when you put them out, different types of content can appear on them and do different things.

Tish Shute: Yes, that’s nice! And the other thing I noticed was the virtual buttons. How well developed is that?

Jay Wright: The idea behind virtual buttons is, in addition to supporting augmentation, we want to support interaction. And we think there are going to be different types of user interaction with augmented reality content. It may be hand tracking and finger tracking, but another compelling form we’ve identified so far is the ability for me to touch particular surfaces and have an event fire within the application..

So virtual buttons are rectangular areas on image targets that a developer can define, and they serve as buttons. So you can create a target that is a game board, for example, and define certain regions. And when the user covers that region with his hand, like pushing a button, your application can detect that event and take some action.

Tish Shute: Nice! And what is the documentation on these capabilities that is offered by Qualcomm…For example Yohan Baillot, who is interested in integrating eyewear-based AR systems with smartphones asked. How deep does this go? Will there be full documentation on Snapdragon, people who want to work at that level? Is there a chip SDK?

Jay Wright: . Qualcomm’s model is to work with providers of the operating systems and deliver functionality of the chip through the operating system. So many operating systems APIs will take advantage of functionality that’s in the chip. But there is no separate chip SDK per se.

Tish Shute: I suppose that does come up a little bit with one of Anselm Hook’s questions, because there is some overlap with Google Goggles here, isn’t there, in terms of what you’re doing, right? Are you going to work closely with Google Goggles ?

Jay Wright: Google Goggles is performing what we’ve described ‘visual search’. So the idea is you take a picture, send it to the cloud and identify it and the results come back. I think if we see Google Goggles go in a direction where there’s an AR experience that would be a good area for us to collaborate with Google.

Tish Shute: Anselm Hook is very interested in having some kind of open standard around this physical tagging of the world, right, – the physical world as a platform. But I suppose that’s down the road but is there a plan to start talking about open standards here – visual search with image recognition? That’s a very powerful combination. (see my interview with Anselm Hook here).

Jay Wright: I think it is. And we’re very interested to hear from developers and others that have ideas about how they would want to integrate with the functionality that we have to best enable those kinds of combined experiences.

Tish Shute: Well, I know Anselm has a lot of very important ideas on that.

Jay Wright: I’d be very interested in hearing those because we want to do everything we can to enable the maximum number of applications and best user experience for anything that people want to do.

Tish Shute: Let’s go back to some specific questions about the platform, right? For example Yohan Baillot asked, “Is it arbitrary image/tag recognition supported? Is the tag / image specifiable by user? Is face recognition supported?” Not yet, face recognition, right?

Jay Wright: Not yet.

Tish Shute: What are the plans with that?

Jay Wright: I think we’ve identified it as an interesting area and something that there’s some interest in, but have not made a decision on a particular technology direction.

Tish Shute: You’ve answered some of these but 3D model based vision tracking. Yohan’s question was, “Is 3D model based vision tracking supported (that is recover the pose of the camera using a known 3D model and a 2D camera view of this model)?”

Jay Wright: That’s something we’re looking at very closely, but again, don’t have a plan, or don’t have a future date for.

Tish Shute: And you said with the natural landmark tracking that’s not supported, right?

Jay Wright: I don’t know if I know what that means, Tish. But we don’t have any APIs that provide compass or GPS functionality other than already exists in the operating system. So if you want to take advantage of the compass or other sensors, you can absolutely do that, but the SDK does not currently provide anything different or anything more than already exists in the OS.

Tish Shute: This is an interesting question, “Is Snapdragon offloading some processing to the GPU, if any?”

Jay Wright: Certainly rendering functionality that utilizes OpenGL is being offloaded to the GPU. We’re currently in the process of determining multiple methods for offloading functionality between both symmetric and heterogeneous cores on Snapdragon. Which would include the GPU, the apps processor, and DSPs.

Tish Shute: No one has truly solved optimizing the GPU/CPU for mobile AR yet have they?

Jay Wright: That really gets to the heart of the optimization here. Which pieces ought to be operating on which cores and when, and why? And that’s something that we’re looking at very closely.

Tish Shute: Right. The only AR – that is truly 3D media tightly registered to the physical world has been done for military and medical (and that has often been with a locked of camera!). But to take mobile AR to the next level I think many developers would like access to the CPU/GPU, for example a developer interested in the future of eyewear like Yohan?

Jay Wright: We’re very interested in hearing what kinds of tools developers would like to see.

Tish Shute: What is the best forum for discussing feature specifics?

Jay Wright: To provide feature requests to us?

Tish Shute: Yes. And discuss them.

Jay Wright: if people go to qdev.net/AR there’s an application up there for the private beta program. So if people do have ideas about features or other things they would like to see, they’re welcome to submit [their requests and ideas] there.

Tish Shute: I also have some questions about the specifics of the competition. Some people are a little confused about some things. Yohan asked, “What is the expected form of the project? Lab demonstration? Specific capability? Complete end to end system?”

Jay Wright: The only requirement is that they submit an Android application that we can then get running on a device. So if it has a backend component or backend server that it works against, great. If it does, it does. If it doesn’t, it doesn’t. But that’s really it. There’s no limit to the application category. It can be a game, it can be a museum tour, it can be a children’s learning game or learning experience. It can really be anything. The idea is we want to find experiences for which AR delivers some unique value. We’ll be announcing more specifics about the competition in the near-future.

Tish Shute: Right, because some people weren’t sure about the Unity being separated whether it was biased towards games. And it’s not really, is it?

Jay Wright: Unity is a bias toward just rapid development for 3D, I think. It’s most commonly associated with games, but there are also a lot of Unity customers that use it for medical simulations and other types of applications that aren’t really games at all.

Tish Shute: Yes. It’s very flexible, I know. You did bring up the backend services again. Are you thinking of offering any of that?

Jay Wright: There is a backend tool that we offer. And the backend tool is what you use to generate your targets. So if you want to create or use a particular image for a target in your application, you upload it to our target management application, and then it will evaluate that target and tell you how well it will work. So as you know, certain images are more likely to be recognizable than others. And so there’s metrics in that application that will give you some feedback.

And then you can download your target resource from the website that you can then incorporate into your application project.

Tish Shute: So this is available at the moment to people who are in the private beta and not to…you know, all of this information and documentation, right?

Jay Wright: That’s correct.

Tish Shute: So that’s an incentive. Now, just to encourage people to submit to the private beta is the other thing that people seem confused about. In one part you say 25 developers. And some people have thought that meant it was limited to 25 individuals. And some people have like maybe four people on their team, so they were going, “Well, are we going to be accepted because we have four developers, or do we count as one because we are all working at the same project?”

Jay Wright: it’s just 25 companies.

Tish Shute: OK. I think we’ve gone through the questions. Just to clarify and maybe give some incentive for people to apply to the private beta…the big advantage of getting in the private beta, aside from getting a month’s start on the competition, is that you get a chance to input, right?

Jay Wright: Yes. A chance to provide feedback, get early access to the technology. And then we are also providing a free HTC phone.

Tish Shute: Oh, yes. I forgot the phone. Yes, right. In the requirements, though, you basically seem to be asking for sort of a full app…some people get reticent about delivering their full application plan, right?

Jay Wright: Yes. I understand that. People should just reveal what they are comfortable talking about. Just so you understand the constraint on this end, this is early technology and we’re trying to understand exactly what the support requirement is going to be. And we have limited supported resources at this time, so we want to make sure that we can focus the resources that we have on folks that are really going to use the technology and have a sound plan to actually build something. So that’s really the motivation behind limiting the size of the private beta.

Tish Shute: OK. Yes, it’s good to reiterate that. We’re down to the last question that I have, and then I’ll ask you if there is anything that I missed. You say you are partnering with Mattel. Who are the developers? Because I mean Mattel isn’t an augmented reality development team.

Jay Wright: Mattel used a subcontractor, Aura Interactive.

Tish Shute: Nice. But that’s your only partner that I saw, right? Why Mattel?

Jay Wright: Well, to launch a new technology, companies will often find showcase partners to demonstrate compelling uses of it. And we thought Mattel and the Rock’em Sock’em™ toy was a great example of combining augmented reality with an existing toy.

Tish Shute: And I think people agree with you on Rock’em Sock’em (see Chris Cameron’s RWW post).

Jay Wright: And there’s other showcase partners and applications that we will continue to work on to kind of spur the ecosystem and show what is possible.

Tish Shute: OK. Now, is there anything I’ve left out that you think? What’s the core of this narrative that we need to get across, and if I’ve left anything out that is a key piece?

Jay Wright: I think you’ve done an excellent job of covering all the bases, Tish.

Tish Shute: [laughs]

Jay Wright: I think the important overriding message to get across is that we really see ourselves in an enablement role here, and that we are trying to provide….we’d like to provide fundamental technology that helps all developers build content for the real world.

Shortly after Augmented Reality Event – are2010, I talked with Bruce Sterling on skype and in gdocs about his experience there.  I am posting the conversation in two parts to make it a more blog friendly length!

The picture above is the Auggie Award for the best AR demo (above) designed by Sigal Arad Inbar.  It was won by YDreams! See, Ivan Franco recounts the team’s ARE 2010 experience, and winning the event’s first-ever Auggie Award, for more.   The video below was shot at the YDreams booth by Bruce Sterling.

“The Hotness” – YDreams rocking it at ARE2010 from brucesflickr

Rudy Rucker, who was hanging out with Bruce Sterling, captured the are2010 buzz and some great images in his post, Augmented Reality, Painting, Twitter. As Rudy put it:

“AR is hoping to be a next big thing, a cozier and more commerce-driven cousin of the old VR, or virtual reality.”

Bruce Sterling’s opening key note is up, ARE 2010 Keynote by Bruce Sterling: Bake a Big Pie!, and also the ARE 2010 Keynote by Will Wright: Brilliant Inspiration for the Augmented Reality Community with more videos from are2010 on the way.  One must read post on are2010 is Chris Cameron’s post, Augmented Reality’s Next Steps: Sitting Down with the Titans of AR.

Talking with Bruce Sterling, Part 1

The Auggie panel, Bruce Sterling, Jesse Schell, and Mark Billinghurst inspect the award.

Tish Shute: In your keynote at the 9am of the augmented reality industry you asked some questions of the are2010 audience: “What’s the mission statement?  You’re the world’s first pure play experience designers, except that user experience it’s mostly futuristic hot air.  But run with that, right?  What are your tactical steps?  You should get dressed, have a coffee, have a to-do list.”

How much of that did you see going on in the next two days?

Bruce Sterling: Well, I wasn’t privy to any of the business discussions.  I didn’t think it was an accident that this standard AR enabled tag thing came up from Bruno Uzzan, Total Immersion.  That seemed to me to be a useful thing.  I was always interested in the Augmented Reality Consortium.  It struck me as remarkable that there was this group of people who clearly all knew one another and it had some kind of game plan.  I applaud them for that, because these are not the 1980’s.  [laughs]  You know, it’s just a different world for young startup companies.

Tish Shute: I think you’re right. There seem to be some VC conversations going on, we don’t know what went on in the meetings, but it was noticeable in the atmosphere of excitement, and remarked on by a few people. So I think that kind of was definitely going on.

And, of course, I was so busy I never even got to see the expo properly! You said you wanted to be surprised.

Did anyone surprise you in any of the talks, in any of the expo?

AR used as interfaces for devices

Italian augmented robot from SEAC02 from brucesflickr

Bruce Sterling: I have to say I was a little bit surprised to see Andrea Carignano demoing a robot. I happen to know him because he’s here in Torino. He’s the guy that came out of Fiat and went into AR. I am not a particularly huge robot fan, but I think it’s of great interest that AR is used as interfaces for devices, as opposed to the Jesse Schell idea that AR is all about a “man with the X-ray eyes.”

My suspicion is that a lot of surprises will come out of mashups of AR.

Tish Shute: I didn’t get to see Andrea’s robot.  So what did it do?

Bruce Sterling:  It’s basically a sister device to that little helicopter that those Parrot AR Drone guys were doing.  It’s a little autonomous robot and it runs around with a webcam on it.  You can place video into the acquisition stream coming off the robot.  You can play a game, and blow away imaginary monsters or whatever.

Tish Shute: It’s interesting, because did you notice Will Wright and Patrick O’Shaughnessey, Patched Reality, spend some time hacking the Parrot AR drone in the hallway?  Did you come across them?

Bruce Sterling: Rudy was there with them.  You know, I didn’t want to watch Will Wright hack a robot.

[laughter]

Tish Shute: They seemed to be having fun even though as it turned out the power supply was dead.

Bruce Sterling:  I’m sure Will enjoyed that.  As a game designer, you want to go out and get your hands dirty with a plastic gizmo.

[laughter]

My Swiss Army knife can’t get through airport security, so I really don’t want to strip anything down.  But yeah, what else did I see that was of particular interest?  I was pretty happy about the Korean guys because they are a difficult group to get close to.

AR companies are like mini-global micro-startups.  They’re “glocal”.

“Korean elegance at the Zenitum booth” – from brucesflickr

Tish Shute: Zenitum.  What did you like from Zenitum.  They were one of our sponsors, along with Qualcomm.

Bruce Sterling:  I know that Seoul is like the number one center for augmented reality discussion.  But it’s  difficult to get behind the scenes as a journalist there and  track what’s going on in Korea.  I’m fine with Italian “realtà aumentata.”   And I feel like I’ve got a handle on French “réalité augmentée.”   The Germans were not hard to find, and the Dutch all speak English!  But the Koreans, and whoever the hell it is in Kuala Lumpur…  I have no idea what’s going in Kuala Lumpur, and only the vaguest idea of what’s transpiring in Singapore!  But I know that people there are paying a coherent interest.

So the Koreans show up, and they had some relatively predictable anime style 3D avatar conversion stuff.  But they had a really nice display space.

“Anime figures become three-d smartphone animated avatars,” from brucesflickr

Tish Shute: Ah, So Zenitum created a hot spot at the exhibition?

Bruce Sterling:  Yeah.  The Koreans had  IKEA furniture and some nifty little woven baskets.  They’d really classed up their presentation.  Most Koreans in tech tend to be kind of muscular.  The Koreans are not known for their refined presentations.  On the contrary, they tend to undersell everybody else.  But I don’t know, maybe they’ve been hanging out with Samsung and upgrading their design chops. [laughs]

Tish Shute:  Did you take some photos you could send me?

Bruce Sterling:  I took a few, but  I don’t consider myself a photographer.  They’re all up on my Flickr set. It was interesting to see so many people from so many different nations in such a collegial atmosphere.

Tish Shute: Yes – there were many different countries represented at are2010

Bruce Sterling:  It’s the beginning… and so global at such a young stage.

Tish Shute: Yes. As you said, it was 9 AM, so everyone was actually super excited to be gathered together from across the globe to start a new day together.  As you mentioned, there was a very warm affirmative vibe – everyone sharing a passion.

Bruce Sterling:   They have an online commonality. They seem to be aware of one another’s work through the Internet.

Clearly they had all heard about one another.  That’s a departure from earlier models of tech startup, where you usually have like three hippies in a local garage.  Now you’ve got German-American-Korean outfits like Metaio, and Total Immersion has a Russian affiliate.  They’re inherently multinational, both inside the company and out.

Tish Shute: It was the multinational garage, wasn’t it?

Bruce Sterling:  Yeah.  AR companies are like mini-global micro-startups.  They’re “glocal.” There’s something quite new to me about that.  I don’t find it’s shocking, because in Europe today it’s common to find startup teams who are multinational.  But to see such intense globalism at such an early stage of an industry is really different.

Tish Shute: Yes it made for a fun atmosphere?  It was wonderful running into Iguchi Takahito, Tonchidot.  You have a great rapport with each other despite the language barrier?

Bruce Sterling:  Yeah.  That guy from Tonchidot, he’s very charismatic.  He’s punchy.  That’s reflected in the very strong graphic design from his company.

Tish Shute: Using minimal English to make the case for Sekai No Camera at the Auggies, Iguchi Takahito still got through to the audience.

Bruce Sterling:  Well, his visuals were good.

What AR means for artistic practice…

Picture of Eric Gradman’s Cloud Mirror, from James Alliban post ARE2010 – Augmented Reality utopia in Silicon Valley – see for more on the are2010 ARt Gala

Tish Shute: So before I move on to wider themes, I’m going to wrap up on some of the different aspects of the conference.  I was chairing the technology track but you were more free roaming, was there anything that went on in the sort of hallway discussions and the presentation rooms that struck you?

Bruce Sterling:  Well, I did get collared by artists.   They really wanted to talk to me.  We got into some serious discussions on  what AR means for artistic practice.  How you can do this and reach that, how can one sharpen up one’s presentation?  I mean, they really wanted some art criticism.

Tish Shute: That’s very interesting.  Did you come up with anything that you hadn’t been thinking about already through the conversations?

Bruce Sterling: I’ve seen augmented reality installations before, and I certainly know many electronic artists.  But I don’t know.  People in the AR art space, they are looking for guidance and trying to find fellow spirits.  In their own way, they have the same pioneer spirit as the business people.

Helen Papagiannis shows Iguchi Takahito, Tonchidot, her AR Wonder Turner, an exquisite corpse inspired installation

Tish Shute: Yeah, it’s interesting, because we wanted the art gala to be even bigger, but it turns out, because of the logistics of putting up art in a conference space is fabulously expensive, because it has to be all installed and hung…

Bruce Sterling:  I’m keenly aware of that.  At Share Festival in Turin we bring in six installations, and it’s very heavy work.  It really takes a lot of logistics.  It was like a Battle of the Bands.  It’s like doing a rock concert.

Tish Shute: One of the installations I was really sad to not have there was Uber geeks’  “Steve” H.E.AI.D installation that Brady Forrest & Co. took to Burning Man.

So I was very happy that we actually did get the number of artists we did.

Bruce Sterling:  Well, there aren’t a million AR artists in the world, so it’s hard to judge.   I didn’t see many business people rushing up to have me critique their business plans.

Tish Shute: [laughs]  They were all in the meeting rooms.

Bruce Sterling:  Maybe it’s for the best.

VC and AR Startup Action

The Zenitum Booth, are2010, photo from Chris Cameron’s Flickr stream

Tish Shute: Do you know that why your talk started a few moments late is because we had 50 people who arrived from the Silicon Valley neighborhood I guess!

Bruce Sterling:  Did they not preregister?

Tish Shute: No. They all stood in the line for the same day registration!

Bruce Sterling: It ‘ll be interesting to see what transpires there, if there is a little wave of startup action.  God knows they need some place to put their money, because the VC scene in the US is pretty much moribund.

Tish Shute: Ogmento is the first US AR Games startup to get VC, I think.  I think there was some VC action at are2010 for sure.  And Qualcomm obviously seems to have commercialization plans for their AR technology, and to be scouting talent  and ways to deliver new AR experiences.

Jay Wright, Qualcomm presents Joe Dunn, e23 Games, winner of the are2010 StartUp Launch Pad with a check

Bruce Sterling:  Some  people don’t need venture capital.  I mean, Google Goggles isn’t going to be hurting for VC money, obviously [ see Chris Cameron's RWW post, Google Goggles Coming Soon to iPhone] .  AR may come up through other methods, like people allying themselves with Hollywood, or peeling off of advertising companies.   There’s a lot of outfits who might conceivably want in-house AR skills.  Then when people set up a specialty AR shop,  they  peel off the list of clients.  I don’t know.  Those old days  of Silicon Valley venture capital seem like a lost world.

Tish Shute: Yes.  I, again, didn’t see anything really of the business tracks and production tracks.  Did you get back and forth between the tracks?

Bruce Sterling:  I went to the Hollywood tracks.  I mean, to the extent that I could.

Is Hollywood stirring? Who’s going to have the first breakout AR property?

Tish Shute: So what did you see from…Is Hollywood stirring?  Is it waking up?  I mean I know Kent Demaine, Oooii,  and Brad Foxhoven, Ogmento, spoke about the Hollywood AR scene.

Bruce Sterling:  There were guys there from LA who were sort of saying, look…they are aware of us, but they just want AR to promote their properties to some particular niche.  They realize that AR is potentially a mass medium and that you could do some real AR entertainment.  So they were batting around some ideas as to where that might happen.  Like, could it come out of a console gaming scene?  Who’s going to have the first breakout AR property?  A popular hit AR property, as opposed to like a neat way to sell shoes, or whatever.   Really, anybody’s guess is as good as theirs or mine.  But at least they were actively guessing.

Tish Shute: I know the breaking the fourth wall discussion has been going on for a while and now the question is, whether AR is going to take down the fourth wall and bring interactive storytelling into the mainstream.  Did you hear any of that?

Bruce Sterling:  Well, I always shy away from discussions of that kind because I don’t think there’s any “final thing.”  Practically everything that AR is involved in right now is  a transitional technology. Also, because I am a storyteller, I get alarmed whenever people in technology start saying, “Oh well, it’s all about telling stories.”  Because obviously it isn’t.

People can tell stories perfectly well orally, and absolutely nobody does that.  AR is not at all about telling stories.  It’s about a great many other things, such as user bases, niche audiences,  media saturation, urban informatics,  convergence culture, and the language of digital media.   I could list these factors until the world looks level. It’s really becoming pretty chaotic.  As I was saying in my speech, AR companies are media startups who almost never use the old-fashioned word “media.”

Tish Shute: Oh, that’s interesting.  Yes.  So why do you think that has happened that way?

Bruce Sterling:  Well, it’s because they are trying to do a different thing than media does.  I mean, they are trying to “augment reality.”  They don’t want you to know that you are using a medium.  They don’t want you to realize that you’re watching computer animation overlaid on some video acquisition stream.  That would defeat the whole point of AR.  It’s entirely different from an analog medium like television, where you turn on the television and there’s a constant stream of station identification alerts.   That’s like: “Don’t touch that dial!  You’re on channel 13!  Stay with us!”  Then it’s like, “And now a few words from our friendly sponsors!”  That medium was engineered to keep your eyeballs locked to a single stream that they’re feeding you.

In AR, it’s much more participative, more geolocative.  I’m not particularly interested in station-identification branding from my AR provider. What I really want to see is the interactivity of the augments they’re bringing to me.  It’s like  FlickR, the photo sharing site. You don’t have any TV-style splash page for FlickR.  ”Hi! We’re FlickR! FlickR, bringing your photos to you!” No, FlickR is all about “you, you, you,” your photos, your tags, your friends, your activity around you.   It’s immediately trying to be very participative.

Tish Shute: Will Wright got to that point, didn’t he. He was trying to move us into an idea of blended reality. That the game is about the world, not about the dragons or the overlays per se.

Bruce Sterling: Right. I think that’s true. But see, the world isn’t a medium. A medium is something like this interview,  where I’m connecting to you and there’s a video Skype channel between us.  Whereas AR is more about spatial 3-D,  about 3-dimensional impositions.  Pieces of media: sound, vision, information visualization, tags, floating tags, air tags, icons, arrows, warning signs, warning sounds, tactility, whatever, being brought into the environment around us.

That’s why it’s properly called “augmented reality” instead of just augmented media.   If you call your work “augmented media,” you’re really in trouble. Because if it’s all about augmenting somebody else’s media, why doesn’t that medium just buy you, and augment their own selves?    If you think that way, instead of augmenting the world, you’ll just be a modest little plug-in for old-school media.

The World as the Platform

Blaise Aguera y Arcas, Microsoft, Santa Clara, are2010, photo from Chris Cameron’s Flickr stream

Tish Shute: Yes, which is why Blaise so generously gave the technical underpinning  for augmenting reality in his tech talk – about the trellis and the grapes,  he really explained how the world can become a platform for augmented reality.

Bruce Sterling: I wish I could have seen that. I did not see Blaise’s speech.

Tish Shute: We’re going to put the videos up in better quality.  People in the front row have put it up on the web already.  He really went into some of the challenges of mapping for augmented reality.

Bruce Sterling: His visual-mapping technique is important.  Registration is super important for AR.

Tish Shute: I think it was a really generous talk actually because he went step by step on how we will do this.

Bruce Sterling: I rather imagine that Microsoft has patented those steps.

Tish Shute: Oh, yes, I guess so!

Bruce Sterling: I could be wrong. Maybe they’ll open-source it. You never know.

Tish Shute: You never know. Because the world as a platform isn’t something one company can own, or go it on their own to exploit.

Bruce Sterling: I expect there to be a thorny path, but sometimes I’m surprised. Sometimes people really do try to fertilize the tech field in the hope of getting a good corn crop before they start fighting.

Tish Shute: We’ll I keep hearing that we may even see the unlikely marriage of Apple and Microsoft  – maybe wishful thinking, but there are motivations beyond AR for this unlikely match, and certainly between them these titans have what it takes to realize the grand visions of AR ? [laughs] But who knows…

Bruce Sterling: Well, yeah, it depends on where the thing catches fire.

Tish Shute: Yes. You mean whether AR catches fire in the form of  AR and mapping..

Bruce Sterling: It’s hard to say, but I’m convinced now that there’s more going on than I once thought. I thought that Bruno Uzzan made a very good speech for his company when he talked about how he worked on AR for eleven years.  Eleven years is no flash in the pan.   He has his long list of clients and successful applications. I thought he was right in his impatience with the press for not catching on. It’s gone on for quite awhile. The mere fact that you’re not aware of it, doesn’t mean it doesn’t exist.

The Illusive AR eyewear

My are2010co-chair, Ori Inbar, CEO and co-founder of the hottest new AR game development start-up, Ogmento, donning his goggles to open are2010 – picture from Chris Cameron’s Flickr stream

Tish Shute: Yes. So, the other theme you brought up in your opening keynote and I would be interested to know if anything you saw at are2010 changed your view is the illusive AR eyewear, and  if we actually got AR Goggles that worked they would bring AR’s gothic sister, VR, back from the grave right? [laughs]

Bruce Sterling: Right.

Tish Shute: It took quite a lot of work, but we pulled together a six-company HMD panel, right?

Bruce Sterling: Yeah. I was impressed to see so many of them there.  And I was chagrined to see how prototype-like all their gadgets were. But that doesn’t surprise me, because if any of those head-mounts were remotely working, they would be hyped out the wazoo. Everybody’s been waiting for them and hoping for the best. They’re obviously not ready for prime time. [laughs] Maybe in certain limited applications. Like maybe a diving mask. [laughs]

Tish Shute: No, I think what was nice though they got inspired and they all got together on the last day. I saw them having a meeting about standards. They got inspired to actually work together.

Bruce Sterling: Yeah, well, unless they’re going to invent mechanical eyeballs that those machines can fit onto, it’s going to be tough. OK, I’m a skeptic, but I’m prepared to be surprised. I’m also a skeptic in Artificial Intelligence, but as soon as they bring me an AI that can write a decent novel, I’m going to get it and review that book. [laughs]

Tish Shute: It’s interesting. Re AI, I’m totally in agreement with you. In terms of the way computers turned out, it wasn’t AI per se that they turned out to be good for, not in the way everyone had dreamed of it, rather it was the harvesting of human intelligence that turned out to be the big thing. But what is interesting is that despite all of that, AI or machine learning, as it is now called, permeates our whole society now from the stock market to how many businesses make many of their decisions.

Bruce Sterling: Well, there’s a lot of so-called collective intelligence.  But Marvin Minsky-style hard AI, no way. Alan Turing-style AI, forget about that.

Tish Shute: Yeah. So, that’s an interesting comparison with the HMDs.

Bruce Sterling: People stretch the definitions.  It’s like, well, my car engine is Artificial Intelligence. Yeah, so is your wall transistor. No, I don’t really think so.

And AR is a similarly big tent. I mean, Uzzan had to admit that he had denied that AR was AR, unless it was using his favorite technology. And he felt embarrassed to be rubbing shoulders with people who put AR into cell phones. And I can understand his feeling there, because, gee whiz, that’s certainly not what AR pioneers had in mind. But he had to admit he’d become more ecumenical about it. Obviously, they’re  there and doing business like gangbusters. You can’t very well ignore success, right?

I had a similar feeling about the goggles. Obviously, the goggles would be great, should they work. But if they did work, I rather think virtual reality would come very strongly to the fore.  You’d see people doing all kinds of elaborate immersive-style stuff.   A truly immersive technology doesn’t need to “augment” much of anything.

Tish Shute: Yeah, you’re right.

Social Augmented Experiences

Bruce Sterling: I think many of the most interesting AI aspects are not personal in the way goggles are.  They’re not about guys walking around with personal tech. They’re about big, communal, social-media experiences, like stage shows, and urban informatics, things where large numbers of people can interact with the same augmented reality. The projection mapping, which I go on and on about. Augmented public spectacles.

Tish Shute: Yeah, projection’s our best example of a social augmented experience right now because we are yet to have an easy way to do networked social augmented experiences easily – but that is of course the thrust of my interest in ARWave [see the slides for my presentation, AR Wave:  Federation, Game Dynamics, Semantic Search, Mobile Social Communications here].

Bruce Sterling: I think of Edison’s early days, when he wanted to sell movies to people for a nickel a clip.  You had to bend over and put your eyes on this visor and turn this crank. That coin-op device was easy for Edison to monetize, as opposed to getting a bunch of people to sit in theater seats. But people laugh at movies when they’re together in the seats.   Cinema is a more social, involving experience in a crowd situation.

Tish Shute: But it started with them, didn’t it, Hollywood – the movie biz? Basically Nickelodeons, right?

Bruce Sterling: That’s right. They were Nickelodeons. They were a lot like the goggles because they isolated the user.

Tish Shute: Yeah, that’s a really important point that the goggles are not Nirvana because of this question of whether they actually detract from the social augmented experience and blended realities, by drawing us into VR experiences?

Bruce Sterling: I’m tempted to claim that they’re more a VR technology than an AR technology.

Tish Shute: That’s a very interesting point because…

[thunder]

Tish Shute: Wow! What was that?

Bruce Sterling: Thunder storm.

Tish Shute: Oh, my God, how very Gothic! [laughs]

Bruce Sterling: It can get pretty loud up here in the mountains.

Tish Shute: Oh, you live in the mountains, better still!

Bruce Sterling:  Torino is in the foothills. This is Piemonte. So the Apennines are over there. The Alps are over here. We do get some rather spectacularly unstable weather.

Tish Shute: It sounded like a bomb to my NYC ears. [laughs]

Bruce Sterling: Yeah, it didn’t hit the building, but it was maybe half a kilometer away. I saw the flash.

Tish Shute: Oh, you did?   Well, I hope you don’t lose your power midstream here.    I was really happy to hear of that connection between Rudy Rucker and Layar  [Rudy was touched when Maarten Lens-FizgGerald from Layar said that he met the Layar co-founder at a Rudy Rucker lecture].

Bruce Sterling: That was very fun, yes.

Tish Shute: Wasn’t that wonderful? What was that experience like going around the conference with Rudy?

Bruce Sterling: Well, you know, Rudy’s very into graphics. He’s a mathematician, so he understands the underpinnings of this stuff. But he’s a skeptic. He thinks they’re kid toys. He’s not a gamer. He’s a good old-fashioned computer-science hacker. So he wanted to tell me all about his new eighth-order, fifth-dimensional fractals. He showed me a great many of them. They’re psychedelic. Rudy’s fractals are considerably trippier than most apps that help you find a barber or a train station. [laughs] Rudy really is a visionary. He’s into some very weird stuff.

Gamer Guys at are2010

Brad Foxhoven, Chief Marketing Officer, Co-Founder, Ogmento at are2010

Tish Shute: At are2010 there was a lot of discussion about how game dynamics and AR are going to intersect, right? Anything that you saw of interest there?

Bruce Sterling: Well, obviously, there are gamer guys there. Ori’s a gamer. The gamer guys are getting some money. The big buzz right now in gaming is, of course, social gaming.  Farmville has kicked everybody’s ass because it’s not even a game and yet it has more users than the entire gaming industry.

Tish Shute: I know, right! [laughs]

Bruce Sterling: Obviously that’s kind of humiliating. For a long time, I’ve seen people trying to do giant multiuser games on cell phones. It’s difficult to do because the interface on cell phones is crap, right? People aren’t going to run around responding to SMSs.

I can imagine people running around with little Wii-style bats that have audio and visuals on them. It makes a very large native AR game seem more plausible.

Tish Shute: Yes. that would be cool!

Bruce Sterling: Again, it’s not very gamelike to use those little fiduciary markers.

Tish Shute: No.

Bruce Sterling: Moving little cardboard chips, around like with card games…. It would be pretty easy to set up a little AR chess game.  Star Trek style hologram chess pieces,  and so forth. But it’s just cumbersome.

Tish Shute: And also, from what we’ve seen from things like Foursquare, the proximity based social gaming doesn’t have to offer very much [a crown badge, a mayorship] to get some mind share.. the social is the primary game dynamic…

Bruce Sterling:  I’ve seen a lot of different philosophies of gaming over the years. Who’s to say that Second Life doesn’t have the best idea? They built a little scene and then slammed their gate shut behind them.  But at least they’ve got a really nicely-paying little cult stuck in there. It’s different. And it’s manageable and it’s really theirs, theirs, theirs.  They don’t have to call in outside experts to try and run the monster.   They haven’t blown it up to the scale of Yahoo! where they’ve lost control of the enterprise, and gone into a tailspin of management overhead. Second Life has a very intense, almost a cultish atmosphere among the player-slash-developers.

Tish Shute: One thing that helped them was the thing they were always criticized, that the barrier of entry was so high. But once they got people they never left, right?

Bruce Sterling:  That’s not a bug, that’s a feature.

Tish Shute: One of the best features!

Bruce Sterling: Yeah, it’s like being in Mensa. Why don’t you lower your barriers to entry and get in some interesting stupid people?

Tish Shute: [laughs]

Bruce Sterling: In Mensa, we’d rather sit here making puns about neutrinos and fourth-order quadratic equations. [laughs] OK, that’s a business model, if that’s what you want.

The Man With the X-Ray Eyes!

Jesse Schell’s during his keynote, “Seeing,” at are2010, picture from Chris Cameron’s Flickr stream

Tish Shute: Ok!  Now to unpack the man with the x-ray eyes idea, Jesse Schell’s keynote theme.  This is a root metaphor for AR – making the invisible visible, seeing through walls. To me. I think you kind of wrote the book on this because all my ideas on what radical transparency might be come from you – your idea of Amazon.org is key to how I understand this..

Bruce Sterling: Oh, really? That’s funny.   I was touched that Jesse brought up that famous Corman film, because I was a judge in a fantasy film conference in Trieste earlier this year.  And Roger Corman was there.  He was the guest of honor.   ”X: the Man with the X-ray Eyes” was one of the films shown during the conference, and I saw it.  I even had dinner with Roger Corman.  I had never met him before, so that was quite amusing.  The difficulty with a film of that kind is that what we science fiction writers call a “House of Cards Ending.”  In that story structure,  you ramp the thing up until the protagonist sees God, and then he has to be destroyed by the falling pillars of the temple.  That’s a classic science fiction structure:  like Frankenstein.  For the sake of the drama, Corman evades the issue of what’s really going on. For instance, let’s just suppose “the Man with the X-ray eyes” is not in fact a psychopath.  Let’s say he gets a grant from the Department of Health and Human Services, and he acts like a real scientist, not a stock B-movie “mad scientist.” So he has, like, backup guys, and some placebos, and a large group of people to test it on, trusted colleagues, and so forth.  You wouldn’t get any of that movie’s wild activity out of that.  What you would get is like a 5% improvement to people’s vision.

Then, in a year, there would be a 10% improvement in people’s vision.  There would be a  classic industrial story.  A rising star, you know, a cash cow.   Real tech isn’t done by a single guy as a divine curse.  It’s created by classic  tech startup culture.  So a runaway technology really behaves in the way that personal computers do.

Tish Shute: The things that get me all Utopian and happy about this are the ideas like those you first outlined with the notion of Amazon.org.

Bruce Sterling:  It would be easy to do an entirely different kind of film than “Man with the X-ray Eyes.”  Something much less B movie,  much less pat.  I mean, at the end of the film,  he destroys his own hardware and blinds himself.  Why?  For what rational reason would he do that?  Why doesn’t anybody else know the big secret of what he’s doing?  Why aren’t there Koreans doing it?  Why aren’t there Austrians doing it?  Why aren’t there Italians doing it?  Why?  AR doesn’t behave like that.  It’s not one lone guy with magic eye drops.  It’s entire teams of people that have been working on stuff for 17 years.  They all approach it in different ways.

Now, they are going to get scandals in AR.  I can guarantee you that.  They are going to get into  hot water eventually.  At least some people will surely come out and accuse them of being Roger Corman B movie monsters.  But unless they accidentally discover atomic fission or destroy the Gulf of Mexico with an oil spill [laughs], I don’t think they’re going to be particularly badly off!   The trouble I imagine  for AR people is very typical new media trouble.  It’s like movies being accused of corrupting our morals, or comic books being accused of leading to violence, or Google being accused of making us stupid and warping our brains.

I’m not an alarmist in that sense, but at least I’m concerned about real threats.  Roger Corman is a B-movie director who’s trying to sew up his lost plot ends by destroying his hero and his hardware. That’s not very plausible. It’s a nice science fiction movie device, but technology isn’t a movie.

Tish Shute: Yes. Well, the other thing that you always remind us of with AR is not to be saying it’s going to be this glorious moment when it’s no longer gimmickey, no longer pop culture. You always emphasize that’s actually part of what’s good about it.

Bruce Sterling: It’s not an accident that practically everybody in that audience knew about Roger Corman.  Nobody looked surprised; not the Austrians, not the Koreans. They were all like: “Oh, yes! Roger Corman!  Love him!”

Tish Shute: There were so many Rudy Rucker fans. Were you watching Twitter? People like Eric Gradman were succumbing to fanboyz moments..

Bruce Sterling: “Yeah. Rudy Rucker, he’s the best.”

“Rudy Rucker gripping an Augmented Reality shoe” from brucesflickr

Tish Shute: [laughs]  I noticed you inspired him to join Twitter..

Bruce Sterling: Well, I’ve got 8,000 followers and, obviously, a lot of them are Rudy’s fans.  Of course he’s going to be gang-rushed on Twitter. That’s not really any more surprising than two motorcycle stunt guys at the same attraction. And I’m a big fan of his Rudy’s blog.   He’s always got interesting things to say.

Tish Shute: Yes. AR does seem to bring out some of the coolest smartest people!  This morning I had breakfast with Joshua Kauffman in Central Park.  He is an advisor and entrepreneur working on design in the public sphere.  I was feeling rather brain dead and jet lagged.  I told Joshua I was wondering how to get the cottonwool out of my brains for this interview and he suggested,  the All Souls College one-word question interview!  Have you ever heard of that? – although apparently they recently scrapped it.

Bruce Sterling: Well, I’ve heard of All Souls College there in Oxford. What was their interview question?

Tish Shute: They used to use only one word, so they would only give you one word. It’s not a question. Basically, they throw out the word and then you had to spin off from there.

Bruce Sterling: You’re supposed to free-associate on a single word?

Tish Shute: I guess so. I hadn’t heard about it, but Joshua suggested it.

Bruce Sterling:  Well, it’s possible..

Tish Shute: Joshua came up with some good words..

Bruce Sterling: Right.

Tish Shute: We were talking about these proximity-based social work networks like Foursquare and Gowalla and how they may influence the emergence of social augmented experiences.

So Joshua’s suggestion for the first word was “territorialization” e.g. how do these new mobile social experiences like Foursquare,  and the observation that actually rather than breaking down territorialization, which would be a good thing, tend to support territorialization…but perhaps new forms of territorialization?

Bruce Sterling: Yeah, they’re re-intensifying it in a very odd, electronic fashion.

Tish Shute: Yes.

Bruce Sterling: I have noticed that.  It’s not true of stuff like projection mapping or the webcam fiduciary display stuff. But with the handheld stuff, and especially the urban informatic stuff, it really can’t help but take on a local flavor. Layar is like “Augmented Dutch Reality.”

And TonchiDot really is “Augmented Japanese Reality.” It’s hard to imagine a Layar interface going gangbusters at Tokyo.  Whereas the TonchiDot interface, which is very clearly influenced by Anime and cartoon graphics…. Maybe it could find some niche of hipsters in Amsterdam hash bars…

…to be continued in Part 2