I am super excited to be speaking at SXSW Interactive 2012, as part of Will Wright’s Stupid Fun Club, on “A Lifestyle with a Gaming Sense.” Michael Trice just did a post on our session for SXSW.com, Screen Burn Panels at the Palmer Presents Will Wright’s Stupid Fun Club. The photos of Will Wright, Tish Shute (me!) and Peter Swearengen are by Anya Zavarzina. Thank you Anya for such great photos!

I have been too busy to blog much lately, but there is a lot to unpack in future posts in my quote in Michael’s SXSW post!

“Really we’ve entered a new era where the world has become a platform for storytelling and the goal is to turn everyday life into an opportunity for play, relatedness, and new forms of autonomy and fun. We’ve now come to a point where software has moved out of the computer and into the world. Rather than viewing this process in terms we’ve already grown out of, like gamification, we view this as an opportunity to explore everyday activities as possibility spaces.”

For the complete post, including Peter Swearengen of Stupid Fun Club on StoryMaker, see here http://sxsw.com/node/9969

1) Like most of us I wear a lot of hats. And I frequently work under a designer title. But recently someone said to me, “So you’re a Reality Architect.” I found the suggestion intriguing in part because I have been thinking about what it means to have agency in the algorithmic landscapes of the future that Kevin Slavin describes in his awesome TED talk, How Algorithms Shape the World. And, Reality Architect, if it implies anything, it implies a lot of agency and that is very appealing. But what does a “A Reality Architect do?”

2) When a very brilliant friend came up with this tag line for me, Tish Shute, Reality Architect, “She puts the reality back in Augmented Reality,” I began to become quite enchanted with the idea.

3) My career began with motion control photography creating visual effects for film and television. The Motion Control era which includes Star Wars, Close Encounters of the Third Kind, The Terminator, Star Trek, 2010, brought us many of the early design fictions for augmented reality.

4) With the arrival of smart phones I focused on the mobile local experience and making AR a reality. I co-founded Augmented Reality Event and ARWave – a completely open federated, realtime updating system for geolocated data of any sort.

5) But the AR dream has a dark side. This is a still from Keichi Matsuda’s great dystopic vision of AR’s future. Kevin Slavin pointed out in his talk, Reality is Plenty Thanks, that AR as visual layers over reality can obscure what is best about reality rather than enhancing it.

6) Recently I have been exploring what it means to make reality more interesting. Meet Gatsby is a location aware networking startup that I love. Gatsby orchestrates small world moments and creates contextually aware opportunities and serendipity in real life.

7) But we already have experts at making reality more interesting they are called Reality Stars. And when I say I want to make reality more interesting, I have no ambitions to be a reality star. Technology and Story telling are my passions.

OKCupid is a startup that has been making reality more interesting and solving dating problems with a combination of data, math and story telling.

9) We are entering a new area of social intelligence where people and algorithms are interacting in interesting new ways. OKCupid has been getting a lot of attention for offering social intelligence that can help us play better in our dating lives. And by connecting social graph, interest graph and location Meet Gatsby hopes to creates new opportunities in our daily activities beyond dating.

10) The combination of math, data and story telling is also a key to a new era of corporate intelligence. Quid works with Government and big corporations, “augmenting our ability to perceive this complex world,” to help them make better decisions on big questions in a complex world.

11) Sean Gurley of Quid at Strata NY described understanding complexity as a dimensionality problem. And, where the dimensionality reduction powers of Math meet the human powers of visualization and story telling powers of people is where insight arises. This is where I think, perhaps, the work of a reality architect emerges. An alternate title for a Reality Architect might be a Data Story Teller?

12) There is also a new space of personal intelligence emerging. Quantified Self, Self Tracking and Start Ups like, MyMee – that transforms “symptoms into empowering data,” are giving us new tools to understand ourselves and unravel pressing problems like allergies that frequently leave Drs drawing a blank.

13) Moodscope adds the power of sharing and benchmarking to the personal intelligence equation. “Lift your mood with a little help from your friends?

14) I am beginning to realize I know a lot of Reality Architects. Brian Krejcarek from Green Goose is designing simple fun sensors that turn everyday things into opportunities to play and give us new ways to play life together and be happier people.

15) There is also an interesting community of practice emerging around Habit Design, Nick Crocker demonstrates in, Floss the Teeth You Want to Keep, that there are a bunch of little hacks that exist to improve your ability to change.

16) The wonderful designer Asye Birsel through her project Design the Life You Love (the illustration above is one I did from her recipe) is teaching us organizing your life is not unlike other design problems. If you can visualize it you can change it.

17) With everyone carrying a powerful sensor device in their pockets, the World is Now a Platform for Story Telling. HipGeo keeps track of your movements and then spits out a slick, animated travel diary. Narrative Science is a company that among other things can turn excel spread sheets into compelling stories for executives.

18) But to return to design fictions again. One thing interesting about the HUDs in Iron Man is the emphasis on dialogue, and the sentient portion of the HUD as a character. The Aesthetics of Artificial Intelligence is increasingly directed at the interaction between algorithms and people. SIRI, for example, has a more highly developed character than Google voice. So the Aesthetics of AI is something I think aspiring Reality Architects might want to be think about and will probably play a significant role in future job descriptions and job titles we are yet to think of.

19) There is lots more I could say particularly about the importance of agency and putting people at the center of their data – please check out The Locker Project. But here are some thoughts on what I hope Reality Architects will do.

Create tools (not just maps and visualizations) to make reality more reliable, more constructable, and more useable.

20) Build technology that helps us live extraordinary lives. Situationist is an app that “injects our present lives with the unexpected.”

21) Create more opportunities, for serendipity, and fun in our daily lives. And last, but not least, never forget the potential of the phone toss!

Thank you @chrisgrayson and @kellyhadous for organizing TEDXSiliconAlley – great work!

I’m really looking forward to the O’Reilly Strata events that are coming to NYC in a couple of weeks. I’m fascinated to see where the art, science, and business of data has gone since February, when I attended the first Strata Conference in Santa Clara – a sold out event imbued with an awareness that this was an important gathering of cognoscenti working on the next big thing.

Strata in New York City is a sequence of events,  Strata JumpStart, Sept. 19th, and then The Strata Summit, “The Business of Data,” Sept. 20th & 21st, and followed by the Strata Conference, “Making Data Work,” Sept. 22nd, 23rd.

“The future belongs to those who understand how to collect and use their data successfully.”

Below is a transcript of a conversation I had last Friday with Strata Program Chair, Edd Dumbill about some of the highlights of the schedule from my perspective.  However, I highly recommend taking a good look at all that is planned through the three events because there is a depth and breadth that could not be covered in one conversation.

The video opening this post is from visual.ly.com – a start-up making it easier for people to create, explore, share, and promote data visualizations and infographics.

Talking with Edd Dumbill

Tish Shute: It seems a dialogue between the art of data and the science of data is going to be center stage at Strata NYC, and there will be much discussion about story telling with data.

Is that observation correct or is there something else going on there?

Edd Dumbill: No, I think that’s a great characterization. For the Summit, the core realization for me has been that when you have these tools for getting value from data and when you can drive what you’re doing by data, then actually, the biggest consequences are human ones, and they are organizational ones, and they are strategic ones once you have the technology in place.

So what the summit is doing is really looking at how, in a variety of industries, governments, and within disciplines within those, how the amount of data, the ease of which it can be communicated and mined is changing the way industry is shaped.

Tish Shute: Also, I noticed  that the Strata Summit Schedule (Sept 20th & 21st), and even through to the Strata Conference (Sept 22nd & 23rd), has more of an emphasis on pop culture; sports – baseball, dating – OKCupid, and Narrative Science, all have a place on the schedule, for example?

Is this the culture of New York City being reflected – interests in media and marketing, or is there something else going on?  Has the data tool stack matured since the Strata Conference in Silicon Valley at beginning of the year?

Edd Dumbill: Yes, there’s certainly a different flavor to the event because we’re in New York. And, yes, the tool stack has matured, but it is, by no means mature, and the maturity’s only coming at the lowest level.

I think there’s many years left in maturing the tool stack. But one of the beauties of big data is that once you have the data together, the algorithms to get value from it initially are pretty simple.

So, focusing on the stories of success of being data driven, particularly in the Summit, is important to us because the two questions people are asking are, “One, I’ve got data. Two, What do I do with it?”  We don’t need to make the argument that data is important anymore. But we do need to demonstrate what you can do with it.

The data isn’t necessarily big; it’s just there. It’s about having an analytical approach to your business that compliments your intuition, and compliments your vision.

“One of the most powerful ways of presenting data to people is in a story,” Edd Dumbill

Tish Shute: Yes I can see the emphasis in the schedule on how to tell meaningful stories with data. Narrative Science seem to be doing something very interesting re turning data into stories?

Edd Dumbill: Yes. They absolutely fascinate me with what they do. There’s this kind of hierarchy and sort of chain of needs right now where business is going, “We need data scientists. Find me data scientists. Train me data scientists. Hire me data scientists.” And the data scientists are all going, “I need visualization. I’ve got this data, I now need to turn it back into a story that’s going to be useful to people or provide interfaces that are going to help people understand and explore this,” because it doesn’t scale to have to have an interpreter all the time between the data and the results.

You need to be able to present it in a way that means something to people.

People can look at a graph and get many things out of it, maybe not even get anything at all out of it if they are not used to it. But particularly for digesting certain kinds of high-level summaries and results, if you can put the data back into prose, it makes it very accessible to people.

Tish Shute: Natural Language Generation from data really opens up so many possibilities..

Edd Dumbill: Yes, it’s interesting. I think it’s a very novel use. A lot of people would consider that the end result of their data was a spreadsheet or a graph that they are processing.

But if you turn that back into a story, I think there’s a lot of potential of helping executives understand what’s going on. It makes it possible to use language to understand the results.

Tish Shute: I am really excited to see the emphasis on stories, data design and visualization, and the way we experience data is as much part of The Strata Summit and The Strata Conference as some of the more hardcore big data challenges and analytics stuff.

Edd Dumbill: Yes. We are definitely ramping up on visualization. And I think that’s going to become more important. Having a fundamental grasp of how to use graphics and charts is still incredibly core to what we’re saying. But I’m also interested in ways that go beyond, because at least 50% of the point of visualization is to help people understand the dynamics of the data, to really augment their senses with the results of the computation.

You know, the people who are some of our best leaders, the ones who know how to ask the right questions of the data, have a sort of indefinable fingertip feel that you get for numbers when you live around them for a while. And anything we can do with interfaces to accelerate this is going to be very beneficial, whether it comes to being visual and flying through the data or hearing it in natural language.

Tish Shute: Have I missed anything in that in terms of what you’ve got on the schedule re visualization? VisualizingData.com published an ideal schedule from the visualizing data perspective. But have you added anything recently?

Edd Dumbill: Well, there’s one event which isn’t actually listed on the schedule yet, which is on Tuesday night. There’s a venue called EyeBeam in New York; we’re having a visualization showcase that evening. So there will be stuff to walk around and then a few talks, really from some of the most interesting companies doing viz and viz approaches. So that’s not up on the schedule yet, but that will be in addition. It gives a nice focus on Tuesday night.

Tish Shute: Oh, that’s super awesome. I’ll definitely go to that.

Tish Shute: I am very interested in mobile social communications and augmented reality – especially augmented reality that feels different, not just looks different, as Kevin Slavin puts it.

I am excited to see people thinking about data not just in terms of visualization, but in other ways too that we can feel it through our secondary senses as well (see Mike Kuniavsky’s talk at ARE2011, “Somatic Data Perception”).

Edd Dumbill: Yes, absolutely. That is where we view this as going. I will be incredibly depressed if I’m still looking at the world through a glowing rectangle in 10 years time.

Tish Shute: Yes, it would be! I am looking forward to see the new data start ups too.

Edd Dumbill: Yes, there are a variety of interesting startups, that I feel are particularly important in the data space. Media Sift and Data Sift, for example, Data Sift is doing a lot of real time processing on the Twitter fire hose. They provide real time analytics on Twitter, which I think is very important.

Tish Shute: In terms of using data to provision mobile experiences, real time is massively important, isn’t it?

Edd Dumbill: Absolutely. Yes.

Tish Shute: But real time data is still a big challenge, isn’t it?

Edd Dumbill: Yes. I mean right now, our focus on real time is probably at the technology level. Looking at real time, people are kind of building out the frameworks, companies like Media Sift and Data Sift creating parts of the experience.

And yes, our Where 2.0 conference will be focused more on the mobile experience.

Tish Shute: Re mobile experiences, I am very excited about Infochimps and their new geo APIs, and sensor data is becoming such a big part of the picture now too. But the Kinect has also opened up a whole set of possibilities for the future of sensor data!

Edd Dumbill: Yeah. I still think Kinect is probably one of the most exciting things going down because of the democratization of that kind of capability. Interesting things happen when the sensors become cheap, right?

When alongside a little camera in your iPad you have a Kinect sensor equivalent. That’s become extremely interesting because everybody has it with them and can do things based off it.

So the things that always fascinate me are when it becomes cheap and hackable.

Tish Shute: And if Kinect went mobile, that would be exciting?

Edd Dumbill: I think it’s entirely likely in the next couple years, yes.

The more sensors we can start instrumenting our mobile and personal devices with, I think it’s going to always result in some much more novel uses that we ever dreamed of.

Tish Shute: There was a lot of hoo-ha about Color when they launched this year. They were unable to capture a user base, but if they had issues of privacy might have come to the fore because they were really collecting more sensor data than any other app, right?

We are still waiting to see a breakthrough app in that area in terms of using all the phone sensors in ways that will really enhance a user experience rather than just the aims of data mining, aren’t we?

Edd Dumbill: Yes. I think this is one of the things where, in parallel, we’re really learning out the social and privacy implications of this kind of technology. It seems to me the focus has shifted from the tech in the second half of the year too. Frankly, everybody getting kind of freaked out about the amount of data that’s being mined and, you know, what’s acceptable use for that.

But on a slightly more prosaic level, there are some rather fabulous things being done. If you look at the Google Maps navigation experience on an Android phone. For instance, there’s some very practical applications of sensors collecting data with traffic and a variety of other augmentations going in that to actually do something useful.

So maybe we’d like to think we carry our sixth sense around with us in our pocket, and maybe we will. But we certainly can in our car right now with all the automatic rerouting and so on. That’s slightly more prosaic, but I think a lot more significant in terms of a pattern of how that can be applied.


Tish Shute: One of the Startups that really excited me in February at Strata, Santa Clara was Singly and The Locker project. They are really thinking innovately in the area of putting people at the center of their data.

I am looking forward to seeing the fruition of that work. And, while I’m enjoying Google +, it seems, we are just sort of holding up our hands and saying, “Well, there’s only one business model for data, and that is a centralized Fort Knox,” isn’t it? Or is there something that I’m missing?

Edd Dumbill: You’re right. I mean I think Google +, for instance, is rather the walled garden is a hedged garden. You know, there is a certain barrier there that I think is more about the fact that you need to put certain barriers up to actually create a decent user experience in the first place. I think user experience is one of the BIG problems with open data, and private data, to be honest.

There’s a reason we are not all writing PGP encrypted emails to each other, right? Because it’s so hard to make a UI for encryption that’s safe. Most people don’t use passwords properly. And I think a lot of the same user experience considerations come into this whole data thing.

Facebook can get away with anything they want to because have you ever tried using their privacy settings? Google, I think, more than anybody has tried to address this issue using sensible defaults, making the explanations clear. And they probably succeeded for a geek tech audience.

So I honestly think, probably, Locker’s biggest challenge, in that kind of approach, is definitely UI and giving the concept to the users so they can understand it.

But there’s certainly a very useful contribution to this conversation.

I think there are parallels in blogging, actually. There is a case where people have information they want to disseminate. And do you choose to do in on your own website, set everything up, publish for yourself, host for yourself, so you have complete control, or do you cede, for convenience, control to Blogger or Tumbler, knowing that you are being monetized somehow and that you’re playing in somebody else’s walled garden and don’t have that control?

So I haven’t really expanded that thought too much, but I think there’s something there in following that along and seeing where that actually leads.

But, you know, there is a whole technical challenge as well.

I really like the idea of being able to give permission to people. Being able to say, well, “I’m engaging you to do X,Y,Z in return for such and such. That seems like a good bargain to me. Giving up my data is a decent bargain for the services I’m getting back.” I mean that’s generally the contract we make in real life with people anyway.

That’s another thing re Google+, –why it’s a promising approach. At least in their rhetoric, they’re trying to say, well, “We’re trying to model this on the real life economy, the economy of real life interactions.”

Tish Shute: Yes. Any movement towards saying, well, “I’m not just collecting your data randomly, I’m collecting this data because I want to give something back to you that will enhance your interactions,” definitely feels like an improvement, doesn’t it?

Edd Dumbill: Yes. I think that bargain is clear. I’m just fascinated by who could be trusted and… I do actually wonder if there will be some kind of, rather than necessarily everything being decentralized like Lockers suggests, there might be an idea of a variety of inter-operating, trusted identity brokers. People who we would actually trust. Banks, right? We do that right now. Banks are pretty much our identity brokers. Who knows?

Tish Shute: I think, that is where the Locker project’s going with Singly, isn’t? Isn’t Singly the trusted broker for the Lockers, right?

Edd Dumbill: Yes. Now the question is whether you trust a startup with that or whether you’re going to trust… I mean, who knows? Trust levels are at such all-time lows with everybody right now. People in America won’t trust the government. I think Google are probably one of the most trusted brokers out there online.

Tish Shute: Perhaps, that’s interesting, isn’t it?

Edd Dumbill: I did write a piece, which kind of speculated that Google may become some sort of center brokering of social information and kind of a platform.

Tish Shute: Oh, yes, “Google+ is the social backbone” – a very thought provoking piece! It deserves an interview on it’s own!

But back to the Strata schedule! I notice you have DePodesta doing the Moneyball talk, right? What’s the 2011 twist on Moneyball?

Edd Dumbill: I think the twist on that is that the’re a lot more people can play now, really, which is why we’re having Strata in the first place. That 10 years ago the people doing this kind of stuff are McDonalds and Walmart and sports teams. Everybody, where there was large money, they could afford to gather the data. Maybe they could try this service out in making decisions based on it.

Well, we’re now in a very instrumented society where every business, every person has instrumented data about their interactions. I think the kind of resistance and dynamics and opinions that Moneyball brought up are the ones that people are going to be facing again right now as they seek to be more data-driven in what they’re doing.

It’s also very interesting to know 10 years on, what do you think? You’ve had 10 years of this, of sort of sabermetrics and so on. Have you matured in your view, have you softened?

What I’m endlessly and ultimately fascinated by is, where does this fit in the decision process and in the organization tree? Where does it mesh with vision?

Steve Jobs achieved it perfectly. He had vision and all kinds of things for his products. But Apple succeeded through a relentless operational efficiency. Absolutely relentless in their suppliers, their supply train, their manufacturing lines down to their detail. They are an utterly data-driven, process-driven organization at the same time as melding that with vision, design values and good quality. That’s a case where it worked together.

I’m eager to try and tease it out, figure out how that really works and how those things come together.

Tish Shute: And that’s another thread I see being explored at Strata, NYC. It’s not human versus machine or machine trumps human, but it’s human with machine. This is another theme, isn’t it?

Edd Dumbill: Exactly. We all operate by feedback loops. Really, what machines are doing enables us to get better quality data and in a tighter feedback loop.

Tish Shute: One feedback loop that we’re finding machines very useful for is understanding how we feel. I think that’s really interesting.

Edd Dumbill: Yes. I’m very fascinated by all the quantified-self stuff and where that can take us. At the end of the day, we have a very personal little organization to deal with, which is ourselves.


Quid: Building Software and Mathematical Solutions 
to Simplify Complex Decisions

Tish Shute: Yes! But the thing is we don’t understand ourselves in isolation, do we? I am definitely going to attend the session by Sean Gourley, CTO of Quid, on semantic clustering analysis. It seems like sentiment analysis is going big-time now, isn’t it?

Edd Dumbill: Yes. I mean, sentiment analysis is actually becoming a checkbox feature in databases now. The latest release of Greenplum has it built it. It’s that kind of level of feature that people want as social data is so important. Of course a lot of this is being driven by marketing and advertising.

Tish Shute: Yes but even re marketing data story telling has been taking some interesting and quirky turns hasn’t it?

Edd Dumbill: Yes, absolutely. I think there’s a lot of interesting research ahead of us there as well.

OKCupid Trends

Tish Shute: OkCupid is a very interesting example of data story telling that leverages our desire to know ourselves, and ourselves in relation to others.
.
Edd Dumbill: Yes. I mean they’re an example of a shift that’s happening in the PR industry, actually, which is companies understanding that telling marketing stories with data is very, very compelling. OkCupid really used that to hit well above their weight. Of course they got acquired as a direct result of that and their profile.

Tish Shute: I know OKCupid got acquired by Match.com, but you were saying they hit above their weight by using this analysis? How did that work?

Edd Dumbill: I think a lot of it’s down to their blog. That they analyze these things, publish them on their blog. It got a lot of attention, generated a lot of media stories, which brought them to Match.com’s attention. There’re millions of – well a large number of dating sites. But they differentiated themselves through the smart use of their data.

Tish Shute: Data and Games is an area I am very interested in. Zynga changed the game with game analytics and social games. And now we are seeing Rovio partner with Medio for analytics, (see Green pigs and data). But I noticed that you don’t have games as a strong theme on the schedule?

Edd Dumbill: I think you’ll see more of that on the West Coast to be honest. It’s not that we’re not interested. I just feel that the center of gravity to that topic is probably back on the West at the moment.

Tish Shute: So what’s after Zynga in terms of game analytics? A nice easy question!

Edd Dumbill: Sure. Let me predict the future for you.

Tish Shute: Yes please do!

Edd Dumbill: I don’t know, to be honest. One of the very interesting things about games is that it helps us understand the real world by modeling and playing around. I’m highly fascinated to see some more of those things played out through real life actors. There’s been some examples right out of Scavngr and whatnot. But if any of those techniques can really start to make a way into mobile technology, that’s one interesting thing.

What lessons can we take from what we’ve actually learned in game analytics that are reproducible and useful elsewhere?

Gamification is a bit of a trend right now. I am slightly skeptical… But I am fascinated by a lot of systems that are having these game elements added to them. And so the second question is, if you’re having games added to things, like losing weight or saving money or writing a book, I’ve seen that too, what can you apply from the analytics world on top of that, and learn about systems and tweak them?

I don’t have that good of an answer for you. How my game is, is not steeped in that. But I am aware that there’s probably a lot of progress in games that has yet to be applied anywhere else.

Zynga and whatnot, is kind of a space race, isn’t it, to monetize that. Space races generate technologies that can be applied in a variety of places.

What are the spinouts of game analytics that we can actually use elsewhere?

“These Bloom Instruments aren’t merely games or graphics. They’re new ways of seeing what’s important.”

Cartagr.am by Bloom

Tish Shute: Last February, at Strata, I was very struck by the new work by Ben Cerveny and Bloom on “pop cultural instruments for data expression” (also see Ben Cerveny’s talk at ARE2011).

Edd Dumbill: Yeah. I love every time the visualization comes onto a tablet….there’s an interesting back channel there.

And Google has done this in extreme to add to their great advantage. There’s a potential when you read an E-book, or you interact with the visualization of a tablet, that it can learn from your interactions.

If you read an E-book, and the book is instrumented and sends stuff back, then the book can read you at the same time that you’re reading it. That kind of collective intelligence can then be harnessed.

So what if Bloom’s pop culture visualizations are instrumented so that they know how people are using it? Well what can they learn about that? About either the quality of the visualization, about what’s interesting to data and back at the same time?

This is what the fundamental principles I think even of Web 2.0 and definitely in this era of big data that we’re in, is that the secondary signals, the exhaust from any electronic product, can be incredibly valuable.

We know that every time you run Google you are probably a part of at least one experiment that they are running to determine an optimal, and optimize their product through that. And how can you turn this up to generalize that out?

Tish Shute: I agree.  This is at the core of the art, science and business of data.  I hear your phone ringing, but do I have time for one more quick question?

Edd Dumbill: Oh yes.

Tish Shute: So it sort of follows on from my previous question.  The relationship between the crowd sourced intelligence and machine intelligence has played a huge role in making data work and  solve real world problems – Crowd Flower, for example.

Where are we at now with this relationship between crowdsourcing power of, for example, Crowd Flower and Mechanical Turk when combined with machine intelligence. Is there anything new going on here?

Edd Dumbill: What we’re actually starting to do is learn where to apply these tools. We’re reaching a point of understanding what crowd-sourcing is for, how to better design crowd-source tasks and so on in innovative uses.

One of the things I am particularly excited about is Natala Menezes who was at Amazon working on Mechanical Turk, she’s now moved to a company called GigWalk, which is a Turk platform that’s mobile.

So if you want to assign tasks that depend on people being in particular places and being able to do particular things, this is a platform for turking using that, which I think is fascinating. That’s definitely a new approach.

Tish Shute: Yes GigWalk is awesome – I saw that Photosynth is partnering with GigWalk. That is interesting – perhaps a step towards strong AR! ( see Read Write World and Blaise Aguera Y Arcas’s work on Photosynth was big news at ARE2011).

Edd Dumbill: Natala will be talking about GigWalk. I think the session is called quirky crowdsourcing. I want to call it Quirky Turks.

Tish Shute: [laughs] I like that.

Interview with Vernor Vinge

Tish Shute: Many of the pioneers of the emerging AR industry who will be speaking at, and attending Augmented Reality Event, consider “Rainbows End” one of their key inspirations. [Note: If you want to attend ARE2011 readers of this post can use my discount code TISH295 ($295 for two days, or for one day only TISH1DAY11 for $149]

What is the best and worst, in your view, about the way Augmented Reality is emerging from science fiction into science fact?

Vernor Vinge: Progress that sets the stage:
The worldwide market penetration of cellphones in the era 2000-2010 was of a size and speed that would have counted as foolish implausibility even in science-fiction of earlier times. More than half the human race suddenly had access to knowledge and comms. Being in the middle of this firestorm of progress, we can’t really judge ultimate effects, but I expect that smart phones and the empowering aspects of social networks and AR are still massively underhyped. (This is not to say that individual innovation enterprises can’t fail; the treasure is there for those who dare, and ultimately the whole human race can benefit.)

But I can still whine:
Some — mostly political/legal — issues are disappointing. These affect AR but also the broad range of our progress with technology:
o Software patents and some styles of cloud computing are blunting the ability of average people to innovate. In the 2010-2020 era, average people should have the building blocks to empower them to create (and throw away at the end of the workday) tools that in olden times would have been the whole purpose of a business startup.
Unfortunately, some companies restrict and compartmentalize their releases like we’re still living in the twentieth century.
There are also some mostly tech issues that I’m impatient with (speaking as a never-satisfied consumer and fan:)
o The low pixel counts in contemporary head up displays.
o The poor position coordination in current HUDs.
o The lack of mass market acceptance of HUDs.
o The lack of progress in distributed store-and-forward between
mobile devices (sub-femtocell, ad hoc and transitory forwarding).
o The lack of progress in uniform solutions to centimeter-scale
localization.

Tish Shute: What do you feel will be the most impactful application of AR in people’s everyday lives?

Vernor Vinge: There are nebulous and fairly high likelihood answers: AR apps that let each person/team see those aspects of physical reality that are important for their current activity. Pointing technologies that coordinate with that AR vision. The combination is a revolution of interfaces, and the probable physical disappearance of more and more of the gadgets that twentieth century people associated with high tech.

There are also more specific, spectacular, and necessarily uncertain impacts (that depend on social acceptance and the development of network infrastructure for consensual sharing of local imagery).
o Economic disruption of the trend toward huge, expensive display devices.
o Bottom up social networking, arising from GPL’d tools. I see this as very disruptive, in good, bad and arguable ways, as illustrated by descriptive terms such as “consumer protection clubs”, “belief circles” and “lifestyle cults”. Some of these could be as public as our topdown social networks. Some might be quiet and widespread, perhaps growing out of pre-existing groups that already have a lot of intermember trust. (See:http://www-rohan.sdsu.edu/faculty/vinge/C5/index.htm)
o More farfetched, but in the tradition of the last 50 years: the digitization of external visual design: building architecture could give less priority to physical appearance and more to cheap physical strength, network access support, and physical modifiability.

Tish Shute: I interviewed Bruce Sterling earlier this week – http://www.ugotrade.com/2011/05/06/augmented-reality-transitioning-out-of-the-old-fashioned-legacy-internet-interview-with-bruce-sterling/.  And, I’m really looking forward to your “fireside chat” with Bruce at the end of Augmented Reality Event to sum up the event [see the full schedule for ARE2011 here].  But was there anything that particularly rung a bell for you in my conversation with Bruce?

Vernor Vinge: Bruce says:  “… it’s pretty clear that the people who would weep for joy to have Augmented Reality are people whose reality is already damaged. People who need reality augmented as a prosthetic …” This really rings a bell with me. And social networks with AR may have a special impact at small sizes, even just _two_ players. At such a scale, they might be better called “joint entities” than “social networks”. For example, two differently disabled persons, where one is mobile. There’s a lot more that could be said about this, including applications that could be done (maybe are being done) already.

Picture via IEEE Spectrum: Augmented Reality in a Contact Lens

Tish Shute: As Jesse Schell pointed out last year at ARE2010, “The whole point of AR is to see things from a different point of view … How can there be a more powerful art form than one that actually changes what you see?”

The magic lens of the smart phone, screens – large and small, projection, audio and sensory devices are mediating our AR experiences today. Bruce pointed out last year in his opening keynote, that these less immersive forms of AR have their own merits.

But eyewear has always been integral to the big vision of AR. Do you see some interesting futures for AR without eyewear? And, How long before AR eyewear is part of our everyday lives?
Vernor Vinge: This importance of vision is a visionist claim , but for the majority of us who have sight, binocular vision is by far the highest bitrate input we have, and we have enormously sophisticated wetware for analyzing what we see. Current display tech is far short of fully exploiting this input channel.

Along the way to this goal, I expect we’ll pass through mini-eras of exploiting the best-available tech. Right now, that is the tablet and the smartphone. Sometimes I almost wish for slower progress: in the nineteenth century, you could profitably spend your tech lifetime mastering one mechanism (for instance, black-and-white silver halide photography). The whole world would benefit from your career. Now, we rattle through the mini-eras so fast that we never fully exploit what’s zooming past before we’re on to the next stage.

How fast (or if) HUDs like in Rainbows End show up will probably depend on network and localizer tech as much as the HUDs themselves, with clear generational differences within such eyeware. In fact, it’s fun to imagine the mini-eras you could get with different combinations of HUDs tech, localization, and networking.

(Aside, a quibble: I think AR should not be restricted to visual only. There are tactile and kinesthetic possibilities, at least.)

(Aside, a whine: If only we had an output channel with the bitrate and flexibility of vision! Wearables plus voice and gesture could do some of that. Going further might involve scary human re-engineering. In Fast Times at Fairmont High, I speculated that a small re-engineering (eidetic memory) could give a form of highrate output,
simply by allowing selection from very large menus.)

Tish Shute: Augmented Reality and Ubiquitous Computing are intimately connected. Is a distinction between AR and Ubicomp still useful? (This recent PARC blog post: http://blogs.parc.com/blog/2010/03/defining-ubiquitous-computing-vs-augmented-reality/ takes a look at the definitions.)

Vernor Vinge: In a literal sense there is a distinction, and there is enough technical challenge in AR to justify specialists spending all their time with AR. But Augmented Reality’s importance to humanity is in its role as a portal to the power of ubicomp and human cooperation.

Tish Shute: Augmented Reality, as we understand it now, is a human centered experience. But even now some of the most important aspects of our lives are governed by machine to machine intelligences that operate for the most part beyond the reach of human perception, e.g., the trading bots of Wall Street. What role can augmented reality play in better mediating between human intelligence and machine to machine intelligence? Does AR hasten the arrival of the technological singularity?

Vernor Vinge: I see four or five concurrently active paths to the Singularity:
a) Artificial Intelligence: We create superhuman artificial intelligence in computers.
b) Digital Gaia: The worldwide network of embedded microprocessors, sensors, effectors, and localizers becomes a superhumanly intelligent entity.
c) Internet Scenario: Humanity with its networks, computers, and databases becomes a superhuman being. (Bruce’s story “Maneki Neko” is a beautiful and subtle illustration of this possibility.)
d) Intelligence Amplification: We enhance individual human intelligence through human-to-computer interfaces.
e) Biomedical: We directly increase our intelligence by improving the neurological function of our brains. (I regard this last item to be the weakest of the possibilities.)

AR is central to progress with possibilities (c) and (d).
If we humans want to keep our hand in the game, AR is an important thing to pursue.

Tish Shute: Powerful computer vision apps are emerging for smart phones and face recognition technologies are beginning to appear in consumer apps. Do you think we need a major shift in the way we handle data ownership? And, is “there is a real risk of our augmented reality world being owned by interests which are not our own?” (see my conversation with Anselm Hook last year. http://www.ugotrade.com/2010/01/17/visual-search-augmented-reality-and-a-social-commons-for-the-physical-world-platform-interview-with-anselm-hook

Vernor Vinge: Yes, there is such a risk. (See also my political/legal comments in response to your question (1).)
More broadly, I see DRM and the Law being used to reify our intellectual heritage as permanent private property. If this could work, it would be the biggest grab in history — and a major roadblock on human progress.

But even setting aside all the open/closed/free ideological questions, there is another important issue here: anytime laws are passed making popular and easily accomplished behavior illegal, things get very ugly. It may seem frivolous to compare this to the first stages of the War on Drugs, but that’s where serious enforcement would lead.

Tish Shute: We have seen gestural interfaces go mainstream in the last year. What are the most interesting innovations with gestural interfaces that you have seen in recent months? What sessions will you go to at ARE this year?

Vernor Vinge: I’m way behind the curve as to what is happening right now. Collecting data points on real hardware and applications is a high priority for me in attending ARE 2011.

Tish Shute: Are you reading/writing any new fictional literature about AR? And/or, What design fictions for AR are most interesting to you in the moment?

Vernor Vinge: As to writing: My novel The Children of the Sky should come out this October from Tor Books. It’s set in the far future and is the sequel to A Fire Upon the Deep. Alas, the story has only indirect connections to our present technological interests.

As to reading: I got a big kick out of Daniel Suarez’s duology Daemon and Freedom(TM).

Planetary from Bloom Studio, Inc. on Vimeo.

It is just over a week until Augmented Reality Event, and I know there are a lot of people, including me (full disclosure I am co-chair and co-founder) who are totally psyched to see what unfolds there this year.   Bruce Sterling, Vernor Vinge, Blaise Aguera Y Arcas,  Jaron Lanier, Will Wright, Marco Tempest and Frank Cooper will join 107 speakers from 76 augmented reality companies on a single stage (see my previous post) to tell a momentous story of a technology of our time (also see my previous post here).

As Bruce Sterling points out, Augmented Reality is “truly a child of the twenty-teens, a genuine digital native,” and one visible indication that:

..the Internet really could look like a “legacy.” The Legacy Internet as an old-fashioned, dusty, desk-based place best left to archivists and librarians, while the action is out on the streets (see the full interview below).


(photo by Jasmina Tesanovic)

Opening this post is a video of Ben Cerveny’s Planetary app, which “turns your music into a universe,” and enchants all who try it.  Planetary shot into #3 on the Top Ten Free ipad app list soon after its release.

Ben Cerveny’s talk at Augmented Reality Event will be one of the must attend talks (see the full schedule for Augmented Reality Event here, and note my discount code for Augmented Reality Event, TISH295, is still good, if you want to register).

Planetary, while it is not an AR experience,  points the way for AR to take us out of the old-fashioned, “Legacy Internet.”

“Planetary is just the sort of science fiction experience you expect when using an object from the future like iPad,” developer Bloom Studio writes on the app’s iTunes page. ( from Mark Brown’s Wired post).

In his interview on cnet Daniel Terdiman, Ben describes how popular computing will evolve beyond those, “dusty, desk-based place best left to archivists and librarians,” (Bruce Sterling).

Ben points out:

“The tablet is a total disruption of how we understand popular computing. The next era of experiences will be driven by visceral gesture-based input, and rich fluid responsiveness in native graphics contexts. I see the potential for Bloom to help define a “killer pattern” for application design. Because apps have been deconstructed into discrete tasks that flow across devices….”

Bruce Sterling had some interesting comments on the Bloom app:

I’m a big fan of Ben and his good works in infoviz — and urban informatics, too. I admit I’m not sure the I entirely need the metaphor of a solar system in order to play a few Texas blues tracks. But I could be persuaded. Ben Cerveny is a significant thinker and a very well-spoken guy.

I’m not saying that happens tomorrow; I’m just saying, why not? Why not aspire to that?

I too am a huge fan of  The Bloom team, Ben Cerveny, Tom Carden, and Jesper Sparre Andersen (also see my post here about Fizz, the Bloom team’s app used by The Locker Project for their Strata demo).  And, if you haven’t already heard about The Locker Project and Telehash – get on it!  This is one of the most important projects of our time – an infrastructure for a better future!

Interview with Bruce Sterling by Tish Shute and Ori Inbar

Tish Shute: As you so memorably put it, “AR is a technovisionary dream come true – something really rare, and you have to be really patient for those….”

What is best and worst, in your view, about the way Augmented Reality technovisionary dream is coming true and emerging to flourish in the wild?

Bruce Sterling: The best part is that AR is truly happening and is a lot of fun, and the worst part is that it’s happening in a Depression. If AR had broken loose in the dotcom days when cash flew around like soap bubbles, man, that would have been psychedelic.

AR that is even more of-our-time than “social media.” AR has arisen directly from modern technical factors that just didn’t use to exist. It’s made from shiny new parts, and is truly a child of the twenty-teens, a genuine digital native. It’s a little kid and it has to walk before it can run, but it’s great to see it walking.

Tish Shute: As Jesse Schell pointed out last year at ARE2010, “The whole point of AR is to see things from a different point of view…How can there be a more powerful art form than one that actually changes what you see?” What do you feel will be the most impactful application of AR in people’s everyday lives?

Bruce Sterling: I’m all for impact, but it’s pretty clear that the people who would weep for joy to have Augmented Reality are people whose reality is already damaged. People who need reality augmented as a prosthetic, in other words, so that they can achieve an “everyday life.” This is like the impactful but underappreciated role of the Internet in the lives of people who’ve been shut-in. If you’re laid-up in a hospital bed, a laptop is a revolution in convalescence.

But that kind of “impact” doesn’t sound too exciting or too profitable. My guess would be that the biggest arena for “impactful AR” would be augmenting cityscapes for foreign people who can’t speak the local language, can’t read the signs, and lack time to learn the local reality. Imagine, say, the Brazilian overlay for Moscow. You could show up, read your native Brazilian overlay of that city, do your business, eat, sleep, buy, leave, and scarcely “be in Moscow” at all. Constructed right, the AR Brazilian Moscow might even be a better Moscow — a Moscow that Russians themselves would pay to visit.

Tish Shute: You pointed out last year, in your opening keynote for ARE2010, that less immersive forms of AR have their own merits. We are still not seeing much “head mounted display weirdness” yet, but many other forms of AR are emerging – mobile, webcam, projected video, sonic augmented reality, even sticky light. You noted, practically everything that AR is involved in is a transitional technology. But since you spoke last year at ARE2010, which of these transitional technologies have shown the most promise for AR?

Bruce Sterling: It’s got to be handsets. Smartphones. The stats there are just amazing. The smartphone biz makes the personal computer business look like a Victorian railroad. When I read a guy like Tomi Ahonen, who talks about transitioning out of the old-fashioned “Legacy Internet,” that idea is startling. But AR is one visible indication that the Internet really could look like a “legacy.” The Legacy Internet as an old-fashioned, dusty, desk-based place best left to archivists and librarians, while the action is out on the streets.

Tish Shute: This year we have seen gestural interfaces go mainstream. What are the most interesting directions for gestural interfaces that you have seen emerge in recent months?

Bruce Sterling: To me, the most “interesting” part is seeing people do gestural stuff in public. William Gibson, my fellow author, observes that cellphones have stolen the gestural language of cigarettes. There’s lots of fidgeting, box tapping, ash-swiping, slipping boxes in and out of pockets… People quickly learn to do that without thinking twice, and they forget how weird it looks. It’s “design dissolving in behavior,” as Adam Greenfield puts it.

The gestural hack scene for the Kinect has been amazing. It’s like watching 1950s Beatnik dancing go mainstream.

Tish Shute: You have observed that Augmented Reality is Glocal which not only gives us different flavors of augmented experience but is “a departure from earlier models of tech startups, where you usually have like three hippies in a local garage. Now you’ve got German-American-Korean outfits like Metaio, and Total Immersion has a Russian affiliate. They’re inherently multinational, both inside the company and out.” What flavors of glocalness do you hope/expect to see at Augmented Reality Event this year.

Bruce Sterling: I’d be pretty happy to see some AR input from Brazil, India, and South Africa. I seem to be picking up a lot of followers in my Twitter stream from those locales. If I saw some Augmented Bollywood Reality, that would pretty much make my day.

Ori Inbar: What sessions will you go to at ARE this year? Who do you want to meet at ARE 2011?

Bruce Sterling: I make it my business to hang out with artists, but I’m hoping to drill down more on the technical aspects. For instance, where exactly are the bottlenecks in building animated augments? It looks like we’re about a sneeze away from jamming some crude Hanna-Barbera cartoons into real spaces. But the devil is in the details there.

Ori Inbar: Your commentary about the evolution of the AR industry over the years had significant focus on style. Is the AR industry dressed to kill yet? Any glimpses of promise in that direction?

Bruce Sterling: I’m not “pro-style” in every possible aspect of life, but as an Augmented Reality critic, it’s clear to me that if you claim to “augment” reality, then you should work hard to augment it — struggle to make it better. Otherwise you might as well call yourself “Defaced Reality,” or even “3D Spam.” When I see that kind of crudity and carelessness in AR, I’m gonna call people out on it. I know there will be the AR equivalent of cheesy billboards and gang graffiti, but I never much cared for those, either.

The industry’s videos have improved radically in the past year and a half. It used to be all about “look at my grainy, shaky handheld video of my cool new AR hack,” but nowadays the biz has really pulled its socks up.

If AR is about “experience design,” as I think it basically is, then eventually, as a matter of intellectual consistency and professional pride, everything you create will be considered part of “the experience.” That’s the industry’s way forward — that’s what it would do if it was grown-up.

AR people already look better than most similar geeks in the gaming business, and some day, I really do believe that augmentation people will become glamorous. They won’t be supermodels, but they’ll be about as chic as, say, professional set designers. Because AR is set design, in a way; it’s real-time interactive set-design for three-D spaces.

Ori Inbar: In the Layar Launch in 2009 you said “it’s the dawn of AR…”, at ARE 2010, you followed up on the theme saying “it’s 9am in the AR industry.” What time is it now?

Bruce Sterling: I’d be guessing it’s around 9:30 AM, but come on, that’s just a metaphor! ARE we all gonna blow off at 4:30 PM and have a beer, or is AR one of those cruel tech startups where nobody ever gets a personal life?

Ori Inbar: Are you reading any new fictional literature about AR that inspires you? And/or What interesting design fictions for AR have you come across recently?

Bruce Sterling: Well, I’m always interested in creative people who just plain make stuff up. Because that’s why I commonly do myself. The stuff that “inspires” me is usually stuff that I just didn’t expect to see. But when I don’t expect it, that usually means I wasn’t paying enough attention. I plan to pay a lot of attention to AR this year.

I’m not sure it makes a lot of sense to write fiction nowadays “about AR,” because it’s no longer a fictional topic. It’s become like writing fiction “about cinema.” You can write good fiction about someone who works in cinema, but not fiction about cinema itself. AR is not sci-fi “Augmented Reality” any more, it’s become a real-world phenomenon, a new industry of real augmentation.

With that said, I must remark that I sit up straight whenever I see Marco Tempest do stuff. Magicians are all about mystery and wonder. You wouldn’t see a magician, say, using AR to work an assembly line, or re-order library books, or find a pizza joint in Barcelona. And that’s great. Marco is always gonna do something freaky and out-there, and even though he’s a tech magician, it’s never about the tech first. It’s always about his ingenuity in finding new ways to employ new tools in creating a magical experience for his audience.

Marco’s not an entrepreneur, he’s not gonna revolutionize people’s daily lives or invent Web 4.0, but even if AR becomes “old hat” some day, it’s never going to be old hat when he’s doing it. The guy is a pro, and I’m quite the fan.

Magic Projection Live @ TEDxTokyo 2010 from Marco Tempest on Vimeo.

The prophets of Augmented Reality, Bruce Sterling (best know for his science fiction works, his non-fiction includes “The Hacker Crackdown,” “Tomorrow Now,” and “Shaping Things”), and  Vernor Vinge (author of “Rainbows End” and “The Coming Technological Singularity”) are joining  Will Wright (Legendary game designer of SimCity, The Sims, and Spore), and Jaron Lanier (a computer scientist, composer, visual artist, legend of virtual reality research, and the author of You Are Not A Gadget: A Manifesto) to judge the Auggies at the Second Annual Augmented Reality Event, 2011, Santa Clara, Ca., May 17th, May 18th!  Pictures top row, Bruce Sterling, Vernor Vinge, Will Wright, 2nd row, Jaron Lanier, Blaise Agüera y Arcas, and Frank Cooper III.

Augmented Reality has been prototyped in science fiction writing and films for many years now.  But Augmented Reality Event, the first global event dedicated to the emerging industry of AR,  is your chance to be part of a momentous transformation of science fiction into science fact.   Bruce gave a seminal keynote last year at ARE2010, “At the 9am of the Augmented Reality Industry…” But we are edging closer to high noon this year – see John Markoff’s recent New York Times post, “Augmented Reality Comes Closer to Reality,” and Bruce Sterling’s opening keynote at ARE2011 will raise the bar for us all again.

Bruce Sterling  and Vernor Vinge will wrap up the show with fireside chat.  Jaron Lanier will be keynoting at are2011 for the first time and Blaise Agüera y Arcas (a leader in Microsoft’s mobile strategy), will return to top his Ted talk demonstrating innovations in Bing Maps and Augmented Reality.  Frank Cooper III, who serves as Senior Vice President and Chief Consumer Engagement Officer of PepsiCo, will give, an up to now, missing perspective of  the big brands’ on Augmented Reality (AR).

“The main stage at ARE 2011 will be blessed with science fiction visionaries, ground breaking scientists, mind blowing technologists, and legendary game designers and more!”

Come prepared to Augmented Reality Event.  Read as much Bruce Sterling and Vernor Vinge as you can, and follow @bruces blog on Wired, Beyond the Beyond.  With only a few weeks to go until Augmented Reality Event, myself and co-chairs Ori Inbar, and Chris Grayson, have been working hard on a preliminary schedule ( by the time you click on this link it should be updated again).  You can follow us on twitter for updates as they come in – @tishshute, @comogard, @chrisgrayson,  and please follow @arealityevent and join our ARE facebook posse.  And, of course, don’t forget to register soon while we are still offering discount codes.

Readers of this post can use the code TISH295 for $100 of the already sweet price if $395.

If you joined us for Augmented Reality Event last year, you will know that the”Auggies” – an award for the best AR demo presented live with commentary American idol style,  is a chance to join the best and brightest in AR as they write the next chapter for AR in the sharp and often amusing reparte between judges and contestants.  Submit your demo proposal here under the “Auggies” track.

The picture below is the Auggie Award for the best AR demo designed by Sigal Arad Inbar.  (See, Ivan Franco recounts the team’s ARE 2010 experience, and winning the event’s first-ever Auggie Award, and the video shot at the YDreams booth by Bruce Sterling. “The Hotness” – YDreams rocking it at ARE2010 from brucesflickr)

ARE2011 will include more than 90(!) speakers in 30 sessions organized into 3 tracks: business, technology & programming, and production & design.

It will feature special activities such as a Startup launch pad (submit your proposal here under track: “Startup Launch Pad”), and an ARt Gala with live performances.  There will be displays by top AR artists of AR art projects and live performance art by the world’s best AR artists featuring: Helen Papagiannis, Amir Baradaran, Sander Veenhof, Ina Centaur and more to be announced.

The exhibition area will be open throughout the 2 days of the event and will include the latest product demos by the leading AR companies, as well as a career fair to help grow the fledging industry.  A small number of booths is still available in the exhibition hall ($995 for 10′x10′).Grab them while they last!

A press conference will kick start the event, as a vehicle for AR companies to launch new products and services. Submit your proposal here and add: “Press Conference” in the title. The event wrap up will feature Marco Tempest with a live AR Magic show

I have been so busy working on are2011 (and another project yet to be named -  hint I am focused on social augmentation and the world as a platform for a geo-situated social interest graph), that I have had no time to blog since I was on The Potential for Augmented Reality in Education Panel at SXSW, 2011. But I have posted my slide deck here for my talk, “Enchanted Objects and People: Data Driven AR.”

We are entering a new era of collective engagement possibilities for augmented reality, consumer to consumer brokerages which could unleash the the visions of the collaborative consumption movement and tap into our collective energies in totally new ways.  As Seth Praebatsch, Scavngr, suggested in his SXSW keynote, The Game Layer on Top of the World, the power of communal game play may even help us address even the most intractable problems like failure in education and global warming!!

One of my favorite new AR ventures exploring the potential of new forms of social engagement through AR is Suwappu.  We are working on how to get them across the pond and a few time zones to are2011.

“Dentsu London are developing an original product called Suwappu. Suwappu are woodland creatures that swap pants, toys that come to life in augmented reality. BERG have been brought in as consultant inventors, and we’ve made this film. Have a look!”

Click here or on the image above to watch the video.

“Suwappu is a range of toys, animal characters that live in little digital worlds. The physical toys are canvasses upon which we can paint worlds, through a phone (or tablet) lens we can see into the narratives, games and media in which they live.”

Many new players in Augmented Reality will join the pioneering AR companies, Layar, Metaio, Occipital, Ogmento, Total Immersion, YDreams, and renowned innovators, Mark Billinghurst, Steven Feiner, Blair MacIntyre and the Georgia Tech team and many others will return to surprise and amaze us this year – see the still growing list of speakers here.   Qualcomm, who announced an SDK for vision based augmented reality, and $200,000 developer challenge last year, will also be back with several of their team at are2011!

Visual Search and OCR augmented reality apps have flourished in the last year and they will have a strong presence at are2011.  Long anticipated and oft fretted over face recognition apps are coming to a phone near you soon.  Viewdle, working locally on the phone, is taking a new approach to face recognition.  Google has so far said that they will not use or release the face recognition technology that they apparently already have.  But Shailesh Nalawadi, Google Goggles, will return to are2011 to discuss building out a computer vision search database and 3D Classifiers to solve AR challenges and show us what Google Goggles has been working on recently.  Aurasma, out next month, is perhaps the next big thing in this space, so we are all eager to try it out and see what they bring to ARE2011.

My interview with Anselm Hook, Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, is still a great reference point for important questions in this space.

Wordlens, wowed us all with OCR done locally on the phone, and they will be  showing off new features at ARE2011. Pulling out WordLens to translate at subway poster for the first time is a magic moment, and guaranteed to impress the person sitting next to you too!    Tactical Information Systems, who Fred Wilson said had the potential to be a Shazam for faces at Web 2.0 Expo, NYC, will show us what they have been up to since then. 

Data driven AR is still in the early stages, but it is data that will unlock augmented experiences of the world around us in many powerful ways.   Also, we are seeing exciting new efforts to put people at the center of their data.  Jeremie Miller who ushered in the Twitter era with his invention of XMPP has brought us a new real time protocol, TeleHash and the Locker Project, that will unleash the full potential of communication at the edge of the network.   And don’t miss Ben Cerveny and Bloom,  at ARE2011, with “pop-cultural instruments for data expression and exploration” and “where data visualization meets game design,”  I highly recommend Ben’s talk!

Also I am very excited that Mike Kuniavsky will be speaking at  ARE2011 for the first time.  He  pointed out in 2007 that Magic is a powerful core interaction metphor for Ubiquitous computing [and AR].  Offices. libraries, and the desktop provided us with common metaphors that unleashed the power of the PC through files, desktops and trash cans, browsing, etc.,  but metaphors of magic – “enchanted objects and people,” are keys to augmented reality experiences.   And, if Magic is a core interaction metaphor for AR, kinect hacks have shown us that gesture will be the soul of the AR experience.  Jaron Lanier’s keynote will be a must see and make sure you catch Chris Arkenberg’s presentation on the Future of Hands Free AR.

This post is just a glimpse at some of what will be going on at Augmented Reality Event.  I cannot do justice to everything in a single post, so I hope to see you there!  And, if by any chance you have a few moments to daydream between now and then, feel free to ponder your own script for a Green Lantern movie:  “In brightest day and darkest night, no evil will escape my sight.”  I will too.  As @brendanscully noted, “Green lantern’s power is Augmented Reality!”

Singly’s appearance at the startup showcase at Strata 2011 this week has excited thought leaders across the web since the story got out. Singly is a new startup that exists to provide oxygen and commercial support to the open source Locker Project, and new protocol TeleHash.

With some wonderful serendipity I met Singly on my first night at Strata.  The next day, I talked in depth to Jeremie Miller and Simon Murtha-Smith, two of the three Singly co-founders (see later in this post). I also had the opportunity to ask Tim O’Reilly, Alistair Croll and Roger Magoulas for some of their thoughts on the significance of this project (see below for their comments).

It was a real – pinch myself in case I need to wake up from a dream experience – for me, to stumble across Jeremie Miller with Simon Murtha-Smith sitting behind a hand written sign demoing Singly at Strata (see my  pic opening this post). As Marshall Kirkpatrick notes:

“Jeremie Miller is a revered figure among developers, best known for building XMPP, the open source protocol that powers most of the Instant Messaging apps in the world. Now Miller has raised funds and is building a team that will develop software aimed directly at the future of the web.”

Singly’s appearance at Strata began auspiciously when they won the judges choice award in the startup showcase. And following Marshall Kirkpatrick’s post, Creator of Instant Messaging Protocol to Launch App Platform for Your Life , and The Locker Project: Why Leave Data Tracking to Others? Do It Yourself, Singly have been burning up Twitter.

Singly, by giving people the ability to do things with their own data, has the potential to change our world.  And, as Marshall Kirkpatrick notes, this won’t be the first time Jeremie has done that.

“Pop-cultural instruments for data expression and exploration,” by Bloom

I was drawn over to the Singly table when an awesome app they were demonstrating caught my eye. Fizz, which is running on a locker with data aggregated from three different places is a first glimpse of one of Bloom’s, “pop-cultural instruments for data expression and exploration.”

Fizz is an intriguing early manifestation of capabilities never seen before on the web – the ability for us to control, aggregate, share and play with our own data streams, and bring together the bits and pieces of our digital selves scattered about the web (for more about Bloom and Singly, see Tim O’Reilly’s comments below). The picture below is my Fizz. In Fizz, large circles represent people and small circles represent their status updates. Bloom says:

“Clicking a circle will reveal its contents. Typing in the search box will highlight matching statuses.
This is an early preview of our work and we’ll be adding more features in the next few weeks. We’d love to hear your feedback and suggestions.”

If you are not already familiar with The Bloom team, Ben Cerveny, Tom Carden, and Jesper Sparre Andersen – go directly to their about page and you will understand why the match of Bloom and The Locker Project is a cause for great delight.

The Locker Project – a whole new way to connect from the protocol up

As Jeremie began explaining the depth and breadth of what The Locker Project is facilitating, I was utterly gob smacked. And when the penny dropped and I realized this is the whole 9 yards, bringing awesomeness to people with a whole new way to connect, from the protocol up, all I could think was, OMG finally!

Luckily I have had time to catch up with the whole team since then, and recovered my composure enough to ask some coherent questions. But I can still barely contain my enthusiasm for this project.

Singly, The Locker Project and TeleHash take on, and deliver a simple, elegant, and open solution to some of the holy grails of the next generation of networked communications. I have written on, and been nibbling at the edges of some of these grails in various projects myself for quite a while now. Even if you haven’t been reading Ugotrade, just a glance at the monster mash of my pre Strata post will give you an idea of how important I think Singly is.

My previous post raised the question of how to invert the search pyramid and to transform search into a social, democratic act. But if you are really interested in social search, I suggest staying keyed into what Singly is doing with The Locker Project!

One of Singly’s three founders, Simon Murtha-Smith, was building a company called Introspectr, a social aggregator and search product. Singly’s other founder Jason Cavnar was working on another similar project. And they came together as Singly because social aggregation and search is a very hard problem for one company to solve, and they realized that the basic infrastructure needs to be open source and built on an open protocol.

As Jeremie puts it, “We shouldn’t…(every startup that wants to do something interesting) have to spend this much time aggregating the data, building robust aggregators.”

To me what is so important about the Locker Project is that it is built on a new open protocol, TeleHash. And having the Singly team focused on supplying tools and the trust/security layer for the Locker Project will mean that developers have the whole stack they need to do some interesting stuff very soon.

I asked Jeremy to explain the relationship between TeleHash, The Locker Project and Singly.

Tish Shute: So TeleHash…

Jeremie Miller: It’s a peer-to-peer protocol to move bits of data for applications around. Not file sharing, but it’s for actual applications to find each other and connect. So if you had an app and I had an app, whenever we’re running that app on our devices, we can actually find those other devices from each other and then connect. Our applications can connect and do something.

For the entire edge of the network, basically, out there in the wild, and let those things mesh together.

And TeleHash is actually what has led to the Locker project itself.

Tish Shute: So TeleHash led to the The Locker Project and the Locker Project led to Singly?

Jeremie Miller: Singly is a company who is sponsoring the open source Locker Project…the three of us as founders, [left to right in pic below - Jeremie Miller, Jason Cavnar, Simon Murtha-Smith, ]

I took the pic above of all three founders being interviewed by Marshall Kirkpatrick of Read Write Web for his post, “Creator of Instant Messaging Protocol to Launch App Platform for Your Life.“  I think we will look back on this moment and say it was an inflection point for the web. At least I tweeted that!

Jeremie Miller: TeleHash is a protocol that lets the lockers connect with each other and share things. The locker is like all of your data. So it’s sort of like a digital person…

Tish Shute: A locker for bits and pieces your digital self?

Jeremie Miller: Yes. So TeleHash lets the lockers connect and directly peer-to-peer connect with each other and share things. Singly, as a company, is going to be hosting lockers first and foremost. But the Locker Project is an open source project. You can have a locker in your machine or you can install it wherever you want…

Tish Shute: Yes it’s often too difficult for a lot of people to set up something locally…so Singly makes it easy to have a locker right?

Jeremie Miller: A lot of people see this cool app or this cool thing they want to do, it’s something that you run in your locker that they need to be able to turn on a locker somewhere very easily.

Tish Shute: So Singly will provide the trust layer and hosting?

Jeremie Miller: Yeah, Singly is a company that will host lockers, as well as…when people build applications that run inside your lockers or use your data, you need to be able to trust them. Maybe it’s like social data and you don’t care that much. But especially once you start to get any of your transactions in there, your browsing history, your health data, like your running logs or sleeping…fit-bit stuff…then it’s much more important to be careful about what you’re running inside your locker and sharing.

So Singly will also look at the applications that are available that you can install and actually run them and look at what data they access, and look at who created them, and be able to come back and either certify or vouch for them.

And I hope in the long-run, as this grows and builds, that power users may actually be able to buy a little device that they can plug into their home network that is their locker. Wouldn’t that be cool? This little hard drive or whatever that you plug in.

Tish Shute: Wow – that would be very cool! Architecturally is TeleHash and the Locker Project related to your work on XMPP?

Jeremie Miller: Architecturally, some of the stuff I’ve learned, XMPP, in Jabber it was designed for the specific purpose of instant messaging, but it was still a federated model, in that you still had to go through sort of a central point so you could…a server that lived somewhere. So it was really optimized for like businesses and small groups, teams, as well as big companies out there; ISPs can use it.

So it was designed with that in mind—for the communication path to be routed through somewhere. And where I’ve sort of evolved over the years since then is really fascinated with truly distributed protocols that are completely sort of decentralized so that things are going peer-to-peer instead of actually through any server.

The last 10 years, peer-to-peer has gotten a pretty bad rap with file sharing.

Tish Shute: A really bad rap, yes.

Jeremie Miller: Yeah. And almost because of that, and because it’s really hard to do, that it hasn’t gotten…the potential for it’s awesome. There’s so many really good things that can be done peer-to-peer. And it hasn’t gotten used very much.

But the other side of the peer-to-peer thing that I think is critically important, look at the explosion of the computing devices around a person anymore, both in the home and on our person. We have one, two, three, four even. And the number of devices that are online for you that are yours…I look at my home network router and I’ve got 30 devices in my house on Wi-Fi. What the heck? That’s a lot of devices.

But right now, all of those devices, for me to work with them, I’m almost always going through a server somewhere, through a data center somewhere, which is ridiculous at face value. You go five, 10 years out from now, there’s probably going to be 300 devices on me in some form.

Tish Shute: So we need a peer-to-peer network just to manage our own devices?

Jeremie Miller: A peer-to-peer, yes. You know, my phone should be talking straight to my computer, or to the iPad, or to the washing machine, or refrigerator. The applications in my TV, or whatever, they should all be talking peer-to-peer. And it should be easy to do that. It shouldn’t be that the only way you can do that is to go through a data center somewhere.

[Our conversation continued, but to sum things up, for now, here is the final question I asked Jeremie which pretty much packs in everything I would like to do with TeleHash, the Locker project, and Singly tools/trust layer all in one!]

Tish Shute: How can TeleHash, the Locker, and Singly help people combine personal data from different sources – web and mobile for example, so the data locked up in our social graph on the web can be integrated with, for example, the location data and “the data wake” from our cell phone sensors, to know not only where we have been but to give us more ways to know where we are going?

Jeremie Miller: That’s a pretty packed question, but here’s my simple answer, hopefully just seeds the right discussion

Telehash is the protocol that lets the apps (mobile, sensor, or anywhere) talk to a locker as well as lockers talk to each other, it’s the chatter, moving the bits around the network. The locker is the storage for a person’s data and the crunching ability to analyze it or trigger actions from it. Singly is the company sponsoring the project(s) and helping anyone dev apps atop it. We’re going to build the platform and looking to the world to create some amazing things on top of it (we have lots of our own personal ideas we already want to create, hah!).

The Locker Project is not just “one more rebel army trying to undo these big data aggregations,” Tim O’Reilly

@lockerproject: “We’ll be posting updates on the Locker Project (https://github.com/quartzjer/Locker) here as we make progress, very awesome stuff “

During the Strata Media Conversation I asked Roger Magoulas about Singly and The Locker Project because Roger played Yentl and brought Singly and Bloom together!  Although there was not much time to discuss it, the relationship of TeleHash, The Locker project and Singly to the social network encumbents, came up, and Roger Magoulis and Tim O’Reilly gave some very insightful comments on this when I talked to them afterwards (see below).

Roger Magoulas pointed out:

“I think Singly has Facebook like aspects, but I think a better description is an app platform that integrates your personal and social network data – including data from Facebook. Sing.ly is likely to have challenges with some of their data sources, particularly if Sing.ly gains traction with users.

I like the app platform business model, although they face risks getting critical mass and app developer attention, and I like how they plan on using open source connectors to keep up with changing social network platforms. Jeremie has credibility with the open source community and is likely to find cooperating developers. The team seems to bring complementary strengths to the project and you can tell they all work well together. ”

And Tim O’Reilly went on to elaborate the awesome potential of this platform to bring something new to the ecosystem, and to comment on just how interesting Bloom’s insight into, “data visualization as a means of input and control” is.

Talking with Tim O’Reilly

Tish Shute: So will the Locker Project be able to break the lock of Facebook’s and other big sites’ control of everyone’s data. Sometimes I feel we are stuck in the era of Zyngification, where you have to do what Zynga did and leverage the system in order to gain traction or do anything with social data?

Tim O’Reilly: I don’t think that is the objective of the Locker Project —to break the Facebook lock, because I tend to agree, the value of Facebook is having your data there with other people’s data. What Singly may be able to accomplish is to give people better tools for managing their data. Because if you can actually start to abstract the data from various sites and you can set it and manage it yourself, then you can potentially make better decisions about what you’re going to allow and not allow. Because right now, the interfaces on a lot of these sites make it very, very difficult to understand exactly what the implications are.

And I think all this done right will create a marketplace where people will build better interfaces that will give people more control over their data. They’ll still want to put it on those sites, because why do you put your money in the bank? You know, because it’s more valuable being with other people’s money.

And I think that to conceive of it as one more rebel army trying to undo these big data aggregations is just the wrong way to frame it.

Tish Shute: Yes and framing the question the way you just did – that this is not just one more rebel army, might mean that the stage at Strata will be filled with new startups next year! That’s what I thought when I found out what The Locker Project and Singly was about – that we are about to see an explosion of creativity with personal and social data.

Tim O’Reilly: Yeah, sure. I mean, because at the end of the day, if you can start to extract your personal data in ways that make it more useful, you can potentially create the ability for people to build better interfaces. It’s not just Facebook.

You know, you think, “Oh, wow, I’d really like to have a management console for all my contacts.” And you go, “Well, I’m stuck with, I can use Facebook, I can use LinkedIn, I can use my address book in Outlook or Gmail or whatever, or on my local machine.” The tools are pretty primitive. And if we get a better set of tools, I think we’ll see a lot of innovation.

Now, some of those startups might well be acquired by a Facebook or a Google. But it if moves the ball forward in giving people better visibility and control over their data, that’s a good thing.

Bloom’s insight, “data visualization will become a means of input and control.”

Tish Shute: I loved the marriage with Bloom, which is interesting, because Ben and the Bloom team haven’t really talked a lot about Bloom yet, but I gather Bloom is moving to consumer facing work with data?

Tim O’Reilly: What’s really interesting about Bloom is the notion…You know people think of data visualization as output. And the insight that I think Ben has had with Bloom is that data visualization will become a means of input and control.

Tish Shute: Right, very cool.

Tim O’Reilly: So I’ve started to feel like data visualization as a way of making sense of complex data is kind of a dead-end. Because what you really want to do is to build these feedback loops where you actually figure something out, some particular atomic action well enough that you can create an application that let’s somebody actually do something with it. But the idea of visualization as a way of manipulating the data in real-time, data visualization as interface rather than as a report, it’s a small but subtle shift that I think becomes kinda cool.

Talking with Alistair Croll

Sing.ly – Join or Die from Singly Inc on Vimeo.

Alistair Croll: So I’m a big fan of Singly. They were my choice for the Startup Showcase. I think it’s certainly the right time – the team can execute on it. But the thing I like the most is I thought back to the early days of Photoshop.

So, Photoshop was a neat application that could take data in the form of an image and manipulate it. But the real value from Photoshop came from these amazing plugins. Like, there’s a company called Kai’s Power Tools that made these things that would allow you to do manipulations. Today, commonplace things that are built in. But at the time, they were things like building bubbles, and spheres, and drop shadows and stuff like that, cutouts, in amazing ways.

Another company, I think, called Alien Skin that made these things. There’s whole ecosystems of plugins. So you could go and get a plugin and transform that original data in ways you hadn’t thought of. And eventually, there was a macro language for scripting how you could do those things, and that found its way into the Photoshop environment.

But you think about the transformation of digital design from Photoshop, I think if you can take that same pattern of you create the basic ecosystem of a few tools, and then you allow people an open system on top of that, that’s unprecedented. I think it really does allow you to take ownership of that.

And then when you allow people the proper tools to federate that information. I was actually thinking of starting a company a couple of years ago based on data federation like that. But what you really want to say is I’ve got a pattern…It’s almost like a multi-channel mixer. You’ve got a band that is your health, your weight, your blood pressure, family photos, words you’ve used. You know, the more data I record when I carry my phone around with a headset of whatever, all of that stuff goes in, all my searches, everything.

And then I say, “Ah, I want to federate height, weight, and blood pressure with my doctor. I want to federate sleep cycles and nutrition with my child’s teacher,” and so on and so on. And you start to create these federated sources of data where now you have a teacher data mining, in a safe manner, the sleep and health habits of all the students along with report card information. And you suddenly realize that Johnny is borderline diabetic and falls asleep at recess.

That’s something that never would have happened. And that happens when you have tools to federate data and then compute on top of them. So this idea of, like, lifestreaming or life logging, this is a logical consequence of the whole lifestreaming movement; that whole recorded future stuff.

Tish Shute: Yes it really is a wonderful fruition to the visions of the lifestreaming movement [see this interview with David Galernter]. And best of all it sits on a new open protocol, TeleHash and the open source Locker Project that will give tools to everyone to work with these data streams.

Alistair Croll: Exactly. This is the toolset that sits on top of that stuff. Because once I’ve life-streamed everything, great, I have this bucket of stuff that I did that I never look at again. But if I can suddenly unlock that with data mining tools and analyze patterns, all of a sudden that life logging has a reason to have existed.

The biggest problem we have with data right now is we don’t have apriori knowledge of what will be useful. We could have been recording crime reports in the city of Chicago, and a year later it turns out that data is really useful for predicting diabetes in the city, but we didn’t know it was related.

So the problem, and one of the things I think that distinguishes big data from traditional data, traditional data is collected to some apriori knowledge of how it will be used. Big data tends to be collected for the sake of…it’s almost collected on faith that later on it will be useful for something.

Tish Shute: I am very interested in this idea of federation, I actually went as far as to deep dive into Wave servers because of this….

Alistair Croll: Yeah, Wave was a great example of federation, just too complicated. When it was canceled, both users [and developers] were furious.

Tish Shute: Yeah, I suppose you could see Google Wave as a bit of an Icarus project, right? I am so excited by Singly because it is coming sort of bottom-up – a very different approach…

Alistair Croll: And remember, Facebook didn’t work before Friendster. The only difference between being wrong and being too early is that too early costs a lot of money. So it may be that this is an idea that works now, but a couple years ago didn’t work.

Alistair Croll, co-chair of Strata 2011, in his post, reframed the question, “Who Owns Your Data?” as, “It’s not who owns the data, it’s about who can put the data to work.”

And I am sure there will be many more people able to put data to work, and into play, in a multitude of  interesting ways, now we have TeleHash, the Locker Project, and Singly.

Photo by Duncan Davidson.
Strata 2011 is presented by O’Reilly Media. Produced by Good Company Communications.

I think the photo above gives a good idea of how I felt on the last day at the Strata conference. Yup – like the cat who got the cream!

And in case you are wondering where AR is in this story – it is everywhere!  Below is a pic of the AR concept designs that were omnipresent in the media communications at Strata.  The one below I snapped off the job board.  But as Sophia Parafina noted,  “AR is maturing from displaying last year’s text bubbles and dinosaurs to big data overlaid on the world.”

The Missing Manual for The Future (or The Future: The Missing Manual)

O’Reilly Media, is famous for is producing  “missing manuals” for new technologies, but thinking of O’Reilly as just a publisher of books would be like saying Facebook is just a website (this came up in the discussion at Media Round Table at Web 2.0 Expo, NY, 2010).   In recent weeks, I managed to catch Tim O’Reilly at several events, Maker Faire, Web 2.0 Expo, Hadoop World, and the free webcast Tim did with John Battelle on The Battle for the Internet Economy (although Tim spoke several other times during this period!).

It occurred to me, as I immersed myself in the depth and breadth of innovation showcased and discussed at these events that Tim O’Reilly, and the O’Reilly team, are creating, The Missing Manual for the Future.

As Tim puts it, we are “changing the world by spreading the knowledge of innovators.” Tim uses a quote from William Gibson to illuminate what is at the heart of the O’Reilly project:

“The Future is here, it is just not evenly distributed yet.” (William Gibson).

But Tim O’Reilly makes another point about the future when he speaks.  The future unfolds unexpectedly – so we must invent for an unknown future not a known future, or as Alex Steffen put it so well in his post, Why Our Bright Green Futures Will Be Weirder Than We Think, – “The world we need is one we’ve never yet seen.” The magic of attending an O”Reilly event is that it gives you a chance to work on this koan in interesting ways, and to take more responsibility for how things turn out.

Tim O’Reilly also urges that we think more deeply about what we are doing.  His keynote for Hadoop World , NYC, billed as, “The Business of Data” turned towards “The Consequences of Living in a World of Data.” The 900 strong crowd at Hadoop World was probably one of the most savvy crowds in the world about the business of data, so this was a nice turn.

Web 2.0 Expo with the theme, Platforms for Growth, was a deep dive into the business of innovation.  Tim O’Reilly’s keynote at Web 2.0 Expo,  “Thinking Hard About The Future” (or rather “thinking a little bit creatively or differently about the future)  – see video here, developed the call he made at Web 2.0 Expo 2008, to “work on stuff that matters,” into a Four Cylinder Engine for Innovation.   The first of the four cylinders in the firing order is, “Having Fun!” But,  at Maker Faire, Web 2.0 Expo, and Hadoop World I got an inside look at the workings of all four cylinders, and there is more to come, I am sure, as the other O’Reilly events unfold over the coming months including,  Web 2.0 Summit, Strata (a new O’Reilly conference on The Business of Data), and Where 2.0, 2011.

In a free webcast, last week (recording here), previewing Web 2.0 Summit, John Battelle and Tim O’Reilly discussed the Points of Control Map which is developing into a fun and useful tool to examine a very serious topic, “The Battle for the Internet Economy,” and how the “increasingly direct conflicts between its major players” could effect “people, government and the future of technology innovation.”   In my previous post, Platforms for Growth and Points of Control for Augmented Reality, I had a great conversation with Chris Arkenberg using this map as a springboard.  More on Points of Control later in this post.

The Four Cylinders of Innovation

click to enlarge

From Jet Ponies to Jet Packs: The First Cylinder of Innovation – “Have Fun”

The “maker” energy and its spirit of play, and the courage to create, hack, reinvent and re-purpose everything and anything, is a quintessential example of the first cylinder of innovation firing big.  Many “maker” projects also go on to fire on all four cylinders.   But the Maker forte definitely is in the first cylinder zone (and safety third as some of the rides, including Jet Ponies, warned).  The photo opening this post by Marc de Vinck – for more pics see here, is of Tim riding The Jet Ponies at Maker Faire which took  the New York Hall of Science by storm in late September – see The New York Times coverage here.  The ride was “built by the dastardly danger-hackers at the Madagascar Institute.“ See this wonderful interview with Hackett on his work to design “our specific jets from a patent that was filed in 1960s by a Mr. Lockwood, for Valveless Pulse Jets.” Hackett points out:

“Louder than god, glowing white-hot and looking like the trombone of the Apocalypse, pulse jets are also really shitty, inefficient engines,”

But, he adds:

“I have always wanted a jetpack, and one of the reasons I learned to build these things was to further that goal.”

This grand vision behind the Jet Ponies is a key to firing, The Second Cylinder of Innovation,  “Hey, we can change the world!”

But Jet Ponies, as a stepping stone to jet packs, also really struck a chord for me as I have been devoting a lot of time lately to the emerging Augmented Reality industry, a technology which was lumped in the same category of sci fi chimera as jet packs until very recently.

Data is the Gasoline

“The faces are coming from the sky.  The locations are coming from the sky.  All these apps depend on something, somewhere up.  And that, to me, was always the heart of Web 2.0.  And I am so delighted that people are finally getting it.  Because for a long time, people thought, ‘Oh, Web 2.0, it’s about lightweight advertising supported in a web start up.’  So I went, ‘No, no, no. It’s about the fact that we’re building these giant database subsystems in the sky that are going to drive applications.’  And now, of course, the same application is on your PC, it’s on your phone, it’s on you iPad.  And clearly, the applications are just sort of an interface to something that is being driven from the cloud, and that is fabulous.  That’s the difference.  People get it now.” (Tim O’Reilly, said this as part of a response to the first questioner at the Media Round table Web 2.0 Expo)

Answering questions about the importance of “Having Fun” to innovation doesn’t look quite as fun as riding Jet Ponies! Photo above from luca.sartoni’s Flickr stream

“ the data that is generated by the sensors and the applications that use that data is going to be where people are going to be innovative.” (Tim O’Reilly)

During the Media Round Table, I had a chance to ask Tim more about the role of bottom up innovation in a world where big data is the gasoline for increasingly sophisticated engines – platforms integrating machine to machine intelligence and real time analytics.

Tish Shute: You brought up Maker Faire in your keynote, and again now.  I was there, which not many people in the audience were  [not too many hands went up when Tim asked during his keynote].  But I think one of the things that struck me was the jet ponies – they were just earthshaking to stand near.  They made the ground tremble; they made the world shake.  Yet, most of your keynote, and most of what’s on our minds here, at Web 2.0 Expo, is extracting intelligence from the big data [in the sky], and algorithmic intelligences are the jet engines of the internet.  And of course, not to be forgotten, as we are here in New York City, where the trading markets are creating the air we breathe  [although we probably don't realize it until we lose our mortgage or something] and these algorithmic economies or “robot casinos” as Kevin Slavin put it, are all about speed – it’s not just real-time, issues of latency are  so critical that co-location is key to winning the game of the markets.  [Kevin Slavin brilliantly unpacks this in his talk, "Loitering on the Motherboard." For more in this see my conversation with Kevin Slavin below].

So my question is, who’s making the jet ponies for the algorithmic economies in the sky that you just described?   How can we make a play from the bottom up?  I always feel Ushahidi is one of the jet ponies of the data algorithmic space [because of their great work to bring human and machine intelligence together to solve problems in crisis situations].  But who do you think is doing exciting work and how can we ensure that this powerful world of data and algorithmic intelligences does not become hidden in a closed black box [only really accessible to elite players like the NYC trading markets]?

Tim O’Reilly: “Well, I think there’s certainly a lot of interesting things happening in, say, the financial services that a lot of, kind of, the Internet folks are kind of blind to.  I think that there are companies like Next Jump which are really good with data and good with algorithms. But kind of speaking specifically to the maker side of this, that whole sensor enabled world which is going to produce data is in its infancy.  What we have that I think is so powerful right now is we have the first portable sensor platform.  I said in my talk the other day, you know, your phone has ears, it has eyes, it has a sense of where it is.  And these are all available to application developers. You know, you can compare, say, Dodgeball to Foursquare, you can see how different… Dodgeball is Foursquare in the tele-type era.  Foursquare is now possible because there are so many more capabilities on the phone.

And I think that we are going to see a lot of other areas that are revolutionized by the sensors in the device.  It could well be that some of them will come explicitly out of the maker kind of projects, or it could just be that make is sort of a proxy for them.  So yeah, Arduino is this great maker sensor platform, but hey, here’s a consumer sensor platform [holding up phone]. Maybe we vaulted past the maker stage already and we just didn’t know it.

And that’s not entirely true, because Arduino is building a whole economy of special purpose devices.  But it feels a little bit like the days when people rolling their own PCs coexisted with the rise of Dell, who was a kid in his college dorm room who made his own PCs and sold them on the net, but figured out how to scale it pretty quickly and get good at it. But there were still a lot of garage shops, you know, ‘I’ll make a PC and sell it to you’ people for probably a decade before there was really a clue that that was a commodity industry.  In fact, I do think the sensor platforms are going to become a commodity industry.  And the data that is generated by the sensors and the applications that use that data is going to be where people are going to be innovative.”

The internet operating system is a data operating system and it is happening in real time (Tim O’Reilly)

click to enlarge the image above  – a slide from Mike Olsen’s  (CEO of Cloudera) keynote at Hadoop World

Not only do we have a portable sensor platform in our pockets  but developers also have powerful platforms and tools to make sense of data that fuel our apps.   Opensource Hadoop makes available, to anyone with some data munching chops, the power to work with giant unstructured databases and do the kind of real time analytics previously only available to giants like Google.  Big players like Yahoo, Facebook, and Twitter use Hadoop (Jonathon Gray from Facebook noted they add 10TB a day).   But, as this great roundup of Hadoop World points out, while Hadoop gets the press for handling petabytes of data , Mike Olsen (CEO of Cloudera) noted, the fastest growing area of users are working with clusters smaller than 10TB and over half of the Hadoop clusters were under 10TB in size.

Four Square: A Platform for Growth with an ecosytem built on top of data that exists in the real world

As an augmented reality enthusiast it is not hard to guess that one of my favorite platforms for growth is Four Square.  See Dennis Crowley’s keynote at Web 2.0 Expo here.  The Four Square API has been available to developers since November 2009,  and there are already a number of  interesting applications, and there will be many more to come.  The screen shot above is of geopollster – see the gallery of Four Square apps here.

@dens tweeted recently  “Politics + @Foursquare = @GeoPollster” http://geopollster.com <- I love love love that people are using 4SQ to think about election tools

As Kati London pointed out in her keynote, Four Square is the “kind of augmented reality that is aimed at shifting or changing a person’s social reality, e.g. the mayor badges in Four Square that change my relationship to the people and the place I am in, and augment engagement and reputation through socially driven consumer tie ins.” We are already see augmented reality developers beginning to work with the Four Square API – see here, Foursquare + Augmented Reality + Virtual Graffiti = ARstreets.

As augmented reality development tools mature, Four Square will, increasingly, become an important platform for creative AR developers interested in integrating the power of this platform for augmented engagement and reputation with “device aided augmented reality that can shift visual experiences of situated geolocal experiences.” With the Qualcomm vision based augmented reality SDK now available for download, and Wave in a Box soon? to be released, and an ARWave client working on Android (almost!), I have been exploring the Four Square API in my non existent spare time!!

The Four Square API also offers some interesting possibilities for exploring games that take the complex economy of Four Square – not personal data but aggregates of behavior, as their subject matter (for more on this see my conversation with Kevin Slavin later in this post and in an upcoming post).

I took this picture of Dennis at Where 2.0, 2009 at the beginning of Four Square’s phenomenal growth (they are at 4 million plus users now).

Pachube (Patch-Bay): a web service for storing and sharing sensor, energy and environmental data

Eighteen months ago, I interviewed Usman Haque (architect and director, Haque Design + Research) and founder of Pachube – see Pachube, Patching the Planet.   Usman pointed me to this wonderful evocative image from T.R. Oke’s book, “Boundary Layer Climates” (original photo source Prof. L. E. Mount’s The Climatic Physiology of the Pig).  “It’s the same piglets, in the same box, but on the right hand side the temperature has been increased. This small change in how the space is “programmed” has dramatically changed the way the ‘inhabitants’ relate to each other and how they relate to their space.”

The Challenge of Connecting people and environments.

At Web 2.0 Expo, I got the opportunity to talk with Usman Haque again.   Pachube, is becoming an established platform now, Usman explained.  They have a development team of eleven and robust back end.  And, they will now be spending some more time on the front end, including a redesign of the website, making “it a lot easier to widgetize the entire website so that you will be able to take almost any element and embed that into your own website.” And, as Usman mentioned in his presentation, they are working on an augmented reality interface, Porthole, for facilities management and, “as a consumer-oriented application that extends the universe of Pachube data into the context of AR – a ‘porthole’ into Pachube’s data environments..  Usman is also contributing to the AR standards discussion and on the program committee now for the W3C group on augmented reality.

Click to enlarge the image above from Chris Burman’s paper for the W3C, Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual”

Pachube, is sometimes described as the Facebook for Data or an analogy Usman prefers, a Twitter for Sensors.  At Web 2.0 Expo, I had an amazing opportunity to hear from Twitter and Facebook about their strategies as platforms for growth.  This gave me lots of fuel for questions about Pachube’s approach to developing their platform.  Simplicity was a theme that Facebook  and Twitter both affirmed as a key.  One of Pachube’s challenges will be to deliver ease of use, and the equivalent of Facebook’s “like” and  Twitter’s “follow” to gain mass appeal.

Here is a brief excerpt from my upcoming conversation with Usman:

Tish Shute: So as a platform you see Pachube as having more in common with Twitter – a Twitter for Sensors. In what ways is Pachube similar to Twitter?

Usman Haque: Well we are the Twitter of sensors, devices & machines in the sense that, really, the API that enables all this communication is important, much more so than the website itself. It is where, basically, most of the millions of our hits actually go, is to the backend. And we’ve now got dozens of applications built on top of the system, a little bit like Twitter’s applications; you know, all the apps are the important part.

But we are actually going to be doing some quite exciting things with API keys that we haven’t really spoken that much about in public. But we have come up with a pretty innovative solution to make almost every resource have granular privacy options on it, now discussed here.

At Hadoop World, Tim O’Reilly also raised some interesting broader questions that are very relevant to Pachube’s vision to “patch the planet”, e.g, the problem of digital identity in the age of sensors? (Smart phones already know their users by the way they walk!) And, “How should we think about privacy in a world where data can be triangulated?”

Usman talked about Pachube’s approach to both the technical aspects of how to build a massively scalable system, and the conceptual aspects of how people connect to each other, and what they might do with these new opportunities to connect environments and sensor data  (see my earlier talk with Usman, Pachube, Patching the Planet, for a detailed explanation of some of the concepts behind Pachube).

I look forward to posting this conversation. Pachube is growing, and Usman always goes beyond the familiar tropes of connecting human and machine intelligence.

2nd Cylinder of Innovation: “Hey Can We Change the World!”

The possibilities for reimagining of the role of data in healthcare produced some of the most powerful “Hey Can We Change the World” moments for me at both Web 2.0 Expo and Hadoop World.  The slide above is from Esther Dyson’s brilliant Ignite presentation, “What you can and can’t learn from your genes?” are here,   Tim O’Reilly also brought up the powerful role real time data analytics can play in improving healthcare in his Hadoop World Keynote.  Also see Alex Howard’s post, 10 Lessons for Gov 2.0 from Web 2.0 for some more great, “hey we can change the world moments” at Web 2.0 Expo.  The keynote from Lukas Biewald of CrowdFlower and Leila Chirayath Janah of Samasource (screen shot below) in particular, is a provocative exploration of the future of work in the new ecologies of human and machine intelligence.

Changing the World When Our Lives Are Increasingly Shaped by Forces Invisible To Us?

Click to enlarge

Mike Olsen, CEO of Cloudera, noted that “the largest area of data growth does not come from humans interacting with machines; rather, it’s from machines interacting with each other” (see here in Minor Technical Difficulties).   One of the most interesting presentations at Web 2.0 Expo was Kevin Slavin’s, “Loitering on the Motherboard,” which, as Tim O’Reilly pointed out in his keynote at Hadoop World, is a talk that raises all kinds of questions about a system where big players are gaming the data for their own ends.

Kevin Slavin, a founder of Area/Code, notes the operating system of our mortgage, life insurance, the operating system of currencies and gold is now governed by machine to machine intelligence and algorithimic economies outside of human cognitive processes.  The markets are now legible only to bots in an algorithmic arms race with bots surveilling bots, and throwing off false information in a bid for counter-surveillance.  He showed some slides of the eery but beautiful visualizations of traces of the trading bots created from the Nanex API.

The screenshot above is from the Nanex: Crop Circle of the Day – Quote Stuffing and Strange Sequences.  “The common theme with the charts shown on this page is they are all generated in code and are algorithmic. Some demonstrate bizarre price or size cycling, some demonstrate large burst of quotes in extremely short time frames and some will demonstrate both…” This one is a zoom of the NSDQ “Wild Thing.”  Wild price/size repeater from NSDQ running at 1,000 quotes per second, effecting the BBO along the way (I love the great names Nanex gives the different patterns and traces produced by the trading bots).

Nanex supplies a real-time data feed comprising trade and quote data for all US equity, option, and futures exchanges. They have archived this data since 2004 and have created and used numerous tools to “sift through the enormous dataset: approximately 2.5 trillion quotes and trades as of June 2010.” May 6th 2010 (day of the flash crash), had approximately 7.6 billion trade, quote, level 2, and depth records.

Kevin points out that our lives are being shaped by criteria invisible to us and the old hackneyed tropes of machine to machine intelligence such a robots reading HUDs in English are long worn out.  The latter point is, perhaps, something for us augmented reality geeks absorbed in ideas of “making the invisible visible” to chew on.

Changing a world shaped by forces that are, increasingly, invisible to us presents a huge challenge.

But I had the glimmer of a, “Hey Can We Change the World” moment, when I attended Kevin Slavin founder of Area/Code’s presentation and had a conversation with him after his talk.  Could games take these complex economies as their subject matter?  The economies of  Farmville and games like WoW are not opaque at all, and these are environments with complex economic behavior, “where you can actually have enough data to understand what it is” – “it’s not so much about personal data.  It’s more about, like, aggregate behaviors.” “Games that can really model those, and play with those, and take those as the subject the way that Monopoly takes Monopoly as a subject could be really interesting.” Kevin made many fascinating points – more to come on this topic.

Photo by James Duncan Davidson, of Kevin Slavin speaking at Web 2.0 Expo NY, 2010, from the O’Reilly Conferences Flickr stream

Here is the beginning of our conversation:

Talking With Kevin Slavin

Tish Shute: You began your talk today about visibility and where some of the algorithmic masters of disguise went to work, after they had solved the math behind stealth bombers.  I thought perhaps you were leading into ideas about a reverse surveillance society.

But you surprised me, as I felt you made visibility itself kind of a non-issue by the end of your presentation and that counter surveillance became basically a time and speed issue.  Now I am not sure quite how to imagine a counter-surveillance society, something I try to think about…

Kevin Slavin: Well, let’s see.  There’s a couple ways to think about it.  I think one point is just that when we talk about counter-surveillance, we usually locate that as something that comes from  the bottom up, something that comes from the population. Think about the way the plane spotters discovered the CIA black rendition flights.

I think in general, when people talk about counter surveillance, or sousveillance, they imagine it as an inversion of the traditional relationship between the people and the state.

But that’s what’s interesting. What’s happening now, is that there are forms of surveillance and counter-surveillance that are in play beyond any human perceptual horizons. These forms are at their most sophisticated in financial services, in the markets.

If you were a bot, and could read the market legibly (which humans cannot), what you would see, effectively, are bots that are surveilling bots. Then you have bots that are throwing off false information in a bid for counter-surveillance. Many of the bots are, themselves, surveilling other bots; each one of them is trying to figure out what all the other ones are going to do. In essence, it’s an algorithmic arms race, and game theory has become concrete, since the theories are code, the code is action, and the action affects, let’s say: your mortgage.

And so, basically what you have is you have this series of algorithms that are all looking to discern each other, while also trying to prevent themselves from being discerned. I think of the tunnels under the trenches in WWI, tunnels to surveil the trenches, and then, later, tunnels to surveil the tunnels. There’s a few examples of this kind of thing.  But It’s especially strange when it’s computer code, and at the magnitude we’re seeing today.

All of it, as noted in the talk, accounting for 70% of all the trades in the market. 70% of the market trades are never touched by human hands or even seen by human eyes; they don’t move through a conventional cognitive process.  And that’s why you get things like the Credit Suisse algorithm, it was buying, selling 200,000 shares of stocks to itself over and over and over again. It was a bug and it slowed the market to a crawl.

Credit Suisse was fined, in essence, for failing to control an algorithm. Maybe that’s the first time an algorithm was treated like a human, in a way. As if the algorithm broke the law, and Credit Suisse was responsible for letting it do so. For me, that feels like a threshold event.

It’s not that humans never made mistakes when trading on the market. But when algorithms err, they err with magnitude.

The idea that we now have bugs in the United States market economy is really worth looking at.  If Apple can’t keep code bugs from the most simple iPhone apps in a closed and regulated ecosystem, I’m pretty certain we’ll have a lot more Credit Suisse type bugs in the future.

And that will be pretty interesting. There will be viruses, and the operating system they will operate on will be the operating system of the United States. The operating system of your pension, your house, your life insurance. The operating system of currencies and gold.

Tish Shute: I was hard-pressed by the end of your talk to think of like, “Well, what would be the equivalent of, sort of a people’s uprising to create a better fairer society in this kind of world where, really, the things that affect the key aspects of lives most are going on beyond human perception at an algorithmic level?”  But you made a pretty radical suggestion at the end…

Kevin Slavin: Well I think increasingly the markets have become delaminated from anything meaningful. First from goods, then from fundamentals, and now finally from homo sapiens. So that’s hard to fight.

It’s the race towards abstraction that makes it impossible to simply “resist.” The latest version in the long series of fiscal catastrophes was based on Wall Street finding goods that could be rolled up and sold with false valuations, but goods that would take a long time to fail. Mortgages are handy like that. It’s the tradition of extending the abstraction as long as possible, until finally the bill arrives and the banks fail. I don’t know if that’s something to rise up against or not. It’s like a rally against evil.

But really, I think the point is that it won’t be the people that rise up. It will be the financial services themselves that rise up. They’ll just detach completely.

That was harder to do with cotton or with wheat, with simple futures; they keep financial services tied to the ground.  So what we’re doing is creating increasingly complex financial instruments that are further and further removed from anything you can touch.  Like the way a mortgage is abstract. But, of course, the bottom line is that at the end of that mortgage lies someone’s home.

It’s said that Wall Street is now moving onto life insurance, because that’s going to take even longer to fail.  They’re doing the exact same thing. The word is that they are rolling up CDOs made out of crap life insurance policies, same way they rolled them up with crap mortgages a few years ago.

And those will probably take, I don’t know, 15 or 20 years to unwrap and unravel.  But what you see in the meantime, is that they are looking for things that are increasingly abstract, intangible, removed as far as possible from the experience of everyday life.

So maybe this is good. Maybe that’s financial services rising up. Lifting off. I think best case scenario now is that they actually leave humans alone altogether.  That, someday, they are just trading, effectively, completely arbitrary goods, the stocks could be anything at all, maybe for crops that no longer exist, and I’m just saying that then these bots would no longer affect what we do and what we are, it would just be a robot casino, an invisible paradise in the air.

People are the platform: How Games Can Be Engines of Innovation in Our Lives

See the video of Games that Know Where We Live here (screen shot above)

Kati London, Senior Producer, Area/Code, in her keynote showed how games that know where we live can shift players perspectives – from device aided augmented reality that can shift visual experiences of situated geolocal experiences to a kind of augmented reality that is aimed at shifting or changing a person’s social reality, e.g. the mayor badges in Four Square that change my relationship to the people and the place I am in, and augment engagement and reputation through socially driven consumer tie ins.

Area/Code has recently developed two games for the Knight Foundation that take people as the platform.  Macon Money, uses very simple games dynamics (for more see the video of Kati’s keynote) in a game designed to help “Knight’s continuing efforts to support revitalizing Macon and creating a vibrant college town.”

The other game that Area/Code has designed with the support of the Knight Foundation  is for the Biloxi and Gulf Coast community, a game called Battlestorm.  “The game’s purpose is to increase awareness about natural disasters and change the way people prepare for them.”

3rd Cylinder of Innovation: Build products, business models and entire industries.

Glympse – real-time, private location tracking

Julianne Pepitone, Yahoo! Finance, nailed the essence of Web 2.0 Expo, NYC, this year in her post, Web 2.0 Expo startups are big on neighborhoods, storytelling.  She writes:

“At the Web 2.0 Expo in New York City this week, executives from big sites like Facebook, Twitter and Pandora all spoke about industry trends. But the showcase of 27 startup tech companies stole the show.”

Listen carefully to Tim O’Reilly and Fred Wilson, Union Square Ventures, question their picks from the startup showcase at Web 2.0 Expo.  Also see this video of Fred and Tim discussing their conversations with all the start ups.  This  is one of the clearest public windows onto both how to present your company to VC, and how to figure out what are the most important questions for you as an entrepreneur  building a business in a world of data.

Glympse successfully pitches their “jet pony” strategy for a location based business, and is Fred’s pick.  They hold up well under pressure and answer Tim and Fred’s hard questions about how their start up will not get overtaken by an encumbent player with resources and market share before they can gain traction.   food52 responds to Tim’s probing about their strategy for business data analytics that he points out are vital if they want to survive with the small margins of ecommerce.  There is a list of all the participants in the start up showcase in Brady’s post here. hour.ly was the audience pick.

Shazam for Faces!

My favorite start up was a biometric service doing face, iris, and finger print matching, Tactical Information Systems.

Tim and Fred also liked them, and they have an interesting discussion about the merits or not of approaching your platform through a narrow first application as Tactical Information Systems are with WanderID -  an application to help identifying lost Alzheimer patients.  As Fred pointed out, they are potentially the Shazam for faces, so why start so small?

I  had asked TIS the same question when I met them in the “speed dating” session.  This is just their first toe in the water as they are a two person company at the moment. Their vision for their platform is big.  Mary Haskett and Dr Alex Kilpatrick, the founders of this quintessential jet pony for the algorithmic economies in the sky, are not only a partnership with the credentials to do a  Shazam for faces – see their bios here, they are the people I would want to be running a Shazam for faces!  They really get the consequences of living in a world of data – check out Dr Kilpatrick’s absolute killer Ignite talk, “Defeating Big Brother.” (screenshot below)

How Can Augmented Reality Add Value to the Real Time Internet/Data Operating System?

planefinder.net – an augmented reality app that lets you find information about planes by pointing your phone at the sky, “including flight number, aircraft registration, speed, altitude and how far away it is” (via MacLife).

The new opportunities in the algorithmic economies in the sky were center stage at Web 2.0 Expo and there are some interesting AR apps for the real time internet/data operating system emerging, like planefinder.net.  But Augmented Reality was still pretty low profile at Web 2.0 Expo (except that NVidia augmented reality demo attracted a lot of attention at the sponsors expo).  However, everyone working in the emerging industry of AR should recognize that apps big on “neighborhoods and story telling” are heading right up the AR street, and that platforms like Four Square and Pachube present enormous opportunity to explore the possibilities of AR.  And if augmented reality enthusiasts are not already paying attention to real time data analytics, and Hadoop, they should be (see this post for an excellent round up on Hadoop World).

At Hadoop World, Tim O’Reilly referenced the great tagline from the  IBM commercial:

“Would you be willing to cross the street — blindfolded — on data that was five minutes old? Five hours? Five days?”

As I have noted in several earlier posts – see here and here for starters,  we are just seeing the tools  for developing near field, vision based, mobile, social AR become widely available to developers, so there should be a new level of AR apps emerging through 2011.  There is a wonderful discussion in the comments of this post by Mac Slocum, “How Augmented Reality Apps Can Catch On,” between Mac, Raimo one of the founders of Layar, and Chris Arkenberg on what constitutes a platform for growth for augmented reality.

Mac’s post, the comments and Chris Arkenberg’s post on the latest edition of the Gartner Hype Cycle, that rather curiously placed Augmented reality almost at the peak of inflated expectations. really got me excited about exploring an idea I have been thinking about for a while, which is to get the AR community to discuss the Points of Control map.    See my discussion with Chris Arkenberg here, Platforms for Growth and Points of Control for Augmented Reality. The recording of  John Battelle’s and Tim O’Reilly’s webcast on Points of Control is posted here.

The interactive Points of Control map is an amazing tool to think with! Check it out in movements, territory and movements, acquisition mode.  There is a competition for the most interesting comment and most interesting acquisition suggestion.  The prize is a ticket to Web 2.0 Summit!

What is the Future of Social?

The recent “defection” from Google to Facebook – see Lars Rasmussen, Father Of Google Maps And Google Wave, Heads To Facebook,  is as MG Siegler of TechCrunch points out, “the biggest one since Chrome OS lead Matthew Papakipos made the same jump in June” (TechCrunch also notes “current Facebook CTO Bret Taylor was heavily involved in the launch of Google Maps”).

These moves have drawn my particular attention as did Bret Taylor’s response in his conversation with Brady Forrest at Web 2.0 Expo to Brady’s question, “How soon until we get the Facebook firehose?”

If you have been reading Ugotrade you already know how important I think an open, distributed, standard for real-time communications such as the very innovative Wave Federation Protocol could be for AR development  -  see ARWave and my presentation at MoMo13, Amsterdam last year, The Next Wave of AR: Mobile Social Interaction Right Here, Right Now!

The anticipated release of  Wave in a Box, has raised hopes in the developer community that  WFP will soon become easier to work with, and hopefully more widely adopted.  Like many others, I wonder what will happen to Wave in a Box now?

But the innovation of Wave is deep and broad (and as many have pointed out hugely ambitious).  Perhaps the boldest attempt yet to innovate both at the low level of architecture (where Google is so powerful) and at the high level of the Mark Zuckerberg, “big idea,” which as Tim O’Reilly notes is, “What is the future of social?” MG Siegler noted Facebook Groups Is Sort Of Like Google Wave For Human Beings.

But I deeply hope that the open, distributed standard part of the Wave big idea is not lost in the mix here.

Fourth Cylinder of Innovation: Keep the Ecosystem Going, Create More Value than You Capture

The Points of Control map is interactive, so please click here or on the image above for the full experience.

Tim O’Reilly points out that there is a worrisome dark side to the Points of Control Map – see Tim’s keynote here.  To paraphrase some or his points:

There are companies on the map that are forgetting to think about creating a sustainable ecosystem.  Rather than growing the pie, they are trying to divide up the pie and that threatens to cause the fourth cylinder of innovation to misfire.  This fourth cylinder is essential to the ecosystem.

Tim O’Reilly looks back to the lessons of the personal computing industry which was incredibly vital and creative, and lots of people made money until a couple of big players “sucked all the air out of the ecosystem” and innovation had to go elsewhere.

The Power of Platforms is to create value not just for your company but for other people.   Create value for yourself by creating value for other people.  Tim O’Reilly used the wonderful example of  Henry Ford inventing the weekend so that there would be enough people with time and money to buy his mass produced cars.  Think about building the ecosystem that will support the future your are going to build.  Grow the pie rather than cut up the pie.  This will be the vital fourth cylinder of innovation in a Web Squared world.

Tim O’Reilly has long proposed that  Web 2.0 is all about harnessing collective intelligence,  But as Gartner predicts, “By year end 2012, physical sensors will create 20 percent of non-video internet traffic.” Yet another previously unevenly distributed future is going mainstream, and if you haven’t read it already, now is the time to read this paper by Tim O’Reilly and John Batelle, Web Squared: Web 2.0 Five Years On.

The Consequences of Living in a World of Data

To bring this very long post to a close!  Here are just a few of the key questions re The Consequences of Living in a World of Data that Tim O’Reilly raised during his keynote for Hadoop World:

“How would we solve the problem of digital identity in the age of sensors? (Our smart phones are able to know their users by the way they walk – their gait!)

“How will we input data when our devices are smart enough to listen on their own?”

“How should we think about privacy in a world where data can be triangulated?”

“We are moving to a world in which every device generates useful data, in which every action creates information shadows on the net.”

“Shouldn’t we regulate the misuse of data rather than the possession of it?”

“How do we avoid a data arms race?”

“Create more value than you capture.”

The Points of Control map is interactive, so please click here or on the image above for the full experience.

Today at 4pm EST, 1pm PDT John Battelle and Tim O’Reilly will discuss the Points of Control map and The Battle for the Internet Economy in a Free Webcast:

“More than any time in the history of the Web, incumbents in the network economy are consolidating their power and staking new claims to key points of control. It’s clear that the internet industry has moved into a battle to dominate the Internet Economy.

John Battelle and Tim O’Reilly will debate and discuss these shifting points of control as the board becomes increasingly crowded. They’ll map critical inflection points and identify key players who are clashing to control services and infrastructure as they attempt to expand their territories. They’ll also explore the effect these chokepoints could have on people, government, and the future of technology innovation.”

I’ve been wanting to start a discussion on the  Points of Control map in the Augmented Reality community for a while now, and Chris’ recent post on the latest edition of the Gartner Hype Cycle, “Is AR Ready for the Trough of Disillusionment?” and this post by Mac Slocum, “How Augmented Reality Apps Can Catch On,” and the conversation in the comments between Mac, Raimo (one of the founders of Layar), and Chris, all prompted me to get a conversation started…(see below for all that followed!).  Chris put me on the hot seat back in June when he did this very generous interview with me on Boing Boing, so it was time to turn the tables.

Tim O’Reilly, in his  keynote for Web 2.0 Expo, pointed out there is both a fun and a dark side to the Points of Control map.  There are companies on this map, he noted, that rather than “growing the pie,” are trying to divide up the pie, and they are forgetting to think about creating a sustainable ecosystem. I expect the conversation between Tim O’Reilly and John Battelle to dig deep into this Battle for the Internet Economy.  If, like me, you have another engagement at the time of the webcast, you can register on the site to receive the recording.

AR is still too young to figure in the battles of the giants, but there will be a lot to be learned from this conversation.  And, The Points of Control map is good to think with from the POV of AR in many ways.  As Chris Arkenberg observed:

“When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?”   You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development.  And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.”

Chris Arkenberg is a technology professional with a focus on product strategy & development, specializing in 3D, augmented reality, ubicomp and the social web. He uses research, scenario planning, and foresight methodologies to help organizations anticipate change and adopt a resilient and forward-looking posture in the face of unprecedented uncertainty. His personal work is collected at urbeingrecorded, and his professional profile is here.

He is also one of the founder/organizers of AR DevCamp which is currently scheduled for Dec. 4th (somewhere in SF or The Valley!)  Chris said, “No further details atm (still trying to find a venue and get sponsors) but please direct people to http://ardevcamp.org for upcoming information.”

Talking with Chris Arkenberg

Tish Shute: I know some people thought the positioning of AR by Gartner near the peak of the hype cycle was misguided, and based on a very narrow understanding of AR as used in marketing apps. But reading your post I thought you made a lot of good points.

Chris Arkenberg: It’s tracking hype, right? It’s not necessarily tracking the growth of the technologies or their maturation so much as it’s tracking the general attention level. And what’s interesting to me is that tends to affect the amount of money that goes into those technologies.

Tish Shute: I was particularly interested in your post because I have been writing a post about two recent O’Reilly events in NYC, Maker Faire, Web 2.0 Expo, and then Hadoop World, where Tim gave a very interesting 45 minute keynote.   AR was pretty low profile at all three events.   But the NVidia augmented reality demo attracted a lot of attention at the sponsors expo, and Usman Haque, Founder of Pachube announced in his presentation, they are working on an augmented reality interface for Pachube called Porthole, its designed for facilities management and, “as a consumer-oriented application that extends the universe of Pachube data into the context of AR – a ‘porthole’ into Pachube’s data environments.. “  Usman also mentioned, when I talked to him, that he is contributing to the AR standards discussion and on the program committee now for the W3C group on augmented reality.  For more on this standards discussion and the Pachube AR interface, see Chris Burman’s paper for the W3C, Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual.”

I think pioneers in the augmented reality commmunity should pay attention to these wider conversations about the Battle for the Internet Economy, and the exploration of the  “Platforms for Growth” theme at Web 2.0 Expo is very important- this is a course also a nudge to read my upcoming post on these O’Reilly events!

Also I have another project I have been chewing on that I would like to talk to you about.   I want to start an AR conversation about the wonderful Points of Control map produce for Web 2.0 summit by John Battelle. [ Note there will be, "Battle for the Internet Economy" free Web2Summit webcast w/ @johnbattelle & @timoreilly Wed 10/27 at 1pm PT http://bit.ly/b46cmb #w2s]

Up to this point, understandably given the immaturity of the technology, AR has little role in the “Battle for the Internet Economy.”  But this doesn’t mean that the map isn’t good for AR visionaries, enthusiasts, entrepreneurs, and developers to think with.   And both you and Tim have pointed out the potential for AR to leverage the giant data subsystems in the sky.   I have to say the positioning of Cloud Computing on the brink of heading down into the trough of disillusionment in this recent rendition of the Gartner Hype Cycle seems ridiculous!

Cloud Computing is already ubiquitous hardly seems credible that it is headed for a trough of disillusionment!

Chris Arkenberg: Yeah, it’s ubiquitous so why even talk about it when it’s your fundamental infrastructure?

Tish Shute: Yeah and I seriously doubt it is imminently headed for a trough of disillusionment….and this brings me back to the Points of Control Map which as John Batelle points out, “aims to identify key players who are battling to control the services and infrastructure of a websquared world” in which the “Web and the world intertwine through mobile and sensor platforms.”  This instrumented world, of course, creates a great deal of opportunity for augmented reality. Have you seen that, that points of control map?

Chris Arkenberg: I think I have, actually.

Tish Shute: There has been much debate about how this intertwining of the web and the world will play out in augmented reality.   Chris Burman points out in his position paper for W3C,  Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual”, that “trying to draw parallels between a browser based web and the possibilities of AR may solve issues of information distribution in the short-term,”  but it must not have a limiting effect in the long-term.   But now we at least have one web standards-based browser for AR thanks to the work of Blair MacIntyre and the Georgia Tech team.  But I think the discussion in the comments of Mac Slocum’s recent post, “How Augmented Reality Apps Can Catch On” is an interesting starting point from which to think about platforms of growth for AR.  I am not sure if I am stretching his meaning but I think Raimo, Layar, is suggesting that what the Point of Control map call the the Plains of Media content is very important to the growth of the fledgling AR industry right now. And I would agree with this, and add that the neighboring terrain of gaming levels will be pretty key as one of my other favorite AR start ups Ogmento hopes to reveal in the near future! But what do you think was most important in this brief but pithy dialogue between you Raimo and Mac?

[The screenshot above is a teaser video the Gary Hayes from MUVEdesign for his upcoming (2011 release date), game called Time Treasure.  See Gary's blog for more and Gary's  post from over a year ago on AR Business models.  Thomas K. Carpenter, on Games Alfresco notes, "I think this is a terrific idea and I find it interesting he’s planning this on a tablet rather than a smartphone."

Chris Arkenberg: The way I took it…And to give a little bit of context, I came from sort of this apprehension of augmented reality as an expression of the existing Internet. So as sort of a visualization layer that allows you to kind of draw out data, and then, with all the affordances of being able to anchor it to real world things.

And my own sort of path has led me to want to really try to understand that and refine it, particularly with respect to the sort of Internet of things and the smarter planet idea of just having embedded systems everywhere. And specifically, what is the value-add for augmented reality as a visualization layer of an instrumented world?

And so that’s caused me to be a bit biased towards that side of AR. And the way I took Raimo’s comment was that he was saying that, “You know, really what we’re interested in is media.” That he was effectively saying that AR for them is really just about that space between the screen and the the world, or between your eyes and the world, and what you can do there.

Certainly I had considered it in the past, but I hadn’t really focused on it or assumed that it was a priority as a business model. And so he kind of reminded me that, actually, there’s a lot of entertainment applications. There’s a lot of, obviously, advertising and marketing applications.
And so I felt that I was being a little narrow in my focus…

Tish Shute: Yes this comes to the heart of what I am interested in about the role AR can play in opening up new relationships to the world of data that we live, not just making it more accessible and useful to us when and where we need it, but AR as a road to reimagining  it..

Have you seen any interesting work yet to explore these great data economies in the cloud through AR. I mean can you think of any others - there is planefinder.net but others?

Chris Arkenberg: I’ve seen a few just sort of skunk works type applications that people have been playing around with, again, to try and reveal things. One of them was similar to the aircraft, but it was more for military use and being able to identify things of interest in the sky. I’ve seen a couple other for navigation, so being able to identify mountain peaks on a visual plane, for example, but this isn’t so much about revealing an instrumented world.

Tish Shute: Yeah, I think that was from the Imagination right? I know that’s an interesting one. Usman at Web 2.0 Expo, in his presentation, mentioned the work Pachube is doing on an Augmented Reality interface. I interviewed Usman again as my last long interview with him was nearly 18 months ago now and Pachube is well on the way to becoming the Facebook of Data or the analogy that Usman prefers - the Twitter of sensors!

Chris Arkenberg: Hmm, interesting.

Tish Shute: And to go back to your comments on Augmented Reality not getting caught in some of the traps that have made virtual worlds lose relevancy I think that is vital that AR developers understand the strategic possibilities of key points of control in the internet economy because the isolation and Balkanization of virtual worlds were certainly a factor in their rapid slide into the trough of disillusionment - although many would argue that a fundamental flaw in the kind of virtual experience that Second Life and other virtual worlds constructed was really the fatal flaw (see James Turner's interview with Kevin Slavin Reality has a gaming layer).

But Second Life’s isolation from the other great network economies of the internet was certainly a limiting factor.

Chris Arkenberg: And that’s been exactly my sense, and I’ve, over the years, tried to encourage development in that direction for virtual worlds. I did work, through Adobe, to help develop Atmosphere 3D back in the the early 2000’s. And we did a lot of work to try and understand the marketplace and the specific value-add of doing things in 3D over 2D.

And this is kind of why I keep referring back to VR and VW’s with respect to augmented reality, is that with immersive worlds, there was this idea…there was this big rush. Everybody was so excited about it. It was obviously the next cool thing. And everybody wanted to try to do everything in it. You could do your shopping in virtual worlds. You could have meetings in virtual worlds.

Tish Shute: and shopping, yes ..that didn't work out so well!

Chris Arkenberg: And everybody was very excited in developing these things. And what it really came down it is, “Yeah, you can, but it’s actually a lot better to do those things on a flat plane or in person.” Meeting Place, WebEx, TelePresence - those tools generally do a much better job at facilitating TelePresence meetings than a virtual world does. The same with TelePresent Education. There are only very specific things that both VR and AR are really good at.

And that’s where I find myself with augmented reality right now, trying to really pick through that and critically look at which uses are really appropriate for an AR overlay. And again, I think that’s why the hype cycle is important, because it reflects back this desire that AR is going to be the next big thing - the be-all, end-all of interacting with data in the cloud - and forces us all to take a critical look at why we should do things in AR instead of on a screen.

AR is not going to work well for most things but it’s going to be very good for certain uses. Right now I’m very keen at trying to understand what those things might be.

Tish Shute: I had this wonderful conversation (more in an upcoming post) with Kevin Slavin one of the founders of Area/Code at Web 2.0 Expo and I think some of what he describes about the data brokerages of High Frequency trading have some interesting implications for AR’s role, say, in ubiquitous computing. The trading markets are now pretty much dominated by machine to machine intelligence; machine to machine brokerages. They are basically game economies on the scale that we can barely wrap our heads around where the speed that bots and algo traders can access the network is the key. We really have no clue what is going on until we lose our house…

Kevin was also interviewed by James Turner on O’Reilly Radar. He talked about how much of the interesting work in location based mobile social apps is defined in opposition to the model of Second Life. He also talked to me about how we are seeing “first life” take on the qualities of “second life.” What goes on the trading floor is largely a performance secondary to a more important world of machine intelligence with giant co-located servers and bots fighting for trading advantages measured in fractions of seconds.

He pointed out how we draw on all these tropes from sci-fi movies, these HUDs based on ideas of machine intelligence where the robot talks to the other robot in English through an English HUD!  Many of our current visual tropes for AR are perhaps just as inadequate for the kind of data driven world we live in.

Of course, when you are thinking of having fun with dinosaurs, or illustrated books, or whatever, this is not, perhaps, an issue.  But if you are thinking of augmented reality interfaces as being important in a battle for network economy, and platforms for growth,  how this new interface helps us live better in a world of data is an important issue.

Chris Arkenberg: Now, does that indicate that the UI just needs more overhaul and innovation, or more that the visual interface for those experiences shouldn’t really leave the screen? It shouldn’t move on to the view plane?

Tish Shute: Yes we have a few concept videos that try and explore this ..

Chris Arkenberg: Well, and I think this will happen at the level of human-computer interface. I mean that’s always been its role, in making coherent the sort of machine mind, for lack of a better term, making it coherent to the human mind. So I mean there is a lot of this sort of machine intelligence, the semantic Web 3.0 revolution, where it really is about enabling machines, and agents, and bots to understand the content that we’re feeding them.

But at the end of the day, they, for now, need to be providing value to us human operators. So there’s always going to be a role for human-computer interface and user experience design to make this stuff meaningful.

I mean, if you look at the revolution in visualization & data viz, this is of incredible value because it takes a tremendous amount of data and collates it into a glanceable graphic that you can look at and immediately comprehend massive amounts of data because it’s delivered in a handy, visual way.

So I see that as a fascinating design challenge, how the user experience of the data world can be translated into meaningful human interaction.

Tish Shute: Yeah. And when we see Stamen Design pursuing a big idea in AR, that’s when we might start to rock and roll, right?

Chris Arkenberg: Yeah. In my article, I sort of jokingly suggested that Apple will create the iShades. But, they’ve got the track record of being way ahead of the curve and delivering the future in very bold forms.

Tish Shute: A key part for the battle for the network economy is to bring the complexity of data into the human realm in a way that increases human agency. Kevin suggests that the giant robot casinos of markets should actually lift off into total abstractions as theses machine-driven trades get back into the human realm in ways that are so damaging to our lives - a lost house or job! The notion of a counterveillence society where people have more agency over the important aspects of their lives, health, housing, job (which I discussed with Kevin - interview upcoming) has gotten pretty tricky!

But I think we will begin to see AR eyewear for specific applications (gaming and industrial) get more common fairly soon - possibly as smart phone accessories.

And it is clear that AR is going to be, increasingly,  a part of our entertainment smorgesborg in coming months. Itouch has a camera (although lower resolution), Nintendo’s are AR-ready and many aspects of the AR vision of hands-free spatial interfaces will go mainstream through Natal.

But we are yet to see an app/platform emerge for mobile. Social AR games that turn every bar and cafe and ultimately the whole city into a gaming venue  -although I think Ogmento and MUVE aim to lead the way here! Will an AR company achieve Zynga level success by using the Foursquare, for example?

My feeling is that the lesson of Zynga is pretty important for mobile social AR games. Could Flash social gaming have taken off without Facebook?

Chris Arkenberg: And that’s the real driver. And again, as you mentioned with Second Life, and this was exactly my own sense, is that they stuck to the closed garden model and didn’t get the power of social and collaboration. They attempted to add some of those affordances within the world, but, you know, ultimately most people aren’t in virtual worlds, and most people aren’t using augmented reality. So leveraging the really predominate platforms like Twitter and Facebook and Foursquare, being able to leverage those affordances, that connectivity, into a platform like augmented reality, I think, is really critical. Because again, you get nothing unless you have the masses, unless you have people present.

Tish Shute: In AR research there is a long history of the notion of powerful AR-dedicated devices, but smart phones and tablets are good enough,  and can launch augmented reality into the heart of the internet economy. I think  the elusive AR eyewear will come to us initially as a smart phone accessory for specific apps.  But, for the moment, most AR apps make little attempt to play in the wider internet economy.

Chris Arkenberg: And I think it’s actually much lower hanging fruit, really, to do gaming, marketing, transmedia. Because then you don’t really care about the cloud, or maybe you only really care about a little part of it that your gaming property is addressing. Then it becomes much more about entertainment, and much more about persuasion, and sensationalism. And if you’ve got dancing dinosaurs on your street, great! It’s entertaining, it’s cool, it’s new. That stuff is fairly straightforward.

I keep coming back to this idea of, you know, the instrumented city. What sort of data trails do you get out of a fully instrumented city? So maybe you get traffic patterns, maybe you get geo-local movements of masses, maybe you get energy usage, that sort of thing, all the, sort of heat maps you can generate from a city. But then what good does it do to be able to have that on an augmented reality layer versus just looking at it on a mobile device or looking at it on your laptop?

Tish Shute: Of course the use cases for “magic lens” AR are different from the kind of hands free, 360 view with tightly registered media, that a full vision of AR has always promised. The 360 view is quite a different metaphor from the web and mobile rectangular screens.

Chris Arkenberg: Yes, yes.

Tish Shute: Did you see that great parody of Michael Jackson's “Beat It” with the iPads versus the iPhones, right?

Chris Arkenberg: Oh, really?

Tish Shute: I tweeted it cos i thought it was quite funny and a little close to the bone!
[laughter]

“ur wanna an ipatch 2 b the new fad?” #AR gets cameo in Twitter, iPads & iPhone’s Michael Jackson-Inspired Parody via @mashable

It is hard to get away from the importance of eyewear when discussing AR!

Chris Arkenberg: Yes, so the hardware, to me, is a big stumbling point right now, or it’s a large gating factor, I think, for realizing what an augmented reality vision could really be like. That it really does need to be heads up. This holding the phone up in front of you is fun to demonstrate that it’s possible, and it’s valuable in some ways…

Tish Shute: And it’s particularly nice in some applications like the planes app, the Acrossair subway app where you hold the phone down and get the arrow, right?

Chris Arkenberg: Yeah, the way-finding stuff I think is really valuable…

Tish Shute: Sixth Sense really caught people’s imagination because it managed to deliver the gesture interface with cheap hardware, even if projection has limited uses (no brightly lit spaces or privacy for example!).

The other important and as yet unrealized part of the AR dream is real-time communications. Many interesting uses cases would require this. As you know that is my chief excitement, along with federation, in the Google Wave Servers for (which should soon be released at Wave in a Box) for ARWave.

Chris Arkenberg: Well my sense of Wave is that it was a ChromeOS protocol that they instantiated, or that they exhibited in the public deployment of Google Wave. That that was a proof of their sort of low level architectural solution. Because, you know, they’ve been rumored to be working on this cloud OS for some time. And so my sense is that Wave is actually one of their core components of that cloud OS, and that it just happened to incarnate for the public in a test run as Google Wave.

Tish Shute: I do hope that Wave In the Box will lower the barriers to entry to people experimenting with this technology. The FedOne server was just way too hard for most people to take the time to set up. Of course, it is the brilliance of the Wave Operational Transform work that also poses problems in terms of ease of use. But Wave Federation Protocol is pretty innovative. And could even play an important role in a real time communications for AR eyewear connected to smartphones. The challenges that Wave takes on re real-time communications, federation, permissions and filters are pretty important ones for AR…

Chris Arkenberg: Especially when you’re trying to federate a lot of permissions and filter a lot of data, which all of that gets even more important when you have a visual layer between you and the real world.

Tish Shute: You got it. Yeah!

Chris Arkenberg: I think that’s really valuable real estate, both for third parties that want to get access to your eyes, as well as for you, as the user, who still needs to navigate through the phenomenal world and not be occluded by massive amounts of overhead data.

Tish Shute: Yes, I am sure Google has big plans for the next level of cloud computing and Wave looks at some key challenges. I suppose federation poses some key business problems. I think it was Michael Jones who said to me that it was a bit like socialism in that you have to be willing to give something up for the greater good.

Perhaps federation does not present enough appeal because of its challenges re business models?

Chris Arkenberg: Well, I wonder. I mean there’s got to be some value for their ad platform as ads are moving more towards this personalized experience. Advertising is becoming less of a shotgun blast and more of a very precise, surgical strike. So being able to track user data to such a fine degree to mobilize the appropriate ads around them wherever they are, on any platform, is certainly very valuable to Google and their ad ecology.

Tish Shute: Many people have high hopes that HTML 5 by lowering the barrier of entry for  browser style AR could also pave the way for some interesting AR work..

Chris Arkenberg: Well, as much as I would hope that all the different players are going to come together and establish some shared set of standards, really, what’s happening is it’s a rush to the finish line to be the first…to get the most penetration in the marketplace so that Layar, for example, can say, “It’s official. We’re the platform.” And then the consolidation that will follow, where the Googles and the other big players like Qualcomm say, “OK, it’s mature enough. We’ll start buying up all the smaller companies.”

And that’s where the real challenge is right now is that there are no standards. It’s such an immature technology that you have a lot of different players trying to establish the ground rules. And again, this is one of the challenges that faced public virtual worlds, is that you had a lot of different virtual worlds that weren’t talking to each other in any particular way, and that they each had their own development platform. And so you end up with a very fractured ecosystem or set of competing ecosystems, which is kind of what’s happening with AR right now, where a developer has to choose between a number of different new platforms or hedge by deploying across multiple platforms. Basically, the web browser wars are set to be recapitulated by the AR browsers.

Among them, Layar and Metaio seem to be getting the most traction. But there’s still not a really strong case for a unified development ecosystem to emerge.

Tish Shute: So a discussion of ecosystem development brings us back to the Points of Control Map I think. So what do you see as key points of interest for AR developers to watch in the Points of Control Map? And where do you want to sort of put your bets, right? We are still really waiting for mobile social AR to emerge into the mainstream.

Chris Arkenberg: Yes. And that’s primarily the shortcoming of the hardware itself, but also of the accuracy of current GPS technology. That’s another kind of gating factor, because again, AR wants to be able to express the data within a distinct place or object.

So in a lot of ways, other than kind of what we’ve allowed for the broader entertainment purposes, for AR to really work, there needs to be more resolution in GPS location. So for it to be truly locative…because it’s OK to tell Foursquare that you’re in Bar X. But if you want to be able to draw data directly on a wall within that bar, or do advertising over the marquee on the front, you need more factors to accurately register those images on a discreet location. So that’s another, sort of, aspect of the immaturity of AR, is that it’s still very hard to register things on discreet locations without employing a number of diverse triangulation methods.

Tish Shute: Right. The mobile AR games we see at the moment are really just faking a relationship to the physical world unless they rely on markers or some limited form of natural feature recognition which is really just a more sophisticated form of markers. But the Qualcomm SDK does offer some opportunities to tie AR media to the world more tightly as does the Metaio SDK. But in terms of a mobile social AR game that could be like the Cape of Zynga to FourSquare in Location Basin [see the Points of Control map]… We haven’t seen anything close yet.

AR should be able to bring the check-in mode to any object in our environment.

Chris Arkenberg: Yes, yes. And that’s actually one of the early interests I had in the notion of social augmented reality. I wanted a way to tag my community with invisible annotations that only certain people could read, and found pretty quickly that that’s very difficult to do. I mean you can kind of do some regional tagging, like on a beach, for example, but if you wanted to tag the bench that was on the cliff above the beach, it’s very difficult to do that using strictly locative reckoning.

There’s all sorts of really cool social engagement that can be revealed when people are allowed to attach things to the world around them, to the streets they normally pass through, or the points of interest that they normally engage in. To be able to author on the fly on the streets and attach it discreetly to an object effectively.

Tish Shute: And yes we do have all kinds of markers and QR codes. But Erick Schonfeld of Tech Crunch made a good point that QR codes: “Until QR code scanners become a default feature of most smartphones and they start to become actually useful enough for people to go through the trouble to scan them, they will remain a gee-whiz feature nobody uses.”

Chris Arkenberg: So again, this gets back to competing standards and who gets access to the phone stack, the bundle. Who gets the OEM deal…?

Tish Shute: Yes, the battles for the networks on the Handset Plains are pretty important for AR!
[laughter] I think Layar have made some smart moves on The Handset Plains.

And there are a lot of acquisitions of nearfield technology to look at.  If I remember rightly Ebay bought the Red Laser tech from Occipital – now there’s any interesting company. Their panorama stuff rocks!

Chris Arkenberg: Right. There’s a lot of nearfield stuff that’s supposed to hit all of the major mobile platforms in the next year or so.

I mean I think where this is heading, in my mind, is basically smart motes. You know, little nearfield wide-range RFID’s that are the size of a small, tiny square that you could attach to just about anything and then program it to be a representative of your establishment or of an object, that then you can start to tag just about anything. I mean you can’t rely on geo to do it, but if you have a Nearfield chip there that costs maybe like two cents to buy in bulk, and you can flash program it, then you can start to attach data to just about anything.

Tish Shute: Yes ‘cos some things still remain very difficult for near field image recognition technologies like Google Goggles.

Chris Arkenberg: Well, if your phone can interrogate for Nearfield devices, and it detects a chip in its near field, it can then interrogate that chip. The chip may contain flash data on itself, or it may contain the local server in the establishment, or it may go to the cloud and get that data back.

Tish Shute: Yes there is moverment from the top and open source hardware like Arduino has created an opportunity for all sorts of creativity with instrumented environments.  And the handheld sensors in our pockets – our smart phones create a lot of opportunity for bottom up innovation too.

Chris Arkenberg: I mean that’s my guess. If you look at what IBM is doing with their Smarter Planet initiative, they’re partnering with a lot of municipalities, and obviously with a lot of businesses and their global supply chains.

But they’re basically working with municipalities and all these stakeholders to instrument their territory, their business, or their city, as it were. So they’re working to provide embedded sensors and the software necessary to read them out and run reports & viz. And presumably that software can extend to include some sort of mobile device to interrogate the sensors and read the data.

That’s kind of a top-down approach of a very large global company working with top-down governance bodies to do this. Simultaneously you have the maker crowd experimenting with Arduino and such to build from the grassroots, the bottom up approach.

And that’s primarily gated by the amount of learning it takes to be able to program these devices, to be able to hack them. Typically, the grassroots creators who make these devices don’t have the luxury of very large budgets to make things highly usable and Wizywig.

So the bottom up community is a sandbox to create tremendous amounts of innovation, because they are unconstrained by the very real financial needs of the top down innovators. And so you get a lot of fascinating innovation, a very rich ecology from the bottom-up approach, but you don’t get a lot of wide distribution. But that does filter up to and inform the top down approach that has a lot more money to put into this stuff. And it ultimately has to respond to the needs of the marketplace.

I mean if there’s an answer to the question of whether something like AR will succeed through the bottom-up grassroots approach or the top-down industry approach, I would say it would be both. That handsets will be hacked to read the bottom up innovations of the maker community, and handsets will be preprogrammed to read the top down efforts of the IBMs of the world.

Tish Shute: Yes but i have to say it is very time-consuming hacking phones (I have just seen a few days suck up in this myself so that I could upgrade my G1 to try out the new ARWave client!). I mean Android has obviously been the platform of choice because of openness but the business model of iPhone and its market share in the US sure make it important for developers.  It’s like you don’t exist if you don’t have an iphone app for what you are doing.

Chris Arkenberg: Yeah, and that’s the challenge, because at the end of the day developers prefer not to work for free and a solid, reliable mechanism to monetize their efforts becomes very appealing.

When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?” You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development. And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.

Tish Shute: Some of the important platforms for AR don’t appear to have spots on the map like Google Street View and other mapping technologies that hold out so much hope for AR, or am I missing something?

Chris Arkenberg: You mean on the map?

Tish Shute: Yes for the full vision of AR we need sensor integration, computer vision and cool mapping technologies to come together. Do you see where Google Maps and Google Street View… Where would they be?

Chris Arkenberg: Yeah, I mean it’s certainly content, it’s location…

Are you familiar with Earthmine?

Tish Shute: Yes, yes I am, definitely. Earth Mine, Simple Geo, Google Street View, user generated internet photo sets like Flickr all of these could be very important to AR, potentially.

Chris Arkenberg: Well, and the interesting thing about Earthmine is that they’re effectively trying to do an extremely precise pixel to pixel location mapping. So they’re taking pictures of cities just like Street View, except they’re using the Z axis to interrogate depth and then using very precise geolocation to attach a GPS signature to each pixel that they’re registering in their images. Effectively, you get a one-to-one data set between pixels and locations. And so you can look at something like Google Street View, and if you point to the side of a building, in theory, it should know exactly where that is.

They’re rolling this out with the idea of being able to tag augmented reality objects in layers directly to surfaces in the real world. So that’s another approach to trying to get accurate registration and to try and create what are essentially mirror worlds. Then your Google Street View becomes a canvas for authoring the blended world, because if you plop a 3D object into Street View on your desktop, and then you go out to that location with your AR headset, you’ll see that 3D object on the actual street.

Tish Shute: There was some experimental work with Google Earth as a platform for a kind of simulated AR but I suppose Google Earth doesn’t figure in the battle for the network economy as it never got developed as a platform.

Chris Arkenberg: It hasn’t tried to become a platform, to my knowledge. I mean I know some people are doing stuff with it, but as far as I know, Google owns it, they did it the best because they have the best maps, and there’s not a huge ecosystem of development that’s based around it other than content layers.

And my sense of everything else on the Points of Control map is they’re looking more at these sort of platform technologies that…

Tish Shute: Yes, re platforms for growth for AR. Gaming consoles will probably emerge as a significant platform for AR this year.

Chris Arkenberg: There will be much more of a blended reality experience in the living room for sure, and with interactive billboards. Digital mirrors are another area. So I mean if we kind of extend AR to include just blended reality in general, you know, this is moving into our culture through a number of different points. As you mentioned, it will be in the living room, it will be in our department stores where you can preview different outfits in their mirror. We’re already seeing these giant interactive digital billboards in Times Square and other areas.

It’s funny. I mean for me, the sort of blended reality aside, the augmented reality, to me, is actually a very simple proposition in some respects. When I look at this map, augmented reality is just an interface layer to this map in my mind, just as it’s an interface layer to the cloud and it’s an interface layer to the instrumented world. It’s a way to get information out of our devices and onto the world.

Tish Shute: The importance of leveraging existing platforms has become pretty clear but it is interesting Facebook definitely gave Zynga the opportunity but would Facebook be so big without Zinga’s social gaming boost?

Chris Arkenberg: I feel that Zynga has definitely helped its growth…But I think Zynga has benefited a lot more from Facebook than Facebook has from Zynga.

Tish Shute: Zynga certainly proved you could build a profitable business on Facebook’s API!

Chris Arkenberg: They did. And they also really validated the Facebook ecosystem and the platform. They really extended it… Zynga benefited from the massive social affordances that Facebook had already architected and developed. They brought gaming directly into Facebook, and particularly, this emerging brand of lightweight social gaming that when you sit it on top of a massive global social network like Facebook, it suddenly lights up.

Tish Shute: AR pioneers should quite carefully go through this map. There is so much to think about here. I’m a kind of fanatic about Streams of Activity in AR. Real time brokerages and their potential for AR is something I am fascinated by. That is one reason I love the ARWave project.

Anselm Hook, to me, is one of the great thinkers in this area of real time brokerages – with his project Angel, and the work of Ushahidi, which is now the platform for augmented foraging (see here). Anselm is now working on AR at PARC which is exciting.

Chris Arkenberg: Well, there are some challenges working with data streams. Presentation and filtering I think is a big challenge with any sort of stream. Because obviously, you have a lot of potential data to manage, to parse, and to make valuable and comprehensible. So I think this is bound very closely to being able to personalize experiences, or having very discreet valuable experiences. Disaster relief, for example, I think is an interesting idea that ties into the Pachube type of work. Where, if you had the headset and you were a relief worker, and you had immediate lightweight, non-intrusive, heads up alpha channel overlay, waypoint markers showing you all of the disaster locations or points of need, AR becomes extremely valuable, because it’s a primarily hands-free environment. This is why the military stuff is so interesting.

Tish Shute: Ha! We are running into the eye patch/shades/goggles/sexy specs thing again. But filtering and making streams of activity relevant will be very interesting for AR.  Again that it why I love the Wave Federation Protocol work because what they have built into their XMPP extensions. You can have your real-time personal data streams, or community streams, or broadcast publicly – the permissions are built.

And Thomas Wrobel’s original vision of these layers and channels is only fully expressed if you have the eyewear.

Chris Arkenberg: Well, and it becomes redundant if it’s on a mobile. To use a very basic example, Twitter, obviously there’s an app you can view those streams of activity on the camera stream. But you can view that real time data on the screen. Why do you need to see it heads up?

The reason I really pay attention to what the military is investing in, one, because they have a ton of money, but also because they tend to represent the core bio survival needs of the species…So, when I look at computing, I see this very obvious trend of computers getting smaller and smaller and closer and closer to us because they’re so valuable to our success. They give us so much valuable information for engaging our world on a moment by moment basis. So, of course now we have these tiny little handheld devices that give us access to the global knowledge depositories of human history, because it’s so useful to have that stuff right at hand.

The only impediment now is that it takes one of our hands, if not both of them, to access it. So if you are in the natural world, which we are all always in the natural world, ultimately, you want your hands free in order to engage with the world on a physical level.

I see computation, or rather, our access to computation is just going to get thinner and thinner, and we’ll very soon move into eyewear, and inevitably, we’ll move into brain computer interface in some capacity.

So when you’re the disaster worker, or a deployed soldier, or the extreme mountain biker, or the heli-skier, or just an adventurer, there are a lot of very practical reasons to have access to information on a heads-up plane. I see AR as being so profound and so valuable, but we’re getting a glimpse of it in its infancy, and it’s got a ways to go to be able to really contain what it is we’re reaching for.

Tish Shute: I agree.

Chris Arkenberg: And that’s been a big criticism I’ve had with all the existing AR implementations that I’ve seen, is that the UI really needs a revolution. It’s very heavy handed. It is not dynamic, even though it’s supposed to be. It does not take advantage of transparencies. It treats the screen like a screen. It doesn’t treat the screen like a window onto the real world. When you’re looking on the real world, you don’t want a lot of occlusion. You want very soft-touch indicators of a data shadow behind something that you can then address and then have it call out the information that’s important to you.

Tish Shute: Now, that’s a very nice kind of image you’ve conjured for me there. Do you see that more could be done on the smartphone than is being done within that? Or are we like waiting for the old ishades?

Chris Arkenberg: I think there’s definitely a lot of room for improvement on the smartphone UI. Nobody’s really played around with it much. And again, I think that’s in part that there hasn’t been a really established platform with enough money to fund interesting UI work. We see it in some of the concept demos that float around every now and then.

I guess it’s both a blessing and curse that I’m always five steps ahead of where I’m trying to get to.

Tish Shute: Yeah, I am familiar with that feeling!

Chris Arkenberg: So I’m always trying to reach for the vision even though it’s a bit distant. I think there’s going to be a lot of development on the handsets. But again, I think we need a lot of refinement. We need a lot of real critical analysis of why this is a good thing.

To get back to the original point of Raimo’s comment, it struck me. And I knew it, but I just had set it aside as gimmickry. But he’s right. Content is a huge driver for this. Just stuff that’s engaging, and fun, and cool, and shows off the technology so they can get enough money to make it through whatever Trough of Disappointment may be waiting.

Tish Shute: Yeah, don’t underestimate the Planes of Content!  They are a great place to get interest and money to keep AR technology moving on, right?

Chris Arkenberg: Yeah, yeah. Because, you know, there’s a lot of freedom there. And you can piggyback on all the rest of the content that’s out there and jump on memes and marketing objectives, etc…

And there’s a lot of stuff…I’m blanking on some of the names, but some of these historical recreations of city streets. There’s a street in London where they overlaid historical photos in a really compelling experience. [Museum of London - http://www.museumoflondon.org.uk/] Again, I’m completely forgetting the attributions, but hose are the type of things that can really be pursued on the existing platforms. There is stuff that’s really compelling and really cool.

I heard of another interesting use case – and I should say that I can’t find attributions to this anywhere on the web and I may be paraphrasing or mis-representing the actual work, but I think the concept is worth exploring anyway. But the idea was that you could take the locations of border checkpoints and conflict sites in Palestine and Israel and visually overlay them on an AR layer in San Francisco. And it would do some sort of transposition where you could virtually view these things in San Francisco with the same locational mapping superimposed. So you could see where the checkpoints where. You could see where the wall was. You could see where suicide bombings were and where there had been conflicts. [I cannot find any citations for this!]

Tish Shute: But with an AR view? But why would you use an AR view if you are in San Francisco, then?

Chris Arkenberg: Because it superimposes two realities, translating the Gaza conflict into San Francisco as you are walking around. You can interrogate the world. There’s a discoverability aspect where you’re using the headset to reveal things, or the handset rather, to reveal things that you could not see otherwise in your city. It was done as an art piece, but as a provocative, obviously political art piece.

Tish Shute: Very interesting. I’d love to see that. Because that’s interesting to get away from this idea that you actually have to sort of have this one to one relationship between the data and the world is kinda nice, isn’t it? Well, not one to one, but a very literal…getting away from that literalness is kind of good.

Chris Arkenberg: And that’s a possibility of virtual reality and augmented reality merging, that maybe virtual reality is actually going to do best by coming out of the box and writing itself over our reality, so that as you are walking around, you are no longer seeing San Francisco, but you are seeing part of Everquest or World of Warcraft.

Tish Shute: Well this is where Bruce Sterling gets to that point he made in his keynote for are2010, that if we actually have viable AR eyewear, then you get the gothic stepsister of AR, VR rising from the grave!  He asks whether the very charm of augmented reality, is in fact that it adds rather than subtracts from your engagement with the world and that getting get sucked back into the black hole of VR might not be so great.

Chris Arkenberg: And then you get all sorts of interesting challenges to social cohesion if you have a lot of different people experiencing very different worlds, effectively. That if there is no real consensual reality and a majority of your local populous is, in fact, experiencing very different and unique versions of the world, what does that do to social cohesion? How does that reinforce tribalism, for example, when only you and certain others get to opt in to a particular layer view of the world?

Tish Shute: Yes Jamais Cascio wrote an interesting piece on that issue on AR and social cohesion a while back.

An eye patch is a more logical vision than the goggles in many ways but I suppose the loss is stereo vision?

Chris Arkenberg: And actually, there were developments in military helicopter technology many years ago that used a single pane square of glass over the eye mounted to the helmets of pilots. And then they drew various bits of heads-up information on it. So that ensures that you’re having a real strong engagement with the real world, which, obviously, when you’re a helicopter pilot is quite important. But you still have access to the data layer of the invisible world.

Tish Shute: I just went to Hadoop World and I have to say, I was awestruck about how big that’s got. I mean Hadoop has gone from like zero to huge in just a few years. I mean it’s just like now everyone has the power of the Google big table at their fingertips.

What’s the play for AR in the land of search?

I could imagine Hadoop being very powerful tool for AR analytics?

Have you got any thoughts on the land of search and AR? Of course visual search is proceeding at a fast pace and there is a lot of promise for integrations with AR in the future but the latency for visual search is still pretty high?

Chris Arkenberg: In the near term, not a lot. In the medium term, there’s a larger trend towards virtual agents that you can program or teach to keep watch over things for you as an effort to scale down the data overload. So search is something that’s going to become more personalized and more active. There’s a movement to make it so people can essentially deputize these agents to be always searching for them; to be out there looking for the things that they have told these agents are important to them.

So active search for AR I think presents some challenges, obviously because you need to do text input, typically, or voice input. Voice input, I think, is much more achievable than text input for AR. But I can certainly imagine an AR layer that is being serviced by these agents that we have roaming around the web for us reconciling their visual view of the world with our personalizations. AR apps are contextually aware so it knows that if you’re downtown, it’s not going to be giving you a ton of information about Software as a Service infrastructure, or what have you. But that, instead, it’s going to be handing you little tidbits about a particular clothing brand you’ve opted in to follow and information about music venues & schedules, for example. Or perhaps you’ll be on the lookout for other users that have opted in to publicly tag themselves as a member of this or that affinity.

I keep coming back to this idea of AR as really just a simple visualization layer that all of these other technologies can potentially feed into. So in that sense, search becomes a passive thing that AR is just simply presenting to you in a heads-up, hands-free, or potentially hands-free environment.

Tish Shute: Yes, the big challenge is the stepping stones to that point! Small steps that keep interest going into developing the underlying technology (and not just in research labs!) that will bring us that interface.  We have seen some movement already with Qualcomm.

Chris Arkenberg: And there are bandwidth issues as well, as we can see with the Google Goggles, which is a great idea of visual search. But you have to take a picture and send it to the cloud and wait for your results. It’s not a real-time dynamic interrogation of the world.

Tish Shute: Yes we are really only at the very beginning of AR being ready for prime time.. it would be interesting to ask AR developers how many of them use AR on a daily basis.

Chris Arkenberg: I think a lot of us, we’re just informed by the sci-fi myths and fascinated with the potential now that’s it’s starting to become real. But I think we all kinda get that it’s still extraordinarily young. I mean the web is extraordinarily young. And AR is itself far younger in a lot of ways in its implementations.

Everybody has a lot of excitement about all of the great potentials that are being unleashed by this great wave of the Internet and the web and ubiquitous mobile computing. So that’s why, you know, you look at that map and we talk about AR and you can’t talk about any of the stuff without talking about all of it, in a lot of ways, particularly with something like AR where it’s so ultimately agnostic and could be completely pervasive across all of these layers.

So my fascination is with the future, and I measure our progress towards it by the young nascent offerings from the platform players and the developers. And yeah, a lot of it is…it’s akin to getting that first triangle on the screen in 3D. You know, when the renderer finally works and you get a triangle on the screen, and you go, “Oh my God, it renders.” And then you can start to really build polygons and build objects, and start doing boolian operations, and get light and rendering in there, and textures, and on, and on, and on.
So I’m fascinated by the Layars and the Metaio’s…
[laughter]

Tish Shute: Yes and hats off to all the players in the emerging industry, Layar, Metaio, Ogmento, Total Immsersion, and all the others who are finding clever ways to bring fun aspects of AR into the mainstream, and fuel interest to take the technology to the next level.

Chris Arkenberg: Absolutely. And the hype cycle is very valuable. It has really helped launch the AR industry. It’s brought a lot of eyes, and it’s brought a lot of money into the industry. And it’s forcing people like us to have these conversations to understand how to refine its growth and really focus on the potential in all these different venues, whether it’s trying to save lives, or better understand your city, or have really compelling entertainment experiences.

Everybody’s excited, and everybody’s sharing, and everybody’s trying to move it forward in a way that’s the most productive.

Social Augmented Experiences leveraging geoawareness and human and machine intelligence to create real time information brokerages, combined with an augmented reality view, can create a new opportunities to reimagine our relationships with each other and our environment.

This Summer, I have been on a blogging hiatus, which has meant I haven’t been sharing as frequently and, unfortunately, the second half of two conversations I had earlier this year, both of which have much influence my thinking on social augmented reality, have languished in private mode -  part 2 of my talk with Bruce Sterling (see Interview with Bruce Sterling, Part I: At the 9am of the Augmented Reality Industry, are2010, and part 2 of my conversation with Anselm Hook - Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, Part 1. Time to get caught up on some blogging!  The lightly edited transcript of Part 2 of my conversation with Bruce Sterling is posted in full below.

Bruce Sterling has been blogging all the key developments in augmented reality (amongst other topics of interest!) on his Wired Blog, and he brought my attention to Boskoi the Ushahidi based app for Android phones, augmented foraging pictured in use above – for more pics see fightthegooglejugend.

Augmented Reality and Real Time Information Brokerages

Picture above is the path the “nomads” took through the Westhaven cryptoforest with Pieter Bol,co-auteur of the book Biological Globalisation and Theun Karelse of Urban Edibles Amsterdam “who presented his ‘augmented foraging’ app Boskoi.“  For more see, The Cryptoforests of Utrecht and, Westraven Psychogeography, 6 June 2010. Note: Cryptoforests: 1) Urban forests hidden from view 2) Urban fallows that might or might not be considered as forests 3) Gardens gone wild)

My interest in the Ushahidi family of ideas was already fired up by a conversation with Anselm Hook early this year.  We discussed a number of Ushahidi related projects, Swift, Crisis Filter and Anselm’s project Angel, Augmented Reality, and my own keen interest in an open, real time, distributed platform for augmented reality – ARWave.

The Ushahidi platform and the related project Swift has pioneered the real time brokerage of information with people acting in curatorial roles or matchmaking roles coevolving with machine assisted matching to connect wants to haves.  Ushahidi uses multiple gateways including SMS, and Twitter.  But the Ushahidi family of ideas is extremely interesting when combined with augmented reality and suggests many new possibilities for social augmented experiences, as Anselm pointed out, for human to human communications, human to civilization communication, and human to environment communications (e.g., perhaps, how machine intelligence can help bridge the difference in time scale that Kate Hartman explores in her, Research for Glacier-Human Communication Techniques).

Ushahidi, which means “testimony” in Swahili, is a website that was initially developed to map reports of violence in Kenya after the post-election fallout at the beginning of 2008. It is now an open platform with a wide range of applications and growing developer community.  See What is the Ushahidi Platform? from Ushahidi on Vimeo.

Swift - a project that emerged from the Ushahidi dev community, is a human sensor/real-time brokerage for dealing with emergencies, enabling the filtering and verification of real-time data from channels such as Twitter, SMS, Email and RSS feeds.

Boskoi – augmented foraging is the first app,  I have seen, to begin linking Ushahidi with augmented reality – although I don’t think there is a full augmented view for Boskoi developed yet?

“The whole point of AR is to see things from a different point of view…”

Click to enlarge poster from upcoming ARWave demo at Software Freedom Day – for more see below

I am often asked what augmented reality brings to the table with respect to location based social networking, which is on the verge of going mainstream in smart phone apps like Four Square. While the first part to my answer is usually to explain what is unique to augmented reality.

As Bo Begole notes, the full vision of AR requires machine perception technologies to detect the identity and physical configuration of objects relative to each other to accurately project information alongside/overlaid with a physical object (see this post on the PARC Blog by Bo Begole on the difference between AR and ubiquitous computing – thank you Rouli for bringing my attention to this).

But it is only in recent months that we have begun to see the kind of tools that make this possible become freely available to developers – see my interview with Jay Wright of Qualcomm here.   Also see this post on Bundler: Structure from Motion for Unordered Image Collections an open source system that allows the creation of 3D point clouds from unordered image collections, e.g. internet image collections.  We now have many tools available to move mobile augmented reality beyond the recent crop of apps relying on GPS and compass alone for positioning into a new era of vision assisted AR apps that will increasingly bring the full vision of AR into our daily lives.

Further, the integration of visual search applications like Google Goggles and Kooaba which can detect the identity of particular objects will add another vital tool to machine perception technologies enabling AR “checkins” on potentially anything in the physical world around us, and more fuel to the Gamepocalypse (e.g. it would be easy to turn every trash can in the city into a basketball hoop as we discussed at the ARNY meetup last month).  And soon, the Pandora’s Box of  facial recognition (Google Goggles have the capability though it is not released to the public yet) will open up.

Jesse Schell described the importance of AR in a nutshell in his keynote for are2010:

“The whole point of AR is to see things from a different point of view…How can there be a more powerful art form than one that actually changes what you see?”

But how AR matures as a social experience will be the key to Jesse’s suggestion that:

“Augmented Reality will be one of the things that fundamentally define the 21st century”

There are many interesting forms of AR that are not reliant on a tight registration between media and physical objects – several are put forward by Bruce in the convo below.  And, it is likely we will see AR eyewear as an occasional useful accessory to a smart phone long before we have the sexy, affordable augmented reality eyewear worn that we wear throughout the day.   These speech to text glasses would be a very useful and viable accessory to a smart phone right now for the hearing impaired.

For the moment, as Bruce notes, some of the most interesting and useful augmented experiences to date have not been in the cell phone space:

“There are other aspects of AR besides the cell phone space. There’s Total Immersion’s big display screens. There’s the web-based fiduciary stuff. And there’s projection mapping. And then there’s experience design just for people who need their reality augmented for whatever personal or social reason.”

On of my favorite social AR experiences is this SMS Slingshot.

But I have been excited for a long while about the intersection of mobile social augmented reality, real time communications, and ubiquitous computing see Total Immersion and the “Transfigured City:” Shared Augmented Realities, the “Web Squared Era,” and Google Wave.  And I have described in many places why I think real time, open, distributed communications for AR are so important to developing social augmented experiences – see the slides for my talk at Augmented Reality Event here, here and here for starters.

ARWave at Software Freedom Day 2010, September 18th 2010

Thomas Wrobel and Bertine van Hovell will demo the first ARWave Android client at Software Freedom Day this weekend!

A number of people have asked me, (including Bruce), What will be the future of ARWave now that Google Wave is no longer a stand alone application?  Yes, the recently announced release of Wave in a Box (see here and here) is very exciting for the ARWave team.

The ARWave Android client is the first open AR client built on an open, real time, distributed platform -  based on a server that anyone can download and set up, currently the “FedOne” server but Wave in a Box, hopefully, will be even easier to deploy.  Wave in a Box seems perfect for ARWave’s needs – for more see the WiaB Google Group here.  And for more information on the ARWave client -  click to enlarge the poster below, see the ARWave concept video here, and for more, and how to get involved see arwave.org.   Props to Thomas Wrobel and Bertine van Hovell (posters below from demo for Software Freedom Day), Mark Evin, Davide Carnovale, and Markus Strickler, for all their hard and brilliant work on ARWave.  Also to JCPT the open Android 3D engine that has saved a lot of work!

click to enlarge slide

Social Augmented Experiences that Matter

My ideas on the future of social augmented experience have been deeply informed by the the conversations I had with Bruce Sterling and Anselm Hook this year.

Bruce Sterling notes in the conversation below, location based social apps like, Four Square, are interesting because they are not “urban geography like Google’s satellite stare from above,” but rather “groups of citizens are doing portraits of their own region.” Augmented Reality, with its of lauded power to make the invisible visible is, of course, is the ideal tool for “citizen portraits”  to the next level.  Cory Doctorow described to me three years ago (see here) an “inverse surveillance society,” enabled by an augmented view  – “where all the data from the positional and temporal characteristics of all the objects that we own were in aggregate visible and available so that we can mix and match them remix them understand them and have more agency in the world.”

It is very cool to go back to reread this conversation now that it is becoming possible to build the kinds of apps Cory described, and Bruce Sterling envisioned in Shaping Things (see Amazon.org  page 111).

click to enlarge

My  conversation with Bruce earlier this summer (see below) took place on the heels of are2010 – Augmented Reality Event.   See the video of Bruce’s keynote, “Bake a BigPie,” here,  and the final keynote, “Seeing,” by Jesse Schell (see video here) in which Jesse riffed on AR and the man with the X-ray eyes.  Both these awesome talks are still fresh in my mind.  Bruce noted how we should pay attention to augmentations for people and situations that could really use some augmentation… and not get too fixated on the coming of AR Goggles.  He elaborated on this in our conversation (again full transcript below):

“Well, it’s a matter of deciding whose reality it is that you’re trying to augment. I’m not trying to be a bleeding heart about it, but obviously there are people in our society right now with reality that could really use some augmentation. They are mostly disadvantaged people. They are vision impaired, or maybe they have autism. They might be senile and just can’t remember where they put their shoes. These are people who could really use some help, right?”

So, start with people who really need sensory or cognitive help. Before you turn our geeks into Superman, why don’t you try turning some people who are harmed into more functional individuals? Then you’ll be able to learn how to do that. Then maybe you can ramp it up to these Nietzschian heights of the superb Man With the X-ray Eyes. Whatever.”

What will make AR interesting and useful long before and long after we see the full vision of AR eyewear manifest is its social aspects.  Bruce points out:

“My argument would be that if you want people to be more sensitive toward certain, say, issues and problems, it’s better to find the people who are already sensitive to those issues and problems, and give them a bigger stake in your augmentation system.”

“Say that I am really worried about public health. Well, if I have a lot of nurses that are using my system, people who are aware of my issues, then I could be walking around and I’ll see a lot more tags saying, “This is where he got food poisoning!” “In this shooting gallery, many people have caught AIDS!” Or, you know, “Tuberculosis has been spotted over here in this building.”

At that point, I could simply share their knowledge and get some social intelligence. As opposed to trying to amp the basements of my little hacker-mind and drag stuff up that’s escaped my conscious attention.”

Finding new ways to broker information – bring together needs with haves and different participants, empowered and disempowered  is., as Anselm discussed with me, one way to change our view of human to human, human to environment and human to civilization communication (particularly in light of this  “sobering account of how open data is used against the poor in Bangalore” that as @timoreilly noted recently OpenData Empowering the Empowered).

The key idea in a crisis filter, Anselm noted,  was to break up the participants into different kinds, to connects wants with haves:

“There are people who are in  situation.  We call them citizens.  And then there are reporters, people who report situations back to Twitter.  And then there are curators, people that canvas Twitter looking for important Tweets.  And then there are first responders, people who take the curating collection of responses and then act on them.”

This kind of brokerage between people acting in a curatorial role or matchmaking role with each other can be extended into and coevolve with machine assisted matching as Anselm explains.

It is also a vital part of creating social augmented experiences that matter.

One of Anselm Hook’s projects, which is called Angel is the the most radical expression of connecting wants with haves in that the idea is that “you have a situation, you broadcast that situation, and help magically appears.  You don’t even sign up for a service.  You just get help …

As Anselm explains this is the same idea of a brokerage for dealing with emergencies, but applied to the long tail of crisis response.  As Anselm describes it:

“I am interested in personal crisis.  ‘I lost my cat.  Help.  I can’t find where my kid is.  I am out of gas.  I have a flat tire.  My house is on fire.  My aunt is trapped in the bedroom.’  The kind of personal crisis that is just as important, but is not enough to get a national movement to help you…

I will publish this conversation with Anselm in full in an upcoming post.

Zorop – an ARG for World Peace

If you want to be part of a really exciting experiment to reimagine our relationships with each other and can be in San Jose this weekend, I highly recommend exploring this “rabbit hole”.

Thank you Gene Becker, Lightning Laboratories and Ubistudio for sending me this invite:

“Ken Eklund (@writerguygames) is developing a wonderful game for the 01SJ Biennial called ZOROP, aimed at creating World Peace(!). Some of you might know Ken from his work on the amazing ARGs EVOKE and World Without Oil. Anyway Ken, along with his collaborator Annette Mees, are furiously working to get ZOROP ready to go for the Sept 17th premiere at 01SJ.

Are you intrigued? I thought so, and here are your next steps down the rabbit hole:

> Check out http://zorop.org to learn about the game

> Follow @ZoropPrime to watch it unfold: http://twitter.com/zoropprime

> ‘Like’ ZOROP on FB for a different view: http://www.facebook.com/pages/Zorop/141140772593618

> Become one with the game; consider volunteering as a Zoropathian: curious@zorop.org

> Head down to San Jose on the 17th, play the game, and ride the ZOROP Mexican Party Bus. Seriously.”

Interview with Bruce Sterling

Click on image above to see video clip from from brucesflickr

[Note the first part of this interview is here and I broke in anticipation of Part 2 just as I started experimenting with an idea Joshua Kauffman - an advisor and entrepreneur working on design in the public sphere gave me for an interview technique - the All Souls College one-word question interview.  Although apparently they recently scrapped it and I am not very good to sticking to a single word!]

Tish Shute: We were talking about these proximity-based social work networks like Foursquare and Gowalla and how they may influence the emergence of social augmented experiences.

So Joshua’s suggestion for the first word was “territorialization” e.g. how do these new mobile social experiences like Foursquare, and the observation that actually rather than breaking down territorialization – which would be a good thing, tend to support territorialization…

Bruce Sterling: Yeah, they’re re-intensifying it in a very odd, electronic fashion.

Tish Shute: Yes.

Bruce Sterling: It’s not true of projection mapping or the webcam fiduciary display stuff. But with the handheld stuff, and especially the urban informatic stuff, it really can’t help but take on a local flavor. Layar is like “Augmented Dutch Reality.”

And TonchiDot is “Augmented Japanese Reality.” It’s hard to imagine a Layar interface going gangbusters at Tokyo. Whereas the TonchiDot interface, which is so clearly influenced by Anime and cartoon graphics…. Maybe it could find some niche of hipsters in Amsterdam hash bars…

Stuff that’s socially generated by people on the ground, as with Foursquare and Gowalla, is bound to take on a regional influence. Right? It’s like the New York hipsters who were early adopters of Foursquare. They’re not mapping New York! They’re mapping Hipster New York.

It’s all about Williamsburg and places where 24-year-olds go to drink… They found a demographic niche there. These guys are building the service for them. They’re people who are willing to work for Foursquare for free, because they want to wear the little king hat.

Tish Shute: I got the far far away badge ‘cos I live on the Upper West Side!

Bruce Sterling: But that’s not urban geography, right? I mean, that’s not like Google’s satellite stare from above. That’s a group of citizens doing a portrait of their own region. You’re going to see interesting things happen because, of course, people who use Foursquare elsewhere are going to check into New York, and they’re going to look at the “New York Foursquare.” They’re going to be aliens who interact with Foursquare people in New York and annotate what they’re seeing.

Tish Shute: Oh! Yes. Good point.

Bruce Sterling: That Foursquare community has a certain émigré soul. It’s different from the normal émigré soul of simple tourists on New York. So you’re friend there is right about the territorialization.

Tish Shute: Yes, Joshua Kauffman is a smart guy! Yes I am interested to see what interesting kinds of deterritorializations proximity based social networks and the hyperlocal view of augmented reality might bring, not just the new territorializations.

Bruce Sterling: It’s not the intense kind of territorialization, like gangs putting down graffiti markers and beating people up. It’s an inherent regional character that comes with using peer production to build your database.

Tish Shute: We were discussing whether AR could break down the walls between people – people who share the same physical space but actually inhabit different territories even if they are sitting on the table next to you.

Bruce Sterling: You know, I just wrote an article for my Italian magazine column. I think I mentioned this to you – a report about ARE 2010. I titled it, “Chicks Dig Augmented Reality.”

Tish Shute: [laughs]

Bruce Sterling: There is a very heavy social element to AR, and a phone based element. So the question is: Why would a woman wear a fiducial marker? Like our Metaio speaker at ARE2010 who had a fiducial marker on her lapel pin.

Tish Shute: Right. Lisa!

Bruce Sterling: Why would a woman go out in public with her Facebook profile on her body?

Tish Shute: Well I can think of some reasons…

Bruce Sterling: So that men will approach her, of course.

Tish Shute: Yes the core of all successful social networks is always a form of dating app.

Bruce Sterling: You do it a social icebreaker. It’s like: I’m a woman, I’m sitting here alone, and you can sort of glide by and, you know, take a snap of me. Then you retreat and have a beer with your friends and you work up the courage, and then you come and say, “So! Susan! I understand you like bicycling! And, boy, me too!” Right?

Tish Shute: There are all kinds of social barriers between people in cities that AR might be helpful in breaking down. An extreme example is the dilemma you actually quite often face as a New Yorker as you walk around a city. There are people asleep on the pavement and you don’t know if they’re dead or alive.

Bruce Sterling: Right.

Tish Shute: And you sort of like have this awful ethical dilemma of like, “Am I walking by someone I should be shaking by the shoulder, right, to wake them up so they don’t die, right?”

Bruce Sterling: Yes.

Tish Shute: You said in your keynote that we should pay attention to augmentations for people and situations that could really use some augmentation..

Bruce Sterling: Right. There actually is such an app in Britain right now. I posted about it: two Augmented Reality schemes for rubbish and hobos.

Tish Shute: Right. Yes I saw that!

Bruce Sterling: “Any sufficiently advanced technology is indistinguishable from garbage and hobos.” You don’t need to personally find out whether this hobo is worth your help. What you need is a good way to report the hobo to a hobo check-up service. They come in, and they look on their own database or supply a database to you, or a facial recognition unit, whatever. The service says: “Oh, well. That’s Fred. He’s a paranoid schizophrenic. He always sleeps in that alley. Let him be.”

The same goes for the rubbish — although I don’t want to compare rubbish to hobos. In fact, people do go out with their AR kits and take pictures of abandoned garbage bags and broken glass. They upload them with geolocated tags for the local garbage guys. Guys who are sitting around doing pretty much nothing because they don’t know where the rubbish is.

And they will come out and get the rubbish! I mean, they just deputize guys to go out and follow these alerts. Right?

But nobody predicted — least of all me — that you were going to have a high-tech Augmented Reality system that consisted of removing rubbish and derelicts. Right? But rubbish and derelicts always go profoundly under-reported. It’s just hard to get people’s attention.

But it’s very easy to set up a system so that, if you get ten reports on the same piece of rubbish, that’s going to work its way to the top of the stack. That’s why I was trying to get AR people away from the romance of the hottest app for the shiniest machine. More toward a design stance that’s more user-centric.

Where are the actual problems about stuff that we perceive? Stuff we can’t do anything about? Or people whose mechanisms of perceptions are harmed. They could be doing good work, being more participative, if they didn’t, basically, walk around without their glasses on.

Tish Shute: Well this leads well into the second word, Joshua suggested was interesting spring board – sensitivity.

On the one hand we can do these things for people who maybe need the augmentation because they have difficulty with one or another sense, e.g., their eyes are not functioning, or their ears are not functioning. But on the other hand, we can’t cross the social bridge to communicate with people who are temporarily disempowered in relation to the rest of society e.g. hobos and people who sleep on the streets of New York City.  And even though Augmented Reality could potentially be helpful it can even be more disempowering to the already disempowered.

Bruce Sterling: Right.

Tish Shute: But re “sensitivity” – does augmentation increase or decrease our sensitivity? This is a problem that Will Wright brought up [see video of Will Wright's keynote at are2010], e.g, the problem of parking HUDs getting in the way of your intuitive parallel parking skills. The Lexus that takes driving control from you when you look back, ‘cos it knows that you’re looking at the road, and it starts to brake. Right?

Bruce Sterling: Right.

Tish Shute: The fact that the problem with technology is that it makes us less sensitive, right, augmentations sometimes get in our way?

Bruce Sterling: I suppose that’s true. But I’ve heard that said about practically every medium. Especially television.

Everybody wants to blame machinery for their lack of morality. It’s hard to top something like the Kitty Genovese killing in New York. This sort of legendary New York horror story from the 1960s. A woman is stabbed to death in public, no one does anything.

Tish Shute: Right.

Bruce Sterling: I don’t think that our media is making us any less humane or more callous.

Tish Shute: All right. Oh no! I see what you’re saying. Perhaps I misrepresented what Will was suggesting by putting it that way. The question is perhaps more how do we get the sensitivity into the technology. Human bodies are fantastically sensitive and sensory.

Bruce Sterling: Right.

Tish Shute: And we have these like sensitivities. For instance, How could augmentations of reality be like a blush ? You definitely want an interaction that’s not just this data being pushed at you. But what is the data that counts, right? Will shows a slide often of an iceberg with the tip of the iceberg which is the conscious mind.

Bruce Sterling: Oh, I see. Yeah.

Tish Shute: And underneath it is all the preconscious stuff that really counts, right? Any thoughts on that?

Bruce Sterling: I did take interest in that. Will has obviously been spending a lot of time studying cognition.

Tish Shute: Yes.

Bruce Sterling: I’m not convinced that AR has got a lot to do with that. There is certainly a trend there. There are a lot of people who want to do body hacks and brain hacks. I can imagine AR being used for that purpose, but it seems like a niche application. What is the point of our accessing even more stuff that’s outside of our consciousness?

Tish Shute: One of the things he is talking about is game dynamics, is it? The role of the imagination in play. For example, he shows the high dynamic range photos that make the world magical. Something you want to engage with playfully. This he points out increases a sense of agency because you are encouraged to engage and to play with the world.

Bruce Sterling: Well, I’m a literary guy. Italo Calvino did a lot of writing about this. He talked about the classics of literature. Why do we read the classics? Calvino said we do not read, but reread the classics. And the reason we do that is that, at first, we read a classic book and we think, “Boy, this book is really good.” Then, five years later, we read it again and we think, “Boy, this is a really good book, and it’s got so much more in it than I thought it had when I was 18.” Then we read it again at 28, and it’s like, “OK, now I really seem to understand this book, and it means something to me now that I didn’t know when I was 18 and 25.”

What you are doing through that access is learning something about yourself. So Will is arguing is what I really need is like a better augmentation. So that I can go in there and sop up the book all at once. I can grab every cultural nuance in it, instead of the stuff that’s sliding past me because I’m 18 and kind of young and hasty. Maybe I could have certain words and phrases helpfully underlined, that are like, “OK, well, this part is problematic for you.” In some sense, that’s not allowing me to be 18.

I’m never going to have the experience of my own maturation against this text, because I’ve devoured it all in one gulp.

My argument would be that if you want people to be more sensitive toward certain, say, issues and problems, it’s better to find the people who are already sensitive to those issues and problems, and give them a bigger stake in your augmentation system.

Tish Shute: Yes the social augmented experiences are going to be the most valuable.

Bruce Sterling: Say that I am really worried about public health. Well, if I have a lot of nurses that are using my system, people who are aware of my issues, then I could be walking around and I’ll see a lot more tags saying, “This is where he got food poisoning!” “In this shooting gallery, many people have caught AIDS!” Or, you know, “Tuberculosis has been spotted over here in this building.”

At that point, I could simply share their knowledge and get some social intelligence. As opposed to trying to amp the basements of my little hacker-mind and drag stuff up that’s escaped my conscious attention.

Tish Shute: Interesting that seems to bring us to another kind of repetitive theme in AR, the people tend to pigeon hole it as “merely” a visual interface. But actually, it’s the intersection, isn’t it, of social intelligence and augmentation.

Bruce Sterling: Well, it depends entirely on how you design the system. If I’ve got a military augmented reality, I would expect that to be mostly about urban fighting. It’s going to be about kicking in a door and shooting terrorists. If I pull that helmet off my head and put that on the head of an emergency worker or a cop, I’m going to get a militarized cop or a militarized emergency worker.

Tish Shute: Well the histories of the two great mass media of the twentieth century – TV and the atomic bomb were intertwined, and I suppose the evolution of ubiquitous media, augmented reality and urban warfare is already intertwined too.  So how can we encourage augmented realities to move beyond military roots that is common to much technology and into more peaceful urban realities?

Bruce Sterling: Well, it’s a matter of deciding whose reality it is that you’re trying to augment. I’m not trying to be a bleeding heart about it, but obviously there are people in our society right now with reality that could really use some augmentation. They are mostly disadvantaged people. They are vision impaired, or maybe they have autism. They might be senile and just can’t remember where they put their shoes. These are people who could really use some help, right?

So, start with people who really need sensory or cognitive help. Before you turn our geeks into Superman, why don’t you try turning some people who are harmed into more functional individuals? Then you’ll be able to learn how to do that. Then maybe you can ramp it up to these Nietzschian heights of the superb Man With the X-ray Eyes. Whatever.

Tish Shute: Did you notice that a couple of apps actually like TagWhat have apps geared towards people with disabilities – I haven’t had a chance to check it out.

Bruce Sterling: I’m sorry, I wasn’t looking at their tags.

Tish Shute: I was discussing this with Joshua who mentioned Zachary Lieberman’s Eye Writer, which is for people with locked-in syndrome. Do you know that?

Bruce Sterling: Sure. And people appreciate that because the poor guy, he’s laid up with Lou Gehrig’s Disease. Now they’ve given him a way out. AR is like a spark of new hope that gives his life meaning. What’s wrong with that?

Tish Shute: Yeah. And Tim Byrne using Sixth Sense for Autism is interesting.

Bruce Sterling: Let’s consider it the other way. Let’s say this graffiti writer there, instead of him being sick and weak, let’s say he’s an athlete. So I want to make him into a super-human graffiti writer. I want him to run around graffiti-tagging the entire town before dawn. Is that a good idea? Do we need that? Super human, super taggers? What if he’s going to spray up stencils of Nietszche? I kinda wonder whether the game is worth the candle.

Tish Shute: Yes I suppose it is not a great social scenario to be always augmenting the lives of the elites! Hmm, the third single word interview question is “homophily,” and earlier were you’re saying that we’ve kinda got to accept this is very much part of AR – as how it works, because hyperlocal experiences gets created by local communities – that up to know have tended to be homophilies.

Bruce Sterling: Well, I think that’s easily handled with some design thinking. You’ve got to do some user observation and show some sympathy with the user, and to be aware that you’re designing for the user and you’re not designing for yourself.

In a field as young as this, it’s mostly geeks building cool stuff for geeks. In a lot of ways, it’s a “can you top this” contest. That’s OK, but it’s not good design to be your own client all the time. It’s like writing novels to amuse yourself, or sitting on the porch singing the blues on your own guitar with only yourself to hear.

Tish Shute: What will it take for AR mature out of this “geeks building cool stuff for geeks” phase do you think?

Bruce Sterling: It’s necessary to master some of the tools first. I think of the way the web has developed over the years. When the World Wide Web first appeared, it was just for physicists, and was all line commands and quite unstable and difficult. Then you got usability studies, and things like Ajax and so forth. It’s a very painstaking thing.

We’re not best at building interfaces for the best computer scientists. Web 2.0 was built from things like watching people cry while they were trying to fill out insurance forms. “Well, why are you so upset?”

“Well, I got to the end of the webpage, and then it said I took too long, and it cut me off and now I have to start all over!” Jesse James Garrett, right? Benefactor of mankind.

If you’re experienced, you think: “Why don’t I build a little module here, and kind of move the form over here, then I’ll periodically update it with some asynchronous Java and XTML.” And people are like, “Gee, how odd.” But that really works for real people. It comes from studying what people want to do. Whereas, the current AR approach to a problem like the insurance form would be like, “I will give you the ability to record the entire insurance form, and it will flash before your eyes!” OK great, that’s a cool hack, but I don’t really need X-Ray Eyes to fill out my insurance form. What I need is a more user friendly interface.

Tish Shute: Well it seems like we are moving into the terrain of Joshua’s fifth word “ventilation,” – if I understand it rightly – it is at least partially the antidote to territorialization because it’s this idea that a place needs air so we come out of our hermetically sealed boxes of the way we relate to a place and what kind of augmentation would bring more oxygen to that space.

There was an interesting moment in the Auggies because when Maarten Lens-FitzGerald presented the guerrilla shopping Layar and basically Mark Billinghurst and Jessie Schell who spoke first didn’t seem too impressed. They didn’t want to walk to shopping – that was what web shopping did, it saved us from walking to shop… but I felt, to me you picked up on something which might have some bearing on “ventilation” in that this AR shopping Layar was kind of squatting Prada – a favela chic AR shopping thing?

Bruce Sterling: I wasn’t sure if I was interpreting what Maarten had in mind by that. But I think Maarten sees his structure accurately as an experience thing rather than a mapping thing. I think he’s proudest of things like the Berlin Wall app on Layar, as opposed to Layars that help you go get a hamburger. It’s like…so when Layar inserts parasitic augmented shopping over other people’s real shopping? That was rather a subversive thing.

I think the key there is that his client is called “Hostage T-shirts,” right? I mean it’s actually kind of a transgressive little hippy T-shirt store that Layar can dump anywhere in the world. Layered right over, say, Versace and Prada. I don’t know what becomes of that effort. And I’m not sure about the term “ventilation,” because that’s a term of art I haven’t heard much.

Tish Shute: Maybe it’s like in a cafe. Ventilation would mean we were able to communicate with all these different categories of people that we normally would be unable to connect to, even though we might be sitting only a few feet apart.

Bruce Sterling: So it means ventilation in the bottles of our homophilies.

That’s not a personal problem for me. I commonly live in foreign cities and, you know, and spend a helluva lot of time talking to strangers at conferences. So I don’t think I’d have that particular tight little social island problem.

Tish Shute: Of the three judges at the Auggies, you seemed most enthusiastic about the Layar entry.

Bruce Sterling: It may be they’re not as familiar with the business models of locative AR as I am, and as Maarten is. It was kind of a subtle in-joke he was making about Layar’s own business model there.

Tish Shute: How do you explain that?

Bruce Sterling: Well, you know, Layar’s in the business of selling software to make mapping and urban structures into ecommerce.

The ideal way to do that obviously would be to move the richest customers into the most expensive shops in the most rapid way possible. Or at least distribute them in the directions they want to go, a la Google. Whereas this app that Maarten was talking about puts big barnacles in the way that are selling punk t-shirts.

Tish Shute: Right! Right!

Bruce Sterling: The Dutch are a bit subtle in their humor. I rather imagine there’s a lot of discussion in Layar’s inner circle about exactly what they want developers to do with their platform. They’re going to have considerable political difficulty deciding who can have a Layar key and how you discipline people when they start doing weird stuff. “The Oakland Medical Marijuana layar.”

Tish Shute: Well, finding nudists is one of the top layars at the moment.

Bruce Sterling: You know, obviously so. And finding narcotics in Amsterdam, or a prostitution layer. I warned them nine months ago this was bound to happen. I’m sure they’re aware of it. I don’t think Layar wants Google’s style of cool, technocratic detachment.

Tish Shute: But that’s pretty difficult to do in current augmented reality because we don’t have all the mathematical voodoo for full on AR search yet, do we?

Bruce Sterling: Well, you can hire it out. Somebody’s going to do it, if they get interested enough. There’s Nokia-Yahoo. Nokia-Yahoo! just did a big corporate deal…involving Nokia’s mapping system and Yahoo’s localization. So the Nokia-Yahoo! mash-up is called Nooo! Or could be called Yahno. Yakia! Unfortunately ridiculous names.

Tish Shute: It’s interesting because you mentioned the spiders’ mating problem at Google. They’ve got all the pieces to make this kind of level of AR obviously right now. But they actually haven’t done it yet.

Bruce Sterling: There must be at least some discussion in Google, but the same goes for Microsoft. I’m frankly baffled by Microsoft, because it’s just full of insanely brilliant people. What the hell are they doing in there? Name one serious innovation that’s come out of their labs in five years. They make Integral Research look dynamic. It’s really kind of sad.

Tish Shute: It’s a very curious situation with AR though, because AR more than any new technology relies on these big hordes of data particularly for the mapping, right? And only the big four have the data – although we are beginning to see upstarts, Earth Mine, Simple Geo… Did you get a chance to meet Di-Ann Eisnor from Waze – real-time maps and traffic information based on the wisdom of the crowd.  Waze is a very interesting project that is a potential giant killer.

Bruce Sterling: No, I didn’t talk to them. I’ve seen people speculate that Earthmine and Apple are going to make an allegiance. I guess if you’re thinking that urban informatic mapping is a super big thing for AR, that must be true. But I’m not convinced that’s necessarily the case. People have pointed out that you can just use Google Maps, and you don’t have to walk around with a little visor. There are other aspects of AR besides the cell phone space. There’s Total Immersion’s big display screens. There’s the web-based fiduciary stuff. And there’s projection mapping. And then there’s experience design just for people who need their reality augmented for whatever personal or social reason. [dog barking]

Tish Shute: Right. Oh, I’m in the middle… My son’s come. What a good hair cut!

Bruce Sterling: Hi, there.

Tish’s Son: Hi.

Bruce Sterling: How’s it going, sir? Good to see you…

Tish’s Son: Good.

Tish: [laughs]

Bruce Sterling: Yeah. Nice looking shirt. I like the back of it.

Tish Shute: That’s from the American Shaolin Temple. [laughs]

Bruce Sterling: All right. Awesome. Kung Fu geek shirt.

Tish Shute: Yup he is a bit of Kung Fu Geek. He and his dad did an iPhone app on it for Yu-Gi-Oh, for Yu-Gi-Oh scoring.

Bruce Sterling: Awesome. Plenty of Pokémon-style combat in Yu-Gi-Oh.

Tish Shute: Yeah. Well, it’s interesting because you’ve talked about this aspect. That all of this, the Pokémon aspect of AR hasn’t kicked in yet. But it’s obviously a match made in heaven to some degree, isn’t it?

Bruce Sterling: One would think so, yeah. The whole little kid gaming thing. What does that have to do with Google or Bing? You don’t need a massive database for stuff like that.

Tish Shute: Yeah, you’re right. But good tracking, mapping and registration requires a lot of mapping…

Bruce Sterling: Well, our current tracking, mapping and registration requires that. Maybe there’s some other way to hack it that we don’t know about yet.

Tish Shute: That’s a very interesting point. We always have to stretch the way we think about mapping… perhaps its a real-time understanding of the location you’re in…perhaps the map is being negotiated through several social processes?

Bruce Sterling: There are maps, and then there are maps. There’s a kind of artillery map where you need to know the precise location of target spaces. And then there’s the kind of social map where I’m really looking for the IN-N-OUT Burger where my sister went last Tuesday. That’s a different system.

Tish Shute: And I think AR, at the moment, we’re getting the most out of the social maps certainly. And the other [machine perception technologies to detect the identity and physical configuration of objects relative to each other to accurately project information alongside/overlaid with a physical object] is still kind of the big dream, isn’t it?

Bruce Sterling: They say that men never ask for directions and women never read maps. Clearly, the genders have different ways of navigating the world. Who’s to say what manner of augmenting our experiences is hottest? I’m not convinced that today’s rather rigid geolocativity is really what our society wants from that particular service. Maybe what we want is something more folksy. Some useful nudge in the right direction as opposed to grids with 200 meters here and instructions to turn such-and-such.

Besides, there’s other hacks we haven’t considered. We’re very dependent on GPS, but just suppose all those satellites are blown out of the sky in a solar storm. Would we really want to give up mapping? Wouldn’t we just come up with some other nifty hack? Radio beacons, let’s just say. Atomic clock timers in towns. Or maybe just little QR codes on lampposts that give you the exact location of that lamppost, and just click the thing and have it calculate where you are.

Tish Shute: Yes the NextHope OpenAMD project had a clever way of triangulating location indoors.

Bruce Sterling: Well, GPS is there and people all want to use it. It’s got good API, so of course you want to. And the guys who are good at doing it are real geolocative freaks. But the mere fact that we can do it this way, and that you can make it pay, doesn’t mean that it’s the ultimate way to provide that service to people. It’s like saying that Egyptian hieroglyphics must be the greatest way to write, because we’ve got a lot of them and they’re hard to learn. What if somebody comes along with an alphabet? It’s going to be a little embarrassing.

Tish Shute: Yeah, that’s a very good point. Now, this is a more simple ordinary question about the event. YDreams went off the map in the Auggie voting, and walked away with The Auggies. No one doubted that that was the most…

Bruce Sterling: I don’t know. I thought those Occipital guys with the panoramic painting…. That was hairy. I would have been tempted to give them the prize myself, actually.

Tish Shute: And what did you like best about that? Because I agree. I love Occipital.

click to enlarge

Bruce Sterling: I thought it was a more technically difficult stunt than the hand registration thing. Using a hand as a 3-D cursor is hot, but not like painting a panorama in 3-D in real time. That was an impressive technical feat.

Tish Shute: And they hinted at the 2.1.1 AR, more AR version of that. What do you see coming out of that as possibilities?

Bruce Sterling: Well, I’d heard of YDreams, so I wasn’t stunned. But I’d never heard of those guys. I wonder what else the heck they’ve got in the attic.

Tish Shute: very cool stuff…

Bruce Sterling: Well, more power to them. But clearly YDreams was the popular favorite. And who couldn’t like it? It was just so AR.

Tish Shute: It was so AR and so gorgeous.

Bruce Sterling: It was pretty, actually.  Except for their ugly menu button and poor font choice.

Tish Shute: Oh, yes. You didn’t like that, did you? [laughs] But with the Occipital panorama, what do you see the next stage of that?

Bruce Sterling: Well, obviously quicker and faster. Quicker and faster and more accurate in a network. Let’s just say I’m in New York and you’re in New York and I’m calling you for help. And you say where are you? I just whirl around like this and I mail it to you on a Google Wave. And you whirl around like that, and then we compare the two panoramas and do an instant triangulation. And you say: I’m over here on this red dot of your screen.

Tish Shute: Yeah, exactly.

Bruce Sterling: We’re navigating with panoramas by having two connected panoramas and considering the difference.

Tish Shute: Yeah, very interesting…

Bruce Sterling: Not shabby, right?

Tish Shute: Not shabby at all.

Bruce Sterling: If you could do it in real time.

Tish Shute: Then the other thing I missed because I was going to meet Will was I missed the Launch Pad competition. Did you catch that?

Bruce Sterling: I didn’t see it either. I thought of another app though.

Tish Shute: Oh!

Bruce Sterling: You’ve got a panorama maker in your home office, and it just scans the office 24 hours 365 and tags anything that moves, right? OK, where’s the clipboard?  At 8:15 it was over here. Now it’s vanished. Now another object is viewed over here. So, logically, ping, you hit it with a sticky light and there it is, right?

Tish Shute: Oh, that’s cool also knowing what has changed in any environment would be a big enabler for a lot of AR visions.

Bruce Sterling: I’m sure there are many other things you could do with panoramas.

Tish Shute: My jet lag is beginning to kick in big time – so many ideas to pursue from are2010 – those panoramas are very exciting though.

Bruce Sterling: Oh, well, it’s all right. We can augment reality! I’ve got three heads and six hands!