I’m really excited that we opened a call for proposals today for Augmented World Expo (registration opens February!).  Our edgy conference on augmented reality has morphed into the world’s first Expo about the augmented world.  If you loved ARE you are going to find Augmented World Expo the most important event of 2013, and if you never got a chance to attend before register early to reserve your spot!

“The way we experience the world will never be the same. We no longer interact with computers. We interact with the world. A set of emerging technologies such as augmented reality, gesture interaction, eyewear, wearables, smart things, cloud computing, and ambient computing are completely changing the way we interact with people, places and things. These technologies create a digital layer that empowers humans to experience the world in a more advanced, engaging, and productive way.

Augmented World Expo will bring together the best in augmented experiences from all aspects of life: health, education, emergency response, art, media and entertainment, retail, manufacturing, brand engagement, travel, automotive, and urban design. It will be the largest ever exposition demonstrating how these technologies come together to change our lives and change the world.

Registration will open in February.”

I am super excited to be speaking at SXSW Interactive 2012, as part of Will Wright’s Stupid Fun Club, on “A Lifestyle with a Gaming Sense.” Michael Trice just did a post on our session for SXSW.com, Screen Burn Panels at the Palmer Presents Will Wright’s Stupid Fun Club. The photos of Will Wright, Tish Shute (me!) and Peter Swearengen are by Anya Zavarzina. Thank you Anya for such great photos!

I have been too busy to blog much lately, but there is a lot to unpack in future posts in my quote in Michael’s SXSW post!

“Really we’ve entered a new era where the world has become a platform for storytelling and the goal is to turn everyday life into an opportunity for play, relatedness, and new forms of autonomy and fun. We’ve now come to a point where software has moved out of the computer and into the world. Rather than viewing this process in terms we’ve already grown out of, like gamification, we view this as an opportunity to explore everyday activities as possibility spaces.”

For the complete post, including Peter Swearengen of Stupid Fun Club on StoryMaker, see here http://sxsw.com/node/9969

The Points of Control map is interactive, so please click here or on the image above for the full experience.

Today at 4pm EST, 1pm PDT John Battelle and Tim O’Reilly will discuss the Points of Control map and The Battle for the Internet Economy in a Free Webcast:

“More than any time in the history of the Web, incumbents in the network economy are consolidating their power and staking new claims to key points of control. It’s clear that the internet industry has moved into a battle to dominate the Internet Economy.

John Battelle and Tim O’Reilly will debate and discuss these shifting points of control as the board becomes increasingly crowded. They’ll map critical inflection points and identify key players who are clashing to control services and infrastructure as they attempt to expand their territories. They’ll also explore the effect these chokepoints could have on people, government, and the future of technology innovation.”

I’ve been wanting to start a discussion on the  Points of Control map in the Augmented Reality community for a while now, and Chris’ recent post on the latest edition of the Gartner Hype Cycle, “Is AR Ready for the Trough of Disillusionment?” and this post by Mac Slocum, “How Augmented Reality Apps Can Catch On,” and the conversation in the comments between Mac, Raimo (one of the founders of Layar), and Chris, all prompted me to get a conversation started…(see below for all that followed!).  Chris put me on the hot seat back in June when he did this very generous interview with me on Boing Boing, so it was time to turn the tables.

Tim O’Reilly, in his  keynote for Web 2.0 Expo, pointed out there is both a fun and a dark side to the Points of Control map.  There are companies on this map, he noted, that rather than “growing the pie,” are trying to divide up the pie, and they are forgetting to think about creating a sustainable ecosystem. I expect the conversation between Tim O’Reilly and John Battelle to dig deep into this Battle for the Internet Economy.  If, like me, you have another engagement at the time of the webcast, you can register on the site to receive the recording.

AR is still too young to figure in the battles of the giants, but there will be a lot to be learned from this conversation.  And, The Points of Control map is good to think with from the POV of AR in many ways.  As Chris Arkenberg observed:

“When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?”   You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development.  And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.”

Chris Arkenberg is a technology professional with a focus on product strategy & development, specializing in 3D, augmented reality, ubicomp and the social web. He uses research, scenario planning, and foresight methodologies to help organizations anticipate change and adopt a resilient and forward-looking posture in the face of unprecedented uncertainty. His personal work is collected at urbeingrecorded, and his professional profile is here.

He is also one of the founder/organizers of AR DevCamp which is currently scheduled for Dec. 4th (somewhere in SF or The Valley!)  Chris said, “No further details atm (still trying to find a venue and get sponsors) but please direct people to http://ardevcamp.org for upcoming information.”

Talking with Chris Arkenberg

Tish Shute: I know some people thought the positioning of AR by Gartner near the peak of the hype cycle was misguided, and based on a very narrow understanding of AR as used in marketing apps. But reading your post I thought you made a lot of good points.

Chris Arkenberg: It’s tracking hype, right? It’s not necessarily tracking the growth of the technologies or their maturation so much as it’s tracking the general attention level. And what’s interesting to me is that tends to affect the amount of money that goes into those technologies.

Tish Shute: I was particularly interested in your post because I have been writing a post about two recent O’Reilly events in NYC, Maker Faire, Web 2.0 Expo, and then Hadoop World, where Tim gave a very interesting 45 minute keynote.   AR was pretty low profile at all three events.   But the NVidia augmented reality demo attracted a lot of attention at the sponsors expo, and Usman Haque, Founder of Pachube announced in his presentation, they are working on an augmented reality interface for Pachube called Porthole, its designed for facilities management and, “as a consumer-oriented application that extends the universe of Pachube data into the context of AR – a ‘porthole’ into Pachube’s data environments.. “  Usman also mentioned, when I talked to him, that he is contributing to the AR standards discussion and on the program committee now for the W3C group on augmented reality.  For more on this standards discussion and the Pachube AR interface, see Chris Burman’s paper for the W3C, Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual.”

I think pioneers in the augmented reality commmunity should pay attention to these wider conversations about the Battle for the Internet Economy, and the exploration of the  “Platforms for Growth” theme at Web 2.0 Expo is very important- this is a course also a nudge to read my upcoming post on these O’Reilly events!

Also I have another project I have been chewing on that I would like to talk to you about.   I want to start an AR conversation about the wonderful Points of Control map produce for Web 2.0 summit by John Battelle. [ Note there will be, "Battle for the Internet Economy" free Web2Summit webcast w/ @johnbattelle & @timoreilly Wed 10/27 at 1pm PT http://bit.ly/b46cmb #w2s]

Up to this point, understandably given the immaturity of the technology, AR has little role in the “Battle for the Internet Economy.”  But this doesn’t mean that the map isn’t good for AR visionaries, enthusiasts, entrepreneurs, and developers to think with.   And both you and Tim have pointed out the potential for AR to leverage the giant data subsystems in the sky.   I have to say the positioning of Cloud Computing on the brink of heading down into the trough of disillusionment in this recent rendition of the Gartner Hype Cycle seems ridiculous!

Cloud Computing is already ubiquitous hardly seems credible that it is headed for a trough of disillusionment!

Chris Arkenberg: Yeah, it’s ubiquitous so why even talk about it when it’s your fundamental infrastructure?

Tish Shute: Yeah and I seriously doubt it is imminently headed for a trough of disillusionment….and this brings me back to the Points of Control Map which as John Batelle points out, “aims to identify key players who are battling to control the services and infrastructure of a websquared world” in which the “Web and the world intertwine through mobile and sensor platforms.”  This instrumented world, of course, creates a great deal of opportunity for augmented reality. Have you seen that, that points of control map?

Chris Arkenberg: I think I have, actually.

Tish Shute: There has been much debate about how this intertwining of the web and the world will play out in augmented reality.   Chris Burman points out in his position paper for W3C,  Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual”, that “trying to draw parallels between a browser based web and the possibilities of AR may solve issues of information distribution in the short-term,”  but it must not have a limiting effect in the long-term.   But now we at least have one web standards-based browser for AR thanks to the work of Blair MacIntyre and the Georgia Tech team.  But I think the discussion in the comments of Mac Slocum’s recent post, “How Augmented Reality Apps Can Catch On” is an interesting starting point from which to think about platforms of growth for AR.  I am not sure if I am stretching his meaning but I think Raimo, Layar, is suggesting that what the Point of Control map call the the Plains of Media content is very important to the growth of the fledgling AR industry right now. And I would agree with this, and add that the neighboring terrain of gaming levels will be pretty key as one of my other favorite AR start ups Ogmento hopes to reveal in the near future! But what do you think was most important in this brief but pithy dialogue between you Raimo and Mac?

[The screenshot above is a teaser video the Gary Hayes from MUVEdesign for his upcoming (2011 release date), game called Time Treasure.  See Gary’s blog for more and Gary’s  post from over a year ago on AR Business models.  Thomas K. Carpenter, on Games Alfresco notes, “I think this is a terrific idea and I find it interesting he’s planning this on a tablet rather than a smartphone.”

Chris Arkenberg: The way I took it…And to give a little bit of context, I came from sort of this apprehension of augmented reality as an expression of the existing Internet. So as sort of a visualization layer that allows you to kind of draw out data, and then, with all the affordances of being able to anchor it to real world things.

And my own sort of path has led me to want to really try to understand that and refine it, particularly with respect to the sort of Internet of things and the smarter planet idea of just having embedded systems everywhere. And specifically, what is the value-add for augmented reality as a visualization layer of an instrumented world?

And so that’s caused me to be a bit biased towards that side of AR. And the way I took Raimo’s comment was that he was saying that, “You know, really what we’re interested in is media.” That he was effectively saying that AR for them is really just about that space between the screen and the the world, or between your eyes and the world, and what you can do there.

Certainly I had considered it in the past, but I hadn’t really focused on it or assumed that it was a priority as a business model. And so he kind of reminded me that, actually, there’s a lot of entertainment applications. There’s a lot of, obviously, advertising and marketing applications.
And so I felt that I was being a little narrow in my focus…

Tish Shute: Yes this comes to the heart of what I am interested in about the role AR can play in opening up new relationships to the world of data that we live, not just making it more accessible and useful to us when and where we need it, but AR as a road to reimagining  it..

Have you seen any interesting work yet to explore these great data economies in the cloud through AR. I mean can you think of any others – there is planefinder.net but others?

Chris Arkenberg: I’ve seen a few just sort of skunk works type applications that people have been playing around with, again, to try and reveal things. One of them was similar to the aircraft, but it was more for military use and being able to identify things of interest in the sky. I’ve seen a couple other for navigation, so being able to identify mountain peaks on a visual plane, for example, but this isn’t so much about revealing an instrumented world.

Tish Shute: Yeah, I think that was from the Imagination right? I know that’s an interesting one. Usman at Web 2.0 Expo, in his presentation, mentioned the work Pachube is doing on an Augmented Reality interface. I interviewed Usman again as my last long interview with him was nearly 18 months ago now and Pachube is well on the way to becoming the Facebook of Data or the analogy that Usman prefers – the Twitter of sensors!

Chris Arkenberg: Hmm, interesting.

Tish Shute: And to go back to your comments on Augmented Reality not getting caught in some of the traps that have made virtual worlds lose relevancy I think that is vital that AR developers understand the strategic possibilities of key points of control in the internet economy because the isolation and Balkanization of virtual worlds were certainly a factor in their rapid slide into the trough of disillusionment – although many would argue that a fundamental flaw in the kind of virtual experience that Second Life and other virtual worlds constructed was really the fatal flaw (see James Turner’s interview with Kevin Slavin Reality has a gaming layer).

But Second Life’s isolation from the other great network economies of the internet was certainly a limiting factor.

Chris Arkenberg: And that’s been exactly my sense, and I’ve, over the years, tried to encourage development in that direction for virtual worlds. I did work, through Adobe, to help develop Atmosphere 3D back in the the early 2000’s. And we did a lot of work to try and understand the marketplace and the specific value-add of doing things in 3D over 2D.

And this is kind of why I keep referring back to VR and VW’s with respect to augmented reality, is that with immersive worlds, there was this idea…there was this big rush. Everybody was so excited about it. It was obviously the next cool thing. And everybody wanted to try to do everything in it. You could do your shopping in virtual worlds. You could have meetings in virtual worlds.

Tish Shute: and shopping, yes ..that didn’t work out so well!

Chris Arkenberg: And everybody was very excited in developing these things. And what it really came down it is, “Yeah, you can, but it’s actually a lot better to do those things on a flat plane or in person.” Meeting Place, WebEx, TelePresence – those tools generally do a much better job at facilitating TelePresence meetings than a virtual world does. The same with TelePresent Education. There are only very specific things that both VR and AR are really good at.

And that’s where I find myself with augmented reality right now, trying to really pick through that and critically look at which uses are really appropriate for an AR overlay. And again, I think that’s why the hype cycle is important, because it reflects back this desire that AR is going to be the next big thing – the be-all, end-all of interacting with data in the cloud – and forces us all to take a critical look at why we should do things in AR instead of on a screen.

AR is not going to work well for most things but it’s going to be very good for certain uses. Right now I’m very keen at trying to understand what those things might be.

Tish Shute: I had this wonderful conversation (more in an upcoming post) with Kevin Slavin one of the founders of Area/Code at Web 2.0 Expo and I think some of what he describes about the data brokerages of High Frequency trading have some interesting implications for AR’s role, say, in ubiquitous computing. The trading markets are now pretty much dominated by machine to machine intelligence; machine to machine brokerages. They are basically game economies on the scale that we can barely wrap our heads around where the speed that bots and algo traders can access the network is the key. We really have no clue what is going on until we lose our house…

Kevin was also interviewed by James Turner on O’Reilly Radar. He talked about how much of the interesting work in location based mobile social apps is defined in opposition to the model of Second Life. He also talked to me about how we are seeing “first life” take on the qualities of “second life.” What goes on the trading floor is largely a performance secondary to a more important world of machine intelligence with giant co-located servers and bots fighting for trading advantages measured in fractions of seconds.

He pointed out how we draw on all these tropes from sci-fi movies, these HUDs based on ideas of machine intelligence where the robot talks to the other robot in English through an English HUD!  Many of our current visual tropes for AR are perhaps just as inadequate for the kind of data driven world we live in.

Of course, when you are thinking of having fun with dinosaurs, or illustrated books, or whatever, this is not, perhaps, an issue.  But if you are thinking of augmented reality interfaces as being important in a battle for network economy, and platforms for growth,  how this new interface helps us live better in a world of data is an important issue.

Chris Arkenberg: Now, does that indicate that the UI just needs more overhaul and innovation, or more that the visual interface for those experiences shouldn’t really leave the screen? It shouldn’t move on to the view plane?

Tish Shute: Yes we have a few concept videos that try and explore this ..

Chris Arkenberg: Well, and I think this will happen at the level of human-computer interface. I mean that’s always been its role, in making coherent the sort of machine mind, for lack of a better term, making it coherent to the human mind. So I mean there is a lot of this sort of machine intelligence, the semantic Web 3.0 revolution, where it really is about enabling machines, and agents, and bots to understand the content that we’re feeding them.

But at the end of the day, they, for now, need to be providing value to us human operators. So there’s always going to be a role for human-computer interface and user experience design to make this stuff meaningful.

I mean, if you look at the revolution in visualization & data viz, this is of incredible value because it takes a tremendous amount of data and collates it into a glanceable graphic that you can look at and immediately comprehend massive amounts of data because it’s delivered in a handy, visual way.

So I see that as a fascinating design challenge, how the user experience of the data world can be translated into meaningful human interaction.

Tish Shute: Yeah. And when we see Stamen Design pursuing a big idea in AR, that’s when we might start to rock and roll, right?

Chris Arkenberg: Yeah. In my article, I sort of jokingly suggested that Apple will create the iShades. But, they’ve got the track record of being way ahead of the curve and delivering the future in very bold forms.

Tish Shute: A key part for the battle for the network economy is to bring the complexity of data into the human realm in a way that increases human agency. Kevin suggests that the giant robot casinos of markets should actually lift off into total abstractions as theses machine-driven trades get back into the human realm in ways that are so damaging to our lives – a lost house or job! The notion of a counterveillence society where people have more agency over the important aspects of their lives, health, housing, job (which I discussed with Kevin – interview upcoming) has gotten pretty tricky!

But I think we will begin to see AR eyewear for specific applications (gaming and industrial) get more common fairly soon – possibly as smart phone accessories.

And it is clear that AR is going to be, increasingly,  a part of our entertainment smorgesborg in coming months. Itouch has a camera (although lower resolution), Nintendo’s are AR-ready and many aspects of the AR vision of hands-free spatial interfaces will go mainstream through Natal.

But we are yet to see an app/platform emerge for mobile. Social AR games that turn every bar and cafe and ultimately the whole city into a gaming venue  -although I think Ogmento and MUVE aim to lead the way here! Will an AR company achieve Zynga level success by using the Foursquare, for example?

My feeling is that the lesson of Zynga is pretty important for mobile social AR games. Could Flash social gaming have taken off without Facebook?

Chris Arkenberg: And that’s the real driver. And again, as you mentioned with Second Life, and this was exactly my own sense, is that they stuck to the closed garden model and didn’t get the power of social and collaboration. They attempted to add some of those affordances within the world, but, you know, ultimately most people aren’t in virtual worlds, and most people aren’t using augmented reality. So leveraging the really predominate platforms like Twitter and Facebook and Foursquare, being able to leverage those affordances, that connectivity, into a platform like augmented reality, I think, is really critical. Because again, you get nothing unless you have the masses, unless you have people present.

Tish Shute: In AR research there is a long history of the notion of powerful AR-dedicated devices, but smart phones and tablets are good enough,  and can launch augmented reality into the heart of the internet economy. I think  the elusive AR eyewear will come to us initially as a smart phone accessory for specific apps.  But, for the moment, most AR apps make little attempt to play in the wider internet economy.

Chris Arkenberg: And I think it’s actually much lower hanging fruit, really, to do gaming, marketing, transmedia. Because then you don’t really care about the cloud, or maybe you only really care about a little part of it that your gaming property is addressing. Then it becomes much more about entertainment, and much more about persuasion, and sensationalism. And if you’ve got dancing dinosaurs on your street, great! It’s entertaining, it’s cool, it’s new. That stuff is fairly straightforward.

I keep coming back to this idea of, you know, the instrumented city. What sort of data trails do you get out of a fully instrumented city? So maybe you get traffic patterns, maybe you get geo-local movements of masses, maybe you get energy usage, that sort of thing, all the, sort of heat maps you can generate from a city. But then what good does it do to be able to have that on an augmented reality layer versus just looking at it on a mobile device or looking at it on your laptop?

Tish Shute: Of course the use cases for “magic lens” AR are different from the kind of hands free, 360 view with tightly registered media, that a full vision of AR has always promised. The 360 view is quite a different metaphor from the web and mobile rectangular screens.

Chris Arkenberg: Yes, yes.

Tish Shute: Did you see that great parody of Michael Jackson’s “Beat It” with the iPads versus the iPhones, right?

Chris Arkenberg: Oh, really?

Tish Shute: I tweeted it cos i thought it was quite funny and a little close to the bone!
[laughter]

“ur wanna an ipatch 2 b the new fad?” #AR gets cameo in Twitter, iPads & iPhone’s Michael Jackson-Inspired Parody via @mashable

It is hard to get away from the importance of eyewear when discussing AR!

Chris Arkenberg: Yes, so the hardware, to me, is a big stumbling point right now, or it’s a large gating factor, I think, for realizing what an augmented reality vision could really be like. That it really does need to be heads up. This holding the phone up in front of you is fun to demonstrate that it’s possible, and it’s valuable in some ways…

Tish Shute: And it’s particularly nice in some applications like the planes app, the Acrossair subway app where you hold the phone down and get the arrow, right?

Chris Arkenberg: Yeah, the way-finding stuff I think is really valuable…

Tish Shute: Sixth Sense really caught people’s imagination because it managed to deliver the gesture interface with cheap hardware, even if projection has limited uses (no brightly lit spaces or privacy for example!).

The other important and as yet unrealized part of the AR dream is real-time communications. Many interesting uses cases would require this. As you know that is my chief excitement, along with federation, in the Google Wave Servers for (which should soon be released at Wave in a Box) for ARWave.

Chris Arkenberg: Well my sense of Wave is that it was a ChromeOS protocol that they instantiated, or that they exhibited in the public deployment of Google Wave. That that was a proof of their sort of low level architectural solution. Because, you know, they’ve been rumored to be working on this cloud OS for some time. And so my sense is that Wave is actually one of their core components of that cloud OS, and that it just happened to incarnate for the public in a test run as Google Wave.

Tish Shute: I do hope that Wave In the Box will lower the barriers to entry to people experimenting with this technology. The FedOne server was just way too hard for most people to take the time to set up. Of course, it is the brilliance of the Wave Operational Transform work that also poses problems in terms of ease of use. But Wave Federation Protocol is pretty innovative. And could even play an important role in a real time communications for AR eyewear connected to smartphones. The challenges that Wave takes on re real-time communications, federation, permissions and filters are pretty important ones for AR…

Chris Arkenberg: Especially when you’re trying to federate a lot of permissions and filter a lot of data, which all of that gets even more important when you have a visual layer between you and the real world.

Tish Shute: You got it. Yeah!

Chris Arkenberg: I think that’s really valuable real estate, both for third parties that want to get access to your eyes, as well as for you, as the user, who still needs to navigate through the phenomenal world and not be occluded by massive amounts of overhead data.

Tish Shute: Yes, I am sure Google has big plans for the next level of cloud computing and Wave looks at some key challenges. I suppose federation poses some key business problems. I think it was Michael Jones who said to me that it was a bit like socialism in that you have to be willing to give something up for the greater good.

Perhaps federation does not present enough appeal because of its challenges re business models?

Chris Arkenberg: Well, I wonder. I mean there’s got to be some value for their ad platform as ads are moving more towards this personalized experience. Advertising is becoming less of a shotgun blast and more of a very precise, surgical strike. So being able to track user data to such a fine degree to mobilize the appropriate ads around them wherever they are, on any platform, is certainly very valuable to Google and their ad ecology.

Tish Shute: Many people have high hopes that HTML 5 by lowering the barrier of entry for  browser style AR could also pave the way for some interesting AR work..

Chris Arkenberg: Well, as much as I would hope that all the different players are going to come together and establish some shared set of standards, really, what’s happening is it’s a rush to the finish line to be the first…to get the most penetration in the marketplace so that Layar, for example, can say, “It’s official. We’re the platform.” And then the consolidation that will follow, where the Googles and the other big players like Qualcomm say, “OK, it’s mature enough. We’ll start buying up all the smaller companies.”

And that’s where the real challenge is right now is that there are no standards. It’s such an immature technology that you have a lot of different players trying to establish the ground rules. And again, this is one of the challenges that faced public virtual worlds, is that you had a lot of different virtual worlds that weren’t talking to each other in any particular way, and that they each had their own development platform. And so you end up with a very fractured ecosystem or set of competing ecosystems, which is kind of what’s happening with AR right now, where a developer has to choose between a number of different new platforms or hedge by deploying across multiple platforms. Basically, the web browser wars are set to be recapitulated by the AR browsers.

Among them, Layar and Metaio seem to be getting the most traction. But there’s still not a really strong case for a unified development ecosystem to emerge.

Tish Shute: So a discussion of ecosystem development brings us back to the Points of Control Map I think. So what do you see as key points of interest for AR developers to watch in the Points of Control Map? And where do you want to sort of put your bets, right? We are still really waiting for mobile social AR to emerge into the mainstream.

Chris Arkenberg: Yes. And that’s primarily the shortcoming of the hardware itself, but also of the accuracy of current GPS technology. That’s another kind of gating factor, because again, AR wants to be able to express the data within a distinct place or object.

So in a lot of ways, other than kind of what we’ve allowed for the broader entertainment purposes, for AR to really work, there needs to be more resolution in GPS location. So for it to be truly locative…because it’s OK to tell Foursquare that you’re in Bar X. But if you want to be able to draw data directly on a wall within that bar, or do advertising over the marquee on the front, you need more factors to accurately register those images on a discreet location. So that’s another, sort of, aspect of the immaturity of AR, is that it’s still very hard to register things on discreet locations without employing a number of diverse triangulation methods.

Tish Shute: Right. The mobile AR games we see at the moment are really just faking a relationship to the physical world unless they rely on markers or some limited form of natural feature recognition which is really just a more sophisticated form of markers. But the Qualcomm SDK does offer some opportunities to tie AR media to the world more tightly as does the Metaio SDK. But in terms of a mobile social AR game that could be like the Cape of Zynga to FourSquare in Location Basin [see the Points of Control map]… We haven’t seen anything close yet.

AR should be able to bring the check-in mode to any object in our environment.

Chris Arkenberg: Yes, yes. And that’s actually one of the early interests I had in the notion of social augmented reality. I wanted a way to tag my community with invisible annotations that only certain people could read, and found pretty quickly that that’s very difficult to do. I mean you can kind of do some regional tagging, like on a beach, for example, but if you wanted to tag the bench that was on the cliff above the beach, it’s very difficult to do that using strictly locative reckoning.

There’s all sorts of really cool social engagement that can be revealed when people are allowed to attach things to the world around them, to the streets they normally pass through, or the points of interest that they normally engage in. To be able to author on the fly on the streets and attach it discreetly to an object effectively.

Tish Shute: And yes we do have all kinds of markers and QR codes. But Erick Schonfeld of Tech Crunch made a good point that QR codes: “Until QR code scanners become a default feature of most smartphones and they start to become actually useful enough for people to go through the trouble to scan them, they will remain a gee-whiz feature nobody uses.”

Chris Arkenberg: So again, this gets back to competing standards and who gets access to the phone stack, the bundle. Who gets the OEM deal…?

Tish Shute: Yes, the battles for the networks on the Handset Plains are pretty important for AR!
[laughter] I think Layar have made some smart moves on The Handset Plains.

And there are a lot of acquisitions of nearfield technology to look at.  If I remember rightly Ebay bought the Red Laser tech from Occipital – now there’s any interesting company. Their panorama stuff rocks!

Chris Arkenberg: Right. There’s a lot of nearfield stuff that’s supposed to hit all of the major mobile platforms in the next year or so.

I mean I think where this is heading, in my mind, is basically smart motes. You know, little nearfield wide-range RFID’s that are the size of a small, tiny square that you could attach to just about anything and then program it to be a representative of your establishment or of an object, that then you can start to tag just about anything. I mean you can’t rely on geo to do it, but if you have a Nearfield chip there that costs maybe like two cents to buy in bulk, and you can flash program it, then you can start to attach data to just about anything.

Tish Shute: Yes ‘cos some things still remain very difficult for near field image recognition technologies like Google Goggles.

Chris Arkenberg: Well, if your phone can interrogate for Nearfield devices, and it detects a chip in its near field, it can then interrogate that chip. The chip may contain flash data on itself, or it may contain the local server in the establishment, or it may go to the cloud and get that data back.

Tish Shute: Yes there is moverment from the top and open source hardware like Arduino has created an opportunity for all sorts of creativity with instrumented environments.  And the handheld sensors in our pockets – our smart phones create a lot of opportunity for bottom up innovation too.

Chris Arkenberg: I mean that’s my guess. If you look at what IBM is doing with their Smarter Planet initiative, they’re partnering with a lot of municipalities, and obviously with a lot of businesses and their global supply chains.

But they’re basically working with municipalities and all these stakeholders to instrument their territory, their business, or their city, as it were. So they’re working to provide embedded sensors and the software necessary to read them out and run reports & viz. And presumably that software can extend to include some sort of mobile device to interrogate the sensors and read the data.

That’s kind of a top-down approach of a very large global company working with top-down governance bodies to do this. Simultaneously you have the maker crowd experimenting with Arduino and such to build from the grassroots, the bottom up approach.

And that’s primarily gated by the amount of learning it takes to be able to program these devices, to be able to hack them. Typically, the grassroots creators who make these devices don’t have the luxury of very large budgets to make things highly usable and Wizywig.

So the bottom up community is a sandbox to create tremendous amounts of innovation, because they are unconstrained by the very real financial needs of the top down innovators. And so you get a lot of fascinating innovation, a very rich ecology from the bottom-up approach, but you don’t get a lot of wide distribution. But that does filter up to and inform the top down approach that has a lot more money to put into this stuff. And it ultimately has to respond to the needs of the marketplace.

I mean if there’s an answer to the question of whether something like AR will succeed through the bottom-up grassroots approach or the top-down industry approach, I would say it would be both. That handsets will be hacked to read the bottom up innovations of the maker community, and handsets will be preprogrammed to read the top down efforts of the IBMs of the world.

Tish Shute: Yes but i have to say it is very time-consuming hacking phones (I have just seen a few days suck up in this myself so that I could upgrade my G1 to try out the new ARWave client!). I mean Android has obviously been the platform of choice because of openness but the business model of iPhone and its market share in the US sure make it important for developers.  It’s like you don’t exist if you don’t have an iphone app for what you are doing.

Chris Arkenberg: Yeah, and that’s the challenge, because at the end of the day developers prefer not to work for free and a solid, reliable mechanism to monetize their efforts becomes very appealing.

When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?” You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development. And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.

Tish Shute: Some of the important platforms for AR don’t appear to have spots on the map like Google Street View and other mapping technologies that hold out so much hope for AR, or am I missing something?

Chris Arkenberg: You mean on the map?

Tish Shute: Yes for the full vision of AR we need sensor integration, computer vision and cool mapping technologies to come together. Do you see where Google Maps and Google Street View… Where would they be?

Chris Arkenberg: Yeah, I mean it’s certainly content, it’s location…

Are you familiar with Earthmine?

Tish Shute: Yes, yes I am, definitely. Earth Mine, Simple Geo, Google Street View, user generated internet photo sets like Flickr all of these could be very important to AR, potentially.

Chris Arkenberg: Well, and the interesting thing about Earthmine is that they’re effectively trying to do an extremely precise pixel to pixel location mapping. So they’re taking pictures of cities just like Street View, except they’re using the Z axis to interrogate depth and then using very precise geolocation to attach a GPS signature to each pixel that they’re registering in their images. Effectively, you get a one-to-one data set between pixels and locations. And so you can look at something like Google Street View, and if you point to the side of a building, in theory, it should know exactly where that is.

They’re rolling this out with the idea of being able to tag augmented reality objects in layers directly to surfaces in the real world. So that’s another approach to trying to get accurate registration and to try and create what are essentially mirror worlds. Then your Google Street View becomes a canvas for authoring the blended world, because if you plop a 3D object into Street View on your desktop, and then you go out to that location with your AR headset, you’ll see that 3D object on the actual street.

Tish Shute: There was some experimental work with Google Earth as a platform for a kind of simulated AR but I suppose Google Earth doesn’t figure in the battle for the network economy as it never got developed as a platform.

Chris Arkenberg: It hasn’t tried to become a platform, to my knowledge. I mean I know some people are doing stuff with it, but as far as I know, Google owns it, they did it the best because they have the best maps, and there’s not a huge ecosystem of development that’s based around it other than content layers.

And my sense of everything else on the Points of Control map is they’re looking more at these sort of platform technologies that…

Tish Shute: Yes, re platforms for growth for AR. Gaming consoles will probably emerge as a significant platform for AR this year.

Chris Arkenberg: There will be much more of a blended reality experience in the living room for sure, and with interactive billboards. Digital mirrors are another area. So I mean if we kind of extend AR to include just blended reality in general, you know, this is moving into our culture through a number of different points. As you mentioned, it will be in the living room, it will be in our department stores where you can preview different outfits in their mirror. We’re already seeing these giant interactive digital billboards in Times Square and other areas.

It’s funny. I mean for me, the sort of blended reality aside, the augmented reality, to me, is actually a very simple proposition in some respects. When I look at this map, augmented reality is just an interface layer to this map in my mind, just as it’s an interface layer to the cloud and it’s an interface layer to the instrumented world. It’s a way to get information out of our devices and onto the world.

Tish Shute: The importance of leveraging existing platforms has become pretty clear but it is interesting Facebook definitely gave Zynga the opportunity but would Facebook be so big without Zinga’s social gaming boost?

Chris Arkenberg: I feel that Zynga has definitely helped its growth…But I think Zynga has benefited a lot more from Facebook than Facebook has from Zynga.

Tish Shute: Zynga certainly proved you could build a profitable business on Facebook’s API!

Chris Arkenberg: They did. And they also really validated the Facebook ecosystem and the platform. They really extended it… Zynga benefited from the massive social affordances that Facebook had already architected and developed. They brought gaming directly into Facebook, and particularly, this emerging brand of lightweight social gaming that when you sit it on top of a massive global social network like Facebook, it suddenly lights up.

Tish Shute: AR pioneers should quite carefully go through this map. There is so much to think about here. I’m a kind of fanatic about Streams of Activity in AR. Real time brokerages and their potential for AR is something I am fascinated by. That is one reason I love the ARWave project.

Anselm Hook, to me, is one of the great thinkers in this area of real time brokerages – with his project Angel, and the work of Ushahidi, which is now the platform for augmented foraging (see here). Anselm is now working on AR at PARC which is exciting.

Chris Arkenberg: Well, there are some challenges working with data streams. Presentation and filtering I think is a big challenge with any sort of stream. Because obviously, you have a lot of potential data to manage, to parse, and to make valuable and comprehensible. So I think this is bound very closely to being able to personalize experiences, or having very discreet valuable experiences. Disaster relief, for example, I think is an interesting idea that ties into the Pachube type of work. Where, if you had the headset and you were a relief worker, and you had immediate lightweight, non-intrusive, heads up alpha channel overlay, waypoint markers showing you all of the disaster locations or points of need, AR becomes extremely valuable, because it’s a primarily hands-free environment. This is why the military stuff is so interesting.

Tish Shute: Ha! We are running into the eye patch/shades/goggles/sexy specs thing again. But filtering and making streams of activity relevant will be very interesting for AR.  Again that it why I love the Wave Federation Protocol work because what they have built into their XMPP extensions. You can have your real-time personal data streams, or community streams, or broadcast publicly – the permissions are built.

And Thomas Wrobel’s original vision of these layers and channels is only fully expressed if you have the eyewear.

Chris Arkenberg: Well, and it becomes redundant if it’s on a mobile. To use a very basic example, Twitter, obviously there’s an app you can view those streams of activity on the camera stream. But you can view that real time data on the screen. Why do you need to see it heads up?

The reason I really pay attention to what the military is investing in, one, because they have a ton of money, but also because they tend to represent the core bio survival needs of the species…So, when I look at computing, I see this very obvious trend of computers getting smaller and smaller and closer and closer to us because they’re so valuable to our success. They give us so much valuable information for engaging our world on a moment by moment basis. So, of course now we have these tiny little handheld devices that give us access to the global knowledge depositories of human history, because it’s so useful to have that stuff right at hand.

The only impediment now is that it takes one of our hands, if not both of them, to access it. So if you are in the natural world, which we are all always in the natural world, ultimately, you want your hands free in order to engage with the world on a physical level.

I see computation, or rather, our access to computation is just going to get thinner and thinner, and we’ll very soon move into eyewear, and inevitably, we’ll move into brain computer interface in some capacity.

So when you’re the disaster worker, or a deployed soldier, or the extreme mountain biker, or the heli-skier, or just an adventurer, there are a lot of very practical reasons to have access to information on a heads-up plane. I see AR as being so profound and so valuable, but we’re getting a glimpse of it in its infancy, and it’s got a ways to go to be able to really contain what it is we’re reaching for.

Tish Shute: I agree.

Chris Arkenberg: And that’s been a big criticism I’ve had with all the existing AR implementations that I’ve seen, is that the UI really needs a revolution. It’s very heavy handed. It is not dynamic, even though it’s supposed to be. It does not take advantage of transparencies. It treats the screen like a screen. It doesn’t treat the screen like a window onto the real world. When you’re looking on the real world, you don’t want a lot of occlusion. You want very soft-touch indicators of a data shadow behind something that you can then address and then have it call out the information that’s important to you.

Tish Shute: Now, that’s a very nice kind of image you’ve conjured for me there. Do you see that more could be done on the smartphone than is being done within that? Or are we like waiting for the old ishades?

Chris Arkenberg: I think there’s definitely a lot of room for improvement on the smartphone UI. Nobody’s really played around with it much. And again, I think that’s in part that there hasn’t been a really established platform with enough money to fund interesting UI work. We see it in some of the concept demos that float around every now and then.

I guess it’s both a blessing and curse that I’m always five steps ahead of where I’m trying to get to.

Tish Shute: Yeah, I am familiar with that feeling!

Chris Arkenberg: So I’m always trying to reach for the vision even though it’s a bit distant. I think there’s going to be a lot of development on the handsets. But again, I think we need a lot of refinement. We need a lot of real critical analysis of why this is a good thing.

To get back to the original point of Raimo’s comment, it struck me. And I knew it, but I just had set it aside as gimmickry. But he’s right. Content is a huge driver for this. Just stuff that’s engaging, and fun, and cool, and shows off the technology so they can get enough money to make it through whatever Trough of Disappointment may be waiting.

Tish Shute: Yeah, don’t underestimate the Planes of Content!  They are a great place to get interest and money to keep AR technology moving on, right?

Chris Arkenberg: Yeah, yeah. Because, you know, there’s a lot of freedom there. And you can piggyback on all the rest of the content that’s out there and jump on memes and marketing objectives, etc…

And there’s a lot of stuff…I’m blanking on some of the names, but some of these historical recreations of city streets. There’s a street in London where they overlaid historical photos in a really compelling experience. [Museum of London - http://www.museumoflondon.org.uk/] Again, I’m completely forgetting the attributions, but hose are the type of things that can really be pursued on the existing platforms. There is stuff that’s really compelling and really cool.

I heard of another interesting use case – and I should say that I can’t find attributions to this anywhere on the web and I may be paraphrasing or mis-representing the actual work, but I think the concept is worth exploring anyway. But the idea was that you could take the locations of border checkpoints and conflict sites in Palestine and Israel and visually overlay them on an AR layer in San Francisco. And it would do some sort of transposition where you could virtually view these things in San Francisco with the same locational mapping superimposed. So you could see where the checkpoints where. You could see where the wall was. You could see where suicide bombings were and where there had been conflicts. [I cannot find any citations for this!]

Tish Shute: But with an AR view? But why would you use an AR view if you are in San Francisco, then?

Chris Arkenberg: Because it superimposes two realities, translating the Gaza conflict into San Francisco as you are walking around. You can interrogate the world. There’s a discoverability aspect where you’re using the headset to reveal things, or the handset rather, to reveal things that you could not see otherwise in your city. It was done as an art piece, but as a provocative, obviously political art piece.

Tish Shute: Very interesting. I’d love to see that. Because that’s interesting to get away from this idea that you actually have to sort of have this one to one relationship between the data and the world is kinda nice, isn’t it? Well, not one to one, but a very literal…getting away from that literalness is kind of good.

Chris Arkenberg: And that’s a possibility of virtual reality and augmented reality merging, that maybe virtual reality is actually going to do best by coming out of the box and writing itself over our reality, so that as you are walking around, you are no longer seeing San Francisco, but you are seeing part of Everquest or World of Warcraft.

Tish Shute: Well this is where Bruce Sterling gets to that point he made in his keynote for are2010, that if we actually have viable AR eyewear, then you get the gothic stepsister of AR, VR rising from the grave!  He asks whether the very charm of augmented reality, is in fact that it adds rather than subtracts from your engagement with the world and that getting get sucked back into the black hole of VR might not be so great.

Chris Arkenberg: And then you get all sorts of interesting challenges to social cohesion if you have a lot of different people experiencing very different worlds, effectively. That if there is no real consensual reality and a majority of your local populous is, in fact, experiencing very different and unique versions of the world, what does that do to social cohesion? How does that reinforce tribalism, for example, when only you and certain others get to opt in to a particular layer view of the world?

Tish Shute: Yes Jamais Cascio wrote an interesting piece on that issue on AR and social cohesion a while back.

An eye patch is a more logical vision than the goggles in many ways but I suppose the loss is stereo vision?

Chris Arkenberg: And actually, there were developments in military helicopter technology many years ago that used a single pane square of glass over the eye mounted to the helmets of pilots. And then they drew various bits of heads-up information on it. So that ensures that you’re having a real strong engagement with the real world, which, obviously, when you’re a helicopter pilot is quite important. But you still have access to the data layer of the invisible world.

Tish Shute: I just went to Hadoop World and I have to say, I was awestruck about how big that’s got. I mean Hadoop has gone from like zero to huge in just a few years. I mean it’s just like now everyone has the power of the Google big table at their fingertips.

What’s the play for AR in the land of search?

I could imagine Hadoop being very powerful tool for AR analytics?

Have you got any thoughts on the land of search and AR? Of course visual search is proceeding at a fast pace and there is a lot of promise for integrations with AR in the future but the latency for visual search is still pretty high?

Chris Arkenberg: In the near term, not a lot. In the medium term, there’s a larger trend towards virtual agents that you can program or teach to keep watch over things for you as an effort to scale down the data overload. So search is something that’s going to become more personalized and more active. There’s a movement to make it so people can essentially deputize these agents to be always searching for them; to be out there looking for the things that they have told these agents are important to them.

So active search for AR I think presents some challenges, obviously because you need to do text input, typically, or voice input. Voice input, I think, is much more achievable than text input for AR. But I can certainly imagine an AR layer that is being serviced by these agents that we have roaming around the web for us reconciling their visual view of the world with our personalizations. AR apps are contextually aware so it knows that if you’re downtown, it’s not going to be giving you a ton of information about Software as a Service infrastructure, or what have you. But that, instead, it’s going to be handing you little tidbits about a particular clothing brand you’ve opted in to follow and information about music venues & schedules, for example. Or perhaps you’ll be on the lookout for other users that have opted in to publicly tag themselves as a member of this or that affinity.

I keep coming back to this idea of AR as really just a simple visualization layer that all of these other technologies can potentially feed into. So in that sense, search becomes a passive thing that AR is just simply presenting to you in a heads-up, hands-free, or potentially hands-free environment.

Tish Shute: Yes, the big challenge is the stepping stones to that point! Small steps that keep interest going into developing the underlying technology (and not just in research labs!) that will bring us that interface.  We have seen some movement already with Qualcomm.

Chris Arkenberg: And there are bandwidth issues as well, as we can see with the Google Goggles, which is a great idea of visual search. But you have to take a picture and send it to the cloud and wait for your results. It’s not a real-time dynamic interrogation of the world.

Tish Shute: Yes we are really only at the very beginning of AR being ready for prime time.. it would be interesting to ask AR developers how many of them use AR on a daily basis.

Chris Arkenberg: I think a lot of us, we’re just informed by the sci-fi myths and fascinated with the potential now that’s it’s starting to become real. But I think we all kinda get that it’s still extraordinarily young. I mean the web is extraordinarily young. And AR is itself far younger in a lot of ways in its implementations.

Everybody has a lot of excitement about all of the great potentials that are being unleashed by this great wave of the Internet and the web and ubiquitous mobile computing. So that’s why, you know, you look at that map and we talk about AR and you can’t talk about any of the stuff without talking about all of it, in a lot of ways, particularly with something like AR where it’s so ultimately agnostic and could be completely pervasive across all of these layers.

So my fascination is with the future, and I measure our progress towards it by the young nascent offerings from the platform players and the developers. And yeah, a lot of it is…it’s akin to getting that first triangle on the screen in 3D. You know, when the renderer finally works and you get a triangle on the screen, and you go, “Oh my God, it renders.” And then you can start to really build polygons and build objects, and start doing boolian operations, and get light and rendering in there, and textures, and on, and on, and on.
So I’m fascinated by the Layars and the Metaio’s…
[laughter]

Tish Shute: Yes and hats off to all the players in the emerging industry, Layar, Metaio, Ogmento, Total Immsersion, and all the others who are finding clever ways to bring fun aspects of AR into the mainstream, and fuel interest to take the technology to the next level.

Chris Arkenberg: Absolutely. And the hype cycle is very valuable. It has really helped launch the AR industry. It’s brought a lot of eyes, and it’s brought a lot of money into the industry. And it’s forcing people like us to have these conversations to understand how to refine its growth and really focus on the potential in all these different venues, whether it’s trying to save lives, or better understand your city, or have really compelling entertainment experiences.

Everybody’s excited, and everybody’s sharing, and everybody’s trying to move it forward in a way that’s the most productive.

Visual search is heating up, and with it a key stage of turning the physical world into a platform is underway as images become hyperlinks to the world in applications like Google Goggles, Point and Find, and SnapTell – see this post by Katie Boehret.  And while there may be no truly game changing augmented reality goggles for a while, make no mistake, key aspects of our augmented view, factors that will have a lot to do with what we will actually see when an augmented vision of the world is a commonplace, are already in the works.  And, as Anselm Hook (pic above from @caseorganic’s flickr) notes:

“There is a real risk of our augmented reality world being owned by interests which are not our own. There is a real question of when you hold up that AR goggle, what are you going to see?”

Cooperating services, e.g., Google Earth, Maps, Streetview, Google Goggles, and leader in local search like Yelp (see here) would have an enormous ability to filter and control a mobile, social, context aware view of the physical world, and Google themselves see an ethical quandary.

“A Google spokesperson says this app has the ability to use facial recognition with Goggles, but hasn’t launched this feature because it hasn’t been built into an app that would provide real value for users. The spokesperson also cites “some important transparency and consumer-choice issues we need to think through” (quote from Wall Street Journal Column by Katie Boehret).

Anselm Hook and Paige Saez, with great prescience, have been advocating a social commons for the placemarks and imagemarks to our physical world platform through a number of pioneering projects, including imagewiki.   I have interviewed both Anselm and Paige (upcoming) in depth, recently.  My talk with Anselm was nearly three hours long!  So I am publishing the transcript in two parts.

Understanding what it means to have a social commons for  our physical world platform, and augmented reality, are key questions for all of us to think about, but especially important for those of us involved in the emerging industry of augmented reality.

Anselm notes :

“The placemarks and imagemarks in our reality are about to undergo that same politicization and ownership that already affects DNS and content. Creative Commons, Electronic Frontier Foundation and other organizations try to protect our social commons. When an image becomes a kind of hyperlink – there’s really a question of what it will resolve to. Will your heads up display of McDonalds show tasty treats at low prices or will it show alternative nearby places where you can get a local, organic, healthy meal quickly? Clearly there’s about to be a huge ownership battle for the emerging imageDNS”

The mobile internet is moving beyond the internet in your pocket phase of mobility with mobile, social, proximity-based, context aware networks like FourSquare, Gowalla, Brightkite and GraffitiGeo (see Smart Data Collective) likely, soon, to start to take precedence over other forms of social network.

Regardless of the timeline for true augmented reality – 3D images & graphics tightly registered to the physical world,  proximity-based social networking and real time search are already taking us into a hyper-local mode and the realm of augmented reality which is “inherently about who you are, where you are, what you are doing, and what is around you” (Robert Rice – see here). The ground is being prepared for augmented reality now.

If you have been reading Ugotrade, you will know I have been actively involved in developing  an open, distributed AR platform/mobile social interaction utility for geolocated data based on the Wave Federation Protocol – AR Wave a.k.a Muku – “crest of a wave” (see my posts here, here and here for more on this project, and the AR Wave Wiki here).  Federation is, I believe, one vital aspect to developing a social commons for augmented reality and the physical world platform.

Also, a bit of news, I am co-chairing the upcoming Augmented Reality Event (are2010) with Ori Inbar of Games Alfresco and Ogmento, whurley.  Sean Lowery, Prospera, is the event organizer, and are2010 has the support of the AR Consortium.   The are2010 web site is live and there is an Open Call For Speakers.  You can submit your proposals and demos for one of the three tracks, business, technology, or production on the web site here.

Bruce Sterling “prophet” of  augmented reality and more, “will deliver the most anticipated Augmented Reality keynote of the year.”

It didn’t surprise me when Anselm mentioned that Bruce Sterling was a key influence for his work on the geospatial web and augmented reality.  Anselm explained:

“I’d seen a talk by Bruce Sterling at an event called Planetwork [May, 2000]. And that event was, for me, a turning point where I decided to focus full time on exactly what I cared about instead of doing things that were kind of similar to what I cared about. So, his influences is a pretty significant one to me at that exact moment.”

For more see viridiandesign.org -  seems it is time for a “Neo-Viridian,” revival!

This post by Bruce Sterling on Pachube Feeds, and Thomas Wrobel’s prototype design for open distributed augmented reality on IRC, were key inspirations for me when I began thinking about the potential of Google Wave Federation protocol for augmented reality.  I had been exploring Pachube and deeply interested in the vision of Usman Haque, but I had a real aha moment when I read this :

“(((Extra credit for eager ubicomp hackers: combine this [pachube feeds] with Googlewave, then describe it in microsyntax. Hello, 2015!)))”

I think the AR Wave group will earn the extra credit and more very soon!  Davide Carnovale, need2revolt, and Thomas Wrobel have been leading the coding charge, and there will be a very early AR Wave demo soon, perhaps as soon as the Feb 16th ARNY Meetup. 

Open access to the creation of view that will eventually find its way into AR goggles, will depend not only on the power of  an open distributed platform for collaboration like the AR Wave project.  Our augmented reality view will be constructed through complex “hybrid tracking and sensor fusion techniques” (Jarell Pair), cooperating cloud data services, powerful search and computer vision algorithms, and apps that learn by context accumulation will drive our augmented experiences, and at the moment, these kind of resources, at least at scale, are for the most part in private hands.

In the interview below, Anselm’s discusses  how trust filters, and being able to publicly permission your searches so that other people can respond and so that people can reach out to you, and the democratization of data in general, are even more of a concern with augmented reality and hyper local search. The task of understanding what it means to have  a social commons for the outernet remains an open, and pressing question.

Anselm explains (see full interview below):

“as we move towards a physical internet where there’s no clicking and there’s no interface and the computer’s just telling you what it thinks you’re looking at, translating, you know, an image of a billboard to the name of the rock star who’s on that billboard, or translating the list of ingredients on a can of soup to the source outlets where it thinks that, those ingredients came from. When you have that kind of automated mediation, the question of trust definitely arises.

And we haven’t seen the Clay Shirkys or the Larry Lessigs of the world start to talk about this yet.  Although I suspect that in the next four or five years that the zero click interface will become the primary interface, that we’ll have…we’ll come to assume that what we see with the extra enhanced data we get projected onto our view is the truth. Yet, at the same time, there is just no structure or mechanism even being considered for a democratic ownership of it.”

Augmented Reality will emerge through sensor fusion techniques & cooperating cloud services

In 2010, sensor fusion techniques, computer vision technology in conjunction with GPS and compass data will create data linking that can enable the kind of augmented reality that has been the stuff of imagination for nearly four decades (see Jarrell Pair’s post).

Putting stuff in the world in 3D is of course key to the original vision of augmented reality, and one of its biggest challenges.  Augmented reality is going to be implicated in a real time mapping of the world at an unprecedented scale and granularity.  We have barely an inkling of the implications of this now.

Anselm and Paige have been working in the heart of the social cartography movement for nearly a decade.  The vision and experience of this community is vital to understanding how augmented reality and the world as a physical platform can evolve into something that benefits people and allows them “to have a better understanding of the opportunities around them.”

We have been hacking maps for millenia –  “from conceptual story mapping, to colloquial mapping in European development and the cartographic renaissance created by the global voyages and rediscovery of Ptolemy’s maps” (Andrew Turner).  And, recently, initiatives on a public-provided GIS, like OpenGeo, have led the way toward more open, interoperable, geospatial data.

Mapping takes on a new an crucial role to augmented reality.  Nokia’s ImageSpace is beginning to do what many thought Microsoft would do with photosynth two years ago.

And, if we see these kind of projects developed into a “photo-based positioning systems” -  “3d models of the environment to cover every possible angle, and then software that can work out in reverse based on a picture precisely where you are and where your facing” (Thomas Wrobel), we would find augmented reality leap forward over night.

It is time to take very seriously the vast opportunities and potential pitfalls of an augmented world.

“when you are mediating the translation layer between the image and the data, then there is an opportunity for you to control it, and that opportunity is hard to resist.  It is hard to choose not to own that opportunity. It is an advertising opportunity. It is a revenue opportunity. It is a chance to send a message and a tone.

I know that Google and companies like that are keenly aware of the kinds of roles they don’t want to hold, but it is sometimes seductive to think about them. And I am afraid that we, as a community, need to assert an ownership, kind of a commons, over how computers will translate what they see to information that we perceive.”

There are some initiatives emerging.  Tonchidot (who closed on $4 million of VC for augmented reality last December) has helped create the AR Commons in Japan.  CFO of Tonchidot, Ken Inoue explained in an interview with me in September 2009.

“We feel that public data, such as landmarks, government facilities, and public transport should be shared. We see an AR world where people can readily and easily access information by just seeing – quick, easy, and efficient.  And because of this ease and intuitiveness, children, the elderly and handicapped will surely benefit.  AR could help create a safer society.  Warnings, alerts, and safety information could save lives and avoid disasters.  These are what we, and AR Commons would like to tackle in the not so distant future.”

But the task of building a social commons for the physical world platform has only just begun.

Interview with Anselm Hook

photo from Anselm’s Flickr stream here

Tish Shute: We first met last year at Wherecamp. The start of 2009 was I think the “OMG finally” moment for augmented reality and in less than a year AR, at least in proto forms, AR is breaking into the mainstream now! You are one of the founding visionaries/philosophers/hackers of the geo web and you have been thinking about geo web and AR for a long time – all the way back to the legendary Head Map Manifesto, and before.  Mostly recently you led the way in the very successful ARDevCamp in Mountain View. Could you start by telling me a little bit about the history of your pioneering work with geolocated data?

Anselm Hook: I am a long time Geo fanatic. I’m really interested in social cartography and what some people call public-provided GIS, that’s some language that people use. Anyway, my personal interest, when I talk to people who are non-technical (and it’s been a long term interest in the way I phrase it) is that I want to help people see through walls. So, the goal is very simple. I want people to have a better understanding of opportunities around them, the landscape around them. I always get frustrated when people make bad decisions because of a lack of information, especially when it’s related to their community and related to their environment. But, plainly put, I really just want “to help people see through walls”. It’s a very simple goal.

Tish Shute: I know you worked on Platial, which is really one of my favorite social mapping applications. It really broke new ground. What was the history of that? How did you get involved with Platial?

Anselm Hook: That’s an interesting question. It actually started at around 2000 when I saw Bruce Sterling talk. I had been writing video games for many years, and I was quite good at it, and I enjoyed it. But, the reasons I was doing it diverged from why the industry was doing it. I was making video games because I like to make shared spaces for my friends to play in and to share experience. I really enjoyed making shared environments. I worked on BBS’s and my friends and I were always making these collaborative shared environments.

Once the video game industry kind of started to take off, I started to do high performance, 3D interactive video games and making compelling shared spaces, and it was a lot of fun. But, the frustration for me was that there was a huge industry growing around it and became very commercial. Although it paid well, it started to diverge from my values which were more centered around community environments, and shared understanding.

Tish Shute: Yes very rapidly, the big games kind of devolved from the social aspects and became more and more into single player really, didn’t they?

Anselm Hook: It was the way, actually, because even though often you were in a many player world, you weren’t collaborating, everything else became just a target.  I liked the idea of deep collaboration that calls the kind of playful space you see in IRC, or in the real world, where people are solving real world problems.

And I grew up in the Rockies, and I was always had a lot of access to the outside. So, I saw shared spaces and collaboration as a way to protect our environment. [ To step back ] I think people used different metrics for measuring their choices in the world and many people have a value system centered around minimization of harm: making sure that the people are not hurt. But, my value system is different. I personally believe that protecting the planet is more important: to maximize biodiversity. I feel like protecting people around me comes from protecting the ecosystems they live in.

Tish Shute: That’s interesting, isn’t it, because the history of Keyhole was really that, wasn’t it.  Keyhole later became Google Earth, but I mean it began out of a project to look at what was going on in the ecosystem over Africa at that time, didn’t it?

Anselm Hook: Yes, in fact many people’s projects are stemming from an environmental concern. Mikel Maron’s works for example – he’s doing Map Kiberia, and he also worked on OpenStreetMaps.

Tish Shute: Map Kiberia – that is the new project?

Anselm Hook: Oh, yes his project is called Map Kiberia. He’s mapping a city in Africa.
[For more see Map Kiberia’s YouTube Channel – photo below from ricajimarie ]

Tish Shute: Right, great!

Anselm Hook: When I started to look at GIS and mapping I started to meet people who had a very similar background. What happened to me is I kind of stepped away from games around the year 2000. I’d seen a talk by Bruce Sterling at an event called PlaNetwork. And that event was, for me, a turning point where I decided to focus full time on exactly what I cared about instead of doing things that were kind of similar to what I cared about. So, his influences is a pretty significant one to me at that exact moment.

[For more see viridiandesign.org – seems that it is time for a “Neo-Viridian,” revival.]

Tish Shute: It’s interesting because now your paths are crossing again with augmented reality. You are on the same wavelength again.

Anselm Hook: It’s funny, actually, I’ve had a couple of brief overlaps in that way.  Well, so in 2000 I went to see this talk and I did a small project called — well, I called it SpinnyGlobe. What I did is I mapped protests from a number of websites onto a globe to show the level of community opposition to the pending war in Iraq. It was the first time there had been a protest before a war. So, it was very interesting to me. [ See http://hook.org/headmap ]

Tish Shute: That’s really fascinating. Do you have any pictures of that you could send me?

photo from anselm’s flickrstream

Tish Shute: Yes, I’ll definitely look SpinnyGlobe up. It sounds very interesting.  One of the aspects of your work on geo-located data projects like this and Platial is that you really started to develop this idea of a culture of place, about how people make place. This was the wake up call to me regarding the power of networks combined with geo-data.

We are hoping to extend this idea into augmented reality with the an open distributed platform for AR so that we can collaboratively map our worlds from the perspective of who we are, where we are, and what we are doing.  I know you’ve just done some work recently in augmented reality.  I know you put the code up already.

By the way, I love the way you take your philosophy into the way you make code – the practice of making some code, trying some things out, making it all public and publishing your findings, you know, your comments on that experience.  Perhaps you could recap sort of how you picked up recently on the state of play with augmented reality and what aspects you looked at, and what came out of that experience?

Anselm Hook: So, it’s a very simple trajectory. Coming out of the work I had done, Platial, among other projects and I started to just look at the hyper-local and I suddenly realize that even those services weren’t really speaking to living, and how to really see and solve local problems. What was missing was a sense of context.

The map doesn’t know how you’re feeling, it doesn’t know if you’re in a hurry, it doesn’t know what you want, it’s very static. Even the web maps are very static. And augmented reality for me I started to recognize as a combination of — well — it’s probably collision of many forces, many forces that we’re all a part of. We’ve also didn’t realize that the real-time web is really important, it’s part of what AR is about.

We have all started to realize that the context is important. You know, your personal disposition, your needs, if you want to be interrupted or not. That is the kind of thing that the ubiquitous computing crowd has talked about. We started to recognize that there are sensors everywhere, and the ambient sensing communities talked about that. So what is funny for me about augmented reality is I started realizing it is just a collision of many other trends into something bigger.

Everything else we thought was a separate thing is actually just part of this thing. Even things like Google Maps or mapping systems we think are so great are really just kind of almost an aspect of a hyper-local view. You actually don’t really care what is happening 10 blocks away or 100 blocks away. If you could satisfy those same interests and needs within a single block, one block away, you would probably be really happy. You really just want to satisfy needs and interests, find ways to contribute, or get yourself fed, or whatever it is you want. And AR seemed to be the playground to really explore the human condition.

Tish Shute: Anyway, I think one of the things that has been very amazing this year is we to have the good mediating devices that, for the first time, give us compasses, GPS, and accelerometers. But one sort of missing pieces with AR at the moment is [tracking, mapping, and registration] – the kind of things colloquial mappings of the world could be of great help with.

We have seen mapping coming out of the Flickr data, e.g., the University of Washington, put the maps together from the geo-tagged Flickr photos. Now if we could have that linked up with AR, then we have the kind of mapping we need to kind of really hook the geo-data onto the world in a way that goes beyond…you know, what compass and GPS can really deliver is pretty minimal at the moment.

Anselm Hook: There is a real risk of our augmented reality world being owned by interests which are not our own. There is a real question of when you hold up that AR goggle, what are you going to see? Are you going to see corporate advertising? Are you going to see your friends’ comments or criticisms? It is going to be an Iran or a democracy, right? It is unclear.

Right now there are some disturbing trends I have noticed. I am a big fan of Google Goggles. I think it is a great project. But when you are mediating the translation layer between the image and the data, then there is an opportunity for you to control it, and that opportunity is hard to resist. It is hard to choose not to own that opportunity. It is an advertising opportunity. It is a revenue opportunity. It is a chance to send a message and a tone.

I know that Google and companies like that are keenly aware of the kinds of roles they don’t want to hold, but it is sometimes seductive to think about them. And I am afraid that we, as a community, need to assert an ownership, kind of a commons, over how computers will translate what they see to information that we perceive.

Tish Shute: Yes. And this is how we met, again, recently [over the project to create an open, distributed platform for AR using the Wave Federation Protocol]…

This is something I feel really deeply is that, you know, basically we need the physical internet to be as open as, as the, as the internet, as the end-to-end internet has been. Or more so, actually, because the end-to-end internet has seen the trend has been to walled gardens.  Basically Facebook became enormous, an enormous walled garden which, I think, was despite, our predictions about them, [walled gardens] are the social experience really on the web.  It’s very much in walled gardens still and I, and I really feel that with the physical internet, we need to make great efforts not for it not just to be a series of small pockets of privately funded walled gardens.

There needs to be some kind of communications infrastructure that keeps it open so that was when I got interested in looking at the Wave Federation Protocol because it was a real time, you know, an open real time protocol that could possibly be a basis for that. But I think the point you’ve talked to just now, the mapping of the world and who has the “goggles”, i.e., the image data, image databases, that make the world meaningful is really, that’s still a, it’s still a BIG question [i.e. who controls the view?].

When I saw ImageWiki, [I realized] that is a piece that is vital for, for augmented reality. We need to have a huge social effort to be involved in this,  linking in and creating the  physical internet, in creating the image hyperlinks that will make that meaningful.

Anselm Hook: I think that’s a great point. The search interface, the kind of Internet that we’re used to, the way we talk to the network now, is fundamentally open end to end. Yes, you can have your oligarchies inside of it, as we see with Facebook, but you can always start your own venture up and you can do a search on something, and you can find that, that website and you can join it or you can put up your own webpage and people can find it.

The translation layer, the idea of text search and the ability to discovery power and the serendipity and the openness of that discovery, it’s pretty open right now. We do have some serious boundaries of language, which is one of the reasons I was working at the Meedan.org [hybrid distributed, natural language translation] for a couple of years, trying to bridge that issue.

But here, as we move towards a physical internet where there’s no clicking and there’s no interface and the computer’s just telling you what it thinks you’re looking at, translating, you know, an image of a billboard to the name of the rock star who’s on that billboard, or translating the list of ingredients on a can of soup to the source outlets where it thinks that, those ingredients came from. When you have that kind of automated mediation, the question of trust definitely arises.

And we haven’t seen the Clay Shirkys or the Larry Lessigs of the world start to talk about this yet.  Although I suspect that in the next four or five years that the zero click interface will become the primary interface, that we’ll have…we’ll come to assume that what we see with the extra enhanced data we get projected onto our view is the truth. Yet, at the same time, there is just no structure or mechanism even being considered for a democratic ownership of it.

We have with DNS, for example, the idea that you can register the domain name and people can search for it, and find it, and go to it. There’s no such thing as an Image DNS, or an image translation to DNS right now. What does it mean when everything is just “magic”, when there’s no way for you to be a part of the conversation, where you’re just a consumer of what people tell you, or of what one company right now, tells you, is reality? That’s a real concern.

Tish Shute: This, to me is the most important question at the moment. I mean, it’s the big one and it’s the place to put energy if you love the Internet [and what it can now become] right. You’ve got to put a lot of energy into this because this [a democratized view of the physical world as a platform] won’t just happen, because there’s a lot of momentum already for it to be heavily privatized, partly because, one reason is, some of the computer vision algorithms that, say, make sense of things like the geotag photographs are not open.  I mean, for example, the beautiful maps that have been made from the University of Washington [from Flickr geotagged photo sets], that isn’t in the public domain.

Anselm Hook: Right. Tish, and in fact you’re referring to [with the maps from the Flickr photos] to ordinary maps and the fact we’ve already seen that maps lie, we’ve already, seen how much maps are reflecting a certain truth that becomes the normative truth. Google maps reflects roads, because this is roads and cars, right? Only recently have they thought about buses and walking. So the normative view that people assume is the reality, is showing off you know Starbucks, and roads, and cars, that becomes the default, those prejudices are just assumed, you know, the truth. But they’re not the truth at all.

I was talking to a friend of mine in Montreal, [Renee Sieber], and she said that their Indian portage routes are a bridge across land and water, they don’t think of a piece of land and a piece of water as being different things, they think of them as one thing: a route. It’s already a different kind of language we can’t even reflect it.

So not only is there this kind of formal, anthropological lie, in a sense, but there’s this way that we deceive ourselves because of our own prejudices.

Tish Shute: Yes I agree and that’s why I think when I saw some of the things you had written on the ImageWiki point clearly to the need to create a social commons. We need a social commons for the real-time physical internet, we need it for the image hyperlinks that make sense of that.

And it’s a complicated thing in a sense, though, because we don’t actually have a good distributed infrastructure for AR yet, and I found exploring AR Wave, that at last we have the suggestion of an open, federated protocol for real-time communication – the wave federation protocol. [Real time communications is a very important part of AR].  It isn’t an actuality yet where lots of people are able to use it, set up their own servers, and there’s not a standard all the way through  [there is not a standard for how data is sent between the client and the server].

But Wave Federation Protocol does make possible truly distributed social AR.  I started thinking when I saw ImageWiki that to bring ImageWiki together with the social collaborative power of distributed AR.  This really would be the basis of creating a social commons for augmented reality and the physical world as a platform – the start of a bottom up with deep social collaboration on how we create augmented reality colloquial maps that can inform a hyper-local of the world.

Anselm Hook: Yes. When Paige Saez, John Wiseman, and myself, and a few other folks… You know, Benjamin Foote, Marlin Pohlmann, and a couple other people started to play with this, we quickly found that… We started to realize, “Oh, this kind of thing will be at least as popular as IRC. There will be at least as many people doing this as chatting in little virtual spaces. There’ll be at least as many people decorating the world with augmented reality markup, and maybe using the real world as a kind of barcode for translating what you’re looking at into an artifact, a digital artifact.

And that the size of that space was going to be huge, basically. Maybe not quite as commodifiable as Twitter, but certainly very energetic.

Many of the projects we did were just kind of looking at these kinds of issues sort of from an artistic, technical, and political point of view. We weren’t so much posing complete solutions, but simply using a praxis to explore the idea with an implementation, as a foundation for this discussion. So I think we sort of opened that can of worms for sure.

Tish Shute: Did you actually set up ImageWiki to be working as a location based app yet?

Anselm Hook: It is a location based app. It collects your longitude, latitude, and the image and stores it. And then it uses that as a way to translate that image to anything else. It could be a piece of text or a URL.

Tish Shute: So there is a smartphone app, but you didn’t take it as far as an AR app yet?

Anselm Hook: No. We didn’t do a heads-up view. There are apps on the iPhone store that do that, but they don’t do the brute force image recognition that we were using. We used a third party off the shelf algorithm that we found on Wikipedia and downloaded the source code, and threw it on the server. And John Wiseman in LA wrote the scalable database backend so that we could scale the actual…

Tish Shute: So how did you set the iphone app up to work?

Anselm Hook: The iPhone side was very simple. You take a picture of something and it tells you what it is. That is all it did. We would take the location, but the client side, the iPhone side, just rendered, returned to you…It said, “Someone said that this picture of a barking dog is an advertisement for a local band.”

Tish Shute: Right. So basically it was a geo-tagged?

Anslem Hook: Yes. We are just collecting the geo information. Actually, there were a whole lot of technical challenges. The whole idea of ImageWiki is actually kind of beyond our technical ability for a small team like us. It really does take a team, a group like Google, to do this kind of thing in a scalable way.

Tish Shute: Why is that?

Anslem Hook: There are two sides. There is the curating the images. I think that is the job of groups like us – open source groups who can curate images that are owned by the community. And then the searching side, the algorithm side, where you are actually matching the fingerprint of one image to images in your database, that takes a much more…that is much more industrial.  We get both sides, ours is not a scalable solution. It is mostly…proving that it could be done was important.

Tish Shute: In terms of hooking Imagewiki up to the collaborative possibilities of AR Wave wouldn’t federation pose some interesting possibilities for scaling search algorithms and all that?

Anselm Hook: Yes. And what is funny also, incidentally, is that, nevertheless, we did look for some financial support for it, but we couldn’t…we just didn’t find the investors to scale it. Now, other companies like SnapTell took a shot at it. And they have an app in the iPhone store where you can point at a beer bottle and get back the name of the beer bottle.

The classic example everyone uses is a book. Amazon has all the image jackets of all their books. You can point SnapTell at almost any book and get back links to buy that at Amazon, the price of the book, and user comments on the book. So they are treating Amazon as the canonical voice of the book, for better or worse. That is the state of the art so far, up until Google Goggles came out a little while ago, which actually blows it out of the water. But, that is where we are now.

Tish Shute: Right. But the point you raise about how when something like Amazon comes canonical of what is book, right, this is the whole point, isn’t it?

Anselm Hook: Is Amazon truth? It’s not bad. Jeff Bezos seems like a nice guy, but, you know.

Tish Shute: And this is the point of having these open infrastructures for this.  And this should be obvious in a way, but it comes back to the thing about what made the Internet great was the fact that even though as you note, you get an oligarchy like Facebook, but people always could just go off and do something else, right? Because the fundamental infrastructure was basically open and designed to be available for everyone. And many people have championed that and fought for it hard [to maintain this openness] haven’t they? They have devoted their lives to keeping it that way, even if the oligarchies have done their thing.

Anselm Hook: Yes. There are really some things that are underneath all of this that haven’t been solved yet.

One is that the trust in social networks has not been built yet, so we can’t do peer based recommendations very well. We can’t filter noise by peers. Twitter kind of is moving there, but I don’t just want to listen to my Twitter friends. I want to listen to my friends of friends. If I am getting truth from somebody, I want to get that truth from people my friends say that they trust.

Then the second problem is that there is a search business. My friend Ed Bice, who owns Meedan, always says that a search itself, a search request, is an opportunity to make…is a publishing moment. It is an opportunity to say what you think. In the real world, if you are just hanging out with humans and you look somewhere, other people might look at your gaze and they might look at what you are looking at. Your gaze itself is a public act.

Gaze is a soft act, but it is one that is visible. With Google, the gaze of four billion people is invisible. We don’t what people are looking at, there is no opportunity to participate. Let me give you a real example.  I have taken a image of something of the bust of figure or a statue.  Why can’t the museum in Cairo look at my request and tell me oh yeah that is Tutankhamen, or that is Nefertiti right? Why can’t they have a chance to participate in the search and respond to me?

Right now the the only person that responds is Google when I do a search. We need to invert the search pyramid and open up search, so that search is a democratic act, so that you can publicly permission your searches so that other people can respond and so that people can reach out to you, not just you having to do a dialogue.

The common example of this.. and we see this everywhere: I am looking for a slice of pizza right, now I am hungry I want some pizza. I have to ask Google, look find twelve websites, call twelve phone numbers, and talk to each of the twelve stores, and ask them are they open late, is the food organic, is the food in any good, do my friends like it.

Whereas what I should be able to do is just say it’s a search moment and I am interested in pizza. If those pizza places my criteria like you know my friend’s like them and they are organic, they are open, then that pizza place can call me. I have the money why should I do the search? So the whole business of search, the whole structure of search is predicated around a revenue model, but its a really short-sighted revenue model, its not a brokerage.

Search isn’t search, search is hand waving.  These should be moments for us to have a discourse. So problem we are seeing in AR with communication of the right information is actually underneath AR, at the level of the whole infrastructure.

Search needs to be inverted, trust filters need to be built. We need to democratically own our data institutions.  We don’t right now.  That will be more of a concern, especially with AR.

Tish Shute: Yes, especially with AR, which is this why got all excited about federation.  Do you think federation has the potential, an opportunity to create [the new infrastructure you describe?]

Anselm Hook: Absolutely,  its absolutely what we must do. It is much harder to do. It is absolutely critical.

Tish Shute: And why is it much harder to do? Could you explain that?

Anselm Hook: Well, it’s very easy for a bunch of hackers to build a service that you log into and fetch some data, it’s a single thing. They don’t have to talk anybody, they can use their own protocols, they can hack it, it’s a big black box, behind the scenes. There’s running back and forth in a giant Chinese room delivering manuscripts and scrolls to you. Whatever is behind the black box, you don’t care, it just works.  But when you federate, you need to actually publish and have standards, and then you’re talk about semantic, everyone starts getting really excited and wave some hands. It becomes a disaster. It’s, at least, another power order, more difficult than DIY, build it yourself.

Tish Shute: So, in terms of what Google Wave have done with their approach to federation, what do you think have been their achievements and what do you think is their obstacles? What do you think are the failings of the Wave? Because it’s the first big public major player backed approach to something federated, isn’t it? In real time.

Anselm Hook: Yes. I think the most important non-federated service on the planet today is Twitter.  Identi.ca it’s not getting any traction with respect to Twitter. [ Even though ] Identi.ca is a federated version of Twitter and is very good. [ Identica is now Status.net ] . So, we see already there that small players aren’t being competitive. Then look at other services like IRC. IRC is the secret backbone of the Net. All the open source projects, all the teams, all the people that work on opensource projects are all on IRC. It’s the only way they get anything done.

With Google Wave, and the protocols underneath Google Wave, we see an attempt to build a similar kind of real time, but distributed protocol. I think it’s the right direction. I think, people should pick up the offering and make their own servers. I think that protocol is really great, I think the fact that is compressed, its high performance, it is small, real-time of blobs of data flying around, all exactly the way it should be done. It is getting close to this kind of rewrite of the Internet that people keep talking about, because, you know, the net protocols are so bad, it is starting to treat the idea of intermittent exchanges being more transitory, volatile, and not heavy.

….to be continued.  Part 2 coming soon!

It is now nearly two weeks since the Google Wave preview launch and I am happy to say we have some AR Wave news. The diagram above shows Thomas Wrobel’s basic concept for a distributed, multi-user, open augmented reality framework based on the Google Wave Federation Protocol and servers (click on the image to see the dynamic annotated sketch or here).

Even in the short time we have had to explore Wave, some very exciting possibilities are becoming clear. Thomas puts some of the virtues of Wave as an AR enabler succinctly when he writes:

“Wave allows the advantages of both real-time communication, as well as the advantages of persistent hosting of data. It is both like IRC, and like a Wiki. It allows anyone to create a Wave, and share it with anyone else. It allows Waves to be edited at the same time by many people, or used as a private reference for just one person.

These are all incredibly useful properties for any AR experience, more so Wave is open. Anyone can make a server or client for Wave. Better yet, these servers will exchange data with each other, providing a seamless world for the user…..a single login will let you browse the whole world of public waves, regardless of who’s providing or hosting the data. Wave is also quite scalable and secure…data is only exchanged when necessary, and will stay local if no one else needs to view it.

Wave allows bots to run on it…allowing blips in a waves to be automatically updated, created or destroyed based on any criteria the coders choose. Wave even allows the playback of all edits since the wave was created.

For all these reasons and more, Wave makes a great platform for AR.”

There will be much more coming soon on Wave enabled AR because the Google Wave invites have begun to flow out to a wider community now. This week, many of our small ad-hoc group looking at the development challenges and implications of Google Wave for AR actually got into Wave for the first time.

Many thanks to all the people who have contributed to this discussion so far including: Thomas Wrobel, Thomas K. Carpenter, Jeremy Hight, Joe Lamantia, Clayton Lilly, Gene Becker and many others.

We will be setting up some public AR Framework Development Waves this week.  If you have any trouble finding them, or adding yourself to it, please add Thomas and I to your contact list.  I am tishshute@googlewave.com  Thomas is darkflame@googlewave.com  The first two are currently called:

AR Wave: Augmented Reality Wave Framework Development (developer forum)

AR Wave: Augmented Reality Wave Development (for general discussion)

The discussion so far has been in two areas. On the one hand, it is gear-heady and focused on the Google Wave Federation Protocol, code, development challenges, and interfacing to mobile, while on the other hand people have been looking at use cases and questions of user experience.

Distributed, “shared augmented realities,” or “social augmented experiences” – that not only allow mashups, & multisource data flows, but dynamic overlays (not limited to 3d), created by users, linked to location/place/time, and distributed to other users who wish to engage with the experience by viewing and co-creating elements for their own goals and benefit – are something very new for us to think about.

As, Joe Lamantia, puts it, now:

“there’s a feedback loop between which interactions are made easy by any given combo of device;/ hardware / software / connectivity, and the ways that people really work in real life (without any mediation / permeation by tech).”

Joe Lamantia whose term, “social augmented experiences” I borrow for this post title, has done some thinking about “concepts and models for understanding and contributing to shared augmented experiences, such as the social scales for interaction, and the challenges attendant to designing such interactions.” Check out Joe Lamantia’s blog for more on this later this week.

It is very helpful, as Joe points out, to shift the focus  back and forth between the experience and the medium.

It is super exciting to have clear evidence that shared augmented realities are no longer merely possible, but highly probable and actually do-able now.

I should  be absolutely clear about what Google Wave does to enable AR because obviously Wave plays no role in solving image recognition and tracking/registrations issues.  But, for example, Wave protocols and servers do provide a means to exchange, edit, and read data, and that enables distributed, social augmented realities.

Thomas explains how the newly named “AR Blip” works as:

“An AR Blip is simply a Blip in wave containing AR data. Typically this would be the positional and url data telling a AR browser to position a 3d object at a location in space.

In more generic terms, an AR Blip allows data of various forms (meshes,text,sound) to be given a real-world position.”

I have mentioned in other posts (here and here) that Wave can be used for AR as precise or as loose as the current generation devices can handle. And as the hardware and software for the kind of AR that can put media out in the world to truly immerse you in a mixed space, the framework  should  be able to handle this too.

(a note on the Wave playback feature – this opens up a whole new world of possibilities.  Check out this post on some of the implications of playback for writing!)

The use cases we have been coming up with are too numerous to go into in detail this post.  The open nature of an AR framework/Wave standard will lead to many new applications we have barely begun to imagine.  As Thomas points out, different client software can be made for browsing, potentially allowing for various specialist browsers, as well as more generic ones for typical  use. The multitudes of different kinds of data in/output that could be integrated in an open AR framework as it evolves are mind boggling.

But, for now, some  obvious use cases do come to mind:
eg.

- Historical environmental overlays showing how a city used to be/and how this vision may be constructed differently by different communities

- Proposed building work showing future changes to a structure/and the negotiations of this future (both the public and professionals could submit their own comments to the plans in context), seeing pipes, cables and other invisible elements that can help builders and engineers collaborate and do their work.

- Skinning the world with interactive fantasies

I asked Thomas to help people understand how Wave enables new interactions to data by explaining how Wave could enable citi sensing and citizen sensing projects (e.g. this one being pioneered by Griswold):

“Sensors, both mobile and static could contribute environmental data into city overlays;

Having these invisible aspects of the world made visible would create ways to improve sustainability, social equity, urban management, energy efficiency, public health, and allow communities to understand and become active participants in the ecosystems and infrastructure of their neighborhoods.

The key is reflecting this  kind of data back to people “making it not back story but fore story,” right where we are, right where it happens, as well as having it available for analysis.

As well as  creating new opportunities to interact/respond to/and enhance data, making visible the invisible as Natalie Jeremijenko’s work on Amphibious Architecture and Usman Haque’s project Natural Fuse shows, can also create new connections/understandings between humans and the non human’s that share our world, e.g. fish, plants, waterways.

At a more prosaic level  potential buyers of property could see more clearly what they are buying, city planners could see better what needs to be worked on, and environmental researchers could see more clearly the impact people are having on an area.

Also Wave can provide some of the framework necessary to begin to begin to address tricky problems of privacy. Sensitive data can be stored on private waves, e.g. medical data for doctors and researchers, but the analysis of the  data could still be of benefit to all, e.g., if it’s tied disease occurrences to locations and  relationships between the environmental data and health were…quite literally…made visible.

“The publication of energy consumption and making it visible as overlays, could help influence the public into supporting more energy efficiency companies and businesses. It could also help citizens to try to keep their own energy usage down, to try to keep their street in “the green.”

Thomas notes:

“With all of the above, it becomes fairly trivial to write persistent Wave-bots that automatically send notice when certain criteria are met (pollutants over a certain level, for example). On publicly readable waves, anyone can use the data in their local computers, process it, and contribute results back on a new wave. Alternatively, persistent remote severs could run Cron jobs, or other automated processing, using services such as App Engine to run wave robots.

All these possibilities become “free” when using Wave as a platform for geographically tied data.”

But of course this is just the beginning!

Recently, I talked at length with Jeremy Hight who has been thinking about, designing and creating shared augmented realities, that anticipate the kind of dynamic, real time, large scale architecture we now have available through Wave,  for quite some time now.   This is exciting stuff.

Modulated Mapping: Talking with Jeremy Hight about Layers, Channels and  Social Augmented Experiences

image from Volume Magazine (Hight/Wehby)

Tish Shute: I know you have been involved in locative media from its early days. Perhaps we can talk about how AR continues the locative media journey?

Blair MacIntyre gave me this distinction, recently: “AR is about systems that put media out in the world, and immerse you in a mixed space.  Even the current “not really registered” mobile phone AR systems are still “sort of” AR (e.g., Layar, etc).

Locative media/ubicomp/etc are very different, in that they tend to display media on a device (phone screen) that is relevant to your context, but does not attempt to merge it with the world.
The difference is significant, and making it clear helps people think about what they do and what they want to do, with their work. The locative media space though points toward future AR systems (when the technology catches up!).”

Jeremy Hight: The need is to finish the arc that locative media and early AR have started and to now truly return to the map itself, but as an internet of data, interactivity, channels of data , end user options like analog machines once were but in high end tools, a smart AI-ish ability for it to cull data for the user, and to allow social networking to be in real world places on the map both in building augmentation and in using and appreciating it..not hacks..which have their place…but a rhizome, a branched system with shared root,end user adjustable and variable..this is the key.

This takes AR and mapping and makes a possible world of channels in space and this eventually can be a kind of net we see in our field of vision with a selected percentage of visual field and placement so a geo-spatial net, a local to world wide fusion of lm into a tool and educational tool

VR[virtual reality] has greatly advanced, but in nodes as it has limitations…LM [locative media] is the same…AR [augmented reality] is the way.. it now has locative elements and aspects of VR integrated into its functionality and nodes…it is the best option with all of these elements, greater hybridity and data level potential a well as end user and community sourcing potential

I wrote an essay for Archis’ Volume, the architecture magazine on a near future sense of some of this….a visual net on the lens like ar but with smart objects and social networking and dissent.

I also wrote of these things for immersive graphic design, spatially aware museum  augmentation, education through ar and lm and nod to the base interface of eye to cerebral cortex in layered and malleable augmentation in my essay “Immersive Sight” a few years back

image [above] is simple illustration of a possible example on a screen or in front of eye where in a mondrian show..the graphic design of information actually builds as one moves

(key is calibrated spatial intervals and related layers of further augmentation which is logical due to location and proximity)

from immersive sight on immersive graphic design: “The design can work with this in a way that creates an interactive supplemental set of information that is malleable, shifts based on location, builds and peels away as one moves closer to a work and plays with the forms of the works and the elements of the space itself. The sequence can contain many different elements and their interplay (both in the field of vision and in terms of context and layers of information). This is the model of sections of augmentation turning on and off at key points as individual spatial and concepts moments and nodes.

Another interesting possibility is that individual points of augmentation don’t turn off, but instead are designed to build as one moves in a direction toward a specific part of the exhibit. The design can work in a sequence both content wise and visually in terms of a delay powered compositional development and style in which each discreet layer of text and image does not fade out, but builds on each other into a final composition. This can form paintings similar to Mondrian perhaps if it is a show of similar works of that era or it can form something much more metaphorical and open interpretation of the space and content but utilizing a sense of emergence spatially in terms of the composition (pieces laid bare until final approach for effect).

Each section will be well designed, but they build in layers as one moves until finally forming the final composition both visually and in terms of scope of information or building immediacy. The effect can be akin to taking a painting and slicing it into onion skin layers laid out in the air at intervals, each the same dimensions, but only one section compositionally of the greater whole. This has many semiotic applications beyond its potential aesthetically and as spatialized information possessing a sense of inter-relationship as one moves.“

Tish Shute: One of the things I found very inspiring when I read your papers was that your ideas are not all dependent on a model of AR that would necessarily require goggles, back packs and lots of CPU/GPU – not that that wouldn’t be nice, but that even using “magic lens” AR of the kind smart phones has enabled in an open distributed framework would open up a lot of new possibilities for what you call modulated mapping wouldn’t it?  What kind of social augmented realities might be enabled by a distributed infrastructure like this [AR Wave]?

Jeremy Hight: right….I see that as wayyy down the road…most important is the one you talk about as it is more immediate and thus more essential and needed. Eventually the goggles will be like a contact lens and a deep immersive ar version of  this will come, that to me is certain, but a ways down the road.  An incredible amount is possible now, and this is a more pragmatic move as opposed to the more theoretical of what is a few steps from here. Thus it is more important and essential now. Tools like Google Wave are taking what even 2 years ago was more theoretical discussions of what may be and instead introducing key elements to a more immediate, powerful, flexible level of augmentation. What have been hacks and isolated elements are to be integrated and social networking, task completion, shared tools and graphics building and geo-location.

Tish Shute: I think some people question what augmented reality has to bring to the continuum of location based experiences that other forms of interface/mapping do not?

Jeremy Hight: right….and the schism between its commercial flat self and tests with physics etc and in between …there are a lot of unfortunate assumptions it seems as to where ar and lm cross and how ar can be many things beyond deep immersion or the opposite pole of a hockey puck having a magic purple line etc….like lm is seen as either car directions or situationist experiments with deep data…..the progression to me is deeply organic….and now augmentation can be more malleable, variable and end user controlled.

Tish Shute: Yes, it is really exciting time for AR.  Historically AR research has gone after the hard problems of image recognition, tracking and registration because we have had available to us these dynamic, real time, large scale architectures like Wave available (until now!),  so less work has been done on exploring the possibilities for distributed AR fully integrated with the internet and WWW hasn’t it?

A distributed augmented reality framework such as we have envisaged on Wave would  allow people to see many layers from many different people at the same time. ‬And this kind of model has been part of your thinking and fundamental to your work for a while, hasn’t it? But it is a very new idea to most people to think about collaboratively editing layers on the world, and to be able to view  augmented space through channels and networked communities?  Could you explain some of the ways you have explored these ideas and how they could be explored further now to create meaningful experiences for people?

Jeremy Hight: right..exactly…modulated mapping to me can be an amazing tool for students…back end searching data visualizations and augmentations based on their needs…while they do something else on their computer or iphone…that can be amazing..and not deep immersive..The map can be active, malleable, open source fed, and even, in a sense, intelligent and able to adapt. The possibility also exists for this map to have a function that based on key words will search databases on-line to find maps, animations, histories and stories etc to place within it for your study and engagement. The map is thus a platform and yet is active. Community is possible as people can communicate graphically in works placed on the map and in building mode in the tool. All the tropes of locative media are to be in a mapping system of channels of augmentation and a spatial net. The software by design will allow development on the map and communication like programs such as second life but in mapping itself.

image from Parsons Journal of Information Mapping Volume 2 (Hight/Wehby)

I wrote an essay a few years ago for the Sarai reader questioning the traditional map and its semiotics and need to reconsider – then did work looking into it and what those dynamics were and they got into 2 group shows in museums in Russia…so it actually was my arc toward modulated mapping…an interesting way to it! But yes the map itself..this is a huge area of potential and non screen based alone navigation etc. I see now that my 2 dozen or so essays in lm,ar, interface design and augmentation have all also been leading in this direction for about 10 years now

Tish Shute: I  love immersive visualization but can we “return to the map – the internet of data” as you mentioned earlier and produce interesting augmentation experiences that go beyond locative media’s device display mode without having the goggles, for example, through the magic lens of or smart phones?

Jeremy Hight: yes, absolutely.  the map in the older paradigm is an artifice born often of war and border dispute and not of the earth itself and its processes…the new mapping like google maps is malleable, can be open source, can read spaces and can be layers of info in the related space not plucked from it as in the past..this is amazing. the old map also was born of false semiotics/semantics like “discovery of new lands” or ” pioneer”  while the places were there already and names often were of empire…now this is no longer the case

Tish Shute: So geoAR is an a better way to express a new social relationship to mapping? And how does this fit into the evolving arc of locative media that evolves into augmented reality?

Jeremy Hight:…early lm was mostly geocaching and drawing with gps..it took new paradigms to invigorate the field  a lot of folks focus on tools and what already is, cross pollination can ground ideas that are more radical…a metaphor in a sense to place what can be in a familiar context.

Tish Shute: one of the great disappointments in VR has been its isolation from networked computing and also, up to now, augmented reality – to achieve an immersive experience with  tight registration of media/graphics have to create separate system isolated from the internet and power of the web.

Jeremy Hight: yes….this will change. vr is to me an island but ar takes a part of it and shifts the paradigm and new things open this way. Do you know the project “life clipper”? friends of mine..doing interesting things..they are a clear bridge betwen lm and ar….and from vr

in ar augmentation and what is being augmented become fused or in collision or in complex interactions as a means to a larger contextualization and exploration of what is being augmented..this is true in immersive or non ar….huge potential

vr is a space, now can be surgery which is amazing. but not layered interaction, thus an island and graphic iconography on a location can use symbolic icons which opens up even more layers (graphic designer/information designer in me talking there I suppose..)

Tish Shute: Yes !  talk to me more about layers and channels I think this is one of the most interesting questions for me  in augmented reality at the moment – what can we do with layers and channels and the new possibilities on connections between people and environments that these can create?

The ability for anyone to post something is critical to the distributed idea but one of the reasons I am so excited by Google Wave is I am fascinated by the playback function. How do you think this will enable new forms of collaborative locative narratives (nice post on Wave playback here ).

Jeremy Hight: We are in an age of cartographic awareness unseen in hundreds of years. When was the last time that new mapping tools were sold in chain stores and installed in most vehicles? When was the last time that also the augmentation of maps was done by millions (Google map hacks, etc)? The ubiquitous gps maps run in automobiles while people post pictures and graphic pins to denote specific places on on-line maps.

The need is for a tool that combines all of these new elements into an open source, intuitive layered and rhizomatic map that is porous (like pumice, organic in form yet with “breathing room” ),ventilated (i.e: adjustable, a flow in and out), and open (open source,open access,open spatialized dialog).

I wrote of this in my essay “Revising the Map: Modulated Mapping and the Spatial Interface .”( http://piim.newschool.edu/journal/issues/2009/02/pdfs/ParsonsJournalForInformationMapping_Hight-Jeremy.pdf )

image from Parsons Journal of Information Mapping (Hight/Wehby)

Tish Shute: One mapping project I really like is Mannahatta.  How could distributed AR contribute to a project like Mannahatta?

Jeremy Hight: that is a good example..imagine taking manhattan and having channels of options to overlay, that being an excellent option, and imagine being able to even run a few at once with deliniating icons..you can augment a space with history, data, erasure, narrative, scientific analysis, time line of architecture, infrastructure, archaeological record etc….endless possibilities, and this agitates place and place on a map into an active field of information with end user control…and open options for new layers

Tish Shute: and do you think we could do interesting things with AR on a project like Mannahatta even with the current mediating devices we have available – i.e. our smart phones as obviously the rich pc experience of Mannhatta has built for it’s web interface would not be available as AR at this point?

Jeremy Hight: yes….k.i.s.s right?   these projects do not have to only be immersive and graphic intensive……take how people upload photos onto google maps….just make that on a menu of options, there are some pretty cool hacks already..
…options is key, a space can have a community as well, building on it in software, and others navigating it, i see it near future and down the road..always have with ar really

image from Volume Magazine (Hight/Wehby)

Jeremy Hight: and yes, a lot of people focus on ar as its limitations and processing power needs as a major road block

Tish Shute: so do you see AR on smart phones adding any value to a project like Mannahatta?

Jeremy Hight: yes…that it can be integrated into other similar works and even disparate but cloud linked ones…so a place can be “read” in diff ways on the iphone….beyond its map location, and more can be possible if you are there…others away, so it becomes channels of augmentation

Tish Shute: AR like locative media puts who you are, where you are, what you are doing, what is around you center stage in online experience but it also “puts media out in the world” – people I think understand this well as a single user experience but we are only just beginning to think about how this will manifest as a social experience – could explain more about modulated mapping as an experience of social augmentation?

Jeremy Hight: Modulated Mapping is a tool that will allow channels to be run along the map itself. This will allow one to view different icons and augmentations both as systems on the map and in deeper layers of information (photos, videos, animations,  visualizations, etc) that can be turned on and off as desired. The different layers of icons and data may be history, dissent, artworks, spatialized narratives, and annotations developed that are communally based on shared interests, placed spatially and far beyond. The use of chat functionality in text or audio will be open in building mode and in mapping navigation/usage as desired. This also allows a community to develop or augment in the spaces on the earth. These nodes can be larger and open or small and set by groups in their channel. The end result is an open source sense of mapping that will also have a needed sense of user control as one can select which layers of augmentation they wish to see and interact with at any time. It also will incorporate all the functionality of locative media in mapping software and mapping. In building mode and in map mode, icons will be coded to represent within channels (remember that the person using it has selected channels of augmentation from many based on their current interests and needs). Icons will be coded as active to show work in progress in cities and the globe to both invite participation and to further agitate the map from the sense of the static as action is visible even with its icons as people are working and community is formed in common interest/need .

locative media got a buzz for “reading” places…when I helped create locative narrative that was what blew me away back in 2001…that we could give places a voice by placing data from research and icons on a map……this meant lost history or augmentation was possible as kind of voices of a place and its layers…….I called it “narrative archaeology.” We now have tools that can push these ideas and concepts farther..much farther…and with a range beyond what was before, and then the map was just a tool….but now we are returning to the map itself…..and this as place as much as marker..this is where ar takes the ball to use a bad metaphor

also that project could only work if you came to our spot of a 4 block augmentation and with us there to lend you our gear…we are far beyond that now but it had its place

Tish Shute: How do you see “in context” AR and something we might call “context aware” cloud computing models interacting?

Jeremy Hight: sure…and I must add that I have issues with cloud computing as much as it is a good idea...

Tish Shute: because of loss of autonomy?

Jeremy Hight: tivo is simply a hard drive…but it keyword reads and gives  suggestions..that is the is cro magnon link to what can be

Tish Shute: The nice thing about Wave is because of the Federation model, the cloud model and local store ur own data models should work together.

Jeremy Hight: yes..that is better…..loss of autonomy also opens up the arbitrary which is the flaw of search engines as we know it…even Bing fails to me in that sense

Tish Shute: how do you mean, could you explain?

Jeremy Hight: spiders  cull from words  but cull like trawlers at sea …. tested Bing with very specific requests.. it spat out the same mass of mostly off topic results….
I wonder if there is a way to cull from key words and topics from a user…not Orwellian back end of course…but from their preferences, their searches etc..

Tish Shute: did you see the discussion on search in the AR Framework doc? AR search will be a massively important thing that will take a lot of intelligence and all sorts of algorithm development won’t it?

Jeremy Hight:It also has one area of key functionality that moves into more intuitive software. Upon continued usage, the mapping software will “learn” and search based on key words used and spheres of interest the user is mapping or observing as mapped and will integrate deeper data and types of animations, etc. into the map or will have them waiting to be integrated upon user approval as desired. Over time the level of sophistication of additions and of search intuition will increase dramatically. The search can also, if the user wishes, run in the back end while working in the mapping program, or in off time as selected while doing other tasks. It also can never be used if one is not interested. One of the key elements of this mapping is that it is not composed of a closed set or needs user hacks to augment, but instead is to evolve and deepen by user controls and desired as designed. Pre-existing data,visualizations and augmentations can be integrated with relative ease.

Tish Shute: One of the things that Joe Lamantia points out about social augmented experiences is that they will operate across a number of different scales – conversation > product design & build team > neighborhood / town fixing potholes > global community for causes. How do designs for channels and layers change across these different social scales?

Jeremy Hight: quote myself …”The “frontier” is often defined as the space just ahead of the known edge and limit, and where it may be pushed out deeper into the previously unknown. The frontier in the world of ideas is not the warm comfort of what has been long assimilated; and the frontier in the landscape is not of maps, but of places beyond and before them⁄

The border along what has been claimed is not only that of maps – it is of concepts, functions, inventions and related emergent industries. Ideas and innovations are like the cloud shape that briefly forms around a jet breaking the sound barrier, tangible yet not fully mapped into measure. It is when things are nailed down into specific entities, calibrated and assessed, that the dangers may inflict themselves – greed, competition, imitation, anger, jealously, a provincial sense of ownership either possessed or demanded”. (from essay in Sarai reader). Otherwise channels and augmentation do not have to be socio-economically stratifying or defined by them. We built 34n  for almost nothing on older tools.

image from 34north 118west  (Spellman/Hight/Knowlton)

The ar that is not deep immersion can be more readily available and channels can be what end users need like the diversity of chat rooms or range of Facebook users among us.

I had two moments yesterday that totally fit what we talked about.  I went to west hollywood book fair and traditional directions off of mapping for driving directions were wrong and we got lost…our friend could only get a wireless signal to map on itouch and we had to roam neighborhoods then we called a friend who google mapped it and we found we were a block away….so a fast geomapping overlay with an icon for the book fair on some optional grid service or community would have made it immediate.  Then at the book fair talked to a small press publisher who is trying to map works about los angeles by los angeles authors on a map..she was stunned when I told her it could be a kind of google map feature option

it also has great potential to publish and place writing and art in places..both for commentary and access. imagine reading joyce in chapters where it was written about and then another similar experience but with writers who published on a service into their city.

Tish Shute: The challenge of shared augmented realities is not just a matter of shipping bits around, but also of how it we will use channels and layars – to create and negotiate different, distributed perspectives, understand a shared common core/or expressions of dissent (this came up in an email conversation with Simon St Laurent).

Jeremy Hight: well my example earlier could have been communal in a way too..a tribe sort of augmentation channeling ….like subscribing to list servs back in the day but of augmentation communities/channels, and for folks to build and use in shared live form, coordinating too

Tish Shute: one good thing though about building an open AR Framework is that as bandwidth/CPU/hardware gets better shared high def immersive experiences could be supported by the same framework..

Jeremy Hight: excellent

Tish Shute: were you thinking of the image recognition and tracking with this example?

Jeremy Hight: yeah….like scanning across a multi channeled google map augmentation with diff icons and their connected data…and poss social networking and fle sharing even in that mode…and rastering etc….could be cool with google wave - on the map..then zooming in a la powers of ten..(eames film).

-I have pictured variations of this for a few years now in my head like the example of my friends and I yesterday…we could have correlated a destination by icons in diff channels..one being lit events within lit channel in l.a map…maybe things streaming on it too…remote info and video etc… that would be awesome

Tish Shute: So many of the ideas in you paper on modulated mapping (see here) are brilliant use cases for shared augmented realities. Perhaps you could talk more your ideas about locative narrative because this is something I think is at the core of the kinds of experiences that a distributed AR Framework would make possible?

Jeremy Hight: on the project “34 north 118 west” we mapped out a 4 block area for augmentation of sound files triggered by latitude and longitude on the gps grid and map and the map on the screen had pink rectangles that were the “hot spots” where the augmentation had been placed.

image of interactive map with map based augmentation connected to audio augmentation on site for 34north 118west (Spellman/Hight/Knowlton)

We researched the history of the area and placed moments in time of what had been there at specific locations ….I called this “narrative archaeology” as it allowed places to be “read” by their augmentations…info that was of the place beyond the immediate experience (diff types of info) that otherwise would be lost or only found in books or web sites elsewhere. there now are locative narratives around the world but they need to be linked.  from humble origins “narrative archaeology” went on to be recently named of the 4 primary texts in locative media which is pretty amazing to me…but it is growing

- the limitations then were what I called the “bowling alley connundrum” – the specifc data had to reset like pins…..and was isolated….this led me to think about ar back then and up to now.  How these could lead to much more from that point, data that would be more layered, variable , fluid..yet still augmented place and sense of place and social networking within data and software

lifeclipper to me is a bridge

Tish Shute: But Life Clipper is isolated from the internet currently is it?

Jeremy Hight: yes…ours was too.. that is what google wave makes possible.. our project only ran on our gear..in 4 blocks…with additional auxiliary info online, and not malleable..but hey 2001 and all..

Tish Shute: so the sites for 34 north 118 west are still active though?

Jeremy Hight: oh yeah!

Tish Shute: nice I really like sound augmentation – have you seen Soundwalk?

Jeremy Hight: yes, very cool.. we chose sound only as it fought the power of image..instead caused a person to be in a sense of two places and times at once

Tish Shute: and in 2001 that was definitely a visionary project!

You must be very excited that finally the pieces are coming together to make this stuff scale!

Jeremy Hight: I can’t even tell you!! it is funny..i have known that this would come..just waited and waited…

..knew it needed the right people and tools..

..so the bowling alley connundrum led me to develop my project shortlisted for the iss (international space station)  as I thought a lot about how points and works are not to be isolated…but connected  and should be flowing in diff parts of a map….to open up perspective and connected augmentations , but also to think about the map again…not as a base only. then moved into my work with new ways to visualize time and it all really began to gell.  The ideas first were published as an essay (http://www.fylkingen.se/hz/n8/hight.html) and later my project blog (http://floatingpointsspace.blogspot.com/)

Tish Shute: One thing I noticed when I was reading your paper is how you have been exploring non-euclidian geometries.  Could you explain how this is part of your idea of modulated mapping?

Jeremy Hight: Yes, this first came to me when my wife was reading to me from a book on the Poincare Conjecture and I was hit with a new way to measure events in time and after months of sketches, schematics and research came to see how it could also be connected to a geo-spatial web of projects and augmentations.  It was published in the inaugural issue of Parsons School of Design’s Journal of Information Mapping which was an exciting fit. I call it “Immersive Event Time”(http://piim.newschool.edu/journal/issues/2009/01/pdfs/ParsonsJournalForInformationMapping_Hight-Jeremy.pdf)

so the last 3 years I have been working on how it could all work as channels of augmentation, and building and navigation as open and community in a sense as well as ai capability that was the time work especially. how time as experienced within an event is not a time “line”  but points on and within a form….and how this model is better for visualizing events in time and documenting them. it actually sprang form reading a book on the poincare conjecture, popped a bunch of other stuff together so one could visualize an event in time as like being in the belly of a whale..with time as the ribs..and our measure of time as the skin…and moving within it….hoping this will be used as educational tool

and this also can be tied to ar and map again…how documentation of important events can be kept within icons on a google map..then download varying visualizations based on bandwidth and desired format

Tish Shute: I have been thinking about is the new forms of social interaction/agency that these kinds of augmentations of space/place/time will create.  it seems there are two poles – one is the area Natalie Jeremijenko explores of shifting social relations from institutions/statistics to real time/location based/interactions and new forms of social agency.  The other pole completely is more like the cloud based AI and perhaps crowd sourced machine learning.

Your ideas explore the possibilities of both these poles.  And certainly one of the big deals of distributed AR integrated with would be the possibilities it opened up both for new forms of networked social relationships and for new ways to draw on network effects.

Jeremy Hight: and cross pollinations within …that is what my mind goes to

Tish Shute: The other night I met Assaf Biderman, MIT, from the Trash Track team. Trash Track doesn’t utilize AR but I could see that there are possibilites there.
What do you think?

Jeremy Hight: yes, absolutely, there can sort of skins on locations that user end selection can yield …like channels of place….and can range from pragmatic core to art and play and places between….how this recalibrates the semiotics of map…more than just augmentation as seen as a kind of piggy back on map..map becomes interface and defanged platform if you wil, interestingly my more poetic/philosophic writing led me here too

Tish Shute: I know they are at very different poles of the system but I do wonder how AR can bring some of the level of social agency/interaction that Natalie Jeremijenko works on into a productive interaction with the kind of innovations in Machine learning that Dolores Lab style machine learning!!and others are pioneering?

Jeremy Hight: Natalie’s genius to me is in practical functional tech that also opens deeper questions and even new openings of what is needed..amazing layers in her work that way.. succint yet deep..very deep

Tish Shute: Yes – I a just writing a post about her work – I find it deeply moving the way she has delved into the possibilities to using technology to open us up to our world.  One of the reasons I find distributed AR so interesting is because it will make it possible for all kinds of people to create and use augmentation in their lives and communities.

So to return to how a distributed AR framework could contribute to a project like Trash Track?

Jeremy Hight: what about using it for community, dissent and awareness raising then?  like Natalie’s work but building like a communal work of multiple points, like the old adage of the elephant and the blind men  sorry..metaphor – like one of my points in immersive sight was how one could take augmentation as multiple works sort of turning the faces of a thing or place…and how this would make a larger work even in such a flow so people moving in a space could also build..

what of ar traces left as people move calibrated to user traffic and trash as estimated in an urban space…like it goes back to chris burden in the 70′s making you know that as you turn the turnstyle you are drilling into the foundation and may be the one that collapses the building?

so their movements leave trash. Natalie is all about raising awareness to cause and effect and data , space and ecology. love that.  so maybe …
a feedback loop , artifact and user end responsibility can leave traces …trash…

.. cybernetics vs ecology and human waste

Tish Shute: could you elaborate?

Jeremy Hight: brain fart…that the mass of trash people leave is a piece at a tiime….and how like the space shuttle mission when it was argued first true cybernaut occured….one cord to air for astronaut..one for computer on their back to fix broken bay arm…if there is a way to build on that and in relation to the topic…..how this can go further, that machines do not waste as much…as ar is a means to cybernetic raise awareness..eh..In a sense it is like  the space shuttle mission when arguably the first true cybernaut occurred….one cord to air for astronaut..one for computer on their back to fix broken bay arm…if there is a way to build on that and in relation to the topic…..how this can go further, that machines do not waste as much…as ar is a means to cybernetic raise awareness..eh.. hmmm... sensors etc…wearables too – could be eco awareness with data and machine and human

what about a cloud computing system with a slight ai in the sense of intuitive word cloud and interest scans…..so as one moves through say new york they can be offered new ai data and services as they move ? could also be of eco interests? concerns about urban farming, eco waste, air pollution etc….perhaps with (jeremijenko element here)  sensors placed in locations and these also giving data reads in public areas  with no input but hard data itself……hmm..could be interesting

it can also give info of the carbon footprints (estimated prob unless data is public record somehow) of chain businesses  and data on which are more eco friendly as well as an iconography color coded and icon coded to the best places to go to support greening and eco friendly business?  and the companies could promote themselves on this service to attract eco aware customers who would be seeing them as kindred spirits and helping the
larger effort?

kind of eco mapping..and ar on mobile app

what about sensors that read air pollution levels, levels of solar radiation (to aid with skin protection in shifting light values in a city space..ie put on some skin cream now…), light sensors that detect density and over density in public spaces…to use the old trope in art of reading crowds in a space..but instead could indicate overcrowding, failing infrastructure in public spaces (which is a congestion that leads to greater pollution levels as well as flaws in city planning over time..), and perhaps a tie in to wearables……worn sensors  on smart clothes….this could form a node network of people in the crowds ….and also send data within moving in a space…

here is a kooky thought… what of taking the computing power and data of people moving in a space..and not only get eco data and make available to them levels of
data..but make possibly a roving super computer…crunching the deeper data of people open to this……a hive crunching deeper analysis of the space, scan properties from sensors, and even a game theory esque algorithm of meta data if say 40 people out of 50 hit on a certain spike or reading…and even their input…..I worked in game theory for paleontology in this manner for a time as a teen….a private project……   the reading can lead to a sort of meta read by what hits most consistently..as well as in their input..text of what they experienced, observed,postulated,analyzed even…. this could be really interesting…even if just the last part from collected data and not from any complex branching of servers..

I thought at 19 or so that the flaw in paleontology was in how so many larger theories were shifting exhibitions and larger senses of things like were there pre-historic birds that were mistaken for amphibean and then back again….so why not make a computer program and feed all the papers published into it and see what hits were counted in terms of an emerging meta theory…and landscape of key points being agreed upon…this data would be in a sense both algorithmic and a sort of unspoken dialogue …came from a lot of study of game theory one summer…

hope this makes some sense…I forgot to mention that I originally planned to be a research meteorologist and my plan in middle school or so was to get a phd and develop new software to have a global map and then run models of hypothetical storms across it in real time animations of cloud forms, radar and wind analysis/fields, barometric pressure spaghetti charts etc….and to also do 3d cut away models of storm architectures…so been into visualizations of complex data and mapping for a long time!

Tish Shute: Wow let me think about this one!

Nathan Freitas holding a Peek with Oliver+Coady partner David Oliver talking to fans at New York Tech Meetup – Mobile Meets Social

Volunteerism and participation in public life seem to come naturally to Nathan Freitas. Nathan is one of the leading innovators/developers in NYC in mobile strategy/design (for more on his Android development read on). And he is much in demand as speaker who shows others how to realize their mobile experience and design dreams (for upcoming speaking engagements see Nathan’s blog). But also Nathan has spent much of the last ten years working on new ways for causes and non profits to benefit from technology.

Most recently Nathan has started working part time for the NY Senate under, “Albany’s King Geek,” the new CIO Andrew Hoppin:

“The CIO team is organizing training sessions for senators and their staff on social networking platforms and how to pay attention to online feedback. Last week, they hired mobile specialist Nathan Freitas to create new phone applications that will allow citizens to get government news on the go.”

Also, Nathan is currently supporting engineer on, The Extraordinaries, a smart phone application that explores territory “beyond the flattening tendency of online relationships” (see this list from Andy Oram of the Questions on Government participation).  The Extraordinaries is Ben Rigby and Jacob Colker’s prize winning project  – “a smartphone application that delivers volunteer opportunities on-demand.”

Ben’s post, Information Age Volunteerism – Open Sourced! Crowdsourced! and the extensive comments give a detailed analysis and critique of this brilliant and creative new approach to volunteersim in the information age.

Nathan, in my view, is a great example of how to “do well by doing good.” And, I am particularly excited by the work Nathan and his partner in Oliver+Coady, David Oliver, are doing on Android, e.g., Nathan’s new gReporter – opensource, geotagging, media capture report client (you can download the source here).

I first met Nathan when I interviewed him about Cruxy in 2007 (see my post, The Mixed Reality Metarati and “Destroy TV:”  Merging Art, Technology, Politics and Play).  Nathan recently announced that “the fat lady has just uploaded her last song.” Cruxy was an innovative distributed music venture Nathan started with Jon Oakes.  Although, as Nathan explains, Cruxy “never really broke through in the way we hoped.” Nevertheless Cruxy seems to have been a fertile garden for ideas that are coming of age in Oliver-Coady’s current mobile experience endeavors.  As Nathan explains, “the world, including Apple and iTunes, has shifted to embrace some of the ideals we have always had – open formats, more ways to distribute and promote online, more avenues for niche content to be discovered and heard.” Cruxy’s technology platform, built by the incomparable Will Meyer:

“was a great success in my mind, being one of the first to fully embrace Amazon’s cloud and provide a widget-based commerce system that actually worked!”

Nathan has a new company, Oliver+Coady. But Nathan told me that he feels he is over his “start up phase.”

Nathan Freitas: I am just tired of the term “startup.” I’m more interested in being defined as person than a member of a corporation. Also I am more interested in the ideas of cooperatives, and have been working on this idea (see here for more on the New York Creative Cooperative ).

Tish Shute: You do a high percentage of non profit work. Are you still managing to keep the home fires burning in the economic downturn?

Nathan Freitas: There is definitely profit to be made in non-profits because even if you only get paid half of what you get for corporate work, it is worth it in terms of fulfillment, ego, respect, and general contribution back to the planet. However, I’ve also been investing time & energy w/o pay into thinking about how causes can benefit from technology for over ten years. So its not just something you decide to do one day, and suddenly are successful.

Tish Shute: What are some of the highlights of your non profit work recently?

Nathan: Well, The Extraordinaries project is definitely a highlight. It is focused on a whole new approach to volunteering and winning the first prize at the WeMedia Conference for the non-profit tech category was a great validation of the work. I am just a supporting engineer on the effort, which was founded by my good friend Ben Rigby (a longtime non-profit tech guy as well) and Jacob Colker.

Ben wrote this excellent book on mobile tech and organizing, Mobilizing Generation 2.0 He’s done a ton of mobile work with youth voters via his non-profit, Mobile Voter.

The Extraordinaries is really taking all of our joint experience and putting it into a whole new system that is meant to go beyond generic email blasts that just ask you to “send a fax” or “send a link”. it gives people specific tasks they can accomplish on their phone or in their local area using their phone.

Tish: Did you do Twitter Vote Report with Ben too?

Nathan: Oh, no, Twitter Vote Report was with a different group of folks…mostly east coast-based, organized by the TechPresident.com blog. But Ben and I worked on SMS efforts for the 2004 election. We sent 40,000 messages out to SEIU labor members and MoveOn members… really the first time SMS was used in a wide-scale manner to help get out the vote on election day.

Tish: Do you have a new mobilization project planned?

Nathan: Its all about The Extraordinaries right now. We’ve got a big launch coming in June, and are working actively to add more causes that can benefit from volunteers and organizations that have volunteers but don’t know what to do with them.

Tish: I was just looking at your post on Peek too.

Nathan: Yeah… fortunately that is a completely  “for profit” gig.  But I like the company a lot, and think their spirit of providing access to email at a very low cost plays well with the non-profit world.

Tish: So it isn’t just iphone apps that are paying the bills?

Nathan: Nope. iPhone is just an aspect. Everyone is so obsessed with it and how to strike it rich quick, but in the greater scheme of things, there is a huge ecosystem of mobility out there for you to find a niche in, if you are looking.

Tish: Are you able to monetize your work on Android yet?

Nathan: here and there… releasing some for pay apps soon, also including “free” Android ports in some high-profile iPhone apps we hope to have out soon. Some successful iPhone app developers are looking for people to port their apps to Android, as well.

gReporter – opensource, geotagging, media capture report client

Tish: So what are your hopes for Android development in general and your gReporter app in particular?

Nathan: I think Android represents right now what Linux on desktops did in 99 or 00  Though as we all know, cycles of technology seem to speed up. There is huge interest in it at the academic level and there is also a genuine interest in its use by non-profit/development agencies working around the globe.

You have to jump through hoops to get an unlocked, open iPhone w/o contract. Android provides an alternative solution to this, that acts more like a true platform, and not just a consumer product.

Tish: At the moment the Android market place is only for free apps right?

Nathan: No, it now supports paid apps. I just bought one today for $2.99

Tish: What did you buy?

Nathan: An app that allows me to turn my G1 phone into a WiFi hotspot sharing my 3G connection to anyone who connects.

Tish: So what are the most important aspects of Android in your view?

Nathan: There are  two sites to help demonstrate what is really going on with Android that makes it significant

1) Open Intents – this is the ecosystem of developers, all creating services and apps that interoperate, share data, and generally build a very rich Microsoft style platform:
except all these are open-source and built by lots of small developers and not one big corporation.

2) Android on HTC – this is the home for all the efforts to port Android to pre-existing HTC/XDA mobile phone hardware. You can see the status of ports here: http://wiki.xda-developers.com/index.php?pagename=Android_devices  Imagine… taking an old Windows Mobile HTC phone, and then popping in an SD card that reformats it over to Android brand new phone!  For much of Asia, India and Africa, there is huge interest in this.

Tish: Nice! You mentioned earlier that you are thinking of doing SDK for the android sensor API’s?

Nathan: That would be part of the geo report app… expanding it to capture all sensing data and report that when you submit your text, photo or audio report.  Right now it just detects your lat and lon, but no reason it couldn’t also check your compass, altitude and whatever other data the device might offer.

Tish: So what will your geo report do now?

Nathan: It allows you to submit a text, photo or audio report, tagged with geo coordinates, timestamp, and basic user info (name, email, home location, etc) to whatever server it is configured to us. it is the latest release of code used for the TwitterVoteReport and InaugurationReport efforts.

There is also just a lot to learn or use from the code itself, which is available at: http://github.com/natdefreitas/georeport-android

Lots of little lessons learned packaged up into a functioning application

Tish: How many sensor APIs does android have?

Nathan: http://developer.android.com/reference/android/hardware/SensorManager.html

int SENSOR_ACCELEROMETER A constant describing an accelerometer.
int SENSOR_ALL A constant that includes all sensors
int SENSOR_DELAY_FASTEST get sensor data as fast as possible
int SENSOR_DELAY_GAME rate suitable for games
int SENSOR_DELAY_NORMAL rate (default) suitable for screen orientation changes
int SENSOR_DELAY_UI rate suitable for the user interface
int SENSOR_LIGHT A constant describing an ambient light sensor Only the first value is defined for this sensor and it contains the ambient light measure in lux.
int SENSOR_MAGNETIC_FIELD A constant describing a magnetic sensor See SensorListener for more details.
int SENSOR_MAX Largest sensor ID
int SENSOR_MIN Smallest sensor ID
int SENSOR_ORIENTATION A constant describing an orientation sensor.
int SENSOR_ORIENTATION_RAW A constant describing an orientation sensor.
int SENSOR_PROXIMITY A constant describing a proximity sensor Only the first value is defined for this sensor and it contains the distance between the sensor and the object in meters (m)
int SENSOR_STATUS_ACCURACY_HIGH This sensor is reporting data with maximum accuracy
int SENSOR_STATUS_ACCURACY_LOW This sensor is reporting data with low accuracy, calibration with the environment is needed
int SENSOR_STATUS_ACCURACY_MEDIUM This sensor is reporting data with an average level of accuracy, calibration with the environment may improve the readings
int SENSOR_STATUS_UNRELIABLE The values returned by this sensor cannot be trusted, calibration is needed or the environment doesn’t allow readings
int SENSOR_TEMPERATURE A constant describing a temperature sensor Only the first value is defined for this sensor and it contains the ambient temperature in degree centigrade.
int SENSOR_TRICORDER A constant describing a Tricorder When this sensor is available and enabled, the device can be used as a fully functional Tricorder.
float STANDARD_GRAVITY
with a few easter eggs as well
GRAVITY_DEATH_STAR_I
SENSOR_TRICORDER

Nathan: They are all in the API however, there isn’t hardware to support all of them yet… for instance TEMPERATURE is not yet supported
nor is LIGHT.

Tish: and errr what is gravity_deathstar

Nathan: It is a value representing the fictional gravity on the Death Star from Star Wars – geek humour

Tish: That makes me think of this great essay by Julian Bleeker, Design Fiction: A Short Essay on Design Science, Fact and Fiction:

“When you trace the knots that link science, fact and fiction you see the fascinating crosstalk between and amongst ideas and their materialization. In the tracing you see the simultaneous knowledge-making activities, speculating and pondering and realizing that things are made only by force of the imagination. In the midst of the tangle, one begins to see that fact and fiction are productively indistinguishable.“

Picture below is Nathan playing his dream ukulele – designed using the free, open-source Inkscape vector drawing tool (see his open-source Ukulele plans here)
See Nathan’s blog for the whole story of how the Flying V Rockin’ Ukulele Design he posted to Thingiverse a few weeks ago, after being inspired by Bre Pettis’ talk at ROFLThang materialized at the  NYC Resistor “amazing workshop laboratory in Brooklyn where they let anyone come over and hang out at, to learn how to make, build and fabricate pretty much anything. They also have a laser (aka “LAAAZZZOOOR”) which you can think of as an automagic thing cutter-outer!”

so this….

became this …

Nathan and David presented Coovents at NYTM – Mobile Meets Social. They had a large group of questioners surrounding them (see picture below).   I talked to David after the presentation.

David Oliver was a software architect, user experience designer and product manager in the areas of mobile/wireless and electronic payment at IBM for over a decade.  Most recently, he lead the effort to productize a mobile client for IBM’s Lotus Connections enterprise social networking suite.  As a software architect, David was often technical lead for IBM’s business partner relationships with mobile device manufacturers.  Prior to IBM, David was co-founder of the Internet’s first “micropayments” company, Clickshare.

Talking with David Oliver

Tish Shute: How are smart phones are causing us to rethink what networked online relationships are all about.

David Oliver: You know these [mobile] devices are .. there’s a long time we tried to pitch that we’re going to treat them like they’re PC’s, or they’re just like anything else. But they’re really not. It may be the same coding style but the way you think about using them is entirely different. And the way you think about your program. so if you use html, java and that kind of stuff, yes it’s same code type but the way you think about it is entirely different. And to me these little devices make what you said [relationships inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.] a lot more possible than a PC. because in a PC you almost have to sit in front of it and like it controls you. But the device is so little and there’s almost no user interface by comparison. You got to be very smart how you build something so that it’s almost invisible. And of course that’s the beauty of the iphone, Apple will tell you. The idea of ubiquitous computing. Ubiquitous what? Am I really computing? I don’t feel like I’m computing. I feel like I’m interacting or something.

I think twitter is very cool. The real way it’s cool is that there’s no required client. You can access Twitter any way you want.  You can imagine other ways to use it. Tweet Deck happens to be a nice for now. What I like about Twitter is, if you give it a tiny bit of thought, the Twitter network’s complete white noise, just like the internet itself. If you put a probe on the internet it’s all white noise, it’s all unordered packets. It makes no sense. So it’s cool that Twitter is at the level of little bitty conversations, but collectively all white noise. Totally meaningless white noise.  There’s some neat things going on, but I think we haven’t seen barely the first of what you can do with Twitter.

The way I see it is it’s like instant messaging where you don’t instant message to someone you instant message to the network and there are listeners. So normally in the old world of IM like AOL IM I would say Tish let’s talk and I kind of like grab you. Then it’s a narrow pipe you to me. You can add a few people in and make a little group, and that makes a bit of a closed network. But with twitter you just like talk into the air as if I were standing over there and you had a twitter client here, we could have the same interview. Because I would be watching you OH I see Tish’s question. I’d be over there talking  and you’d be picking me up over here. I’ts like you’re talking into white noise, like at this bar. You choose to hear me, this guy is not choosing to hear me right now.

Tish Shute: So what does Android bring to the party?

David Oliver: They have the notion that you have a telephone platform that’s open, and that everybody can use. And it’s got a variety of sensor data – not just location but also accelerometer and compass and more.  So in theory you can almost broadcast that data. It’s connected to a network. It’s easy, open API’s to get at that data. But the question is who are you going to broadcast it to or who are you sending it to. What are they going to do with it? How are you going to control it, and make sure people don’t misuse it? As you heard with the services tonight, there’s a central kind of service necessary to filter and rebroadcast that stuff back out to places that need it, or can use it, or you want to have use it. I think the mobile device is only one piece of this. Nat and I always talk about well we do mobile applications but a portion of it is on the server. And coordinating with the people or the group or the central resource that brings all this data together.

Tish Shute: There seems to be a lot of new location based services – platforms to aggregate location based data being developed (e.g. xtify and viaplace). What do you think about the direction this development is going in?

David Oliver: It’s not conventional wisdom but it’s one of these things where when a crowd of people does something, and that means people themselves are the service providers,  when they all get together the net effect is greater that the individual effect would be. Pooling together makes more sense than doing it individually. Its a little bit like an advanced version of you have to have a password for every single site and you manage your passwords. Location is the same way. If you had to give every single website that you enjoyed your location data or tell them how to get it, what a huge pain. So they’re offering a way to do that in a more general sense. There are humongous privacy issues though. Just like passwords. Would you really trust a place that held all your passwords centrally?

Even with the most basic level of calling. Now that you can call from anywhere. Largely people are getting into a mode where their mobile phone is them. It’s always with them. That’s how you reach me. Forget the home phone, the work phone it’s just a mobile phone. You have an address attached to you, an address I can reach you at that’s location independent. So there some beauty in that and it’s very freeing. It makes your location unimportant, you can call me anywhere. You can text me anywhere, message me anywhere. You can be anonymous. My son told me something recently. “I love going to New York City because I can just walk around and nobody knows me. I’m completely anonymous. That’s the coolest thing”, he says. At one level that is a good thing and a lot of good things can happen that way. But this new thing is sort of the flip side where everybody knows your location. And we haven’t figured out if that’s a good thing yet. But we’re in the throes of that whole changeover happening. And we’ll see. There’ll be some misuse. I’m not an advertising guy, so the fact that everything’s got to be ad supported makes it potentially very creepy and very dangerous. So we’ll see how that evolves.

Is there any model where you can go “Oh this is just like ‘S’”? I don’t see where that’s possible. It’s a new world. Where you’re exposed all the time, potentially. And how do you figure out either as an individual or a larger group, society or whatever, when that works and when that doesn’t. And you know there’s going to be some mis-steps probably. But the tangibility creates some of these interesting opportunities, there are just some amazing things that could happen, really, really good things. But we’re not going to get there in one step.

One of the things that was really a killer for privacy and a killer for in some ways the internet, was during the dot com bust. Prior to the bust, there were web sites that you’d given your name and email, and they said “we promise to preserve this privacy.” But as soon as those companies went bankrupt, their email list was gold. It was value. And a bankruptcy judge, in a court in Delaware, created a legal basis to sell that data. Those things that were formerly private were no longer private – “no no no that’s got value. I’m going to sell it so the shareholders get their money.” So all these web sites who had lists of user names that they promised were private, became public information. That was one of the biggest blows to privacy in the history of the internet. That’s going to happen again and again. Like if MeetMoi goes out of business the likelihood is all your shit’s going to get sold. I’m sorry it’s all going to be sold. It’s all a big joke. And that’s why central services are horrid, and I don’t like anything about a central service.

There are some pragmatic things about the way routing on networks actually works and the fact that the internet has gotten very centralized itself. The core ideas of the early internet which were essentially a survivable telecommunications network, remember it was the defense department that did the original internet? So the original idea of the original internet was survivability. The Russians could bomb the daylights out of the United States, territorial U.S. and we would still have a survivable network. That was the idea. And therefore all the nodes were dispersed and did not count on each other, and could reroute. Well now one company UUNET or whatever they are they own the whole thing. And you can look up all their locations on some internet database. 18 well placed bombs and the whole internet goes down. That’s what happens over time.

Well the whole cloud thing is also kind of a myth. It’s a very neat sounding term, and some aspects of it are different and new. Nate and I do a lot of cloud computing, it’s all on Amazon.

But we’ve always had that. That’s called time sharing. Strictly speaking it’s a thin contractual accompanied by a much much much easier application programming interface. That’s what cloud computing is. It’s a very skinny contract. Timeshare was a  huge contract. Literally it’s legal and a little bit of API ease. It’s just timesharing. But at Amazon and the other ones too, you’re not responsible for your node going down. If it goes down, they push it somewhere else automatically. Your disk goes down. You’re not responsible for backing up your disk, it’s already on 14 copies on 8 continents. They do that. So it’s a higher level of service. Nate and I have this thing called slice host. And we’ll probably build some services on it, and if they get popular, it’s like a vending machine. You just drop in a dime, they give you another slice. No contract at all. It is growth and learning about old ideas. Like this whole idea of software as a service. The company called ADP Automatic Data Processing, who basically in short do payroll for everybody. It’s software as a service. It’s been going on since 1952 or something. It’s more like a reconception using modern tools. It’s like virtual worlds are a different thing. That’s a whole different beast.

Om Malik just wrote that he was surprised by the number of location-aware mobile services being launched in Austin, Texas, SXSW Festival this year in his post, “SXSW, Location Awareness Is The New Black.” I am not surprised.  The Mobile Meet Social, Tech Meetup in New York City in February was abuzz with new location aware applications and platforms for location based services.

I have been fortunate to attend a number of the key conferences this year. I am just back from ETech 2009 brimming with ideas and interviews for posts, and I am just catching up with publishing my interviews from Tools of Change 2009!  At TOC I did an interesting interview with Chris Brogan on how location awareness will change social media.  The web is flowing out into the world and questions of  who YOU are, WHERE you are, WHAT you are doing, WHAT is around you are becoming central to the rapidly emerging internetworked world.  TOC explored the digital future of books and heralded the transition of books into “everyware,” (see my interview with Adam Greenfield author of “Everyware: The Dawning of the Age of Ubiquitous Computing”).

There were many highlights for me at TOC 2009, including Nick Bilton’s keynote, “The Narrative is Changing: Sensors, Social Editors and the New Storytelling.” Nick developed some of these ideas further at ETech where I had the pleasure of playing Werewolf with Nick, Jane McGonigal, and many other ETech presenters and attendees. Nick is seated to the right of Jane (center) – click to enlarge. Jane is demonstrating the attitude of a werewolf to noobs to the game like me. Nick Bilton by the way is an accomplished werewolf (in case one day you find yourself as an innocent villager, seer, or healer defending yourself against him), for more photos of Werewolf at ETech see my Flickr stream.    Also see this excellent write up of Nick Bilton’s talk at ETech, “Sensors, Smart Content, and the Future of News,” by Richard MacManus.

Peter Brantley’s, “Literature as a (Web) Service,” talk at TOC also elaborated a theme, “literature as a service,” that was developed more broadly at ETech where Mike Kuniavsky of Thingm brilliantly presented on things as services (much more coming on this soon!).  Brantley, at TOC, concluded with an invocation for the future of machine parsed books.  And Bob Stein’s, “A Book is a Place”, Tim O’Reilly’s “Reasons to be Excited,” and Google’s  Jon Orwant’s update on Google’s quest for the holy grail, “converting images to original intent XML,” should not be missed (you can watch the videos here). Enjoying the serendipity particular to conferences,  I had a very inspiring long lunch conversation with  Ben Vershbow about “Books as Spimes” – more on this too soon!

Tim O’Reilly’s talks were inspiring at Tools of Change and ETech. His finger is on the pulse as always. Tim O’Reilly’s big vision is richly informed by  his own lived engagement with tools of change. Tim does not merely narrate how Twitter can facilitate publisher/curator relationships with authors and content, he is part of the evolution of Twitter as a publisher’s medium, through his own creative use. Also see these two key posts by O’Reilly Radar’s Joshua Michele-Ross on Forbes, The Rise of the Social Nervous System and a follow up by Tim,  The Social Nervous System Has More Than One Sense.

Tim’s  keynote for ETech is perhaps one of the best invocations and elaborations of a theme he has been developing in recent months – “work on stuff that matters” (see this excellent summary in Venture Beat).

In  my post, “Towards a Newer Urbanism: Talking Cities, Networks, and Publics with Adam Greenfield,” Adam makes some interesting comments on “the networked book” and  “unbooks.” Adam’s upcoming book, The City Is Here For You To Use,  is evolving as somewhat of an unbook.  Unbooks are, he writes:

“a container for long-form ideas appropriate to an internetworked age” and an approach that “can usefully harness the dynamic and responsive nature of discourse on the Web. At the same time, you preserve the things books are really good at: coherence, authorial voice and intent.”

Unbooks draw on lessons from the “open source” approach in software – version control, open-endedness to inform a collaborative development of books (this is an approach Usman Haque has taken to environments and cities – Urban Versioning System, for more see my interview with Usman here).

It seems fiiting that as I began writing this post an email came in from my friend Steve Fagin inviting me to The Last Book Project in LA, April 26th, 2009 – save the date.  Steve, is a brilliant artist, director, and impresario (please check out one of his early films, The Machine That killed Bad People, 1989 which was how I first became aquainted with his work).  Steve writes:

“Our vain glory is the attempt to resurrect the medieval illuminated manuscript through the invocation of our current alchemy, the new technologies, to conjure a future the past in reverse.”

The Last Book project’s illustrious team  include’s China’s “best bad girl novelist,” Mian Mian, as the reader of the Last Book.

The mobile phone is leading the charge into ubiquitious computing (although more immersive forms of experience will not be far behind (see my earlier post here). Note in Japan, “half the top selling books are WRITTEN on mobile phones.” While the “total spimy revolution isn’t here yet” – see “What Bruce Sterling Actually Said About Web 2.0 at Webstock 09,” we are blowing holes in “the spider’s web” of the end to end internet.

Since the NewYorkTech Meeetup – Mobile Meets Social, I have been thinking about the question: “How do timeliness and location-independence affect social media?”

This is a core question as David Oliver pointed out to me after the meetup (interview with David and Nathan Freitas upcoming).  David and Nathan are the principles of  Oliver + Coady a company focusing on Mobile/Social/Architecture  – also see Nathan’s blog for more.

At Tools of Change, I had the opportunity to talk with the social media guru and uber blogger Chris Brogan.

You can catch up on Chris’ TOC workshop, “Blogging and Social Media,” -  here (also see my Posterous here, “Smart Phones the Gateway Toy in Everyone’s Pocket”).

Interview with Chris Brogan

Tish Shute: I have been thinking about the question that David Oliver (principle of Oliver + Coady with Natahan Freitas) put to me: “How do timeliness and location-independence affect social media?” Mobility, is the key as David points out, NOT mobile as in a desktop in your hand, but timeliness (you do things when you need them) and location independence (you do things where you need them.

Chris Brogan: There’s a theory that I’ve been working on and I’ve typed a little bit into my blog from time to time where I call it ‘the secrets of the annotated world’. What I’m saying is, where I’ve experienced differences in Brightkite as applied to the iphone. When I first ever used Brightkite, I didn’t like it. The app didn’t do anything for me on the desktop. But when the iphone app came about, because it sucked my location off of my iphone, it said “can I tell people where you are?” And I’d say, “Why yes.” So what I’d do with it wasn’t so much talk to my friends and have a location element to it.  What I would do is I’d talk of the location. You know, “I’m at the Roger Smith hotel on 47th and Lex and they have a nice quaint little bar.”  And I would note, “try the pear fused gin.”

So what I’ve done is I’ve left a note in time and also in space. So now if another Brightkite user, and this is an example but I think this extrapolates to other places.. and I’ll think about a clearing house of space. What I think is that another Brightkite user who now comes somewhere near that space and who says what information, who said anything nearby where I am right now, will see I was at the Roger Smith Hotel, I had the pear fused gin it’s delicious. Oh! I like gin maybe I’ll try that. I think I’m helping put signposts up in space and time such that other people will come and read these glyphs that nobody else can see. And that’s sort of the imagery I have.

I think there’s an opportunity with this. One I think there’s an opportunity for this data to port across more places. I understand that in the data, if you own the customers then you own the future. But, if you don’t do it then somebody like Google will do it and I think that what will happen is Google maps will own this idea of you can add notes to a place. Google Local does this right now on the desktop,  so if I look up my local pizza place I now have would you like to write a review about it?  Would you add more info to the story?  And I thought, “wow this is getting there, except I still can’t see it nicely on my phone,” but now Google Locate arrives and now maybe I can.

So I think what it adds to social isn’t … Dodgeball tried awhile back and theirs is more like where are you, where are you, where are you. This is great if you’re a very mobile tribe, not everyone is. But from time to time we wander through different pieces of space and we would run across this information that would tie us more to the space. And so I’m passionate about how do I annotate and how do I maybe do more equipping. It’s interesting to me that Yelp as a restaraunt review product doesn’t have input from an iphone.  So in lieu of, I mean it lets me ask what reviews are up on the site, but in lieu of that  I can then put my restaurant reviews in Brightkite and then anyone can come find them based on time and space not based on a fixed site.

Tish Shute: I suppose it is hardly surprising (if disappointing to me) that some of the early location based services are trying to get mindshare by picking up on the glue celebrities give to mass culture. At the last New York Tech Meetup, OMGICU demoed a rather terrifying new pre-launch location based “participatory celebrity gossip application” which seems to combine all the worst features of social media with celebrity stalking, plus a narrative to change the notion of celebrity itself by “turning D listers into A listers.”

Chris Brogan: It raises a really interesting question because I’m willing to divulge my information freely. I’m not willing to divulge my family’s information. I don’t ever point out where they are in the world. One criticism that people have had when I use a product like Brightkite, is they’re saying you’re identifying that you’re not at your home. But I’m also telling people I’m not at conference. There’s lots of things, it’s not Brightkite telling them that. Its everything I do, it’s my blog. I’m not afraid of that. But I am interested in the negative impact of people doing that OMGICU thing because maybe I don’t want people to know I’m in Wyoming working with a client who maybe doesn’t want the world to know I’m working with them. It might be under non-disclosure. Or I might not want people to know which town exactly  I live in or where exactly my house is.

There’s some beautiful things that can be done on location, Wired had a big piece in their magazine about some location things. even that doesn’t get to the crux of the things I would do with it. Look at my analogy of annotating, I’ll put tags in the air that you’ll come and stumble across. Why can’t I find caches of these – like geocaching – why can’t I find caches of data that only exist in place. Why can’t I find data that’s … now I’m talking more like Gibson and Neuromancer.  But why can’t I find pods of data that can only be accessed by being in a locale.

Tish Shute: Have you seen Wikitude?  Something like Wikitude would be really great with some social tools.

Chris Brogan: Anywhere where it is possible to have all those hands raise up the load it is so much better because things get done faster, they get done more effectively. I am never going to get over to Williamsburg in Brooklyn and hang out there.  But if someone else was doing it I could always get the mapping, and I would know where I would want to hang out if I did. So I think these tools cry for this. The hint of this was in the Batman movie when he used eveyone’s phone to make a picture. The technology was obviously more vivid for a film sense.  But that is not unlike what we are doing with things like Twitter.

Tish Shute: One of the things I really like about Twitter is a a sense that i can drop in and out of the stream.  Is Twitter is the best current example of timeliness and social media?

Chris Brogan: I think that Twitter’s a great example, Twitter filters the noise out. If you search twitter, there’s a trending topics thing, that tells us now here are the top 6 trends going on at the moment. And so you can see when something surfaces that matters to you. Or you can search and try to aggregate.  So for example when the Australia fires story was breaking for a little while the only news you got was just links back to other peoples news. Then what we were searching for there was specifically this gentleman named Paul Mooney, and he was searching for first hand accounts. That was harder to get through twitter. But it’s almost like calling it out. It’s imagined in a game sense, we’re all in a big room, it’s almost like Marco Polo in the water. Maybe there’s 10 people and there’s one of us blind and he has to send out a ping, and all 10 ACK that they’re there somewhere. Well I think that the social meets presence type of things deal with, can you bubble that information up to me so that I hear it through the din and can you direct me to things that I shouldn’t normally have access to, that through my own senses cannot find.

Tish Shute: Without underestimating the value of social media  (as a blogger I owe much to social media), do you think though we need to develop different and more nuanced social tools as the web reaches out further into who YOU are, WHERE you are, WHAT you are doing, WHAT is around you?

Chris Brogan: There’s a lot to be said about the whole friends quote unquote equation  But I don’t understand, I mean believe me in the business sense I do, but I do not understand the amnesia that happens. When I go over and jump on any of those new services that you just showed me, I am now forced to go refind my friends. So the very first thing I need to do is cure my amnesia. Oh we’re friends here too? Oh I like that we’re friends. Why not here? Like open social but not like data portability. I need to carry my network with me. And another sort of future trend is that I also want private areas where it’s sort of the velvet rope social network, if you will. Right now we’re in the media area here. If we were doing this out in the hallway, then I’d get however many folk from that session. If we weren’t hitting the afternoon sessions, it’s say “Hey, great speech whatever or horrible speech I think you’re terrible.” We’d be bothered. So I think as we join the stalkerati, and as we know where everyone is all the time, we now often need a sort of “hush mode” that lets us go off and do some things in sort of small clusters.

The one number we’re up against is Dunbar’s number of 150 people. I’m fascinated by that because on twitter I have 40 thousand people, and I follow 36 thousand. I clearly don’t know what 36,000 people are doing. So what I’ve done instead, I’ve sort of wired a phone network. So I can ring you because I have this access to you, but I don’t always listen in on every conversation. Instead I dip in when I need to and otherwise I’ll wait until you come in. I use Tweet Deck like you do. I have one call in for @replies, I have one call in for my last name because sometimes people don’t form it right. And then direct messages and then the regular stream. The regular stream, I’ll see stuff go by and I’ll comment on as, but let’s just do quick math – 40 thousand people 1 percent of that is 400 people. 1 tweet each per day for 1 percent of my people would fill that screen for some time. So I can’t field all your conversations. But what I can do, as I demonstrated in my presentation, I use search a lot and I fall onto those conversations and then talk to people that way. So I’m using Twitter differently than most. And then the goal is to extrapolate that out until we find that there’s a bunch of journalists and people talking about journalism talking on this thing called journchat which is a Twitter group of people, and the only way to do that is to follow a tag. The beauty of such an option is in the way that the web could be TV, and still hasn’t in a meaningful way, in that I can deliver very exactly what someone wants to them. We don’t see the parity of that yet in a lot of spaces, and we haven’t really found the perfect ways to monetize the effort that’s there.

Clearly this is the magic. We were talking about this more than a decade ago. But it never came out the way we said. You were talking about the forward thinking part of it. I was in wireless technology for years. It just never was really there fast enough, and the reason was, of course, because the telco’s don’t really want to innovate. There’s no motivation to. They’re making money as it is. Telecoms adopted when VOIP looked like it was going to crush them. And so they absorbed it. and so they may or may not absorb location as a service. They might absorb presence as a service.

That’s my speech tomorrow. I stole most of my idea from Jeff Holver’s line. Jeff Holver said young people today bring their own dial tones with them. That means if you think of our US election, one person won and the other person lost. and the other person who lost was ringing everybody’s land line home phone numbers. And the person who won was on facebook and twitter and myspace and text messaging and all those sources. So dial tone as we understand it to make a telephone call is nothing like it used to be. When you call a home that’s where you call to ask a question. But that’s not, I mean I have a cell phone, my wife has a cell phone, there’s a land line but we never pick up the land line because it’s usually some calling about a bill we didn’t pay. So we listen to the message, pay the bill and we’re done.

Tish Shute: I have to ask you this as you are THE expert on the business of social media, how will people make money out these new services?

Chris Brogan: Honestly I’m really surprised that more of these things aren’t willing to try a subscription model. At the volume I use twitter, I mentioned if I do one tweet a day it goes out to 40000 people. So every time I do that it writes 40K records in 40K different accounts if you think of it as a database. Twitter technically should charge me. Maybe there’s other services they can offer in sort of a premium model, so there’s a free to premium kind of a plan. That’s how companies can make the money. I don’t know when we stopped feeling like we should pay for software.

I don’t know that we know just yet but I don’t think that there’s a business model for the phone except for subscription. So if twitter is to be the new phone it’s a subscription product. But you know, the iphone store, there’s people selling 99 cent to 5 dollar product and it’s going well. I’ve bought 6 apps now.

I think as we’re defining the etiquette and as we’re starting to understand how social platforms in general do and don’t work. I think what we’re finding is that people are having to relearn their business communications skills, not their marketing or their sales skills, because the old ways that we were trying to market were getting more and more frenetic, and getting more pushy, and more and more the walls came up and no one wanted to pay attention anymore. That’s where we are as a group in the western world for the most part. So now what we’re doing is we’re rediscovering relationships. I’ve talked to people from Tyson Foods. Why should I ever care about a company that sells chicken nuggets, except that I have a small child. So I’m finding conversations like that all the time. I think that’s what’s going to happen. I think there’s sales there. I think the problem is that the numbers are a little more wiley.  And I think that the numbers are a little more hops down the stream instead of direct. I can argue till the cows come home that you buying a post card and shipping out to a pile of people isn’t worth the money. But you at least can say well I got 45 sales out of those 5 thousand post cards that cost me this much. I got 45 sales and that numbers higher than this number, we did well.

Tish: I can see you are being called to your next meeting. Thank you so much for taking the time to answer my questions!

Adam Greenfield’s new book, The City Is Here For You To Use, is coming soon (photo above by Pepe Makkonen is from Adam Greenfield’s Flickr stream). Adam told me:

“I’m aiming at a free v1.0 PDF release on 05 June 2009, with the book shipping as quickly thereafter as humanly possible. There will be a version zero or public alpha in about six weeks.”

I am not good at waiting for books I really want to read to arrive. But, on the upside, it brings out my already pretty highly developed investigative instinct. So when Adam very generously agreed to do an interview, impatience turned into delight in tasting what is to come. And Adam is encouraging this kind of engaged anticipation. He writes (see post) that The City Is Here For You To Use, is shaping up:

“as something of an unbook avant la lettre. It’s why we’ve [Nurri Kim and Adam Greenfield] always insisted on keeping you in the loop as to the book’s fitful progress, it’s why I take every opportunity to test its ideas here, it’s why I make explicit the fact that your response to those ideas is crucial to their evolution and expression. And it’s why, even though the process is inevitably going to result in a static, physical document as one of its manifestations – and hopefully a very nice one indeed – we’ve committed to offering a free and freely-downloadable Creative Commons-licensed PDF of every numbered version of The City, from zero onward.

You buy the book if you want the object. The ideas are free.”

I found the opportunity to ask Adam questions about some of his subtle renderings of technology, culture, and being in urban environments challenging and very illuminating.  Although I definitely get the feeling I am asleep at the wheel on some of the critical areas he is thinking and writing on.

Knowing the depth and range of Adam’s thought in his seminal book, Everyware, and his blog, Speedbird, before I began the conversation I asked Adam to point me to some of his posts that reflect key ideas he is working on at the moment (Adam has recently posted The City Is Here: Table of contents).  Adam directed me to these three posts.

Antisocial networking

More songs about context and mood

Messenger, space, messenger body, messenger mesh

I may ramble and diverge, as is my nature, but these posts inspired many of the questions I ask.

Adam is currently head of design direction for service and user-interface design at Nokia and living in Helsinki, so I did not have the opportunity to do the interview in person. But I have glimpsed Adam’s world through his Flickr stream and some of these images have found their way into this post. But I suggest you browse Adam’s photography for yourself. I cannot do justice to the thousands of nuanced perceptions of cities, networks and publics you will find there. In the meantime, here are three glyphs of Adam Greenfield that I liked a lot.

“My favorite shoes,” “My favorite chair” and photo by Adam Greenfield, “Favoriteplace”

A Conversation (in gdoc) with Adam Greenfield

Tish Shute: Could you explain a little about the evolution of your thoughts on urban environments, ubicomp and interaction design? What shifts in your thinking have taken place over the last few years re the dawning of the age of ubiquitous computing? It is a couple of years now since Everyware, what aspects of the uptake of Everyware have most surprised, disappointed or inspired you? Which of the many thesis you discuss in Everyware have become the most crucial for The City Is Here For You To Use?

Adam Greenfield: You know, there’s a little passage in the liner notes to the second Throbbing Gristle album that I always think of when I’m asked questions along these lines. As part of their stance, they’d adopted the dry tone of a corporate annual report, and the preamble began by saying, “Since our last report to you, many things have changed. Indeed, it would be foolish to assume that it could be otherwise.” And I think that’s just exactly right: the world keeps moving, and the positions we’d staked ourselves to not so long ago may no longer be correct, or even relevant, to the one we find ourselves inhabiting now.

So, first, I think it’s important to cop to all the places in Everyware where I just outright got things wrong. There’s a passage in Thesis 50, for example, where I unaccountably mock the idea that “the mobile phone…will do splendidly as a mediating artifact for the delivery of [ubiquitous] services.” OK, this was admittedly written in a pre-iPhone world – and was correct for that world – but you can really see my parochialism showing here. It took the iPhone to make the proposition as blazingly self-evident to me in North America as it had been for quite some time to folks in Europe and Asia.

Having said that, though, I think I’m justified in taking a little pride in what the book got right. The broader trends the book set out to discuss – the colonization of everyday life by information processing – well, take a good look around you. And so one of the points of departure for the new book is taking everything posited in Everyware as a given: the urban environment, and most everything in it as well, has been provisioned with the kind of abilities you mention. So what now?

How do you go about designing informatic systems so they don’t undermine the wonderful things about cities? How do you design cities so they can incorporate networked informatics to greatest advantage? How, especially, do you accomplish these things when the disciplinary communities involved barely speak the same language? And how do you keep everyone’s eyes on the prize, which is the ordinary human being asked to make sense of these new propositions? These are the questions The City Is Here For You To Use sets out to address.

Adam talking about the “Le Long Ici” in Paris (also see Adam’s post, “The long here and the big now”)

TS: You mention that the hardest parts of  producing The City Is Here For You To Use wasn’t “keeping on top of all the emergent manifestations of urban informatics, or even developing a satisfying spinal argument about their significance” but getting the voice right.  It seems that now is the perfect time for a book that would really speak to a wide audience.  But also it seems that the city that is here for you to use is manifesting quite differently in different parts of the world?  You seem to be somewhat of a nomad, Japan to NYC to Helsinki.  Can putting together different views of urban informatics give us more depth perception on the emergence of ubiquitous computing?

AG: There’s no question in my mind that the long-term experience of everyday life in Tokyo, New York, and now Helsinki has been an invaluable asset to me, as I imagine it would be to anybody interested in thinking or writing about the networked city. It’s given me a certain amount of parallax, you know? And that, in turn, throws a really interesting light onto how the selfsame technology can appear in substantially different guises in different social contexts.

But explaining those things – those complicated, delicate negotiations – getting them right, doing them justice, doing so in a way that doesn’t dumb anything down, and still remaining accessible? It’s a challenge, let me tell you. You want to remain approachable and humane, but you also want to explain things like different jurisprudential takes on property, or how advocates of RESTful architectures think that REST is the reason why Internet adoption spread as rapidly as it did. If you want to enjoy even one chance in a hundred of getting your message across, you’ve got to start with an understanding that those subjects are MEGO territory for most people – whether they hail from Shibuya, Shoreditch or San Pedro.

Everyware icons: Information processing dissolving into behavior (Icons inspired by Timo Arnall; design by Adam Greenfield and Nurri Kim).  [Adam notes on his Flickr page that he tweaked these icons as section headers for Everyware]

TS: Could you explain more about what you term “onto” and “ontome” and how this differs from spimes and spime wrangling?

AG: You know, I never did get to develop that idea as much as I would have liked. In my mind, at least, “ontome” referred to the totality – the global environment of addressable, queryable, scriptable objects. (An “onto,” then, would be any given such object.) I guess I was looking for words that would do two things: allow us to distinguish between the instantiation and the class, and leave us with a better word than “spime.”

TS: When you say better word than spime this is this because….

AG: Euphony, primarily. : . )

TS: When I first used the Android app,  Wikitude, on Broadway, NYC – a street I have traveled thousands and thousands of times, and it offered up new information about itself, it was definitely an “OMG this is big!” moment for me. Like the first time I clicked on a screen and Amazon sent out a book in the early nineties (something so ordinary now it seems impossible that it was exciting but I remember it was to me!). But if I understand your post here correctly, isn’t Android with compass the first easy-to-use context-aware mediator for wrangling onto, ontome and spimes?

AG: Wikitude sure looks pretty impressive, and maybe even useful. But I would never, ever call it “context-aware.”

To my mind, at least two more things would need to happen before we could comfortably think of it a “context-aware spime wrangler.” First, the buildings and other public objects around you would actually have to be spimy – they’d have to report something of their past and current state to the network. And then, some application running on your phone would somehow have to cross-reference that state information with some fact about your current state of being, and deliver you relevant information.

So, let’s take your Wikitude example. You’re walking down Broadway and you pass an unfamiliar building, and for whatever reason you want to know more about it. Your phone pings the building’s dynamic self-description, and it replies to the effect that Andy Warhol had his Factory there between 1973 and 1984. If Wikitude chooses to share this particular piece of information with you, and not some other potentially germane factoid from the building’s history, on the strength of the fact that “The Velvet Underground and Nico” was in your last.fm playlist? That would constitute some small measure of context-awareness.

But you see how hard we had to try just to come up with an example, how forced it is, how so-what. And I have to say that – short of some infinitely supple system that really could model your innermost desires ahead of real time, and present appropriate responses to them – most so-called “context-aware” applications and services are like this. They’re either trivial, or wildly overambitious.

Maybe we don’t need for things to be context-aware for them to be useful, anyway. Certainly a great many objects in the world are starting to report their own status, and many more will do so in the fullness of time. And for the most part, all you’ll need to avail yourself of them is a Web browser running on a device that knows where it is in the world. An iPhone or an Android device will work splendidly – I called the iPhone “the first real everyware device” the day it came out and I was able to play with it for the first time – and in that way, the answer to your question is “yes.” Not to be longwinded or anything. ; . )

This Object has imperceptible properties. [Adam notes on his Flickr page: “This is a custom RFID-enabled transit pass that Timo Arnall had made up for me here in Seoul. I’ve (clumsily) tagged it with the icon that Nurri and I developed to represent just such emergent situations as this in the everyware milieu – that there’s no way for anyone to understand that this object has puissance beyond the obvious simply by examining it.”]

TS: It seems that  we are just at the beginning of understanding how to create networks of spimes (e.g. Pachube). Gavin Starks of AMEE (”the world’s energy meter”) once suggested to me that AMEE could be described as a facilitator of networked spimes (everything will have an energy identity). I think you may be familiar with AMEE because you keynoted next to Gavin at Xtech 2007.

I would be interested to hear your thoughts on AMEE?

When you discussed onto and ontome in this post, you noted:

“The greater part of the places and things we find in the world will be provided with the ability to speak and account for themselves. That they’ll constitute a coherent environment, an ontome of self-describing networked objects, and that we’ll find having some means of handling the information flowing off of them very useful indeed.”

Is the idea of “energy identity” that AMEE proposes an ontome? 

AG: See below for a précis of my feelings regarding environmental/sustainability initiatives, AMEE included. Uh…is AMEE an ontome? No. There’s just one ontome, and it’s coextensive with what folks now call the Internet of Things. It sounds like individual AMEE sensors would be “ontos.”

But I think the difficulty we’re having is a pretty good indicator that the terminology is more trouble than it’s worth. Sometimes a coinage, as satisfying as it may be lexically, just doesn’t work for people. These days I’m trying to get out of the neologism trade.

TS: I know when Usman Haque talks about Pachube he talks about spimes and spime wrangling. I asked Usman for his thoughts on spimes and onto/ontome and he gave me some comments.

Usman Haque: I think I had somehow missed the conversation about onto and ontome but backtracked through blog posts to piece it together (unfortunately some posts at v-2 and Studies & Observations no longer exist!). There are a couple of things that have made me uncomfortable about the word ’spime’: (a) the fact that it might be too easy to confuse with an “object”. A ’spime’ should also encompass relationships between things, and not just the “thingness” itself. (b) the sound of it (as Adam noted above). But then I am reminded of that horrible gooey interface used to plug into people in eXistenZ – it somehow seems appropriate that it should be a horrible gooey word, and not something that can disappear politely… So I like onto/ontome because it speaks to my first concern about ’spime’; but my second concern, it turns out, is not the problem I thought it was, and so onto/ontome might be… ahem… too euphonic! On the question of this thing people are calling the “Internet of Things”, I’ve tried in lectures to reframe it as the “Ecosystem of Environments”. Further, Vlad Trifa makes a delicious point that just as ‘web’ is different from ‘internet’, so too should we consider the “Web of Things” rather than the “Internet of Things”, something I agree with.

TS: It seems like this point about the difference between “the web of things” and the “internet of things” is pretty important?

AG: The parallel distinction between Web and Internet sure is! They’re two completely different things, right? And http is far from the only protocol that runs over the Internet. Now, as to what Vlad means by extending this particular distinction to the domain of networked objects, I don’t yet know, I haven’t had time to check it out. But sure, in principle I’d totally be willing to go along with the idea that there’s a meaningful distinction between two environments named that way.

No information is collected here; network dead zone

TS: I was just going over Greenfield’s principles of ubiquitous computing.  I am not sure that I see any current manifestations of ubicomp that hold to these priniciples yet?

AG: Oh, sure there are. Look at the work Tom Coates has done on Yahoo!’s Fire Eagle; look at Dopplr. And look at some of the steps other, less compassionate developers (e.g. Facebook) have been forced to take by their own users.

Look, those principles are just codifications of common sense and basic neighborly virtues, expressed in language appropriate to the domain of application. The best, smartest and most ethical developers have never needed guidelines to do the right thing. But especially inside companies and other complex organizations, people who want to implement compassion in their design of a technical system may occasionally find it useful to have some color of authority to invoke in their struggles. That’s all those five principles are there for, and I’m well satisfied that people have been able to use them that way.

Boffi’s take on the smart home- photo by Adam Greenfield

TS: In your post, More Songs About Context And Mood, you suggest a direction for interaction design that you point out is not far from Yvonne Rogers’ ideas in “Moving on from Weiser” about a switch in goal of ubicomp from Weiser’s vision of calm living (”computers appearing when needed and disappearing when not”) to engaged living – ubicomp technologies not designed to to do things for people but to help people engage more actively in things that they do (ensembles, ecologies of resources).

You also suggest interaction designers should be:

“parsimonious about the interaction design challenges our organizations do take on, with an eye toward reducing the complications of context (and the attendant opportunities for default, misunderstanding, misfire, time-wasting, and humiliation) to some manageable minimum.”

As you have pointed out, “we don’t do “smart” very well yet.” But paradoxically smart grids, smart homes, smart products etc. etc. are ubiquitously coming to market right now.

Yvonne Rogers suggests interaction designers should be:

moving from a mindset that wants to make the environment smart and proactive to one that enables people, themselves, to be smarter and proactive in their everyday and working practices

What areas might interaction designers most productively direct their attention towards?

AG: You note that things called “smart homes” and “smart products” are coming onto the market, and that sure would seem to be the case. But as to whether or not these things are genuinely smart, we don’t have anything more to go on than the marketing department’s word. I think you can already see that I tend to take language very seriously, and I really don’t uses like the “smart” here, or the “aware” in “context-aware.” They overpromise, they cannot help to set us up for failure and disappointment.

You know what I’d really like to see interaction design wrestle with? I would love to see a rigorous, no-holds-barred examination of the complexities of the self and its performance in everyday life, and how these condition our use of public space (and personal media in public space). I would love to see the development of ostensibly “social” platforms informed by some kind of reckoning with issues like vulnerability, dishonesty, the fact of power dynamics. In other words, before we deign to go about “helping” people, wouldn’t it be lovely if we understood what they perceived themselves as needing help with, and why?

I’d also pay good money to see talented interaction designers turn their efforts toward tools for the support of deliberative democracy, for the navigation of complex multivariate decision spaces, and for conflict resolution.

Locative is a mood – photo by Adam Greenfield

TS: I know you said this would take too long to explain but I couldn’t help noticing that you seem to be, perhaps, skeptical about the role of everyware can play in sustainable living and yet, it seems at the moment, in the hacker and business communities at least, the role of everyware in reducing carbon footprint/energy management etc, is the great green hope?

Will everyware enable or hinder fundamental changes at the level of culture and identity necessary to support the urgent global need – “to consume less and redefine prosperity?”

AG: I’m not skeptical about the potential of ubiquitous systems to meter energy use, and maybe even incentivize some reduction in that use – not at all. I’m simply not convinced that anything we do will make any difference.

Look, I think we really, seriously screwed the pooch on this. We have fouled the nest so thoroughly and in so many ways that I would be absolutely shocked if humanity comes out the other end of this century with any level of organization above that of clans and villages. It’s not just carbon emissions and global warming, it’s depleted soil fertility, it’s synthetic estrogens bioaccumulating in the aquatic food chain, it’s our inability to stop using antibiotics in a way that gives rise to multi-drug-resistance in microbes.

Any one of these threats in isolation would pose a challenge to our ability to collectively identify and respond to it, as it’s clear anthropogenic global warming already does. Put all of these things together, assess the total threat they pose in the light of our societies’ willingness and/or capacity to reckon with them, and I think any moderately knowledgeable and intellectually honest person has to conclude that it’s more or less “game over, man” – that sometime in the next sixty years or so a convergence of Extremely Bad Circumstances is going to put an effective end to our ability to conduct highly ordered and highly energy-intensive civilization on this planet, for something on the order of thousands of years to come.

So (sorry again, Bruce) I just don’t buy the idea that we’re going to consume our way to Ecotopia. Nor is any symbolic act of abjection on my part going to postpone the inevitable by so much as a second, nor would such a sacrifice do anything meaningful to improve anybody else’s outcomes. I’d rather live comfortably – hopefully not obscenely so – in the years we have remaining to us, use my skills as they are most valuable to people, and cherish each moment for what it uniquely offers.

Maybe some people would find that prospect morbid, or nihilistic, but I find it kind of inspiring. It becomes even more crucial that we not waste the little time we do have on broken systems, broken ways of doing things. The primary question for the designers of urban informatics under such circumstances is to design systems that underwrite autonomy, that allow people to make the best and wisest and most resonant use of whatever time they have left on the planet. And who knows? That effort may bear fruit in ways we have no way of anticipating at the moment. As it says in the Qu’ran, gorgeously: “At the end of the world, plant a tree.”

Biowall! - photo by Adam Greenfield

TS: In your post “Antisocial Networking,” you make some telling comments on the sorry state of social networking systems.

But you do also state:

Which of your concerns is social networking more central to than you might wish and why?

AG: Well, you know I’m interested in social interaction, interpersonal behavior, and in how these things play out in networked environments. There’s virtually no way for me to avoid dealing with Facebook, as wretched as I think it is.

Facebook is pretty hegemonic, in that its reach and influence extend further than the universe of people who use it. I bump up against it constantly, in a few different ways. People send me links I can’t access, because I’m not on Facebook. People spend time and energy trying to convince me that I’m really missing out, because I’m not on Facebook. The last few months, there’s even been a few people who feel justified in expressing some kind of exasperation, that they’re really pissed off…because they can’t find me on Facebook. It’s become the sovereign interface to any kind of life in public, and as a result a great many people don’t question its modes, tropes and metaphors.

So when it comes time to build some kind of situated interpersonal mediation framework, some kind of intervention in the fabric of the city, those are the tropes they reach for: accounts, profiles, friend counts, friendings and unfriendings, nudges and pokes. And as a member of a team tasked with the design of such systems, as a potential user of them, and certainly as someone exposed to the social rhetoric flowing downstream from their use, you bet these tropes become central to my concerns.

But what if we admitted that Facebook and the whole paradigm it’s built on are broken? What would things look like if we started from a more sensitive understanding of the interaction between self and others? Say, the understanding Erving Goffman was offering us as far back as the late 1950s? Then you’d understand the need for provisions like a “backstage,” a place to swap out one mask for another, the ability to present oneself differently to different communities and networks. That’s what I’m interested in exploring.

TS: Social networking systems in their current form are crude and express a very narrow bandwidth of human relationship. But already people are connecting everyware’s networked social acts to existing social networking systems. At the ITP winter show there was kickbee – networked fetal communication (and tweetmobile which used twitter as an acctuator for an ambient display) and green everyware (energy monitoring) is showing up in a number forms on existing social networks. But rather than just hooking up everyware to these existing flawed social network systems, does everyware require a reimagining of networked social interactions and social networking systems?

AG: That’s a great question, and I think the answer is clearly “yes.” It’s one thing to confine the consequences of that brokenness to the Web, and entirely another to let it bleed out into the world.

Does that mean any such reimagining is going to happen, that people will somehow refrain from plugging real-world outputs into these terribly flawed frameworks? Not a chance in hell. It’s too late to put a fence on that particular cliff. But maybe there’s still time to park an ambulance in the valley below.

“A graphic representation of a portion of the Earth’s surface, as seen from above” – photo by Adam Greenfield

TS: I saw you tweet that you met Usman Haque from Pachube recently. What do you find most interesting about Pachube and EEML? Will you design a project for Pachube to push the conversation further?  Did Usman ask you to take a role in the future of Pachube. How does Pachube enable the vision of The City Is Here For You To Use? I could go on for ever with questions,  so please do tell!

AG: OK, I should probably reiterate that my fundamental interest is in people, and in what they choose to make and do with technology, not the technology itself. For the last few years, I’ve particularly been trying to understand how people interact with each other and with the urban environments around them when those environments have been provisioned with the ability to gather, process and take action on data. And this is how I come about my interest in what Usman is up to with Pachube, because those “gather,” “process” and “take action upon” functions are generally accomplished by different systems, designed by different groups of people, at different times and to different ends. What Pachube aims to do is make the difficult and not-particularly-glamorous work of connecting these pieces a whole lot easier.

Think of it as a step toward enabling the ontome, this so-called Internet of Things we’ve been talking about, the same way basic protocols like HTTP and HTML enabled the wildfire spread of the Internet we’re familiar with. What Pachube offers is a way – a relatively straightforward and self-explanatory way – to plug any given compatible input into a similarly compatible output. So if you’ve got an air-quality sensor or a soil-pH sensor or a personal biometric monitor, you can plug it into Pachube, and someone else can grab the data those things generate and use it to drive a visualization, or the state of a physical system like a window, or whatever else they can imagine. It’s as close as anyone’s yet come to providing a plug-and-play backbone for the creation of responsive environments.

And I think it’s absolutely brilliant that it’s designed to work with Arduino and Processing, two lightweight, open-source frameworks that hobbyists and researchers (and even one or two more serious developers) around the world are already using to build things. (Arduino’s a kit of parts for doing basic physical computing – using data to drive lights, motors, and other actuators that have effect out here in the world – while Processing is a very accessible language to do dynamic and interactive graphics for screen-based media). Given both its openness and modularity, and its willingness to build on top of the very popular frameworks that already exist, I’m very excited to see what people make of and with Pachube.

I have to be honest and admit that personally, I couldn’t really care less about the environmental angle, for reasons that I went into at embarrassing length above. What I’m engaged by in Usman’s work is the idea that Pachube is helping to create an open platform for people to share data more readily. And while, no, he hasn’t explicitly asked me to take any particular stake in things, I’m always happy to lend a hand in whatever way would be most useful. I think it’s a project worth supporting.

As to how Pachube enables some of the ideas in The City Is Here, the answer has to do with the book’s call for every “public object” – every lamppost, bus shelter, commercial façade, and so forth – to support an open API. Something’s got to string all those objects together, present them to people as resources to be taken up and used, and Usman’s offered us a critical first step in that direction.

TS: Usman suggested, it might be interesting to ask you about “the tension between ‘could’ and ’should.’

Usman Haque: There are a whole bunch of things that we “can” do, technologically speaking; how do we decide what we ’should’ do, as we find ourselves in an age where we can build almost anything we can imagine…? particularly with reference to technology/privacy/security triumvirate. e.g., leaving aside that the majority of the world is *not* in the technology ‘paradise’ that we’re in, here in the west, only a small fraction of people are currently producing the technology that the rest of us use; one aim is to get people more engaged in the productive process, but, in a sense that will also mean the whole wide ecosystem of technology will be even bigger, both “good” stuff and “bad” (that qualification firmly placed on how it’s used), as opposed to now when we can focus on quite specific things that government & industry are doing and saying “that shouldn’t be happening….”. part of this relates to something adam said on his blog  in the comments (see here).” 

AG: I think the first part of answering that question has to involve figuring out who “we” are in any given situation. A “we” composed of seven Helsinki-based Linux developers would most likely arrive at very different answers than the United States Air Force Materiel Command or Samsung’s board of directors, right? So clearly, a first challenge is getting to some kind of pragmatically useful alignment between those local and occasionally even painfully parochial perspectives with what’s best for the Big We. And this challenge is only going to become more vexing as the ability to imagine, design, build and deploy informatic componentry gets more and more widely distributed. In this respect the spread of simple, modular, low-barrier-to-entry tools only makes things worse!

The primary issue that I can see here is that the inherent clock speed of technical development is so very much faster than that of any meaningful deliberative process “we” might bring to bear on it. A concomitant concern is that the sources of technical innovation and production are now so widely distributed that you can be reasonably certain that somebody, somewhere will implement any given technically feasible idea, no matter how offensive, poorly thought-out, socially disruptive or frankly stupid. A public toilet you have to SMS to unlock and use? A “Friend Finder” visualization with high locational precision and no privacy features whatsoever? A first-person rape-simulation “game”? A clunky brown iPod knockoff? Somebody thought each one of these things was worth the time, expense and effort to actually go about making it. They exist.

But I’m pretty old-fashioned in some ways, in that I think the good old Habermasian idea of the public sphere still has some life left in it. And I think it should be self-evident by now that there’s no necessary contradiction between even the newest (cough) “social media” and the formation of such a sphere. So you’ve provided a forum, and in it I get to express my belief that these things are stupid and pointless and probably should not have been built. And if somebody gets all het up about that, they can argue right back at me in comments. And eventually one or another of these positions begins to tell, in terms of regulation, legislation, and other tools of the juridical order, in terms of protest campaigns or organized boycotts or litigation…in terms of nonexistent sales!

There’s nothing new in any of this, of course, though indubitably some of the dynamics are amplified or accelerated by e-mail, Twitter and YouTube. My main contention is that informatic technology now has such deeply pervasive implications, and for things like presentation of self that previous waves of technical development barely touched, that “we” as societies need to be very much more conscious of the consequences before committing to any one course of action.

I should also point out that I do not, at all, believe that we’re “in an age where we can build almost anything we can imagine,” though I might buy “…two or three of almost anything we can imagine.” On the contrary, as I implied above, I think the global constraints on our ability to operate freely are already becoming quite evident, and will continue to grow teeth over the next few decades.

TS: Also Usman  added …

Usman Haque: ..where Adam said: in this regard, I very much *do* have a problem with “just showing up.” — something I feel that as well. but i always wonder: What happens when one appears to be mandating participation…?

AG: Look, I happen to have a strong – maybe some would say obnoxious or hyperactive or overdeveloped – sense of personal responsibility and accountability. I think one is basically committed to some measure of responsibility for the commonweal simply by surviving to the age of majority. The choice of how, particularly, to discharge that responsibility can only be yours and yours alone, but it can’t be ducked or gotten around without severe and entirely predictable consequences. So to Usman I’d respectfully suggest that I’m not the one mandating participation. Life is.

TS: It seems we have grown accustomed to striking a Faustian bargain on the internet today -  in order to share and distribute parts of our identity we are expected to give up key information to one site to store and disperse our data. I took part in a discussion with David Levine, IBM and Eben Moglen on privacy last year.  And Eben Moglen gave a succinct description of the elements of privacy and how they have been treated in the American Constitution that is, I think, relevant to unpacking some of the challenges of ubiquitious computing. Here are some extracts from that conversation where, Eben notes:

there are three elements that are mixed up in privacy and we tend not to notice which one we are talking about at any given moment.

There is secrecy – that is the data should not be readable by or understandable by anybody except me or people I designate. There is anonymity which is the data can be seen by anybody but about whom it is should be knowable only by me or people that I designate. And there is autonomy which isn’t about either secrecy or anonymity but which is about my right to live under circumstances which reinforce my sense that I am in control of my own fate. And this form of privacy is actually the one we talk about in the constitutional structure when we talk about the right to get an abortion or use birth control.

“Anonymity” is a condition that is a deep structuring characteristic of the internet as you, Lessig and others have commented on.  And frequently we are promised (questionably) “secrecy” or anonymity as privacy protection by services handling our data on the internet.  But Eben (one of the US’s great constitutional lawyers) points out that “autonomy” is a key form of privacy in the  US constitutional structure that is often compromised in situations where our digital selves may constrain our non-digital selves.

The real issue here is about the forcing of choices on us…digital aspects of identity can quickly acquire an inflexibilty that constrains our non-digital selves.

I see again and again the ways in which people now find themselves unable to make certain life choices easily because there digital self has acquired an inflexibility that constrains their non-digital self.

As we go beyond the end to end internet and we lose the structuring characteristic that has privileged anonymity: How do you see these three elements of privacy, anonymity, secrecy and most importantly autonomy, being worked out in a networked world beyond the end to end internet?

Are there any new structuring characteristics that could privilege autonomy? (which Eben indicates is linked to having a flexible identity).

AG: If we accept for the moment a definition of autonomy as a feeling of being master of one’s own fate, then absolutely yes. One thing I talk about a good deal is using ambient situational awareness to lower decision costs – that is, to lower the information costs associated with arriving at a choice presented to you, and at the same time mitigate the opportunity costs of having committed yourself to a course of action. When given some kind of real-time overview of all of the options available to you in a given time, place and context – and especially if that comes wrapped up in some kind of visualization that makes anomaly detection and edge-case analysis instantaneous gestalts, to be grasped in a single glance – your personal autonomy is tremendously enhanced. Tremendously enhanced.

But as to how this local autonomy could be deployed in Moglen’s more general terms, I don’t know, and I’m not sure anyone does. Because he’s absolutely right: Bernard Stiegler reminds us that the network constitutes a global mnemotechnics, a persistent memory store for planet Earth, and yet we’ve structured our systems of jurisprudence and our life practices and even our psyches around the idea that information about us eventually expires and leaves the world. Its failure to do so in the context of Facebook and Flickr and Twitter is clearly one of the ways in which the elaboration of our digital selves constrains our real-world behavior. Let just one picture of you grabbing a cardboard cutout’s breast or taking a bong hit leak onto the network, and see how the career options available to you shift in response.

This is what’s behind Anne Galloway’s calls for a “forgetting machine.” An everyware that did that – that massively spoofed our traces in the world, that threw up enormous clouds of winnow and chaff to give us plausible deniability about our whereabouts and so on – might give us a fighting chance.

TS: The concept of autonomy is signaled clearly in the title you have chosen for your next book, The City Is Here For You To Use, and is a theme of all your writing!  While you talk about many of the possible constraints to presentation of self and potential threats to a flexible identity that ubicomp poses, your next book signals optimism. What are your key grounds for optimism?

AG: It’s not optimism so much as hope. Whether it’s well-founded or not is not for me to decide. I guess I just trust people to make reasonably good choices, when they’re both aware of the stakes and have been presented with sound, accurate decision-support material.

Putting a fine point on it: I believe that most people don’t actually want to be dicks. We may have differing conceptions of the good, our choices may impinge on one another’s autonomy. But I think most of us, if confronted with the humanity of the Other and offered the ability to do so, would want to find some arrangement that lets everyone find some satisfaction in the world. And in its ability to assist us in signalling our needs and desires, in its potential to mediate the mutual fulfillment of same, in its promise to reduce the fear people face when confronted with the immediate necessity to make a decision on radically imperfect information, a properly-designed networked informatics could underwrite the most transformative expansions of people’s ability to determine the circumstances of their own lives.

Now that’s epochal. If that isn’t cause for hope, then I don’t know what is.

Newson Obamanook – photo by Adam Greenfield, “The fact that it was one of the happiest days of my adult life may have colored my appreciation of this space. A bit, anyway.”

TS: In your writing you seem to imply that we will not find answers to our new relationship with Everyware by transposing the internet onto things for convenience’s sake but rather like the bike messengers -  we must explore the rich and complex terrain of the city that is ours to use in a give an take relationship.  Through our own exertions we find- how “anything reasonably smooth and approximately horizontal can become a thoroughfare,”  rather than be served up the city as something for us to consume.

You seem to be suggesting our city becomes ours to use because of the way we use it in our personal journeys -like “the messenger subconsciously maps the contours of an economic geography – known sources and sinks of courier assignments, or “tags” – and a threat landscape, this latter comprised of blind corners, cable-car and metro tracks, and traffic lanes.

But bike messengers are the lone ranger of our big cities. Others surf the city in tribes that ride the roiling tides of highly networked information together. How are the “natural” gestures of these tribes, e.g. day traders, who yoked to the tracings of a hive mind, part of the city that is here for us to use?  I thought the comment @ginsudo made shortly after joining Twitter and setting up TweetDeck particularly poignant:

“watching Tweetdeck is like watching stock market of your personality ebb and flow. needs analytics to maximize inherent self-involvement.”

But, for many of us our work has more in common with the day trader than the bike messenger, and are we pretty hooked on the ever growing possibilities for “contact” and identity sharing/construction, social media has produced  (with all the”Here Comes Everybody,” C. Shirky, benefits and risks).  Early theorizing of a “calm,” invisible” ubicomp seems out of synch with the excitable, active, engaged, contact driven, “users” that are watching stock market of their personality (or personal brand) ebb and flow.

How will these excitable/exciting processes of contact and identity sharing that have captured of a pretty large segment of popular imagination (not confined to the West -services like Gupshup does much of the same curating, linking and distributing of identity that web based social media does in SMS) be/ or not be part of The City Is Here For You To Use?

AG: Let’s remember that ubicomp itself, as a discipline, has largely moved on from the Weiserian discourse of “calm technology”; Yvonne Rogers, for example, now speaks of “proactive systems for proactive people.” You can look at this as a necessary accommodation with the reality principle, which it is, or as kind of a shame – which it also happens to be, at least in my opinion. Either way, though, I don’t think anybody can credibly argue any longer that just because informatic systems pervade our lives, designers will be compelled to craft encalming interfaces to them. That notion of Mark Weiser’s was never particularly convincing, and as far as I’m concerned it’s been thoroughly refuted by the unfolding actuality of post-PC informatics.

All the available evidence, on the contrary, supports the idea that we will have to actively fight for moments of calm and reflection, as individuals and as collectivities. And not only that, as it happens, but for spaces in which we’re able to engage with the Other on neutral turf, as it were, since the logic of “social media” seems to be producing Big Sort-like effects and echo chambers. We already “maximize inherent self-involvement,” analytics or no, and the result is that the tools allowing us to become involved with anything but the self, or selves that strongly resemble it, are atrophying.

So when people complain about K-Mart and Starbucks and American Eagle Outfitters coming to Manhattan, and how it means the suburbanization of the city, I have to laugh. Because the real suburbanization is the smoothening-out of our social interaction until it only encompasses the congenial. A gated community where everyone looks and acts the same? That’s the suburbs, wherever and however it instantiates, and I don’t care how precious and edgy your tastes may be. Richard Sennett argued that what makes urbanity is precisely the quality of necessary, daily, cheek-by-jowl confrontation with a panoply of the different, and as far as I can tell he’s spot on.

We have to devise platforms that accommodate and yet buffer that confrontation. We have to create the safe(r) spaces that allow us to negotiate that difference. The alternative to doing so is creating a world of ten million autistic, utterly atomic and mutually incomprehensible tribelets, each reinforced in the illusion of its own impeccable correctness: duller than dull, except at the flashpoints between. And those become murderous. Nope. Unacceptable outcome.

Uncanny Valleys - Adam comments,”Our apartment in NYC as rendered in Google Earth, with realtime traffic, weather, daylight and shadow as well as geodetic, street grid and service overlays. Camera view is South; that’s First Avenue just left of center-screen.”

TS: Smart phone’s are now drawing everyware data into the system and the net is reaching into who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc..

Nathan Freitas says Android: “seems to be the platform most likely to socialize the idea that sensor data could be a piece of every application.” (Android APIs for a wide range of sensor data.)

What in your view will be the most likely platform, Android or what?, to socialize the idea that sensor data could be a piece of every application?

AG: An open platform. A platform with lots of hooks and ways to plug things into it, a strong developer community, a shallow learning curve and/or an easy-to-use, high-level development environment.

I don’t have a dog in this race, mind you. I couldn’t care less who gets there first.

TS: New location based services, e.g., Xtify and ViaPlace, are offering us ways to share location data across lots of different applications (eg Xtify and a dating application like MeetMoi ). In return for services that allow us to share information, we must give up key information up to one site to store and disperse (although there are many differences in approach to our data, from the Twitter stance “show but don’t own” as opposed to Facebook’s stance – “in order to show we must have rights to it”). But the basic model of Twitter – to provide a white noise platform for people to build service on top off seems to be being transposed to location based services. Obvious questions arise like what happens to our data in a start up like MeetMoi if they go belly up?  Apparently in the dot.com bust data was the first thing to go on the auction block in bankrupcy cases.

Also, I suppose it is hardly surprising (if disappointing to me) that some of the early location based services are trying to get mindshare by picking up on the glue celebrities give to mass culture. At the last New York Tech Meetup, OMGICU demoed a rather terrifying new pre-launch location based “participatory celebrity gossip application” which seems to combine all the worst features of social media with celebrity stalking, plus a narrative to change the notion of celebrity itself by “turning D listers into A listers.”

Hopefully location based applications  will not get stuck on “stalker, stalker, stalker” apps like OMGICU .

David Oliver, Oliver Coady gave me a good question: “How does timeliness and location-independence change our ideas of social media?

And how can we design new architectures that can reinforce the sense that I am in control of my own fate?

AG: But we’ve already come so far in terms of turning D-listers into A-listers! On a daily basis, I’m exposed to almost as many cues insisting I attend to nonentities and dullards like Robert Scoble as those insisting I attend to nonentities like Madonna or Thomas Friedman. It’s gotten ridiculous.

Now, how does timeliness and location change our ideas of social media? It makes them dangerous!

Look, even a proud Z-lister like myself – I’m a public person only in the most debased and degraded meaning of that word – I’ve had experiences that shook me up, like having someone approach me while I was quietly hanging out in the back of St. Mark’s Books, and wanting to strike up a conversation based on some talk they’d seen me give a year or so previously. Now part of learning to deal with this kind of thing is shrugging it off, being grateful and flattered that someone thinks you’re interesting enough to single out for that kind of attention, or chalking this up to Sennett’s observation about the constitution of urbanity. Or doing all three at once.

But let’s remember that at the end of the day, a “social network” is nothing but a group of arbitrarily distributed human beings joined by a communications channel, and those people have eyes and ears. The degree to which they recognize some shared interest gives them significance filters. If social capital accrues to those in the network who are able to claim some connection with a “celebrity,” no matter how fleeting, then such connections are going to be mobilized, made explicit. And now say the network has been provided with the tools allowing it to plot the appearances of those putative celebrities in space and time, and what do you get? You get a circumstance in which it is very, very difficult to maintain any membrane between the private self and the world, for anyone who’s even remotely a public figure, whether they particularly want to be a public figure or not. You get network effects that amplify those locational traces, and further undermine any possibility of anonymity, even anonymity-by-suspension-of-interrogative-awareness (which is a clumsy way of referring to that blasé matter-of-factness around famous people that most big-city folks eventually develop).

Am I letting myself off the hook? Not in the slightest. I passed Terence Stamp on the street not so long ago, and you bet I Twittered it. My only excuse was that I Twittered it to a closed loop of no more than a few dozen people. But then, who knows what those few dozen people will turn around and do with that fact, on the open networks to which they in turn belong? And that, too, is my responsibility.

I’m not sure there’s anything to be done about any of this but cultivate our own urbanity, learn to say “so what” when we happen to find ourselves next to Philip Seymour Hoffman in the line at Whole Foods.

TS: Zittrain in The Future of the Internet: And How To Stop It, foregrounds “generativity” and a generative devices (as opposed to appliances) as the most fortuitous starting point for: “tools to bring about social systems to match the power of the technical one.”

Are appliances a threat to the city that is here for you to use? How can generativity ensure The City Is Here For You To Use as Zittrain argues it has ensured, even if imperfectly, that the internet has been here for us to use?

AG: You know, I haven’t read the book, I’ve only heard him give the talk, so it’s certainly possible there’s a subtlety to the argument that I’m missing. But I’m not sure Jonathan isn’t simply wrong about this notion of generativity. Not that the concern is misplaced, but that he’s insufficiently trustful in human agency. Is a car “generative,” by his definition? Certainly not. And yet look at all the cultural production that goes on around “the car,” look at all the assemblages people make with cars, from Beach Boys songs to ghost riding the whip, from J.G. Ballard novels and Herbie the Love Bug to Tokyo Drift.

Or probably more to his point: look at the Japanese mobile-phone market – seemingly one of the most locked-down and unpropitious circumstances imaginable for the production of culture, in technical terms and Zittrain’s both. And yet fully 50% of the bestselling books in Japan last year were written on mobile phones. Not read, which would already be impressive enough (if “impressive” is indeed the word): written. What does that imply for his argument?

So, yes, I think there are grounds for concern in that we don’t allow technologies and frameworks to appear that unduly limit the scope of human creativity. Code is still law. But I also think people are quite amply able to reach into what would appear to be the least propitious technologies and tell their own stories with same.

TS: One aspect of Everyware that seems in need of some visionary yoga is the how we will relate to pixels anywhere.

In Urban Computing and its Discontents you mention how our technological trajectories often make it seem as if we seem to get fixated on particular scenes in movies, e.g., Minority Report. You point out that so many ambient informatics projects seem simply “to expand the reach of signage and advertising in dense urban spaces….as if we’ve become transfixed by the scene from Minority Report where heterosexual cop John Anderton is on the run from his colleagues.”

Ideas from the Minority Report continue to hold sway in designs as we saw in the recent MIT demo of SixthSense at TED.

But visions of augmented reality were pretty high profile in this years Super Bowl commercials this year (including a highly anthropomorphic imagining of ubicomp that was a kind of WoW mashup with a Pixar movie).

What recent movies/commercials have produced scenes mostly likely to be are new fixation fodder for ubicomp and why?

AG: I don’t think I’m qualified to answer that, actually. We don’t have a TV, so I don’t see much in the way of commercials, and most of the films I wind up seeing are the kind that play at Anthology Film Archives. What I can say is that science fiction is currently suffering in toto from an inability or disinclination to posit future scenarios that are any weirder or more visionary than those emerging from other sectors of the culture. And that would be fine, except sf has traditionally been the place where we wrestled with the imaginary.

We need that set of tools, badly. If for no other reason than something I glean from personal experience: essentially my entire professional career has simply been the leveraging of ideas and concepts I originally wrestled with in the encounter with William Gibson and Bruce Sterling when I was 16. Today’s visionary sf means tomorrow’s halfway-competent generalist.

Nurri Kim in the waiting zone – photo by Adam Greenfield

TS: My AR friend, Robert Rice, who is working on a markerless AR platform, notes that data visualization is one of the critical elements of AR in terms of “make or break.” Robert says, “even with the ultimate in ubiquitious data from everything, without good data vis it will all be useless”

Also something Cory Doctorow said to me last year has really stuck in my mind. When I asked him what happens when Cyberspace everts, he talked about a reverse surveillance society:

Cory: Well this is like Spook Country the new Gibson novel – What happens when cyber space everts – hmmm? I’m not sure I have anything very pithy to say on that EXCEPT………
Apart from all the traditional kind of overlay reality stuff, if there is one thing I am actually interested seeing from a virtual world migrating to the real world its instrumentation.
I think lot of things that are characteristic of very successful internet based business is that they are extremely finally instrumented so like Amazon knows in aggregate on a second by second basis how their site is being used by people and they can twiddle the dials in real time.

As users of the world we have very little access to that kind of instrumentation. We don’t even know how the tube is running. The tube knows how the tube is running and we kinda of don’t. I would be really interested in seeing that. You’ve seen Joi Ito’s WoW interface right. Have you seen it …

Joi Ito’s WoW interface seems a long way from the calm, invisible imaginings for ubicomp by early ubicomp visionaries?

AG: Well, he’s got a particular kind of neural wiring. And there’s not a thing that’s wrong with that, except that I’d never, ever want to assert that what’s appropriate for Joi Ito necessarily is or should be understood to be appropriate for anybody else. The point of calling for open systems and frameworks is to allow us maximum scope of diversity in the ways we choose to interface with the world’s richness and complexity.

TS: What new imaginings/possibilites do you see when pixels anywhere are linked to everyware?

AG: Product placement. Commercial insertions and injections, mostly.

Beyond that: one of the places where Mark Weiser logic breaks down is in thinking that the platforms we use now disappear from the world just because ubiquitous computing’s arrived. We’ve still got radio, for example – OK, now it’s satellite radio and streaming Internet feeds, but the interaction metaphor isn’t any different. By the same token, we’re still going to be using reasonably conventional-looking laptops and desktop keyboard/display combos for awhile yet. The form factor is pretty well optimized for the delivery of a certain class of services, it’s a convenient and well-assimilated interaction vocabulary, none of that’s going away just yet. And the same goes for billboards and “TV” screens.

But all of those things become entirely different propositions in everyware world: more open, more modular, ever more conceived of as network resources with particular input and output affordances. We already see some signs of this with Microsoft’s recent “Social Desktop” prototype – which, mind you, is a very bad idea as it currently stands, especially as implemented on something with the kind of security record that Windows enjoys – and we’ll be seeing many more.

If every display in the world has an IP address and a self-descriptor indicating what kind of protocols it’s capable of handling, then you begin to get into some really interesting and thorny territory. The first things to go away, off the top of my head, are screens for a certain class of mobile device – why power a screen off your battery when you can push the data to a nearby display that’s much bigger, much brighter, much more social? – and conventional projectors.

Then we get into some very interesting issues around large, public interactive displays – who “drives” the display, and so forth. But here again, we’ll have to fight to keep these things sane. It’s past time for a public debate around these issues, because they’re unquestionably going to condition the everyday experience of walking down the street in most of our cities. And that’s difficult to do when times are hard and people have more pressing concerns on their mind.

Citywarecrash – photo by Adam Greenfield, “An occupational hazard for urban screens.”

TS: I know in Everyware you mentioned that architects have play an important visionary role in imagining ubicomp and I know you work closely with your wife, artist Nurri Kim.  Robert Rice asked me the following question – which I will in turn ask you: “In terms of augmented reality do you think virtual worlds and virtual reality experts / leaders / are good pioneers for thought and guidance on AR? Or, should we look for new leaders, or where are new leaders emerging? Is the tech similar enough for the old crowd to be useful or is it different enough to be a disadvantage coming from the old models?.”

AG: I should make it clear that I have absolutely no interest in virtual worlds or virtual reality. The so-called virtual worlds I’ve experienced seem sad and really rather tatty – eversions of the most predictable adolescent fantasies of unlimited power, reinscriptions of all the usual politics – and completely lacking in just about everything that makes life resonant, meaningful and awe-inspiring. And anyway, to paraphrase J.G. Ballard, ordinary, everyday life is now far more vividly and fantastically weird than anything you’ll see in Second Life. I mean, Garry Kasparov was heckled by a radio-control dildocopter, Joe the Plumber’s off to Gaza as a war correspondent, a sea of dust-covered BMWs waits in the long-term parking lot at Dubai International for owners who are never, ever coming back.

Look to virtual worlds for insight into the hard work of negotiating the actual, with its physics, its entropy, its suffering, with all its constraints? Oh my goodness gracious, no.
And look to leaders? Never. Leaders are for followers, and who wants to be that? I don’t mean you can’t take inspiration and insight from the work of others – not at all – but use your own imagination, take some personal risk, do your own damn work.

Now, having said that. This opposition of virtual and physical worlds strikes me as increasingly a false one, as it does many people. The hard-and-fast distinction between “the real world” and virtual environments make less and less sense, as righteously satisfying as making it can sometimes seem. There may be attributes of this physical environment that are impossible to see or make use of without access to the networked overlay, and those attributes may in time come to constitute the primary wellsprings of a given place’s meaning. And if you’re offering me some insight that I think could be of utility in resolving the challenge of making this overlay accessible to all, equally, I’ll gladly accept it, no matter what domain or disciplinary background you claim as your own.

Am I aware of any such insight coming out of virtual worlds? No. As Bryan Boyer notes, “If you want to start talking about some serious cross-disciplinary pollination then you better take both sides of that disciplinary divide seriously. When your ubi- runs into my building with its boring HVAC, mundane load paths, typical finished floors, plain old foundations, etc., the transformative powers of comp are bracketed pretty seriously by the realities of the physical world.”

The Cloud Gate has landed – photo by Adam Greenfield, “Tell me this doesn’t look *just* like the descriptions of “stasis fields” in 70s SF. In fact, the picture looks practically CGId to me.”

TS: Some people thought the whole world would have been plastered with RFID by now.  But before that has happened markerless AR seems to be in our sights.

If I understand it correctly marker versus markerless AR has quite different implications for how the cyberspace of ubicomp evolves?  I asked Robert Rice (he is developing a markerless AR platform) to explain some of the differences.  He said:

Robert also pointed out to me that markerless AR may prove even more problematic for privacy:

Do you have any thoughts about the different roles that markerless versus marker techinologies will play in AR and Ubicomp?

AG: I need to admit that I’ve never until this moment heard the phrase “markerless AR,” although I’d think it’s more or less self-explanatory to anyone who’s been following this stuff. Let me make the distinction explicit, shall I, for anyone who hasn’t been? And you or Robert can correct me if I’ve gotten it wrong.

Augmented reality means that I have some mediating artifact that provides me with a visual overlay on the world. This could be a phone, it could be a windshield, it could be a pair of glasses or contact lenses, doesn’t matter. And you’re going to use that overlay to superimpose some order of information about the world and the objects in it onto the things that enter my field of vision – onto what I see. So far, so good: that’s AR 101.

Now where does that information come from?

What you’re calling marker-based AR implies that there’s some reasonably strong relationship between the information superimposed over a given object, and the object itself. That object is an onto, a spime, it’s been provided with a passive RFID tag or an active transmitter. And it’s radiating information about itself that I’m grabbing, perhaps cross-referencing against other sources of information, and superimposing over the field of vision. Fine and dandy.

But there’s another way of achieving the same end, right? Instead of looking at a suit jacket on a rack and having its onboard tag tell you directly that it’s a Helmut Lang, style number such-and-such from men’s Spring/Summer collection 2011, Size 42 Regular in Color Gunmetal, produced at Joint Venture Factory #4 in Cholon City, Vietnam, and packed for shipment on September 3, 2010, you’re going to run some kind of pattern-matching query on it. And without the necessity of that object being tagged physically in any way, you’re going to have access to information about it. But this set of information isn’t, necessarily, what the object itself, or its creators or merchandisers, want you to know about it; it could be derived from online discussion fora or review sites, or blog posts, or whatever. All there needs to be is a lookup table, essentially, that tells you where to find information about any object in the field of vision whose identity can be established.

Do I have that right? And if I do, then as I understand it, the distinction is primarily a pragmatic one: it’s just easier to get to an augmented world, by far, if we don’t actually have to go to all the trouble of tagging everything in the world with its own dedicated RF transponder. Easier, and cheaper, and quicker, and more environmentally sound besides, because the relevant traffic is in bits not atoms.

Unless I’ve missed something, you don’t, then, get the distinction between classes of objects and instances of same. Sometimes, when there’s a 1:1 correlation between the two, that’s not going to matter: I’m walking down the street in Madrid, and my glasses or whatever can easily recognize that this building is the Caixa Forum. There’s only one of it, and I can get a positive ID via pattern recognition. But for some edge cases – twins and lookalikes, mostly – the same thing is generally true of people.

But other times it will matter. Is this specific watch a real, $10,000 Panerai or a $50 Kowloon fakery? How has this black 1998 Honda Civic over here differ from this other one in terms of its use and maintenance history? Does this O-ring gasket need to be replaced? I don’t see how you extract data from specific instances of things without the necessary sensor instrumentation, transmitter, etc., being coextensive with the object in question or very closely colocated with it over time – in the terminology you’re using, a “marker.”

So using these terms, I’d say that “markerless” AR comes first, is relatively easy to deploy, and generates not-insignificant value. But – again, unless I’m missing something – there are some things that it won’t ever be able to do, and for those things you need some provision for self-identification and self-location.

Ultimately I think it’s a distinction without a difference, from the user’s point of view. People will care much more about the source of whatever information shows up on their overlay than the precise technical means used to get it there.

The surrender to cynicism – photo by Adam Greenfield

TS: Much early thinking around ubicomp seems to have come from visionary architects and engineers but recently I was at the O’Reilly Tools of Change for Publishing Conference (publishing in the Digital Age) and I met several book futurists.  It struck me how ubicomp from the perspective of the book created some interesting questions for how particular material cultures will shape and be shaped by Ubicomp differently.

I noted, Google seemed well down the path to holy grail “converting images to original intent XML.” And Peter Brantley talked about machine parsed books.

At TOC there were many suggestions about how books might manifest as everyware. (Although it did not seem that many people felt books had a special relationship to time and history and would not vanish as one of the great metaphors of calm and solitary enjoyment in our culture soon).  Books as everyware will, it seems, include, amongst other things:

books that read books

books that read context

context that reads books

books that read me

books linked to mobility – timeliness and location independence

books that are not books

books becoming babble

books bubbling up from the babble

There is an Institute of the Future of the Book. Will all former material cultures require their own institutes of the future to guide their cultures into everyware?  Do you think books transition into everyware is especially significant and why?

AG: But all objects have a relationship to time and history, no?

TS: Yes! What I meant to convey really was the idea that many people expressed at TOC that books had a privileged relationship to knowledge in our culture that was valuable and related to some aspects of their current form, and that books as everyware, e.g. machine parsed books, and more sociallly generated forms would not replace that entirely.

AG: Gotcha. Well, I certainly agree that books constitute an interesting category unto themselves – I’ve held onto my physical books, and in fact still spend a fortune buying new ones, where I stopped buying music on discs a long, long time ago. But I don’t think this state of affairs can or should obtain forever.

Lately there’s been a good amount of thought around the notion of “unbooks,” which I regard as a container for long-form ideas appropriate to an internetworked age. By building on some of the tropes of software development, mostly having to do with version control, open-endedness and an explicit role for the “user” community, unbooks can usefully harness the dynamic and responsive nature of discourse on the Web. At the same time, you preserve the things books are really good at: coherence, authorial voice and intent.

The important part is in acknowledging two points which have usually been understood as contradictory, but which are actually nothing of the sort: firstly, that the expression of ideas in written form has something to learn from the practices that have evolved around the collaborative creation of dynamic, digital documents over the half-century-long history of software; and secondly, that certain ideas require elaboration in the reasonably strongly-bounded form we know as a “book,” and cannot meaningfully be shared otherwise. A third point, concomitant to the second, is that despite recent technical advances, screen-based media still cannot, and may not ever fully be able to, deliver the extratextual cues and phenomenological traces that support, inform and extend the meaning of written documents.

The unbook lets you have your cake and eat it too. So, for example, when we publish The City Is Here, one of its manifestations will be a static, physical document – and hopefully, if we do our jobs well, a very nice one indeed. But even before that, you’ll be able to download a Creative Commons-licensed PDF of every numbered version of the manuscript, from zero onward. Bottom line: you buy the book if, and only if, you want the object. The ideas are free.

TS: David Brin sees two futures:1) the government watches everybody, and 2) everybody watches everybody (the latter he calls “sousveillance”).  My friend Ben Goertzel says “hooking AI up to a massive datastore fed by ubicomp is the first step toward sousveillance?” What do you think the role of AI in ubicomp will be?  Is it worth thinking about what is the first important “AI meets AR” app is?

AG: I don’t believe that artificial intelligence as the term is generally understood – which is to say, a self-aware, general-purpose intelligence of human capacity or greater – is likely to appear within my lifetime, or for a comfortably long time thereafter.

Having said that, your friend Ben seems to be making the titanic (and enormously difficult to justify) assumption that a self-aware artificial intelligence would share any perspectives, goals, priorities or values whatsoever with the human species, let alone with that fraction of the human species that could use a little help in countering watchfulness from above. “Hooking [an] AI up to a massive datastore fed by ubicomp” sounds to me more like the first step toward enslavement…if not outright digestion.

Sousveillance - the term is Steve Mann’s, originally – doesn’t imply “everybody watching everybody” to me, anyway, so much as a consciously political act of turning infrastructures of observation and control back on those specific institutions most used to employing same toward their own prerogatives. Think Rodney King, think Oscar Grant.

TS: I have one last question from Usman Haque.

Usman Haque: insofar as a lot of what adam describes as desirable could be said to constitute pretty radical socio-political change (or perhaps… “adjustment”) i would be really interested to know how his current work @ nokia is or isn’t able to gel with the themes of his writing. in some senses there’s quite an undercurrent strongly challenging corporate practices, in other senses it could be seen as gentle nudges. how does adam see it? and how about the nokia behemoth? does he have success nudging nokia towards the kind of world he would like to see (i imagine the answer is ‘yes’ otherwise he wouldn’t be doing it…) but i’d love to know more about the limits/challenges.

AG: I am told that Henry Kissinger, on his first trip to China in 1971, asked Zhou Enlai whether he thought the French Revolution had or had not advanced the cause of human freedom.
Zhou thought for a moment, pursed his lips, and replied, “It is too soon to tell.”