The video above, The Imawik commercial, is a collaboration between In The Can Productions and Paige Saez for Makerlab

“The Imawik (ImageWiki) is a visual search tool for mobile devices. It allows for the ability to turn images into physical hyperlinks, conflating visual culture with a community-editable universal namespace for images.”

Paige Saez is an artist, designer and researcher.  In 2007 she founded Makerlab with Anselm Hook, an arts and technology incubator focused on civic and environmental projects.

Paige and Anselm (see my interview with Anselm Hook here, Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook) have been asking a very important question:

“Who Will Own Our Augmented Future?”

But most importantly, they have been actually developing applications (again see my interview with Anselm for more background on this), to allow people to play with, hack and explore and create with the physical world platform, and to imagine new possibilities for physical hyperlinking and augmented realities.  This is pretty important stuff, and kudos to Paige and Anselm for beginning this work before the big players – Google Goggles, Point and Find, and SnapTell came hurtling into the field of visual search and physical hyperlinking  – see this demonstration of translation and optical character recognition in Google Goggle’s.  Also check out Jamey Graham’s (Ricoh Research) Ignite presentation at Tools of Change, 2010 – Visual Search: Connecting Newspapers, Magazines and Books to Digital Information without Barcodes, for more see ricohinnovations.com/betalabs/visualsearch.

We are only just beginning to get a glimpse of how contested the social commons of the physical world platform is going to be – see the Yelp controversy.

As Paige points out:

“The lens that you are actually looking through was as important as what you were looking at. And democratizing that lens became the most important thing that we could possibly do.”

I am in total agreement.  One reason I have so much enthusiasm for ARWave (note: if you are interested in following the developer conversations there are several public Waves) is I see this open framework playing an important role in the democratization of our augmented views, by creating an open, distributed, and universally accessible platform for augmented reality that will allow the creation of augmented reality content and games to be as simple as making an html page, or contributing to a wiki.

Federation, real time collaboration, linked data – ARBlips that contain metadata that is usable for semantic searches, and modified wave servers that can listen to and respond to  SPARQL HTTP requests properly (see Jason Kolb’s many interesting posts on XMPP and Wave).  These are just some of the reasons why ARWave could revolutionize augmented reality searches and more! (see my presentation at MoMo13 – video here)

For more on real time social augmented experiences see our panel, The Next Wave of AR: Exploring Social Augmented Experiences at Where2.0 2010, and don’t miss the Where2.0 conference which has been the crucible for the emergence of location technologies.

Augmented realities, proximity- based social networks, mapping & location aware technologies, sensors everywhere, linked data, and human psychology are on a collision course in what Jesse Schell calls the “Gamepocalypse”   See Jesse Schell’s Dice 2010 talk here, and check out his Gamepocalypse Now blog.  As Bruce Sterling’s notes in his post here:

*Another precious half hour out of your life.  However: if you’re into interaction design, ubiquity, social networking, and trendspotting, in the gaming biz or out of it, you’re gonna wanna do yourself a favor and listen to this.

And don’t forget to register now for Augmented Reality Event (ARE2010 in 2-3 June, 2010 – Santa Clara, CA).

Bruce Sterling, Will Wright, and Jesse Schell will be keynoting, and there is a totally awesome line up of AR innovators and industry leaders, including Paige and Anselm!

And:

You are in luck!

Here is a discount code for the first 100 folks to register to the event (before the end of March). Go to the registration page, type in code AR245 and “you’ll be asked to pay only $245 for 2 full days of AR goodness.”

“Watching AR prophet Bruce Sterling, and gaming legend Will Wright, visionary game designer Jesse Schell deliver keynotes for this price – is a magnificent steal.  And on top, participating in more than 30 talks by AR industry leaders will turn these $254 into your best investment of the year,” as Ori  put is so well on Games Alfresco!

If you want a preview of just how exciting it is to be involved in augmented reality right now check out Ori Inbar’s great round up on our latest monthly Augmented Reality Meetup NY (or as, Ori notes, we fondly like to call it ARNY.)  There is lots of video up now (much thanks to Chris Grayson, who  live streamed it).  Augmented Reality Magician, Marco Tempest, is an absolutely must see.  (developers note this is an awesome use of open Frameworks and OpenCV).   The video of the show includes a rare explanation of how it all works  – see here.

Talking with Paige Saez – “Software is candy now!”

Tish Shute: What interests me about ImageWiki is that you have thought about physical hyperlinking beyond the obvious of where to get your next good hamburger and beer, right?

Paige Saez: Right. It was interesting for me in just thinking about the two things. How do you design a tool to work in a way that people are getting value from it? And also, how do you make it work in a way where people can explore and hack it? I think the most interesting technologies, and this is probably something somebody else said sometime, are the ones that disappear, that we don’t see, instead we see through. They become just the intermediaries.  They don’t interfere with what we are trying to do.

It’s a struggle whenever you are developing a new way for people to get information or make something happen, because you are playing with magic a little bit. And you have to make it vanish the way a good magic trick makes an experience a magical one. But at the same time you also need to reveal just enough that you let people in and they can see how to change it and make it their own. That is the interesting tension for this space right now, the idea of augmented reality begins to lead the idea of a social commons for physical things. The Imagewiki project was a locus of just this tension. Tish you and I have previously discussed how difficult it was to even get people to understand the two concepts independently.

Tish Shute: Right, until recently most people hadn’t even heard the term augmented reality and I am not sure that a particularly high percentage of people would recognize it now despite the recent interest in smart phone apps.

Paige Saez: It’s very difficult to get people to understand the two concepts, and now you are adding in the third level of participation as well. So I don’t think it is impossible, but I do think it requires narrative. It is interesting that you were talking about the stories you heard this morning from the creatives at the event [Tish mentioned David Curcurito, Creative Director, Esquire gave an excellent presentation at Sobel Media event NYC] because it’s narrative and the attention to telling a story that help you walk through all of the ways you can understand how completely expansive this area is right now.

So I think we have to play with it, play with the space and the tools. I think we need to have an idea of what we want people to use the tool for, and we need to not only introduce them to the tool and the technology, but also introduce them to the concepts as well. So I see it as a three part process.

I’m really excited to be there with people, helping them do that. I think we need to do this face to face. I don’t think this can be only through a social network. The ImageWiki website is like one quarter of the entire picture, you know? The website is the resource center and the place where you can see people adding images, but what value is it to you to see an added image? It is more valuable for you to be interacting with the image or interacting with the object in the real world.

Designing for the experience of using the ImageWiki got very complicated very fast. I was trying to figure out the main thrust of the design for the UI for the ImageWiki and at a certain point I had to take a step back and say “Okay, this has to be good enough for now because we can lay it out and prototype as long as we want on the Web or mobile UI. What we need to be doing is going outside and actually aggregating and putting images into the database in order to see what exactly happens when we are adding.”  It’s not just like you are taking a picture of something and adding it to Flickr. Using the tool is very context specific and the information is context specific, and you can’t necessarily make that all happen at the exact same time. I think these are really fascinating spaces to be struggling in and I’m so glad to be working in this space.

Images by Chris Blow of unthinkingly.com

Tish Shute: Could you explain why we need ImageWiki? I mean I think I have ideas on this, but perhaps you can explain to me from you point of view why we need an ImageWiki, as opposed, to say, extending the image space of Wikimedia or something added on to Flickr.  I mean maybe something leveraging the geotagged photos sets and APIs we already have?

Paige Saez: Yes, definitely. It’s a really good question, I mean it really is. Like, do you need an entirely new place to be holding images outside of the places that we are already holding images? That’s a huge question; enormous. Especially when you take a look at the problems around that. Its’ exhausting for an end user. Who the heck wants to go and reload everything into yet another place, right?

Tish Shute: Right.

Paige Saez: Moreover, who is going to really bother? Another problem would be what happens to the existing datasets that people have already committed to? And then of course there is the problem of authority and explanations why….Gaining interest and authority in a space when nobody even understands why that space should exist in the first place. And those are just three, you know, off the top of my head problems with that idea.

And yet at the same time, I don’t actually know how else to go about thinking about the ImageWiki unless I think about it as it’s own thing. Then you start thinking about models of large independant image databases that exist already, examples of this from a product standpoint- references to consider. The Getty Foundation comes to mind. There are many other historical centers that have huge resources and images that are licensed out and used. So here we have a working example of people already doing this. But succesfully? I don’t know. We do have a ton of intellectual property rights and copyright issues and ownership and use issues with images currently. As a working artist these issues for me were a major red flag to consider. Working on the social commons for augmented reality starts paralleling issues found in digital rights management and intellectual property.

Tish Shute: But one good thing about Wikimedia, why I focused on Wikimedia, is Flickr and Wikimedia already use a creative commons licensing, right?

Paige Saez: Creative commons, you know they have their own resource center, too. But you know they haven’t been successful as great databases for images so far.

Tish Shute: What would you like to see that they don’t have? Like say maybe start with Wikimedia, right?

Paige Saez: There’s just still a lot of issues with how to encourage people to want to contribute. It’s hard to show the value to someone who doesn’t already understand the value for some reason. At least for me personally this is something I have run into frequently. I don’t know if it is necessarily what Wikimedia doesn’t have, I think it is a lack of understanding of what creative commons really means. And there is still a very strong sense of ownership and concern about creative property rights. Being paid to be creative is a tremendously difficult thing to do. People fear losing their livelihoods. They think this is possible. Is it? I dunno.

For example : Look at me, I take a photograph of something, I can sell that. And there’s a question about whether or not, as an artist, I want to have my photographs in a pool of images that is open and accessible when I could be making money on it instead. Now that is just an example. Me personally, I can see the value. But that is a common concern. The gist of the question being, ‘what value does it bring to give something away versus holding on to it?’ A hugely popular discussion right now.

This is the same crux of the problem we are dealing with when we talk about thinking about images in the social commons for the real world. It’s a conversation about ownership. It’s about, who does this belong to really? If I take a photograph of a Levi’s billboard, does that photograph belong to me or does it belong to Levi’s? We know the boundaries of that. But when the image becomes a living image, an image capable of transmutation; an image that provokes an action or hyperlinks to a product, experience, information….where are the boundaries in that?

Tish Shute: But how is ImageWiki handling that differently from Wikimedia, I suppose is my question.

Paige Saez: We haven’t solved the problem.

Tish Shute: Yes, I suppose it is not like we have fully solve the problem of a creative commons for images on the internet let alone the issues of a social commons for the real world! So neither one has solved the problem, right?

Paige Saez: Exactly. To be honest, it made my head spin. I realized we were building a web application and a mobile tool doing augmented reality, real time feedback on the world and suddenly we weren’t. Suddenly we were dealing with DNS and talking about physical hyperlinks and ownership and property. And basically at that point you just have to sit and really start looking at catching up on IP issues and figuring out how to deal with that space in a much more wholistic way. It became so important that we had to take a step back and go

“Oh my god I think we have really uncovered a real problem here.”

At the point when we were building out the tools we realized something was really going on with our project. Here we were thinking that this was just a beautiful experience of learning about the world around us. We really…Anselm and I both just really wanted this tool to exist. It was something that we both just really wanted to happen in the world, something that we felt really just thrilled to make. And we looked at and used it and realized that instead of it just being a beautiful experience, it was a fundamental shift in how we understood everything. That it impacted our world in the same way the Internet impacted our world. It was a fundamental shift in understanding. A sea-change.

So I put down the prototype and went back to researching, read a ton of books on IP and went and presented to friends, family, schoolmates and co-workers trying to explain the project and then the larger conceptual framework that had emerged from the project. I began using the metaphor of thinking about Magritte’s “Ceci n’est pas une pipe.” Thinking about a pipe that isn’t actually a pipe.

Tish Shute: Oh, yes!

Paige Saez: ..to try to help explain to people that the image that you see is actually not, you know, it’s not an image of a thing. It’s an image. And that image has a tone and that image has a voice, and that image was chosen. And there were decisions that were made through the interface of the camera, specific decisions that defined the view of what you were looking at. And that that wasn’t being acknowledged and that that was a fundamental part of what the ImageWiki was aiming to do. The lens that you are actually looking through was as important as what you were looking at. And democratizing that lens became the most important thing that we could possibly do.

Tish Shute: So the emphasis for you on ImageWiki was in fact the lens, even though you found obstacles to creating the interface, right?

Paige Saez: Yes. Definitely. That’s what I fell in love with first. I really wanted to be able to use my phone to learn about what kind of tree this was or to buy tickets for the band on the poster I just saw, or see a hidden secret. For me it was very much a story, a narrative experience that I just thought was magical. And that is how I fell in love with it, which is not where I ended up. Where I ended up was realizing it was a fundamental shift in not only my own understanding of how to use the world around me, but in our understanding of looking at the world.

Tish Shute: It would be pretty scary if an image DNS was basically in the hands of either one or very few people, right? I mean even ImageWiki would be stuck with this problem, that if you set up a bunch of servers, you are going to be holding a very, very large image database. I mean, whatever your motivation, right? I think at the minute that is why I am very into seeing everything through the lens of federation, I see that unless we have federation, these giant central, databases are inevitable aren’t they?

Paige Saez: Essentially, yes. I mean I wasn’t able to walk through it as quickly as that. It kind of just overwhelmed me. Looking back on it, it seems perfectly obvious. I was just like “Oh my god, what have we done? Like what is going on?” Particularly for me because so much of my life has been spent in art, it was really easy to immediately understand the connection between the view, the viewer, and what’s being viewed as all just different layers of ownership and understanding that it is a gaze. Right? We know that we are never able to look at something without passing judgment on it, but to see that become a part of the interface in a real-time fashion just blew my mind.

Tish Shute: Yes.

Paige Saez: I think you are right. Getty Images, Flickr images, no matter what you are always holding on to something and you have to be responsible for it. Right? So how do you deal with the responsibility but don’t take on too much ownership? Where is the boundary with that?

Tish Shute: And for me, the simple answer to that is loosely connected small parts, distributed systems and federation. Because there is only one way to be able to utilize these things is to have them distributed so that no one holds all the cards. Right?

Paige Saez: Definitely and I personally agree with you wholeheartedly. However, the idea of distributed power is a concept that most people just don’t know how to deal with.

Tish Shute: And it’s easier said than done because actually the root problems that you are talking about aren’t got rid through federation, because if someone really holds the, sort of, all the good image databases just because they have the potential to be federated, they may not choose to open them up on many levels.

Paige Saez: And even then you have to think about, sort of, like the next level of it, which is we want it to be all open and accessible, but everything is owned by somebody. Like, what really is public anymore, in general?

Tish Shute: And what is interesting though, regardless of what we speculate conceptually on this, we already set off down the road. I mean we have already several large…they are all in beta I suppose, Google Goggles, Point and Find, right? But we have applications that are beginning to implement this. They are beginning to implement search on it, and it is geo-located even if it’s not in an augmented view, right? So it is proximity based.

Paige Saez: Right, right. I mean maybe the solution is that if we follow that line of thinking then Flickr will be partnering with Google Goggles. And then my images would stay under my ownership through the authority of Flickr. And I would use Flickr as my place to add images and they would just be responsive via my devices via AR.

Tish Shute: That’s very interesting.

Paige Saez: Definitely I think so. It is also the shortest distance between things.

Tish Shute: Yes, and as Anselm kept pointing out, basically it is going to happen in the simplest way possible, really, regardless of the implications of that. But OK, getting back to ImageWiki. As you say neither Wikimedia nor Flickr were really designed to take this role, right?

Paige Saez: Right.

Tish Shute: With ImageWiki, you’ve had these ideas and a concern with the social implications of physical hyperlinking in your mind since it’s inception. Are there any design ideas you’ve come up with that you know, as opposed to sort of, as you say, connecting Flickr to Point and Find, or who knows, Google Goggles. How is ImageWiki going to be different, do you think? Is that a hard question at this point?

Paige Saez: It is, and it’s a great question, and it’s a question I really love to think about. I think we have to introduce the politics with the tools. It has to be acknowledged that it’s not just a place to hold information, that’s what I feel in my heart.

At the same time, is that too much for people to really grasp at one time? In my experience it really has been, so the design of the experience needs to allow for an understanding of the power of the tool and the level of authority that the tool offers, while not getting in the way of it; just using it. Because ultimately, at the end of the day, nobody will use anything if it isn’t valuable to them. And so I could talk for miles and miles and miles about how important it is that corporations don’t own all of the rights to all of the visual things in my life, right? For the rest of my life I could talk about that. The idea that advertising is dominating all of our views of anything in the world around us is horrifying. It doesn’t matter unless I can show somebody why it matters to them or how it affects them. It’s just that that is a tremendously difficult thing to explain through a user interface.

And I actually think that it’s great that tools like Google Goggles and Nokia Point and Find are here to do a lot of the hard work of showing people how it works. Recently somebody explained to me their experience of using Google Goggles. They went through this process of saying how the Google Goggles took a picture and then did this really complicated visual scanning thing over the image and it took a full minute.

And I said, “Well of course they did it that way.” And they said, “Well what do you mean?” I said, “Well, what they are really doing there when they are doing all these fancy graphics, is they are showing you how it works.” And even if it isn’t actually related at all to how it functionally works, algorithmically, that’s not the point. The point is that this gesture of the time taken to make it look like it’s scanning an image and going back and forth with pretty colors is giving people the time to process that as an experience. That’s a metaphor for what’s really happening. And these kinds of metaphors are crucial with user experience design. We have lots and lots of examples of them and how they work, and many of them aren’t necessary. Like you know, for example, the bar that shows you the time it’s taking for something to process.There is no relationship between that and reality. But it is really important.

Tish Shute: Yes those bars often have no relationship between the actual time..

Paige Saez: And that’s the thing. Like the idea of time versus our perceived understanding of time. Right? The length of time it takes for your Firefox browser to open and load your last 30 tabs, versus the reality of what’s actually happening. When you are doing that sort of research you are actually accessing millions and millions of places and points of interest all over the world, so we need more of that. We need more of the process shown. Anselm and I worked with a film maker named Karl Lind from In the Can Productions here in Portland to try and make a video about the ImageWiki. We made this little video and I can try to show it to you or send it to you if you want.

Tish Shute: One of the issues with this kind of visual search is that it is inherently dependent on large databases, regardless of where they are federated, are going to be very large. Right? I mean someone is going to have something big, and aggregated there. I suppose someone will figure out the challenges of federated search eventually but that is quite a big challenge!

So I suppose I am still trying to understand what ImageWiki can offer that we can’t get with any other existing service? How will their be a social commons and even a social contract for the world as a platform for computing and physical hyperlinks?

Eben Moglen brought up something when I talked to him about virtual worlds, he said we need code angels to let us know what was going on in the virtual space – who was gathering data and how, for example.

Paige Saez: Tell me more about that, I want to hear more about that.

Tish Shute: Eben suggested this metaphor for when I was asking him about privacy in virtual worlds. The fact that people just didn’t know that when they were pushing avatars around virtual worlds what metrics were being gathered on their behavior. And he basically said that what we need is code angels when we enter these spaces because having the rules of the game buried in a TOC was ridiculous.

Paige Saez: That is a really interesting idea.

Tish Shute: Maybe ImageWiki needs to be our code angel to navigate the augmented world. I mean that’s what I want to see it as. And when I hear you talk, what I hear is you talking in broad categories about what a code angel might be in the space of images and image links to the physical world. I mean that is what I hear from you.

Paige Saez: Yeah. No, I definitely agree with that. It is interesting. In that sense, it is kind of a protection layer. Is that what you are thinking?

Tish Shute: Yes, I suppose because we can’t be navigating a lot of complicated opt-ins and opt-outs just to get around our neighborhood safely (in terms of privacy (also see Eben Moglen’s definition of privacy here…) We will need a code angel that is sort of keeping up with you in real time!

Paige Saez: Right, right. I wonder how that would work in regards to images, though. That is a really interesting thing to try and put on an image. I guess why I am having such a hard time being specific about it, is I am just trying to work it in my head, thinking of a specific use case, like what would be an example of that?

Tish Shute: Well I suppose the example, and this is a crude one, is when you point your Google Goggles to the book jacket, the code angel, this is very crude, would say “You are right now drawing images from the Amazon database – they are collecting data such and such data from your search.

And then of course the ability to have crowd sourced tagging and corrections..

There was a wonderful book that came out last year on how we can have commercial intelligence -Dan Goleman’s new book: “Ecological Intelligence: How Knowing the Hidden Impacts of What We Buy Can Change Everything”…

how corporations various different stakeholders, including their customers will drive corporations to do the morally right thing because they will lose the commercial support of customers who won’t support them unless they are more green, fairer, do the things we would like them to do whatever that happens to be – physical hyperlinking and tagging I guess would be a big part of this.

Paige Saez: Sort of a transparency issue. And that almost becomes a page rank algorithm in and of itself. I mean now we are really talking about search more than anything, and what tool becomes the dominant search tool. Anselm and I talked a lot about one platform… I mean eventually we will have a unified platform. It will…No matter what, for the Internet and for physical objects and visual objects in the real world. It will just be a matter of, literally, who can find the best and most valuable, most relevant information on a thing. Currently we just have it very proprietary.

Tish Shute: Yes.

Paige Saez: That definitely won’t last. It just can’t, because of the exact problem that you are raising. And we already know too much about resources and information as they pertain to products for us to ever go back to a time where we are not considering other ways of getting information about it anyway. Right?

Like I have the same concerns nowadays when I look at fruit. I look at a piece of fruit in the store. I would never just assume that the person who put the sticker on that fruit, anymore, is the ultimate authority necessarily. I would always assume at this point I could go online and go find out more information about a company. Issues about like eco-footprint or how much toxicity, or pesticides or whatnot are now totally accessible already.

So I am thinking when you look at that piece of fruit and that sticker for Google, say what you are describing, do we just go immediately to the company’s website, or is it even more specific? Do we know that the sticker on that piece of fruit is going to tell us specific information about that? Or are we just getting back the nutritional resources, or are we getting a listing of all of the different options out of a page rank algorithm that shows us, “Well this is the website for the fruit. Here is the nutritional information. Here are the last 15 comments on it.” It’s basically just a basic search.

Have you heard of Good Search?

Tish Shute: you mean http://en.wikipedia.org/wiki/GoodSearch

Paige Saez: Right.

Tish Shute: A code angel interface would have to give you options, wouldn’t it on possible views available?

Paige Saez: Yes. You are then talking about filtering your view. Then it really gets really interesting, of course. I don’t even know if we have a choice in that. I think we are really kind of hitting a wall with who owns the space and the platform. Is it just a basic search because we are already familiar with search? If you had an option to choose, say, “I want to look at this apple sticker and I only want to get…programmatically only looking at my friend’s opinions of this company.”

Or I have a safety valve on it that only shows me certain information based on what the code angel knows about me, my preferences, my age, things like that. Then that gets really, really interesting, because we are trying to do all that work right now just with social media and the Internet. We are already overwhelmed with too much information. It is already past the point of comprehension. So to think that we would actually drill down even more specifics is very interesting.

Tish Shute: That was a point Anselm made about the fact that once you are into this mobile, just in time, one view kind of situation, it is quite different than the Internet where you can bring up all these different screens and go to another website.

Paige Saez: Well yes, mobile is a different level of engagement. Very contextual. Much less information. Much more about timeliness. I don’t want to look an apple and get back a Google search. Oh my God no. That’s the last thing I want. I would love to be able to look at an apple and my phone already knows exactly what I want, information-wise, to get back from that apple. But I don’t know. It’s all contextual and personal. So I think the code angle concept you are talking about is really interesting because you still need to think about who is the person that is adding or creating those level filters- is it you, a filtered friend network, an algorithm? How much work is too much work? Where do we draw the line? How much of this are we willing to let the machine do for us?

Tish Shute: Right.

Paige Saez: And then of course once you have those filters in place, you need control over them. You will need to dial them up and dial them down, be able to choose and add new ones, so on and so forth. It becomes very modal at that point. For example, I want to change my view: To walk into a grocery store and instead of finding out information, I’d want to see where the hidden Easter egg puzzles were that my friends left last week because we’re playing a game.

I’m still really attracted to the creative opportunities with the ImageWiki. I’m really attracted to changing this experience from being a one-to-one relationship (from Corporation to Consumer) to an open-ended relationship (From Person to Person). If I look at a book jacket, sure I can find out where to buy the book, but that’s boring. Who cares? I’d like to find out a link to a story or an adventure or a movie or something unthought-of before.

How do we build that in? How do we encourage serendipity? Mystery? I think the ImageWiki is the space for building that in, actually. Not how, that would be the one place, right? That’s my really big fear is that this relationship just stays one-to-one. Click an image of consumable object, get back objects retail value. How completely dull. We have to do better than this.

Additionally, what if I want to take a photograph of a book, an apple, or something and I don’t want to pull back data. Instead, I want to pull back music, or I want to pull back a video, or I want to pull back a song, or lyrics, or a story, or another image. It’s just a hyperlink at the end of the day, you know? That’s all we’re really doing. Hyperlinks can pull back so many different things.

Tish Shute: And that’s one of the reasons I’m into mobile social interaction utility building, because without that, if we don’t have that way to do that in mobile technology…that’s very available on the Internet, as we’ve seen, with Twitter. These applications are very easy to do on the Internet. They’re not easy to do natively in a mobile application..

hey, I’m just promoting AR Wave again. I should shut up.

Paige Saez: Oh, no. I think it’s a fascinating concept, I really do. I totally agree. As we’ve talked about it before, it’s amazing that marketing and advertising are helping push forward AR, and it’s great. It’s fantastic.

But it’s also the worst possible thing that could ever happen because it is such a singular way of looking at an overall ubiquitous computing experience. There are other ways.

The best experience I ever had was trying to explain to people about physical hyperlinks. I had to walk them through it. Good interactive isn’t something you present or show, it’s something you do. Nothing beats just walking around and showing people with a device or a tool or something else.

I mean, God forbid it always stays in our computers and our phones. I really hope we don’t have to be stuck living our entire lives with these horrible interfaces. But for the time being, we will. Having an AR app show you a puzzle, or a mystery, or a game, or an adventure is a magnificent experience, totally overwhelming, and people get it right away. There’s no question; they totally understand.

Tish Shute: Yes, I agree.

Paige Saez: You walk them through the experience with a physical hyperlink and then you say, “Here, I could use this device and I could show you where to buy this thing, or I could use this device and we could start playing a game.” Then everybody gets it.

Tish Shute: So then I have a question, because one of the things Anselm said to me when he wanted me to refer back to you is that he feels that the direction for ImageWiki should be perhaps to focus less on the technology and more on just the actual, I suppose, gathering of the images, how they’re going to be annotated, the metadata, right? But my question to him was, the problem if you do that, without the platform, there’s no experience or motivation for people to do that. Right? Is there?

Paige Saez: Yeah, I agree with you on that one. I’m curious what his…I think the reason why he wants to do that is he wants to be able to show people examples via the resources. Like to be able to show someone a library, essentially, which I think makes sense with some people. I definitely think that some audiences would really relate to that. For me, it doesn’t make sense because I’m just very experiential. I need to do it and I need to show other people how to do it and I need to grow that way. I think that at the end of the day, those are great ways to go about doing it. It’s just it’s a huge thing to do in either direction.

What Anselm’s really thinking on, I believe, is more about exemplifying how we read and understand images culturally. Then you’re really getting into Visual Studies and Critical Theory which is what I did for my Masters at PNCA. I worked on the ImageWiki while I was in grad school, it was something I was doing for fun. Independently of my studies, the project lead to issues on democracy and objects and property and I ended up right smack in the middle of what I was studying; the nature and cultural analysis of images Questions like, ‘what exactly do we get out of images?’ and how all these different things are happening in an image, and people get tons of totally different things out of an image depending on many factors.

The questions I began to ask myself got very philosophical. Questions like “Is this apple red? Is this apple red-orange? Is this a small apple? What’s my understanding of small versus your understanding of small?”

Because you supposed that you needed a text backup to the search, how would I be able to search for an apple? Because what if my understanding of apple is red and your understanding of apple is green. And so if I’m looking for a green apple, am I looking for the same green apple as you? It’s all semantics, sure. But at the same time, it gets bigger and bigger, and it’s fascinating.

Tish Shute: Google Goggles seem to work best on book jackets, basically.

Paige Saez: But book jackets are actually perfect for this. Book jackets are perfect for this problem, because book jackets are specifically designed art. So at the end of the day, we are still talking about creative works, artistic works, that have been designed as a communication tool. But that is not something that people can own. Creative works that are designed are a communication tool, with varying levels of skill to be sure, but still something anybody can do. What we need to do is we need to be using that language. We don’t need to be trying to reach as far as facial recognition. We need to develop our own logos, our own brand, our own…I mean not brand. Brand is a bad way of saying it. Another way of saying it would be like, just use it. Develop a visual language that we can use that is as effective and as well utilized as book jackets or the movie posters or something.

Tish Shute: What are some of the use cases for ImageWiki you would like to develop first?

Paige Saez: My dream…I have like four or five use cases that I want to see happen. One of them is I walk down the street and there is a new poster for my favorite band. And I can just go up to the poster and I use my device, whatever it looks like, and I download the latest album. It’s transactional. I am able to just plug in my headset and walk down the street and the transaction is done. I saw something I wanted. It was beautiful. I was able to get it and I was able to move on in my life. And that is totally possible.

Another one would be I walk down the street and there is a piece of graffiti. And I am able to use my device to find out who the artist was that made it and to give them props, and to point my other friends to the fact that the piece is there and it will most likely be there only for a short period of time- information retrieval and socialization.

Or, use my device to find an Easter egg, to find a narrative puzzle that ends up going on for weeks, and everybody is involved, and we are all playing this game together. Adventure-based, non-linear experiences. I want playfulness, not just purchases.

Tish Shute: Did you think of piggybacking on the Flickr API for geo-tagged photos as a way to work with those databases or not?

Paige Saez: Yeah, we definitely thought about that.

Tish Shute: And why did you decide not to, for any reason or…?

Paige Saez: Ultimately, we just…we were such a small group, we just had to tackle certain things at a certain time.

Tish Shute: Right. And you were so prescient, you were working slightly before we had the mediating devices, weren’t you? You were just before the mobile devices really got adequate for this.

Paige Saez: Yeah. We started on it…I believe it was January…No. December 2007. Basically, the iPhone had just launched like maybe six months prior or something like that.

Tish Shute: But not 3G and not 3GS, right?

Paige Saez: Not 3GS. It was the first generation iPhone. We built the ImageWiki before the App Store existed.

We knew that the App Store was coming out. And we knew that the App Store was going to be the biggest thing in the whole world. I remember getting into multiple fights with friends about how revolutionary the iPhone and the App Store were going to be and people thinking I was totally crazy; people just thinking I was absolutely nuts for being so excited about it.

It sucks that it is a closed proprietary system, but the App Store has done something for software that nothing has ever done in the whole world. Software is candy now. It’s candy. It is like when you are waiting at the grocery store at the checkout line and you are stuck behind somebody, and you have got all these little tchotchka’s, candy bars, magazines, nail-clippers and things. That is the equivalent of software now. It’s become an impulse buy, which is amazing. Nobody would ever have thought…that is actually revolutionary. That’s huge.

Tish Shute: Steven Feiner, who is one of the founding fathers of augmented reality said to me during a conversations at the ARNY meetup that one reason that augmented reality, despite the hype, is manifesting very differently from how virtual reality burst onto the tech scene is that it is about affordable apps on affordable readily available hardware.

“Come to ETech; Experiment with Physical Computing and RFIDs” said Brady Forrest. The ETech RFID tag that I activated at registration is a gateway to several internetworked worlds.  It allows you to check into pulse stations to tell you about people with similar interests to you based on your traffic movements around the conference.  There is a photo booth that allows you to upload photos to Flickr. And even a Fortune Teller from Josh and Tarikh of Uncommon Projects (makers of the awesome Yahoo! geo-bike) that will be arriving tomorrow, and more.

But the first day of ETech 2009 was packed with hands-on workshops.  And I actually managed to make, in Tom Igoe’s, Hands-On RFID for Makers workshop, my first RFID to web interface that could read Etech’s elegant RFID tags (also see my photo set on Flickr to get a glimpse of the action in the workshop). Amazingly it worked perfectly first time (I did have help from the very patient executive editor of Maker Media Books, Brian Jepson. And Tom Igoe’s step by step instructions on his website are invaluable (picture of Tom Igoe below).

It was very exciting to actually get hands-on with the Arduino open source electronics prototyping platform, and Processing – a very accessible language to do dynamic and interactive graphics for screen-based media, . You’ll know how much I love to write about these things if you have checked out some of my previous posts.

My “build” is sitting on the right of my workshop neighbor Ahmed Riaz (ebay) in the photo opening this post. We shared power supplies and a great discussion on interaction and user experience design (see my previous post, Towards a Newer Urbanism: Talking Cities, Networks, and Publics with Adam Greenfield, for an idea of some of the topics that we touched on). We also discovered a shared interest in User Experience Design sketches – see Ahmed’s blog here and his flickr stream for his project on UX sketches. I have reposted here one of my favorite UX sketches done by an eight year old, especially for Ahmed.

If you look closely at the picture below you will see that Ahmed’s RFID to web interface has read my Etech RFID tag and pulled up my Etech Conflink profile and picture.

In the evening, Tom Igoe announced during his Ignite presentation that an Arduino MEGA will be available next week – more pins, more ports, more memory.  I can’t wait to see what people come up with for the MEGA, especially as Pachube (another favorite project of mine – see my interview with founder Usman Haque here) is designed to work with Arduino and Processing.

I think I’m hooked on Maker culture. I can’t wait to check out the Etech Maker Shed that opens today. I got a feel for the excitement of rapid prototyping in the morning doing the LilyPad Electronic Fashion workshop with Leah Buechley, a brilliant and patient teacher. Leah is checking out Raffi  Krikorian and Tom Igoe’s progress in the photo below.

There was some big talent in the Lilypad workshop. The Wattzon team, Raffi  Krikorian and Jeremy Cloud, and Wattzon-phile Tom Igoe stitched and ironed (see my Flickr stream here), and helped out noobs like me. Possibly we will see some programmable T-Shirts displaying carbon footprint data. But certainly you can use Wattzon to compute the embodied energy data of all the Lilypad components.

I was a little hampered by my appalling needlework skills. But Maker culture came to the rescue when I twittered about needlework phobia and LilyPad love. @dpentecost replied in seconds inviting me to “sew and tell” at a NYC Lilypad meetup when I return to NYC. Below is a picture of Jeremy Cloud’s excellent stitching with the challenging silver plated thread.

Adam Greenfield’s new book, The City Is Here For You To Use, is coming soon (photo above by Pepe Makkonen is from Adam Greenfield’s Flickr stream). Adam told me:

“I’m aiming at a free v1.0 PDF release on 05 June 2009, with the book shipping as quickly thereafter as humanly possible. There will be a version zero or public alpha in about six weeks.”

I am not good at waiting for books I really want to read to arrive. But, on the upside, it brings out my already pretty highly developed investigative instinct. So when Adam very generously agreed to do an interview, impatience turned into delight in tasting what is to come. And Adam is encouraging this kind of engaged anticipation. He writes (see post) that The City Is Here For You To Use, is shaping up:

“as something of an unbook avant la lettre. It’s why we’ve [Nurri Kim and Adam Greenfield] always insisted on keeping you in the loop as to the book’s fitful progress, it’s why I take every opportunity to test its ideas here, it’s why I make explicit the fact that your response to those ideas is crucial to their evolution and expression. And it’s why, even though the process is inevitably going to result in a static, physical document as one of its manifestations – and hopefully a very nice one indeed – we’ve committed to offering a free and freely-downloadable Creative Commons-licensed PDF of every numbered version of The City, from zero onward.

You buy the book if you want the object. The ideas are free.”

I found the opportunity to ask Adam questions about some of his subtle renderings of technology, culture, and being in urban environments challenging and very illuminating.  Although I definitely get the feeling I am asleep at the wheel on some of the critical areas he is thinking and writing on.

Knowing the depth and range of Adam’s thought in his seminal book, Everyware, and his blog, Speedbird, before I began the conversation I asked Adam to point me to some of his posts that reflect key ideas he is working on at the moment (Adam has recently posted The City Is Here: Table of contents).  Adam directed me to these three posts.

Antisocial networking

More songs about context and mood

Messenger, space, messenger body, messenger mesh

I may ramble and diverge, as is my nature, but these posts inspired many of the questions I ask.

Adam is currently head of design direction for service and user-interface design at Nokia and living in Helsinki, so I did not have the opportunity to do the interview in person. But I have glimpsed Adam’s world through his Flickr stream and some of these images have found their way into this post. But I suggest you browse Adam’s photography for yourself. I cannot do justice to the thousands of nuanced perceptions of cities, networks and publics you will find there. In the meantime, here are three glyphs of Adam Greenfield that I liked a lot.

“My favorite shoes,” “My favorite chair” and photo by Adam Greenfield, “Favoriteplace”

A Conversation (in gdoc) with Adam Greenfield

Tish Shute: Could you explain a little about the evolution of your thoughts on urban environments, ubicomp and interaction design? What shifts in your thinking have taken place over the last few years re the dawning of the age of ubiquitous computing? It is a couple of years now since Everyware, what aspects of the uptake of Everyware have most surprised, disappointed or inspired you? Which of the many thesis you discuss in Everyware have become the most crucial for The City Is Here For You To Use?

Adam Greenfield: You know, there’s a little passage in the liner notes to the second Throbbing Gristle album that I always think of when I’m asked questions along these lines. As part of their stance, they’d adopted the dry tone of a corporate annual report, and the preamble began by saying, “Since our last report to you, many things have changed. Indeed, it would be foolish to assume that it could be otherwise.” And I think that’s just exactly right: the world keeps moving, and the positions we’d staked ourselves to not so long ago may no longer be correct, or even relevant, to the one we find ourselves inhabiting now.

So, first, I think it’s important to cop to all the places in Everyware where I just outright got things wrong. There’s a passage in Thesis 50, for example, where I unaccountably mock the idea that “the mobile phone…will do splendidly as a mediating artifact for the delivery of [ubiquitous] services.” OK, this was admittedly written in a pre-iPhone world – and was correct for that world – but you can really see my parochialism showing here. It took the iPhone to make the proposition as blazingly self-evident to me in North America as it had been for quite some time to folks in Europe and Asia.

Having said that, though, I think I’m justified in taking a little pride in what the book got right. The broader trends the book set out to discuss – the colonization of everyday life by information processing – well, take a good look around you. And so one of the points of departure for the new book is taking everything posited in Everyware as a given: the urban environment, and most everything in it as well, has been provisioned with the kind of abilities you mention. So what now?

How do you go about designing informatic systems so they don’t undermine the wonderful things about cities? How do you design cities so they can incorporate networked informatics to greatest advantage? How, especially, do you accomplish these things when the disciplinary communities involved barely speak the same language? And how do you keep everyone’s eyes on the prize, which is the ordinary human being asked to make sense of these new propositions? These are the questions The City Is Here For You To Use sets out to address.

Adam talking about the “Le Long Ici” in Paris (also see Adam’s post, “The long here and the big now”)

TS: You mention that the hardest parts of  producing The City Is Here For You To Use wasn’t “keeping on top of all the emergent manifestations of urban informatics, or even developing a satisfying spinal argument about their significance” but getting the voice right.  It seems that now is the perfect time for a book that would really speak to a wide audience.  But also it seems that the city that is here for you to use is manifesting quite differently in different parts of the world?  You seem to be somewhat of a nomad, Japan to NYC to Helsinki.  Can putting together different views of urban informatics give us more depth perception on the emergence of ubiquitous computing?

AG: There’s no question in my mind that the long-term experience of everyday life in Tokyo, New York, and now Helsinki has been an invaluable asset to me, as I imagine it would be to anybody interested in thinking or writing about the networked city. It’s given me a certain amount of parallax, you know? And that, in turn, throws a really interesting light onto how the selfsame technology can appear in substantially different guises in different social contexts.

But explaining those things – those complicated, delicate negotiations – getting them right, doing them justice, doing so in a way that doesn’t dumb anything down, and still remaining accessible? It’s a challenge, let me tell you. You want to remain approachable and humane, but you also want to explain things like different jurisprudential takes on property, or how advocates of RESTful architectures think that REST is the reason why Internet adoption spread as rapidly as it did. If you want to enjoy even one chance in a hundred of getting your message across, you’ve got to start with an understanding that those subjects are MEGO territory for most people – whether they hail from Shibuya, Shoreditch or San Pedro.

Everyware icons: Information processing dissolving into behavior (Icons inspired by Timo Arnall; design by Adam Greenfield and Nurri Kim).  [Adam notes on his Flickr page that he tweaked these icons as section headers for Everyware]

TS: Could you explain more about what you term “onto” and “ontome” and how this differs from spimes and spime wrangling?

AG: You know, I never did get to develop that idea as much as I would have liked. In my mind, at least, “ontome” referred to the totality – the global environment of addressable, queryable, scriptable objects. (An “onto,” then, would be any given such object.) I guess I was looking for words that would do two things: allow us to distinguish between the instantiation and the class, and leave us with a better word than “spime.”

TS: When you say better word than spime this is this because….

AG: Euphony, primarily. : . )

TS: When I first used the Android app,  Wikitude, on Broadway, NYC – a street I have traveled thousands and thousands of times, and it offered up new information about itself, it was definitely an “OMG this is big!” moment for me. Like the first time I clicked on a screen and Amazon sent out a book in the early nineties (something so ordinary now it seems impossible that it was exciting but I remember it was to me!). But if I understand your post here correctly, isn’t Android with compass the first easy-to-use context-aware mediator for wrangling onto, ontome and spimes?

AG: Wikitude sure looks pretty impressive, and maybe even useful. But I would never, ever call it “context-aware.”

To my mind, at least two more things would need to happen before we could comfortably think of it a “context-aware spime wrangler.” First, the buildings and other public objects around you would actually have to be spimy – they’d have to report something of their past and current state to the network. And then, some application running on your phone would somehow have to cross-reference that state information with some fact about your current state of being, and deliver you relevant information.

So, let’s take your Wikitude example. You’re walking down Broadway and you pass an unfamiliar building, and for whatever reason you want to know more about it. Your phone pings the building’s dynamic self-description, and it replies to the effect that Andy Warhol had his Factory there between 1973 and 1984. If Wikitude chooses to share this particular piece of information with you, and not some other potentially germane factoid from the building’s history, on the strength of the fact that “The Velvet Underground and Nico” was in your last.fm playlist? That would constitute some small measure of context-awareness.

But you see how hard we had to try just to come up with an example, how forced it is, how so-what. And I have to say that – short of some infinitely supple system that really could model your innermost desires ahead of real time, and present appropriate responses to them – most so-called “context-aware” applications and services are like this. They’re either trivial, or wildly overambitious.

Maybe we don’t need for things to be context-aware for them to be useful, anyway. Certainly a great many objects in the world are starting to report their own status, and many more will do so in the fullness of time. And for the most part, all you’ll need to avail yourself of them is a Web browser running on a device that knows where it is in the world. An iPhone or an Android device will work splendidly – I called the iPhone “the first real everyware device” the day it came out and I was able to play with it for the first time – and in that way, the answer to your question is “yes.” Not to be longwinded or anything. ; . )

This Object has imperceptible properties. [Adam notes on his Flickr page: “This is a custom RFID-enabled transit pass that Timo Arnall had made up for me here in Seoul. I’ve (clumsily) tagged it with the icon that Nurri and I developed to represent just such emergent situations as this in the everyware milieu – that there’s no way for anyone to understand that this object has puissance beyond the obvious simply by examining it.”]

TS: It seems that  we are just at the beginning of understanding how to create networks of spimes (e.g. Pachube). Gavin Starks of AMEE (”the world’s energy meter”) once suggested to me that AMEE could be described as a facilitator of networked spimes (everything will have an energy identity). I think you may be familiar with AMEE because you keynoted next to Gavin at Xtech 2007.

I would be interested to hear your thoughts on AMEE?

When you discussed onto and ontome in this post, you noted:

“The greater part of the places and things we find in the world will be provided with the ability to speak and account for themselves. That they’ll constitute a coherent environment, an ontome of self-describing networked objects, and that we’ll find having some means of handling the information flowing off of them very useful indeed.”

Is the idea of “energy identity” that AMEE proposes an ontome? 

AG: See below for a précis of my feelings regarding environmental/sustainability initiatives, AMEE included. Uh…is AMEE an ontome? No. There’s just one ontome, and it’s coextensive with what folks now call the Internet of Things. It sounds like individual AMEE sensors would be “ontos.”

But I think the difficulty we’re having is a pretty good indicator that the terminology is more trouble than it’s worth. Sometimes a coinage, as satisfying as it may be lexically, just doesn’t work for people. These days I’m trying to get out of the neologism trade.

TS: I know when Usman Haque talks about Pachube he talks about spimes and spime wrangling. I asked Usman for his thoughts on spimes and onto/ontome and he gave me some comments.

Usman Haque: I think I had somehow missed the conversation about onto and ontome but backtracked through blog posts to piece it together (unfortunately some posts at v-2 and Studies & Observations no longer exist!). There are a couple of things that have made me uncomfortable about the word ’spime’: (a) the fact that it might be too easy to confuse with an “object”. A ’spime’ should also encompass relationships between things, and not just the “thingness” itself. (b) the sound of it (as Adam noted above). But then I am reminded of that horrible gooey interface used to plug into people in eXistenZ – it somehow seems appropriate that it should be a horrible gooey word, and not something that can disappear politely… So I like onto/ontome because it speaks to my first concern about ’spime’; but my second concern, it turns out, is not the problem I thought it was, and so onto/ontome might be… ahem… too euphonic! On the question of this thing people are calling the “Internet of Things”, I’ve tried in lectures to reframe it as the “Ecosystem of Environments”. Further, Vlad Trifa makes a delicious point that just as ‘web’ is different from ‘internet’, so too should we consider the “Web of Things” rather than the “Internet of Things”, something I agree with.

TS: It seems like this point about the difference between “the web of things” and the “internet of things” is pretty important?

AG: The parallel distinction between Web and Internet sure is! They’re two completely different things, right? And http is far from the only protocol that runs over the Internet. Now, as to what Vlad means by extending this particular distinction to the domain of networked objects, I don’t yet know, I haven’t had time to check it out. But sure, in principle I’d totally be willing to go along with the idea that there’s a meaningful distinction between two environments named that way.

No information is collected here; network dead zone

TS: I was just going over Greenfield’s principles of ubiquitous computing.  I am not sure that I see any current manifestations of ubicomp that hold to these priniciples yet?

AG: Oh, sure there are. Look at the work Tom Coates has done on Yahoo!’s Fire Eagle; look at Dopplr. And look at some of the steps other, less compassionate developers (e.g. Facebook) have been forced to take by their own users.

Look, those principles are just codifications of common sense and basic neighborly virtues, expressed in language appropriate to the domain of application. The best, smartest and most ethical developers have never needed guidelines to do the right thing. But especially inside companies and other complex organizations, people who want to implement compassion in their design of a technical system may occasionally find it useful to have some color of authority to invoke in their struggles. That’s all those five principles are there for, and I’m well satisfied that people have been able to use them that way.

Boffi’s take on the smart home- photo by Adam Greenfield

TS: In your post, More Songs About Context And Mood, you suggest a direction for interaction design that you point out is not far from Yvonne Rogers’ ideas in “Moving on from Weiser” about a switch in goal of ubicomp from Weiser’s vision of calm living (”computers appearing when needed and disappearing when not”) to engaged living – ubicomp technologies not designed to to do things for people but to help people engage more actively in things that they do (ensembles, ecologies of resources).

You also suggest interaction designers should be:

“parsimonious about the interaction design challenges our organizations do take on, with an eye toward reducing the complications of context (and the attendant opportunities for default, misunderstanding, misfire, time-wasting, and humiliation) to some manageable minimum.”

As you have pointed out, “we don’t do “smart” very well yet.” But paradoxically smart grids, smart homes, smart products etc. etc. are ubiquitously coming to market right now.

Yvonne Rogers suggests interaction designers should be:

moving from a mindset that wants to make the environment smart and proactive to one that enables people, themselves, to be smarter and proactive in their everyday and working practices

What areas might interaction designers most productively direct their attention towards?

AG: You note that things called “smart homes” and “smart products” are coming onto the market, and that sure would seem to be the case. But as to whether or not these things are genuinely smart, we don’t have anything more to go on than the marketing department’s word. I think you can already see that I tend to take language very seriously, and I really don’t uses like the “smart” here, or the “aware” in “context-aware.” They overpromise, they cannot help to set us up for failure and disappointment.

You know what I’d really like to see interaction design wrestle with? I would love to see a rigorous, no-holds-barred examination of the complexities of the self and its performance in everyday life, and how these condition our use of public space (and personal media in public space). I would love to see the development of ostensibly “social” platforms informed by some kind of reckoning with issues like vulnerability, dishonesty, the fact of power dynamics. In other words, before we deign to go about “helping” people, wouldn’t it be lovely if we understood what they perceived themselves as needing help with, and why?

I’d also pay good money to see talented interaction designers turn their efforts toward tools for the support of deliberative democracy, for the navigation of complex multivariate decision spaces, and for conflict resolution.

Locative is a mood – photo by Adam Greenfield

TS: I know you said this would take too long to explain but I couldn’t help noticing that you seem to be, perhaps, skeptical about the role of everyware can play in sustainable living and yet, it seems at the moment, in the hacker and business communities at least, the role of everyware in reducing carbon footprint/energy management etc, is the great green hope?

Will everyware enable or hinder fundamental changes at the level of culture and identity necessary to support the urgent global need – “to consume less and redefine prosperity?”

AG: I’m not skeptical about the potential of ubiquitous systems to meter energy use, and maybe even incentivize some reduction in that use – not at all. I’m simply not convinced that anything we do will make any difference.

Look, I think we really, seriously screwed the pooch on this. We have fouled the nest so thoroughly and in so many ways that I would be absolutely shocked if humanity comes out the other end of this century with any level of organization above that of clans and villages. It’s not just carbon emissions and global warming, it’s depleted soil fertility, it’s synthetic estrogens bioaccumulating in the aquatic food chain, it’s our inability to stop using antibiotics in a way that gives rise to multi-drug-resistance in microbes.

Any one of these threats in isolation would pose a challenge to our ability to collectively identify and respond to it, as it’s clear anthropogenic global warming already does. Put all of these things together, assess the total threat they pose in the light of our societies’ willingness and/or capacity to reckon with them, and I think any moderately knowledgeable and intellectually honest person has to conclude that it’s more or less “game over, man” – that sometime in the next sixty years or so a convergence of Extremely Bad Circumstances is going to put an effective end to our ability to conduct highly ordered and highly energy-intensive civilization on this planet, for something on the order of thousands of years to come.

So (sorry again, Bruce) I just don’t buy the idea that we’re going to consume our way to Ecotopia. Nor is any symbolic act of abjection on my part going to postpone the inevitable by so much as a second, nor would such a sacrifice do anything meaningful to improve anybody else’s outcomes. I’d rather live comfortably – hopefully not obscenely so – in the years we have remaining to us, use my skills as they are most valuable to people, and cherish each moment for what it uniquely offers.

Maybe some people would find that prospect morbid, or nihilistic, but I find it kind of inspiring. It becomes even more crucial that we not waste the little time we do have on broken systems, broken ways of doing things. The primary question for the designers of urban informatics under such circumstances is to design systems that underwrite autonomy, that allow people to make the best and wisest and most resonant use of whatever time they have left on the planet. And who knows? That effort may bear fruit in ways we have no way of anticipating at the moment. As it says in the Qu’ran, gorgeously: “At the end of the world, plant a tree.”

Biowall! - photo by Adam Greenfield

TS: In your post “Antisocial Networking,” you make some telling comments on the sorry state of social networking systems.

But you do also state:

Which of your concerns is social networking more central to than you might wish and why?

AG: Well, you know I’m interested in social interaction, interpersonal behavior, and in how these things play out in networked environments. There’s virtually no way for me to avoid dealing with Facebook, as wretched as I think it is.

Facebook is pretty hegemonic, in that its reach and influence extend further than the universe of people who use it. I bump up against it constantly, in a few different ways. People send me links I can’t access, because I’m not on Facebook. People spend time and energy trying to convince me that I’m really missing out, because I’m not on Facebook. The last few months, there’s even been a few people who feel justified in expressing some kind of exasperation, that they’re really pissed off…because they can’t find me on Facebook. It’s become the sovereign interface to any kind of life in public, and as a result a great many people don’t question its modes, tropes and metaphors.

So when it comes time to build some kind of situated interpersonal mediation framework, some kind of intervention in the fabric of the city, those are the tropes they reach for: accounts, profiles, friend counts, friendings and unfriendings, nudges and pokes. And as a member of a team tasked with the design of such systems, as a potential user of them, and certainly as someone exposed to the social rhetoric flowing downstream from their use, you bet these tropes become central to my concerns.

But what if we admitted that Facebook and the whole paradigm it’s built on are broken? What would things look like if we started from a more sensitive understanding of the interaction between self and others? Say, the understanding Erving Goffman was offering us as far back as the late 1950s? Then you’d understand the need for provisions like a “backstage,” a place to swap out one mask for another, the ability to present oneself differently to different communities and networks. That’s what I’m interested in exploring.

TS: Social networking systems in their current form are crude and express a very narrow bandwidth of human relationship. But already people are connecting everyware’s networked social acts to existing social networking systems. At the ITP winter show there was kickbee – networked fetal communication (and tweetmobile which used twitter as an acctuator for an ambient display) and green everyware (energy monitoring) is showing up in a number forms on existing social networks. But rather than just hooking up everyware to these existing flawed social network systems, does everyware require a reimagining of networked social interactions and social networking systems?

AG: That’s a great question, and I think the answer is clearly “yes.” It’s one thing to confine the consequences of that brokenness to the Web, and entirely another to let it bleed out into the world.

Does that mean any such reimagining is going to happen, that people will somehow refrain from plugging real-world outputs into these terribly flawed frameworks? Not a chance in hell. It’s too late to put a fence on that particular cliff. But maybe there’s still time to park an ambulance in the valley below.

“A graphic representation of a portion of the Earth’s surface, as seen from above” – photo by Adam Greenfield

TS: I saw you tweet that you met Usman Haque from Pachube recently. What do you find most interesting about Pachube and EEML? Will you design a project for Pachube to push the conversation further?  Did Usman ask you to take a role in the future of Pachube. How does Pachube enable the vision of The City Is Here For You To Use? I could go on for ever with questions,  so please do tell!

AG: OK, I should probably reiterate that my fundamental interest is in people, and in what they choose to make and do with technology, not the technology itself. For the last few years, I’ve particularly been trying to understand how people interact with each other and with the urban environments around them when those environments have been provisioned with the ability to gather, process and take action on data. And this is how I come about my interest in what Usman is up to with Pachube, because those “gather,” “process” and “take action upon” functions are generally accomplished by different systems, designed by different groups of people, at different times and to different ends. What Pachube aims to do is make the difficult and not-particularly-glamorous work of connecting these pieces a whole lot easier.

Think of it as a step toward enabling the ontome, this so-called Internet of Things we’ve been talking about, the same way basic protocols like HTTP and HTML enabled the wildfire spread of the Internet we’re familiar with. What Pachube offers is a way – a relatively straightforward and self-explanatory way – to plug any given compatible input into a similarly compatible output. So if you’ve got an air-quality sensor or a soil-pH sensor or a personal biometric monitor, you can plug it into Pachube, and someone else can grab the data those things generate and use it to drive a visualization, or the state of a physical system like a window, or whatever else they can imagine. It’s as close as anyone’s yet come to providing a plug-and-play backbone for the creation of responsive environments.

And I think it’s absolutely brilliant that it’s designed to work with Arduino and Processing, two lightweight, open-source frameworks that hobbyists and researchers (and even one or two more serious developers) around the world are already using to build things. (Arduino’s a kit of parts for doing basic physical computing – using data to drive lights, motors, and other actuators that have effect out here in the world – while Processing is a very accessible language to do dynamic and interactive graphics for screen-based media). Given both its openness and modularity, and its willingness to build on top of the very popular frameworks that already exist, I’m very excited to see what people make of and with Pachube.

I have to be honest and admit that personally, I couldn’t really care less about the environmental angle, for reasons that I went into at embarrassing length above. What I’m engaged by in Usman’s work is the idea that Pachube is helping to create an open platform for people to share data more readily. And while, no, he hasn’t explicitly asked me to take any particular stake in things, I’m always happy to lend a hand in whatever way would be most useful. I think it’s a project worth supporting.

As to how Pachube enables some of the ideas in The City Is Here, the answer has to do with the book’s call for every “public object” – every lamppost, bus shelter, commercial façade, and so forth – to support an open API. Something’s got to string all those objects together, present them to people as resources to be taken up and used, and Usman’s offered us a critical first step in that direction.

TS: Usman suggested, it might be interesting to ask you about “the tension between ‘could’ and ’should.’

Usman Haque: There are a whole bunch of things that we “can” do, technologically speaking; how do we decide what we ’should’ do, as we find ourselves in an age where we can build almost anything we can imagine…? particularly with reference to technology/privacy/security triumvirate. e.g., leaving aside that the majority of the world is *not* in the technology ‘paradise’ that we’re in, here in the west, only a small fraction of people are currently producing the technology that the rest of us use; one aim is to get people more engaged in the productive process, but, in a sense that will also mean the whole wide ecosystem of technology will be even bigger, both “good” stuff and “bad” (that qualification firmly placed on how it’s used), as opposed to now when we can focus on quite specific things that government & industry are doing and saying “that shouldn’t be happening….”. part of this relates to something adam said on his blog  in the comments (see here).” 

AG: I think the first part of answering that question has to involve figuring out who “we” are in any given situation. A “we” composed of seven Helsinki-based Linux developers would most likely arrive at very different answers than the United States Air Force Materiel Command or Samsung’s board of directors, right? So clearly, a first challenge is getting to some kind of pragmatically useful alignment between those local and occasionally even painfully parochial perspectives with what’s best for the Big We. And this challenge is only going to become more vexing as the ability to imagine, design, build and deploy informatic componentry gets more and more widely distributed. In this respect the spread of simple, modular, low-barrier-to-entry tools only makes things worse!

The primary issue that I can see here is that the inherent clock speed of technical development is so very much faster than that of any meaningful deliberative process “we” might bring to bear on it. A concomitant concern is that the sources of technical innovation and production are now so widely distributed that you can be reasonably certain that somebody, somewhere will implement any given technically feasible idea, no matter how offensive, poorly thought-out, socially disruptive or frankly stupid. A public toilet you have to SMS to unlock and use? A “Friend Finder” visualization with high locational precision and no privacy features whatsoever? A first-person rape-simulation “game”? A clunky brown iPod knockoff? Somebody thought each one of these things was worth the time, expense and effort to actually go about making it. They exist.

But I’m pretty old-fashioned in some ways, in that I think the good old Habermasian idea of the public sphere still has some life left in it. And I think it should be self-evident by now that there’s no necessary contradiction between even the newest (cough) “social media” and the formation of such a sphere. So you’ve provided a forum, and in it I get to express my belief that these things are stupid and pointless and probably should not have been built. And if somebody gets all het up about that, they can argue right back at me in comments. And eventually one or another of these positions begins to tell, in terms of regulation, legislation, and other tools of the juridical order, in terms of protest campaigns or organized boycotts or litigation…in terms of nonexistent sales!

There’s nothing new in any of this, of course, though indubitably some of the dynamics are amplified or accelerated by e-mail, Twitter and YouTube. My main contention is that informatic technology now has such deeply pervasive implications, and for things like presentation of self that previous waves of technical development barely touched, that “we” as societies need to be very much more conscious of the consequences before committing to any one course of action.

I should also point out that I do not, at all, believe that we’re “in an age where we can build almost anything we can imagine,” though I might buy “…two or three of almost anything we can imagine.” On the contrary, as I implied above, I think the global constraints on our ability to operate freely are already becoming quite evident, and will continue to grow teeth over the next few decades.

TS: Also Usman  added …

Usman Haque: ..where Adam said: in this regard, I very much *do* have a problem with “just showing up.” — something I feel that as well. but i always wonder: What happens when one appears to be mandating participation…?

AG: Look, I happen to have a strong – maybe some would say obnoxious or hyperactive or overdeveloped – sense of personal responsibility and accountability. I think one is basically committed to some measure of responsibility for the commonweal simply by surviving to the age of majority. The choice of how, particularly, to discharge that responsibility can only be yours and yours alone, but it can’t be ducked or gotten around without severe and entirely predictable consequences. So to Usman I’d respectfully suggest that I’m not the one mandating participation. Life is.

TS: It seems we have grown accustomed to striking a Faustian bargain on the internet today -  in order to share and distribute parts of our identity we are expected to give up key information to one site to store and disperse our data. I took part in a discussion with David Levine, IBM and Eben Moglen on privacy last year.  And Eben Moglen gave a succinct description of the elements of privacy and how they have been treated in the American Constitution that is, I think, relevant to unpacking some of the challenges of ubiquitious computing. Here are some extracts from that conversation where, Eben notes:

there are three elements that are mixed up in privacy and we tend not to notice which one we are talking about at any given moment.

There is secrecy – that is the data should not be readable by or understandable by anybody except me or people I designate. There is anonymity which is the data can be seen by anybody but about whom it is should be knowable only by me or people that I designate. And there is autonomy which isn’t about either secrecy or anonymity but which is about my right to live under circumstances which reinforce my sense that I am in control of my own fate. And this form of privacy is actually the one we talk about in the constitutional structure when we talk about the right to get an abortion or use birth control.

“Anonymity” is a condition that is a deep structuring characteristic of the internet as you, Lessig and others have commented on.  And frequently we are promised (questionably) “secrecy” or anonymity as privacy protection by services handling our data on the internet.  But Eben (one of the US’s great constitutional lawyers) points out that “autonomy” is a key form of privacy in the  US constitutional structure that is often compromised in situations where our digital selves may constrain our non-digital selves.

The real issue here is about the forcing of choices on us…digital aspects of identity can quickly acquire an inflexibilty that constrains our non-digital selves.

I see again and again the ways in which people now find themselves unable to make certain life choices easily because there digital self has acquired an inflexibility that constrains their non-digital self.

As we go beyond the end to end internet and we lose the structuring characteristic that has privileged anonymity: How do you see these three elements of privacy, anonymity, secrecy and most importantly autonomy, being worked out in a networked world beyond the end to end internet?

Are there any new structuring characteristics that could privilege autonomy? (which Eben indicates is linked to having a flexible identity).

AG: If we accept for the moment a definition of autonomy as a feeling of being master of one’s own fate, then absolutely yes. One thing I talk about a good deal is using ambient situational awareness to lower decision costs – that is, to lower the information costs associated with arriving at a choice presented to you, and at the same time mitigate the opportunity costs of having committed yourself to a course of action. When given some kind of real-time overview of all of the options available to you in a given time, place and context – and especially if that comes wrapped up in some kind of visualization that makes anomaly detection and edge-case analysis instantaneous gestalts, to be grasped in a single glance – your personal autonomy is tremendously enhanced. Tremendously enhanced.

But as to how this local autonomy could be deployed in Moglen’s more general terms, I don’t know, and I’m not sure anyone does. Because he’s absolutely right: Bernard Stiegler reminds us that the network constitutes a global mnemotechnics, a persistent memory store for planet Earth, and yet we’ve structured our systems of jurisprudence and our life practices and even our psyches around the idea that information about us eventually expires and leaves the world. Its failure to do so in the context of Facebook and Flickr and Twitter is clearly one of the ways in which the elaboration of our digital selves constrains our real-world behavior. Let just one picture of you grabbing a cardboard cutout’s breast or taking a bong hit leak onto the network, and see how the career options available to you shift in response.

This is what’s behind Anne Galloway’s calls for a “forgetting machine.” An everyware that did that – that massively spoofed our traces in the world, that threw up enormous clouds of winnow and chaff to give us plausible deniability about our whereabouts and so on – might give us a fighting chance.

TS: The concept of autonomy is signaled clearly in the title you have chosen for your next book, The City Is Here For You To Use, and is a theme of all your writing!  While you talk about many of the possible constraints to presentation of self and potential threats to a flexible identity that ubicomp poses, your next book signals optimism. What are your key grounds for optimism?

AG: It’s not optimism so much as hope. Whether it’s well-founded or not is not for me to decide. I guess I just trust people to make reasonably good choices, when they’re both aware of the stakes and have been presented with sound, accurate decision-support material.

Putting a fine point on it: I believe that most people don’t actually want to be dicks. We may have differing conceptions of the good, our choices may impinge on one another’s autonomy. But I think most of us, if confronted with the humanity of the Other and offered the ability to do so, would want to find some arrangement that lets everyone find some satisfaction in the world. And in its ability to assist us in signalling our needs and desires, in its potential to mediate the mutual fulfillment of same, in its promise to reduce the fear people face when confronted with the immediate necessity to make a decision on radically imperfect information, a properly-designed networked informatics could underwrite the most transformative expansions of people’s ability to determine the circumstances of their own lives.

Now that’s epochal. If that isn’t cause for hope, then I don’t know what is.

Newson Obamanook – photo by Adam Greenfield, “The fact that it was one of the happiest days of my adult life may have colored my appreciation of this space. A bit, anyway.”

TS: In your writing you seem to imply that we will not find answers to our new relationship with Everyware by transposing the internet onto things for convenience’s sake but rather like the bike messengers -  we must explore the rich and complex terrain of the city that is ours to use in a give an take relationship.  Through our own exertions we find- how “anything reasonably smooth and approximately horizontal can become a thoroughfare,”  rather than be served up the city as something for us to consume.

You seem to be suggesting our city becomes ours to use because of the way we use it in our personal journeys -like “the messenger subconsciously maps the contours of an economic geography – known sources and sinks of courier assignments, or “tags” – and a threat landscape, this latter comprised of blind corners, cable-car and metro tracks, and traffic lanes.

But bike messengers are the lone ranger of our big cities. Others surf the city in tribes that ride the roiling tides of highly networked information together. How are the “natural” gestures of these tribes, e.g. day traders, who yoked to the tracings of a hive mind, part of the city that is here for us to use?  I thought the comment @ginsudo made shortly after joining Twitter and setting up TweetDeck particularly poignant:

“watching Tweetdeck is like watching stock market of your personality ebb and flow. needs analytics to maximize inherent self-involvement.”

But, for many of us our work has more in common with the day trader than the bike messenger, and are we pretty hooked on the ever growing possibilities for “contact” and identity sharing/construction, social media has produced  (with all the”Here Comes Everybody,” C. Shirky, benefits and risks).  Early theorizing of a “calm,” invisible” ubicomp seems out of synch with the excitable, active, engaged, contact driven, “users” that are watching stock market of their personality (or personal brand) ebb and flow.

How will these excitable/exciting processes of contact and identity sharing that have captured of a pretty large segment of popular imagination (not confined to the West -services like Gupshup does much of the same curating, linking and distributing of identity that web based social media does in SMS) be/ or not be part of The City Is Here For You To Use?

AG: Let’s remember that ubicomp itself, as a discipline, has largely moved on from the Weiserian discourse of “calm technology”; Yvonne Rogers, for example, now speaks of “proactive systems for proactive people.” You can look at this as a necessary accommodation with the reality principle, which it is, or as kind of a shame – which it also happens to be, at least in my opinion. Either way, though, I don’t think anybody can credibly argue any longer that just because informatic systems pervade our lives, designers will be compelled to craft encalming interfaces to them. That notion of Mark Weiser’s was never particularly convincing, and as far as I’m concerned it’s been thoroughly refuted by the unfolding actuality of post-PC informatics.

All the available evidence, on the contrary, supports the idea that we will have to actively fight for moments of calm and reflection, as individuals and as collectivities. And not only that, as it happens, but for spaces in which we’re able to engage with the Other on neutral turf, as it were, since the logic of “social media” seems to be producing Big Sort-like effects and echo chambers. We already “maximize inherent self-involvement,” analytics or no, and the result is that the tools allowing us to become involved with anything but the self, or selves that strongly resemble it, are atrophying.

So when people complain about K-Mart and Starbucks and American Eagle Outfitters coming to Manhattan, and how it means the suburbanization of the city, I have to laugh. Because the real suburbanization is the smoothening-out of our social interaction until it only encompasses the congenial. A gated community where everyone looks and acts the same? That’s the suburbs, wherever and however it instantiates, and I don’t care how precious and edgy your tastes may be. Richard Sennett argued that what makes urbanity is precisely the quality of necessary, daily, cheek-by-jowl confrontation with a panoply of the different, and as far as I can tell he’s spot on.

We have to devise platforms that accommodate and yet buffer that confrontation. We have to create the safe(r) spaces that allow us to negotiate that difference. The alternative to doing so is creating a world of ten million autistic, utterly atomic and mutually incomprehensible tribelets, each reinforced in the illusion of its own impeccable correctness: duller than dull, except at the flashpoints between. And those become murderous. Nope. Unacceptable outcome.

Uncanny Valleys - Adam comments,”Our apartment in NYC as rendered in Google Earth, with realtime traffic, weather, daylight and shadow as well as geodetic, street grid and service overlays. Camera view is South; that’s First Avenue just left of center-screen.”

TS: Smart phone’s are now drawing everyware data into the system and the net is reaching into who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc..

Nathan Freitas says Android: “seems to be the platform most likely to socialize the idea that sensor data could be a piece of every application.” (Android APIs for a wide range of sensor data.)

What in your view will be the most likely platform, Android or what?, to socialize the idea that sensor data could be a piece of every application?

AG: An open platform. A platform with lots of hooks and ways to plug things into it, a strong developer community, a shallow learning curve and/or an easy-to-use, high-level development environment.

I don’t have a dog in this race, mind you. I couldn’t care less who gets there first.

TS: New location based services, e.g., Xtify and ViaPlace, are offering us ways to share location data across lots of different applications (eg Xtify and a dating application like MeetMoi ). In return for services that allow us to share information, we must give up key information up to one site to store and disperse (although there are many differences in approach to our data, from the Twitter stance “show but don’t own” as opposed to Facebook’s stance – “in order to show we must have rights to it”). But the basic model of Twitter – to provide a white noise platform for people to build service on top off seems to be being transposed to location based services. Obvious questions arise like what happens to our data in a start up like MeetMoi if they go belly up?  Apparently in the dot.com bust data was the first thing to go on the auction block in bankrupcy cases.

Also, I suppose it is hardly surprising (if disappointing to me) that some of the early location based services are trying to get mindshare by picking up on the glue celebrities give to mass culture. At the last New York Tech Meetup, OMGICU demoed a rather terrifying new pre-launch location based “participatory celebrity gossip application” which seems to combine all the worst features of social media with celebrity stalking, plus a narrative to change the notion of celebrity itself by “turning D listers into A listers.”

Hopefully location based applications  will not get stuck on “stalker, stalker, stalker” apps like OMGICU .

David Oliver, Oliver Coady gave me a good question: “How does timeliness and location-independence change our ideas of social media?

And how can we design new architectures that can reinforce the sense that I am in control of my own fate?

AG: But we’ve already come so far in terms of turning D-listers into A-listers! On a daily basis, I’m exposed to almost as many cues insisting I attend to nonentities and dullards like Robert Scoble as those insisting I attend to nonentities like Madonna or Thomas Friedman. It’s gotten ridiculous.

Now, how does timeliness and location change our ideas of social media? It makes them dangerous!

Look, even a proud Z-lister like myself – I’m a public person only in the most debased and degraded meaning of that word – I’ve had experiences that shook me up, like having someone approach me while I was quietly hanging out in the back of St. Mark’s Books, and wanting to strike up a conversation based on some talk they’d seen me give a year or so previously. Now part of learning to deal with this kind of thing is shrugging it off, being grateful and flattered that someone thinks you’re interesting enough to single out for that kind of attention, or chalking this up to Sennett’s observation about the constitution of urbanity. Or doing all three at once.

But let’s remember that at the end of the day, a “social network” is nothing but a group of arbitrarily distributed human beings joined by a communications channel, and those people have eyes and ears. The degree to which they recognize some shared interest gives them significance filters. If social capital accrues to those in the network who are able to claim some connection with a “celebrity,” no matter how fleeting, then such connections are going to be mobilized, made explicit. And now say the network has been provided with the tools allowing it to plot the appearances of those putative celebrities in space and time, and what do you get? You get a circumstance in which it is very, very difficult to maintain any membrane between the private self and the world, for anyone who’s even remotely a public figure, whether they particularly want to be a public figure or not. You get network effects that amplify those locational traces, and further undermine any possibility of anonymity, even anonymity-by-suspension-of-interrogative-awareness (which is a clumsy way of referring to that blasé matter-of-factness around famous people that most big-city folks eventually develop).

Am I letting myself off the hook? Not in the slightest. I passed Terence Stamp on the street not so long ago, and you bet I Twittered it. My only excuse was that I Twittered it to a closed loop of no more than a few dozen people. But then, who knows what those few dozen people will turn around and do with that fact, on the open networks to which they in turn belong? And that, too, is my responsibility.

I’m not sure there’s anything to be done about any of this but cultivate our own urbanity, learn to say “so what” when we happen to find ourselves next to Philip Seymour Hoffman in the line at Whole Foods.

TS: Zittrain in The Future of the Internet: And How To Stop It, foregrounds “generativity” and a generative devices (as opposed to appliances) as the most fortuitous starting point for: “tools to bring about social systems to match the power of the technical one.”

Are appliances a threat to the city that is here for you to use? How can generativity ensure The City Is Here For You To Use as Zittrain argues it has ensured, even if imperfectly, that the internet has been here for us to use?

AG: You know, I haven’t read the book, I’ve only heard him give the talk, so it’s certainly possible there’s a subtlety to the argument that I’m missing. But I’m not sure Jonathan isn’t simply wrong about this notion of generativity. Not that the concern is misplaced, but that he’s insufficiently trustful in human agency. Is a car “generative,” by his definition? Certainly not. And yet look at all the cultural production that goes on around “the car,” look at all the assemblages people make with cars, from Beach Boys songs to ghost riding the whip, from J.G. Ballard novels and Herbie the Love Bug to Tokyo Drift.

Or probably more to his point: look at the Japanese mobile-phone market – seemingly one of the most locked-down and unpropitious circumstances imaginable for the production of culture, in technical terms and Zittrain’s both. And yet fully 50% of the bestselling books in Japan last year were written on mobile phones. Not read, which would already be impressive enough (if “impressive” is indeed the word): written. What does that imply for his argument?

So, yes, I think there are grounds for concern in that we don’t allow technologies and frameworks to appear that unduly limit the scope of human creativity. Code is still law. But I also think people are quite amply able to reach into what would appear to be the least propitious technologies and tell their own stories with same.

TS: One aspect of Everyware that seems in need of some visionary yoga is the how we will relate to pixels anywhere.

In Urban Computing and its Discontents you mention how our technological trajectories often make it seem as if we seem to get fixated on particular scenes in movies, e.g., Minority Report. You point out that so many ambient informatics projects seem simply “to expand the reach of signage and advertising in dense urban spaces….as if we’ve become transfixed by the scene from Minority Report where heterosexual cop John Anderton is on the run from his colleagues.”

Ideas from the Minority Report continue to hold sway in designs as we saw in the recent MIT demo of SixthSense at TED.

But visions of augmented reality were pretty high profile in this years Super Bowl commercials this year (including a highly anthropomorphic imagining of ubicomp that was a kind of WoW mashup with a Pixar movie).

What recent movies/commercials have produced scenes mostly likely to be are new fixation fodder for ubicomp and why?

AG: I don’t think I’m qualified to answer that, actually. We don’t have a TV, so I don’t see much in the way of commercials, and most of the films I wind up seeing are the kind that play at Anthology Film Archives. What I can say is that science fiction is currently suffering in toto from an inability or disinclination to posit future scenarios that are any weirder or more visionary than those emerging from other sectors of the culture. And that would be fine, except sf has traditionally been the place where we wrestled with the imaginary.

We need that set of tools, badly. If for no other reason than something I glean from personal experience: essentially my entire professional career has simply been the leveraging of ideas and concepts I originally wrestled with in the encounter with William Gibson and Bruce Sterling when I was 16. Today’s visionary sf means tomorrow’s halfway-competent generalist.

Nurri Kim in the waiting zone – photo by Adam Greenfield

TS: My AR friend, Robert Rice, who is working on a markerless AR platform, notes that data visualization is one of the critical elements of AR in terms of “make or break.” Robert says, “even with the ultimate in ubiquitious data from everything, without good data vis it will all be useless”

Also something Cory Doctorow said to me last year has really stuck in my mind. When I asked him what happens when Cyberspace everts, he talked about a reverse surveillance society:

Cory: Well this is like Spook Country the new Gibson novel – What happens when cyber space everts – hmmm? I’m not sure I have anything very pithy to say on that EXCEPT………
Apart from all the traditional kind of overlay reality stuff, if there is one thing I am actually interested seeing from a virtual world migrating to the real world its instrumentation.
I think lot of things that are characteristic of very successful internet based business is that they are extremely finally instrumented so like Amazon knows in aggregate on a second by second basis how their site is being used by people and they can twiddle the dials in real time.

As users of the world we have very little access to that kind of instrumentation. We don’t even know how the tube is running. The tube knows how the tube is running and we kinda of don’t. I would be really interested in seeing that. You’ve seen Joi Ito’s WoW interface right. Have you seen it …

Joi Ito’s WoW interface seems a long way from the calm, invisible imaginings for ubicomp by early ubicomp visionaries?

AG: Well, he’s got a particular kind of neural wiring. And there’s not a thing that’s wrong with that, except that I’d never, ever want to assert that what’s appropriate for Joi Ito necessarily is or should be understood to be appropriate for anybody else. The point of calling for open systems and frameworks is to allow us maximum scope of diversity in the ways we choose to interface with the world’s richness and complexity.

TS: What new imaginings/possibilites do you see when pixels anywhere are linked to everyware?

AG: Product placement. Commercial insertions and injections, mostly.

Beyond that: one of the places where Mark Weiser logic breaks down is in thinking that the platforms we use now disappear from the world just because ubiquitous computing’s arrived. We’ve still got radio, for example – OK, now it’s satellite radio and streaming Internet feeds, but the interaction metaphor isn’t any different. By the same token, we’re still going to be using reasonably conventional-looking laptops and desktop keyboard/display combos for awhile yet. The form factor is pretty well optimized for the delivery of a certain class of services, it’s a convenient and well-assimilated interaction vocabulary, none of that’s going away just yet. And the same goes for billboards and “TV” screens.

But all of those things become entirely different propositions in everyware world: more open, more modular, ever more conceived of as network resources with particular input and output affordances. We already see some signs of this with Microsoft’s recent “Social Desktop” prototype – which, mind you, is a very bad idea as it currently stands, especially as implemented on something with the kind of security record that Windows enjoys – and we’ll be seeing many more.

If every display in the world has an IP address and a self-descriptor indicating what kind of protocols it’s capable of handling, then you begin to get into some really interesting and thorny territory. The first things to go away, off the top of my head, are screens for a certain class of mobile device – why power a screen off your battery when you can push the data to a nearby display that’s much bigger, much brighter, much more social? – and conventional projectors.

Then we get into some very interesting issues around large, public interactive displays – who “drives” the display, and so forth. But here again, we’ll have to fight to keep these things sane. It’s past time for a public debate around these issues, because they’re unquestionably going to condition the everyday experience of walking down the street in most of our cities. And that’s difficult to do when times are hard and people have more pressing concerns on their mind.

Citywarecrash – photo by Adam Greenfield, “An occupational hazard for urban screens.”

TS: I know in Everyware you mentioned that architects have play an important visionary role in imagining ubicomp and I know you work closely with your wife, artist Nurri Kim.  Robert Rice asked me the following question – which I will in turn ask you: “In terms of augmented reality do you think virtual worlds and virtual reality experts / leaders / are good pioneers for thought and guidance on AR? Or, should we look for new leaders, or where are new leaders emerging? Is the tech similar enough for the old crowd to be useful or is it different enough to be a disadvantage coming from the old models?.”

AG: I should make it clear that I have absolutely no interest in virtual worlds or virtual reality. The so-called virtual worlds I’ve experienced seem sad and really rather tatty – eversions of the most predictable adolescent fantasies of unlimited power, reinscriptions of all the usual politics – and completely lacking in just about everything that makes life resonant, meaningful and awe-inspiring. And anyway, to paraphrase J.G. Ballard, ordinary, everyday life is now far more vividly and fantastically weird than anything you’ll see in Second Life. I mean, Garry Kasparov was heckled by a radio-control dildocopter, Joe the Plumber’s off to Gaza as a war correspondent, a sea of dust-covered BMWs waits in the long-term parking lot at Dubai International for owners who are never, ever coming back.

Look to virtual worlds for insight into the hard work of negotiating the actual, with its physics, its entropy, its suffering, with all its constraints? Oh my goodness gracious, no.
And look to leaders? Never. Leaders are for followers, and who wants to be that? I don’t mean you can’t take inspiration and insight from the work of others – not at all – but use your own imagination, take some personal risk, do your own damn work.

Now, having said that. This opposition of virtual and physical worlds strikes me as increasingly a false one, as it does many people. The hard-and-fast distinction between “the real world” and virtual environments make less and less sense, as righteously satisfying as making it can sometimes seem. There may be attributes of this physical environment that are impossible to see or make use of without access to the networked overlay, and those attributes may in time come to constitute the primary wellsprings of a given place’s meaning. And if you’re offering me some insight that I think could be of utility in resolving the challenge of making this overlay accessible to all, equally, I’ll gladly accept it, no matter what domain or disciplinary background you claim as your own.

Am I aware of any such insight coming out of virtual worlds? No. As Bryan Boyer notes, “If you want to start talking about some serious cross-disciplinary pollination then you better take both sides of that disciplinary divide seriously. When your ubi- runs into my building with its boring HVAC, mundane load paths, typical finished floors, plain old foundations, etc., the transformative powers of comp are bracketed pretty seriously by the realities of the physical world.”

The Cloud Gate has landed – photo by Adam Greenfield, “Tell me this doesn’t look *just* like the descriptions of “stasis fields” in 70s SF. In fact, the picture looks practically CGId to me.”

TS: Some people thought the whole world would have been plastered with RFID by now.  But before that has happened markerless AR seems to be in our sights.

If I understand it correctly marker versus markerless AR has quite different implications for how the cyberspace of ubicomp evolves?  I asked Robert Rice (he is developing a markerless AR platform) to explain some of the differences.  He said:

Robert also pointed out to me that markerless AR may prove even more problematic for privacy:

Do you have any thoughts about the different roles that markerless versus marker techinologies will play in AR and Ubicomp?

AG: I need to admit that I’ve never until this moment heard the phrase “markerless AR,” although I’d think it’s more or less self-explanatory to anyone who’s been following this stuff. Let me make the distinction explicit, shall I, for anyone who hasn’t been? And you or Robert can correct me if I’ve gotten it wrong.

Augmented reality means that I have some mediating artifact that provides me with a visual overlay on the world. This could be a phone, it could be a windshield, it could be a pair of glasses or contact lenses, doesn’t matter. And you’re going to use that overlay to superimpose some order of information about the world and the objects in it onto the things that enter my field of vision – onto what I see. So far, so good: that’s AR 101.

Now where does that information come from?

What you’re calling marker-based AR implies that there’s some reasonably strong relationship between the information superimposed over a given object, and the object itself. That object is an onto, a spime, it’s been provided with a passive RFID tag or an active transmitter. And it’s radiating information about itself that I’m grabbing, perhaps cross-referencing against other sources of information, and superimposing over the field of vision. Fine and dandy.

But there’s another way of achieving the same end, right? Instead of looking at a suit jacket on a rack and having its onboard tag tell you directly that it’s a Helmut Lang, style number such-and-such from men’s Spring/Summer collection 2011, Size 42 Regular in Color Gunmetal, produced at Joint Venture Factory #4 in Cholon City, Vietnam, and packed for shipment on September 3, 2010, you’re going to run some kind of pattern-matching query on it. And without the necessity of that object being tagged physically in any way, you’re going to have access to information about it. But this set of information isn’t, necessarily, what the object itself, or its creators or merchandisers, want you to know about it; it could be derived from online discussion fora or review sites, or blog posts, or whatever. All there needs to be is a lookup table, essentially, that tells you where to find information about any object in the field of vision whose identity can be established.

Do I have that right? And if I do, then as I understand it, the distinction is primarily a pragmatic one: it’s just easier to get to an augmented world, by far, if we don’t actually have to go to all the trouble of tagging everything in the world with its own dedicated RF transponder. Easier, and cheaper, and quicker, and more environmentally sound besides, because the relevant traffic is in bits not atoms.

Unless I’ve missed something, you don’t, then, get the distinction between classes of objects and instances of same. Sometimes, when there’s a 1:1 correlation between the two, that’s not going to matter: I’m walking down the street in Madrid, and my glasses or whatever can easily recognize that this building is the Caixa Forum. There’s only one of it, and I can get a positive ID via pattern recognition. But for some edge cases – twins and lookalikes, mostly – the same thing is generally true of people.

But other times it will matter. Is this specific watch a real, $10,000 Panerai or a $50 Kowloon fakery? How has this black 1998 Honda Civic over here differ from this other one in terms of its use and maintenance history? Does this O-ring gasket need to be replaced? I don’t see how you extract data from specific instances of things without the necessary sensor instrumentation, transmitter, etc., being coextensive with the object in question or very closely colocated with it over time – in the terminology you’re using, a “marker.”

So using these terms, I’d say that “markerless” AR comes first, is relatively easy to deploy, and generates not-insignificant value. But – again, unless I’m missing something – there are some things that it won’t ever be able to do, and for those things you need some provision for self-identification and self-location.

Ultimately I think it’s a distinction without a difference, from the user’s point of view. People will care much more about the source of whatever information shows up on their overlay than the precise technical means used to get it there.

The surrender to cynicism – photo by Adam Greenfield

TS: Much early thinking around ubicomp seems to have come from visionary architects and engineers but recently I was at the O’Reilly Tools of Change for Publishing Conference (publishing in the Digital Age) and I met several book futurists.  It struck me how ubicomp from the perspective of the book created some interesting questions for how particular material cultures will shape and be shaped by Ubicomp differently.

I noted, Google seemed well down the path to holy grail “converting images to original intent XML.” And Peter Brantley talked about machine parsed books.

At TOC there were many suggestions about how books might manifest as everyware. (Although it did not seem that many people felt books had a special relationship to time and history and would not vanish as one of the great metaphors of calm and solitary enjoyment in our culture soon).  Books as everyware will, it seems, include, amongst other things:

books that read books

books that read context

context that reads books

books that read me

books linked to mobility – timeliness and location independence

books that are not books

books becoming babble

books bubbling up from the babble

There is an Institute of the Future of the Book. Will all former material cultures require their own institutes of the future to guide their cultures into everyware?  Do you think books transition into everyware is especially significant and why?

AG: But all objects have a relationship to time and history, no?

TS: Yes! What I meant to convey really was the idea that many people expressed at TOC that books had a privileged relationship to knowledge in our culture that was valuable and related to some aspects of their current form, and that books as everyware, e.g. machine parsed books, and more sociallly generated forms would not replace that entirely.

AG: Gotcha. Well, I certainly agree that books constitute an interesting category unto themselves – I’ve held onto my physical books, and in fact still spend a fortune buying new ones, where I stopped buying music on discs a long, long time ago. But I don’t think this state of affairs can or should obtain forever.

Lately there’s been a good amount of thought around the notion of “unbooks,” which I regard as a container for long-form ideas appropriate to an internetworked age. By building on some of the tropes of software development, mostly having to do with version control, open-endedness and an explicit role for the “user” community, unbooks can usefully harness the dynamic and responsive nature of discourse on the Web. At the same time, you preserve the things books are really good at: coherence, authorial voice and intent.

The important part is in acknowledging two points which have usually been understood as contradictory, but which are actually nothing of the sort: firstly, that the expression of ideas in written form has something to learn from the practices that have evolved around the collaborative creation of dynamic, digital documents over the half-century-long history of software; and secondly, that certain ideas require elaboration in the reasonably strongly-bounded form we know as a “book,” and cannot meaningfully be shared otherwise. A third point, concomitant to the second, is that despite recent technical advances, screen-based media still cannot, and may not ever fully be able to, deliver the extratextual cues and phenomenological traces that support, inform and extend the meaning of written documents.

The unbook lets you have your cake and eat it too. So, for example, when we publish The City Is Here, one of its manifestations will be a static, physical document – and hopefully, if we do our jobs well, a very nice one indeed. But even before that, you’ll be able to download a Creative Commons-licensed PDF of every numbered version of the manuscript, from zero onward. Bottom line: you buy the book if, and only if, you want the object. The ideas are free.

TS: David Brin sees two futures:1) the government watches everybody, and 2) everybody watches everybody (the latter he calls “sousveillance”).  My friend Ben Goertzel says “hooking AI up to a massive datastore fed by ubicomp is the first step toward sousveillance?” What do you think the role of AI in ubicomp will be?  Is it worth thinking about what is the first important “AI meets AR” app is?

AG: I don’t believe that artificial intelligence as the term is generally understood – which is to say, a self-aware, general-purpose intelligence of human capacity or greater – is likely to appear within my lifetime, or for a comfortably long time thereafter.

Having said that, your friend Ben seems to be making the titanic (and enormously difficult to justify) assumption that a self-aware artificial intelligence would share any perspectives, goals, priorities or values whatsoever with the human species, let alone with that fraction of the human species that could use a little help in countering watchfulness from above. “Hooking [an] AI up to a massive datastore fed by ubicomp” sounds to me more like the first step toward enslavement…if not outright digestion.

Sousveillance - the term is Steve Mann’s, originally – doesn’t imply “everybody watching everybody” to me, anyway, so much as a consciously political act of turning infrastructures of observation and control back on those specific institutions most used to employing same toward their own prerogatives. Think Rodney King, think Oscar Grant.

TS: I have one last question from Usman Haque.

Usman Haque: insofar as a lot of what adam describes as desirable could be said to constitute pretty radical socio-political change (or perhaps… “adjustment”) i would be really interested to know how his current work @ nokia is or isn’t able to gel with the themes of his writing. in some senses there’s quite an undercurrent strongly challenging corporate practices, in other senses it could be seen as gentle nudges. how does adam see it? and how about the nokia behemoth? does he have success nudging nokia towards the kind of world he would like to see (i imagine the answer is ‘yes’ otherwise he wouldn’t be doing it…) but i’d love to know more about the limits/challenges.

AG: I am told that Henry Kissinger, on his first trip to China in 1971, asked Zhou Enlai whether he thought the French Revolution had or had not advanced the cause of human freedom.
Zhou thought for a moment, pursed his lips, and replied, “It is too soon to tell.”