We are in the age of unearthing and uncovering data, and only just at the beginning of the age of processing data and dealing with it (see my interview with Anselm Hook, Part 2 upcoming).  O’Reilly’s Strata Confernence 2011, will explore, “the change brought to technology and business by data science, pervasive computing, and new interfaces.” It is, perhaps, one of the most important events of 2011.

Data is driving a revolution much as coal, oil, and steel powered the industrial revolution.  And the world changing insight from Karl Marx that “the industrial revolution polarized the world into two groups: those who own the means of production and those who work on them,” is taking on on new life, as  Alistair Croll, co-chair of Strata 2011, points out in his post,  “Who Owns Your Data?”

“The important question isn’t who owns the data. Ultimately, we all do. A better question is, who owns the means of analysis? Because that’s how, as Brand suggests, you get the right information in the right place. The digital divide isn’t about who owns data — it’s about who can put that data to work.”

Strata is where a vanguard will be meet, not only to discuss this revolution’s futures, but to define how to create, handle, and build the platforms and experiences that will harness the data. My flight is booked!  (Also check out BigDataCamp which takes place the night before Strata.)

The picture opening this post is from Michael EdgeCumbe’s  Fall 2010: ITP Winter Show Project.  A project exploring ways to intuitively get the feel of what it going on with big data sets using “the gestural manipulation and stereoscopic visualization of complex data to create a meditative state for data analysis.”  Michael project will be part of the Science Fair at Strata.  For more on Michael’s work see Noise Derived. I also have a number of the  interesting new interface sessions at Strata in my schedule.

The daily Strata Gems on O’Reilly Radar are great place to get a gestalt of some of the Strata themes, and this post by Edd Dumbill, program chair for Strata, Three key data trends for 2011, looks at the year ahead.  This week, I got the chance to ask Edd a few of the questions that I will have on mind at Strata – see his responses below.

If you have been reading Ugotrade, you will know I am interested in our mobile social augmented futures and there is no question in my mind that these will be unleashed by our new capacities to work with data (see my post here).

Data is the how.

The pic above is from “Secrets of BackType’s Data Engineers.” This post on ReadWriteHack by Pete Warden, an ex-Apple engineer, and founder of OpenHeatMap, really lives up to its title.  Check it out if you want to know how  “three guys (the BackType team ) with only seed funding process a hundred million messages a day?”

I asked on Quora, “What would be the most important developments for Augmented Reality in 2011,” Michal Avny, Strategist & Real Time search expert, wrote:

“AR strongly relies on localized personalized real time information.

Having a stream of tweets based on keyword search, location or circle of friends doesn’t really make the AR experience; it is the processed real time relevant information that will make AR useful and intensify the experience.”

In 2011 Real Time search and Social Search will drastically change to provide the infrastructure required.”

I followed up on Michal’s Quora answer with some more questions – see below in this post.

Also note the response from David Molnar, here is an excerpt:

“2. A wave of actionable, important data APIs opened up, enabling useful non-gimmicky AR apps for the first time. Think geoloqi.com , or the work Max Ogden has done with Portland civic data. Plus of course face.com , email providers and calendar providers, etc.”

Amber Case, one of the founders of Geoloqi, is on the programming committee of Strata and will be speaking. Be sure to catch her session! Posthumans, Big Data and New Interfaces, and if you haven’t already seen it, Amber’s TED talk is a must see.

Geographic proximity is a powerful filter, as is route, and time. But clearly social proximity, social relevance, and shared tastes are also key dimensions for location based experiences, (see my convo with Schuyler of Simple Geo, upcoming).

While the whole business of location based search and curation of augmented mobile social experiences is still, for the most part, uncharted terrain, the danger of key points of control being only really accessible to elite players looms large. I asked Sophia Parafina, a pioneer in the open geo space for some thoughts on real-time local /geosearch and geomessaging, and the future of openess & big data (see Sophia’s response below).

Is the market ready yet for P2P cloud computing?

This is another question I’m following,  Is the market ready yet for P2P cloud computing? It is one of those questions that we seem to have been asking in various forms for a very long while now, but without a  major shift in sight.  The pic above is from, The Cloud Made Open Source “Invisible” This Year.  But, perhaps, we are at the point when open p2p clouds will find a place in the market because of their potential importance in real time social search and discovery. Borislav Agapiev, Search Entrepreneur and founder of Vast.com, writes on Quora:

“I believe a P2P cloud is ideally suited for social & real-time search and discovery.

Consider MapReduce, a very interesting and popular paradigm for distributed computing. MapReduce is very much about bringing computation to data i.e. doing computation at nodes (map) and then aggregating results through network (reduce).

It is very clear now that user attention data (what they click on) is very valuable for search and discovery, yet a centralized model relies upon uploading all that to a single location and then doing a supposed local MapReduce. Clearly, MapReduce could be done across the network, without any centralized uploads.

In addition to the efficiency argument raised here, it is even more important to consider privacy issues. Uploading massive amounts of user attention data to a centralized location is not something that is going to make users warm and fuzzy as we are increasingly seeing.

In a P2P cloud, there is no big brother watching over anyone, all computation and data storage is done in the cloud, fragmented in many, many small encrypted pieces ala BitTorrent.”

Picture above from Brynn Marie Evans, “It takes two to tango: review of my social search panel“

The Delta of Now – Transforming Search into a Social Democratic Act

Picture of Maneki Neko “beckoning” cats from Journeyetc

New ecologies of human and machine intelligence are beginning to change basic social structures – see the Future of Work (Biewald and Chirayath Janah 2010). And projects like Swift River, using search and machine mining to filter out streams on topics of interest that can then be subsequently curated by human beings. This may be extended to the curation of real-time data streams and employment of machine learning algorithms based upon the explicit relationships.

Augmented mobile social experiences are a new frontier in which ideas and practices from a number of fields collide, including: ambient findability (Morville 2005), urban psychogeography, narrative structures, ambient games and devices, 4d (time-space), explorations of place and memory, enchanted objects and people (Kuniavsky 2010), and designed animism (Laurel 2010), to mention just a few.

Mobile local interaction presents an opportunity to invert the search pyramid and to transform search into a social, democratic act (see my interview with Anselm Hook upcoming).  Up until now search has been predicated around a very narrow revenue model. Google has an implicit model of a B2C – business to consumer brokerage. We are only just beginning to get a glimpse of the disruptive potential of C2C – consumer to consumer brokerages. Mobile local C2C brokerages that allow us to transact in a trustworthy way over our local geography in close to real time (Hook 2010) have the potential to enable new forms of social organization. Bruce Sterling’s short story about a networked gift economy, Maneko Neki, is a brilliant glimpse at the disruptive potential of such re-imaginings.

Augmented experiences that shift or change a person’s situated geolocal experience of social reality, and change our relationship to the people and the place by augmenting engagement in, and reputation through, socially driven consumer tie ins and game dynamics, like Four Square, & Gowalla are beginning to emerge, as Kati London pointed out in her excellent keynote at Web 2.0 Expo. And, while the integration of mobile local interaction and an augmented view that shifts our geolocal experience visually will involve creative solutions to some well churned mobile, tracking, mapping and registration challenges, the exploration and development of new dimensions through which we can filter and create trusted and meaningful augmented mobile social experiences is vital, whether you are considering a mobile screen, map, camera view, or futuristic HUDs and gestural interfaces.

Talking with Edd Dumbill

Picture from O’Reilly Community.

Tish Shute: First congratulations on Strata! On the Strata homepage there is a quote from Jason Hoffman:

“My gut feeling is that we’re going to look back at the upcoming Strata Conference like we do at the Web 2.0 Conference in 2004/2005.”
—Jason Hoffman, CTO/Founder, Joyent, Inc.

Why do you think Jason’s comparison might be prescient?

Edd Dumbill: Web 2.0 is a development that ran through every brand that has a web presence and radically changed the way business is done for many companies and brands.

Strata will have a similar impact: every business has data, every business collects an increasing amount of data. This data is the new oil – a valuable raw material that when refined or combined creates value and opportunity.

Tish Shute: The rise of real time was one of your three key data trends for 2011. Hadoop is bringing the capacity to work with big data to more than just a few elite players. But the challenge is still real time. You mention we will be seeing a hybrid approach to real time and batch MapReduce processing. Will we hear more about these approaches to real time at Strata? And, what do you see as the most important conversations on real time data analytics emerging at Strata?

You point out “open source projects and cloud infrastructure means developers can evaluate and learn to love technologies without requiring support or approval from above.” What are the most exciting developments on the horizon for open source tools?

Edd Dumbill: Here are some projects worth watching, in the key areas of real time, cluster management and Hadoop.

* Cassandra and MongoDB — NoSQL databases that will prove vital for anybody with real time big data needs

* Mesos — a compute cluster management tool, modeled after that which powers Google

* Hadoop ecosystem’s continuing maturation, especially HBase and Hive.

Tish Shute: Do you think the market is ready for p2p cloud computing?

Edd Dumbill: The market is emerging for decentralized and distributed cloud computing, and P2P technologies are one way of achieving that. They key trends will be moving computation nearer the data sets or nearer the point of user consumption of the result.

P2P is a difficult model for anybody wanting to commercialize a service, so I think it will tend to form part of a hybrid solution.

Tish Shute: We have seen enormous strides in our ability to work with giant unstructured databases recently. Do you think, perhaps, that the dream of a web of linked data – “a web of data that can be processed directly and indirectly by machines,” will be attained through brute force – i.e. through our ability to harness the power of massively parallel processing, as much as by Semantic Web approaches focused on machine readable metadata? [Also see my question on Quora, "Is this a good approach (www.dist-systems.bbn.com/people/...) to use Hadoop to build a scalable, distributed triple store?"]

Edd Dumbill: I’ve been an observer of the SW for over a decade and I tend to believe that on the web, data means to you whatever meaning you give it as the consumer. With that model, the links are made by the consumer rather than sitting out there explicitly. Some links become de facto standards, and some very few become web standards.

I think the actuality will be a mix of both explicitly stated metadata and that which is inferred. The Semantic Web is a great framework for certain operations, especially interoperable exchange of metadata. A great many more private meanings, never intended to be shared, will be created by consuming software.

There’s no question that machines will learn how to process most of the Web. Furthermore, machines will learn how to process most of the physical world we’re in. And that by the end of this decade.

Talking with Sophia Parafina

Picture of Sophia at Where 2.0

Tish Shute: Sophia you have worked in the trenches for a long time now to support the growth of open geo data. What do you hope to see emerge in 2011 in the field of geo-data?

Sophia Parafina: Better support for displaying and handling location data across multiple apps. Fred Wilson recently blogged about content-shifting, he talks about overcoming content silos across devices. We’ve worked very hard to reduce data silos via formats, but devices are creating their own silos. I would like to see a standard method for sending geo data and geo information to mobile devices.

Producing content for mobile is different from producing content for a computer browser. Web 2.0 produced a lot of infrastructure for browser based interfaces, but in mobile devices that gap has been filled with apps which is fragmenting how data is handled by various devices. What is even more interesting in the mobile space is that devices can push data back that contains location, user updates, photos and even sensor data.  If mobile data standardizes, it could lead to browser based applications and stem the continued fragmentation of the mobile application market.

Tish Shute: Simple Geo and Factual are startups emerging in the geodata space. What do you see on the horizon in terms of both the growth of business opportunities and an open geo data community?

Sophia Parafina: In the near future think we’ll see startups providing curated data + API and in response we will also see companies that provide a single interface across multiple data providers. We saw this when everyone released a mapping API and companies such as Mapufacture provided a single interface across multiple APIs.

We will see a resurgence in data providers repackaging the the 2010 US Census data in different ways to respond to market segments, some of this will be open data but all of it will be provided through an API instead of file. Additionally, we’ll see more data from outside the US.

Tish Shute: What are the biggest obstacles to having the open geodata sets available that we need to enable mobile local interactions and social augmented experiences?

Sophia Parafina: Licensing for both crowd sourced data and private curated open data will become an issue. We recently seen VLC, the open source video player, pulled from the Apple app store because of licensing issues. Also, licensing of content by geography will be problematic, limiting searches by geographical location. In addition, how will licensing of data that is updated by crowd sourcing work?

Multiple APIs for accessing data sources. The current trend for each provider to create an API for their data sets will result in data silos – there needs to be a single sign-on equivalent for requesting data.

Size of data on the wire, the current models for delivering data is based on broadband connections. However, as mobiles increasingly become the way people use the web, the data needs to be sized accordingly. This also goes for mobile interfaces. Have you tried to shop on a mobile device, or buy a train or plane ticket? It’s frustrating and error prone. There is a large untapped market of people who only use the Internet on mobile devices.

Tish Shute: You pointed me to this link in Strata Gems re “an interesting and pertinent (also a competitor to GeoLoqi),” – the Android Tasker app. What do these emerging services bring to the table in terms of the next generation of location based services?

Sophia Parafina: This app let’s your device interact with the environment. I think that this is a great way of using the sensors on existing platforms to increase interaction and to implement ambient findability. The basic premise of Tasker is that some action happens in response to an event in an application, time, date, location, event, or gesture. Tasker has defined 180 actions that can occur based and number or combination of events. This can provide a basic vocabulary for interaction between the user and the device and more importantly between users. Tasker also can use Android script plugins, which lowers the bar to creating your own ambient application.

Programs such as Tasker can provide a way for people to interact with social networks beyond sending messages. People can use their mobile devices to interact with their surroundings with out having to interact with the device.

Tish Shute: We have had many conversations about emerging ideas of geo-search, geo-messaging and geo-fencing. What are the most interesting developments in these areas and what do you see on the horizon for 2011?

Sophia Parafina: The map will fade into the background and become less important. Display of information will be context aware, that includes location. For example, let’s say I make a grocery list, when I’m at the grocery story, the list will just pop-up without the need for me to find the app that has the list. Or reminders or offers pop-up when you are near a place at a certain time, let’s say you need to buy a present for a birthday party for a child, you could send out a request that you are looking for an item and retailers could offer “on the spot” discounts if you are in the area.

Geo-search, geo-messaging, and geo-fencing are geared to towards mobile devices, so I expect to see them soon as part of apps. Building generic applications that implement geo* will fail because that sort of information is useful only within a context. Geo* apps are solutions looking for an problem. The killer mobile app will use these functions transparently to reduce the cognitive load of the user who is busy moving around in the world.

User data gathered from multiple web applications will become consolidated profiles that will used for context aware applications. For example, there could be a service which matches prices of items that you have shopped for on the web, so for example the service would have access to your cookies, know your favorite retailers, things you have shopped for, your location and activity patters (when you are at home, work, restaurant). When you are in the vicinity of a brick and mortar retailer with the same or similar items, the service can send you alert to match the price of the item you found on line. So your digital life will become more closely linked with your day to day activities.

Talking with Michal Avny

Tish Shute: At Web 2.0 Summit, one of the highlights for me was the, Q&A:The New Search Insurgents lunch where Charlie Cheever of Quora, IMO, stole the show. I tweeted:

“One of my takeaways from #w2s is that #quora points to future of augmented mobile social experiences – a search filter for experience! #AR”

In your view what are the biggest challenges for location Q&A to emerge as a search filter for location based experiences?

Michal Avny: The biggest location Q&A challenges yet to be conquered are immediacy (real time dynamic data), relevancy (strong personalized filters) and user experience (simplified interface).

Location Q&A enables different use cases. The most prominent are Follow (follow places, topics and friends to learn about a location), Interact (meet new people based on common interests), Plan ahead (plan a trip, night out or a shopping day by asking and searching for local information) and On-site (check for recommendations, friends, deals, events and traffic nearby).

Unlike Follow, Interact and Plan ahead that can be added to existing Q&A platforms (such as Quora) by attending location specifics as they share similar characteristics, the on-site mode introduces a completely different experience, first and foremost it requires immediate attention. It is real time based and the nature of the data is dynamic. Traffic updates, current events, nearby friends, all that changes constantly. Posting a location question on-site implies the response should be in real time (e.g. best kid friendly restaurant), the normal Q&A response latency wouldn’t work.

Strong relevancy filters are required to accommodate for the overwhelming flood of information. Moreover, some of the data should be filtered by user behavior and preferences, check in notifications (type of relation), restaurant recommendations (type of food, price level, etc), shopping deals (commercial categories) and more.

Mobile experience requires ease of use and simplicity. A new Q&A interface and query language that allows for posting questions should be defined as well as coherent summarized response interface. User on the go should not have to post lengthy questions, browse through tens of results or search for the right service, but instead use a simple intuitive tool.

Tish Shute: Real- time location based search is in its infancy. Real time questions can be answered using different services such as Yelp, TripAdvisor, Waze, Foursquare, IMDb and more. But what are the challenges to moving forward with aggregating these sources and then into “locals” that are able to process and deal with vast amounts of information?

Michal Avny: Using some of the leading location services to answer question is sufficient to start with.

In order to provide broad coverage (worldwide) and reliable information, aggregation of the different services is required for instance to normalize product and service rank, aggregate classified, and more. This is quite challenging as there is no one standard available.

When location Q&A user base is big enough, I foresee a tendency to rely more on ‘locals’ input as the base of information. As the platform grows, communities will be formed with different cultures, relationships and trust levels, making the information more valuable and customizable. Some of the challenges I already mentioned are implementing filters, query language and interfaces to enable using the vast amounts of real time data in a mobile environment. More of the challenges lying ahead are integrating the ‘locals’ data with location based services as they are integral components of the Q&A ecosystem. Merging trust levels and relationships while adhering to different privacy guidelines is a challenge yet to be explored. (This should be discussed in more detail under the protocols topic).

It is quite evident that Quora is now facing growing pains and is struggling to maintain its character. Same as with Quora, it will also be a challenge to support and maintain the ecosystem while allowing for massive scale-up.

Tish Shute: I have been very interested in exploring protocols that will be enablers to micro local interaction and mobile social interaction for AR – particularly the XMPP extensions and operational transform work of Google Wave (now Apache Wave), and PubSub protocols like PubHubSubbub and Erlang based RabbitMQ. We are beginning to see protocols emerging that could enable new real time local services. What do you think are some of the most valuable use cases for “locals” that this new generation of real time protocols can enable?

Michal Avny: AR is about interacting with digital information; the AR ecosystem is composed of layers and components such as devices, platforms, browsers, applications and content. For the different components to interact new protocols, security guidelines, and privacy policies must be in place. A standard will enable local vendors and service providers to publish specials, deals, updates and events for any application to broadcast, identify people and places by proximity (without having to use the same application or device), local recommendations will be shared by services, devices will be able to interact, location based platforms, such as Q&A, will have access to vast breadth of information, geo aware devices will provide consistent experience globally, and much more.

Tish Shute: What do you think are the biggest challenges to going mainstream for this emerging field of real time social discovery?

Michal Avny: The biggest challenge is building towards real time, geo-aware, localized, personalized ambient data. Discovery is in its infancy, location social based Best, Top, and Trending lists with some basic filtering options are available, and this is great as people are getting accustomed to information surrounding them. To some degree it can intensify the AR experience, for instance suggest the most popular dish in a restaurant, or map the best coffee shops nearby, but it is customized at best by friend recommendations and depends on the coverage and broadness of the specific discovery service.

There is a need for the next generation of discovery, customized geo social aware discovery that filters the vast amount of real time data by learning user preferences and behavior (built on top of the much needed local social real time open protocol)

Tish Shute: Who are your favorite startups/upstarts in the the field of real time search and why?

Micha Avny: My6Sense - My6sense provides a sharper and better way to experience your information from feeds you subscribe to (Social Networks, News, RSS feeds, etc.). It’s personal – Content is ranked based on what’s relevant to you. It learns what’s valuable to you by translating your consumption behavior into a personalized ranking function.
My6Sense – because it is a personalized prediction filter, a critical foundation for AR

Topsy – Topsy is realtime search powered by the social web that finds the most relevant conversations happening online. The site’s underlying technology examines popular links as well as the influence of each person citing a link. Topsy augments traditional search engines by finding information that people are talking about.
Topsy – because its ranking is based on retweets and influencers, a great social experience

Collecta – Collecta is a real-time search engine for the social web. It monitors the update streams of popular realtime blogs and sites like Twitter, WordPress, and Flickr, and shows results as they happen. Results can be filtered by status updates, comments, stories, or photos. The entire engine is built around the XMPP standard, which pushes out data on a continual basis, so that for every search you end up watching a stream that keeps updating itself.
Collecta – because it is built around XMPP, a real time experience

Shortly after Augmented Reality Event – are2010, I talked with Bruce Sterling on skype and in gdocs about his experience there.  I am posting the conversation in two parts to make it a more blog friendly length!

The picture above is the Auggie Award for the best AR demo (above) designed by Sigal Arad Inbar.  It was won by YDreams! See, Ivan Franco recounts the team’s ARE 2010 experience, and winning the event’s first-ever Auggie Award, for more.   The video below was shot at the YDreams booth by Bruce Sterling.

“The Hotness” – YDreams rocking it at ARE2010 from brucesflickr

Rudy Rucker, who was hanging out with Bruce Sterling, captured the are2010 buzz and some great images in his post, Augmented Reality, Painting, Twitter. As Rudy put it:

“AR is hoping to be a next big thing, a cozier and more commerce-driven cousin of the old VR, or virtual reality.”

Bruce Sterling’s opening key note is up, ARE 2010 Keynote by Bruce Sterling: Bake a Big Pie!, and also the ARE 2010 Keynote by Will Wright: Brilliant Inspiration for the Augmented Reality Community with more videos from are2010 on the way.  One must read post on are2010 is Chris Cameron’s post, Augmented Reality’s Next Steps: Sitting Down with the Titans of AR.

Talking with Bruce Sterling, Part 1

The Auggie panel, Bruce Sterling, Jesse Schell, and Mark Billinghurst inspect the award.

Tish Shute: In your keynote at the 9am of the augmented reality industry you asked some questions of the are2010 audience: “What’s the mission statement?  You’re the world’s first pure play experience designers, except that user experience it’s mostly futuristic hot air.  But run with that, right?  What are your tactical steps?  You should get dressed, have a coffee, have a to-do list.”

How much of that did you see going on in the next two days?

Bruce Sterling: Well, I wasn’t privy to any of the business discussions.  I didn’t think it was an accident that this standard AR enabled tag thing came up from Bruno Uzzan, Total Immersion.  That seemed to me to be a useful thing.  I was always interested in the Augmented Reality Consortium.  It struck me as remarkable that there was this group of people who clearly all knew one another and it had some kind of game plan.  I applaud them for that, because these are not the 1980’s.  [laughs]  You know, it’s just a different world for young startup companies.

Tish Shute: I think you’re right. There seem to be some VC conversations going on, we don’t know what went on in the meetings, but it was noticeable in the atmosphere of excitement, and remarked on by a few people. So I think that kind of was definitely going on.

And, of course, I was so busy I never even got to see the expo properly! You said you wanted to be surprised.

Did anyone surprise you in any of the talks, in any of the expo?

AR used as interfaces for devices

Italian augmented robot from SEAC02 from brucesflickr

Bruce Sterling: I have to say I was a little bit surprised to see Andrea Carignano demoing a robot. I happen to know him because he’s here in Torino. He’s the guy that came out of Fiat and went into AR. I am not a particularly huge robot fan, but I think it’s of great interest that AR is used as interfaces for devices, as opposed to the Jesse Schell idea that AR is all about a “man with the X-ray eyes.”

My suspicion is that a lot of surprises will come out of mashups of AR.

Tish Shute: I didn’t get to see Andrea’s robot.  So what did it do?

Bruce Sterling:  It’s basically a sister device to that little helicopter that those Parrot AR Drone guys were doing.  It’s a little autonomous robot and it runs around with a webcam on it.  You can place video into the acquisition stream coming off the robot.  You can play a game, and blow away imaginary monsters or whatever.

Tish Shute: It’s interesting, because did you notice Will Wright and Patrick O’Shaughnessey, Patched Reality, spend some time hacking the Parrot AR drone in the hallway?  Did you come across them?

Bruce Sterling: Rudy was there with them.  You know, I didn’t want to watch Will Wright hack a robot.

[laughter]

Tish Shute: They seemed to be having fun even though as it turned out the power supply was dead.

Bruce Sterling:  I’m sure Will enjoyed that.  As a game designer, you want to go out and get your hands dirty with a plastic gizmo.

[laughter]

My Swiss Army knife can’t get through airport security, so I really don’t want to strip anything down.  But yeah, what else did I see that was of particular interest?  I was pretty happy about the Korean guys because they are a difficult group to get close to.

AR companies are like mini-global micro-startups.  They’re “glocal”.

“Korean elegance at the Zenitum booth” – from brucesflickr

Tish Shute: Zenitum.  What did you like from Zenitum.  They were one of our sponsors, along with Qualcomm.

Bruce Sterling:  I know that Seoul is like the number one center for augmented reality discussion.  But it’s  difficult to get behind the scenes as a journalist there and  track what’s going on in Korea.  I’m fine with Italian “realtà aumentata.”   And I feel like I’ve got a handle on French “réalité augmentée.”   The Germans were not hard to find, and the Dutch all speak English!  But the Koreans, and whoever the hell it is in Kuala Lumpur…  I have no idea what’s going in Kuala Lumpur, and only the vaguest idea of what’s transpiring in Singapore!  But I know that people there are paying a coherent interest.

So the Koreans show up, and they had some relatively predictable anime style 3D avatar conversion stuff.  But they had a really nice display space.

“Anime figures become three-d smartphone animated avatars,” from brucesflickr

Tish Shute: Ah, So Zenitum created a hot spot at the exhibition?

Bruce Sterling:  Yeah.  The Koreans had  IKEA furniture and some nifty little woven baskets.  They’d really classed up their presentation.  Most Koreans in tech tend to be kind of muscular.  The Koreans are not known for their refined presentations.  On the contrary, they tend to undersell everybody else.  But I don’t know, maybe they’ve been hanging out with Samsung and upgrading their design chops. [laughs]

Tish Shute:  Did you take some photos you could send me?

Bruce Sterling:  I took a few, but  I don’t consider myself a photographer.  They’re all up on my Flickr set. It was interesting to see so many people from so many different nations in such a collegial atmosphere.

Tish Shute: Yes – there were many different countries represented at are2010

Bruce Sterling:  It’s the beginning… and so global at such a young stage.

Tish Shute: Yes. As you said, it was 9 AM, so everyone was actually super excited to be gathered together from across the globe to start a new day together.  As you mentioned, there was a very warm affirmative vibe – everyone sharing a passion.

Bruce Sterling:   They have an online commonality. They seem to be aware of one another’s work through the Internet.

Clearly they had all heard about one another.  That’s a departure from earlier models of tech startup, where you usually have like three hippies in a local garage.  Now you’ve got German-American-Korean outfits like Metaio, and Total Immersion has a Russian affiliate.  They’re inherently multinational, both inside the company and out.

Tish Shute: It was the multinational garage, wasn’t it?

Bruce Sterling:  Yeah.  AR companies are like mini-global micro-startups.  They’re “glocal.” There’s something quite new to me about that.  I don’t find it’s shocking, because in Europe today it’s common to find startup teams who are multinational.  But to see such intense globalism at such an early stage of an industry is really different.

Tish Shute: Yes it made for a fun atmosphere?  It was wonderful running into Iguchi Takahito, Tonchidot.  You have a great rapport with each other despite the language barrier?

Bruce Sterling:  Yeah.  That guy from Tonchidot, he’s very charismatic.  He’s punchy.  That’s reflected in the very strong graphic design from his company.

Tish Shute: Using minimal English to make the case for Sekai No Camera at the Auggies, Iguchi Takahito still got through to the audience.

Bruce Sterling:  Well, his visuals were good.

What AR means for artistic practice…

Picture of Eric Gradman’s Cloud Mirror, from James Alliban post ARE2010 – Augmented Reality utopia in Silicon Valley – see for more on the are2010 ARt Gala

Tish Shute: So before I move on to wider themes, I’m going to wrap up on some of the different aspects of the conference.  I was chairing the technology track but you were more free roaming, was there anything that went on in the sort of hallway discussions and the presentation rooms that struck you?

Bruce Sterling:  Well, I did get collared by artists.   They really wanted to talk to me.  We got into some serious discussions on  what AR means for artistic practice.  How you can do this and reach that, how can one sharpen up one’s presentation?  I mean, they really wanted some art criticism.

Tish Shute: That’s very interesting.  Did you come up with anything that you hadn’t been thinking about already through the conversations?

Bruce Sterling: I’ve seen augmented reality installations before, and I certainly know many electronic artists.  But I don’t know.  People in the AR art space, they are looking for guidance and trying to find fellow spirits.  In their own way, they have the same pioneer spirit as the business people.

Helen Papagiannis shows Iguchi Takahito, Tonchidot, her AR Wonder Turner, an exquisite corpse inspired installation

Tish Shute: Yeah, it’s interesting, because we wanted the art gala to be even bigger, but it turns out, because of the logistics of putting up art in a conference space is fabulously expensive, because it has to be all installed and hung…

Bruce Sterling:  I’m keenly aware of that.  At Share Festival in Turin we bring in six installations, and it’s very heavy work.  It really takes a lot of logistics.  It was like a Battle of the Bands.  It’s like doing a rock concert.

Tish Shute: One of the installations I was really sad to not have there was Uber geeks’  “Steve” H.E.AI.D installation that Brady Forrest & Co. took to Burning Man.

So I was very happy that we actually did get the number of artists we did.

Bruce Sterling:  Well, there aren’t a million AR artists in the world, so it’s hard to judge.   I didn’t see many business people rushing up to have me critique their business plans.

Tish Shute: [laughs]  They were all in the meeting rooms.

Bruce Sterling:  Maybe it’s for the best.

VC and AR Startup Action

The Zenitum Booth, are2010, photo from Chris Cameron’s Flickr stream

Tish Shute: Do you know that why your talk started a few moments late is because we had 50 people who arrived from the Silicon Valley neighborhood I guess!

Bruce Sterling:  Did they not preregister?

Tish Shute: No. They all stood in the line for the same day registration!

Bruce Sterling: It ‘ll be interesting to see what transpires there, if there is a little wave of startup action.  God knows they need some place to put their money, because the VC scene in the US is pretty much moribund.

Tish Shute: Ogmento is the first US AR Games startup to get VC, I think.  I think there was some VC action at are2010 for sure.  And Qualcomm obviously seems to have commercialization plans for their AR technology, and to be scouting talent  and ways to deliver new AR experiences.

Jay Wright, Qualcomm presents Joe Dunn, e23 Games, winner of the are2010 StartUp Launch Pad with a check

Bruce Sterling:  Some  people don’t need venture capital.  I mean, Google Goggles isn’t going to be hurting for VC money, obviously [ see Chris Cameron's RWW post, Google Goggles Coming Soon to iPhone] .  AR may come up through other methods, like people allying themselves with Hollywood, or peeling off of advertising companies.   There’s a lot of outfits who might conceivably want in-house AR skills.  Then when people set up a specialty AR shop,  they  peel off the list of clients.  I don’t know.  Those old days  of Silicon Valley venture capital seem like a lost world.

Tish Shute: Yes.  I, again, didn’t see anything really of the business tracks and production tracks.  Did you get back and forth between the tracks?

Bruce Sterling:  I went to the Hollywood tracks.  I mean, to the extent that I could.

Is Hollywood stirring? Who’s going to have the first breakout AR property?

Tish Shute: So what did you see from…Is Hollywood stirring?  Is it waking up?  I mean I know Kent Demaine, Oooii,  and Brad Foxhoven, Ogmento, spoke about the Hollywood AR scene.

Bruce Sterling:  There were guys there from LA who were sort of saying, look…they are aware of us, but they just want AR to promote their properties to some particular niche.  They realize that AR is potentially a mass medium and that you could do some real AR entertainment.  So they were batting around some ideas as to where that might happen.  Like, could it come out of a console gaming scene?  Who’s going to have the first breakout AR property?  A popular hit AR property, as opposed to like a neat way to sell shoes, or whatever.   Really, anybody’s guess is as good as theirs or mine.  But at least they were actively guessing.

Tish Shute: I know the breaking the fourth wall discussion has been going on for a while and now the question is, whether AR is going to take down the fourth wall and bring interactive storytelling into the mainstream.  Did you hear any of that?

Bruce Sterling:  Well, I always shy away from discussions of that kind because I don’t think there’s any “final thing.”  Practically everything that AR is involved in right now is  a transitional technology. Also, because I am a storyteller, I get alarmed whenever people in technology start saying, “Oh well, it’s all about telling stories.”  Because obviously it isn’t.

People can tell stories perfectly well orally, and absolutely nobody does that.  AR is not at all about telling stories.  It’s about a great many other things, such as user bases, niche audiences,  media saturation, urban informatics,  convergence culture, and the language of digital media.   I could list these factors until the world looks level. It’s really becoming pretty chaotic.  As I was saying in my speech, AR companies are media startups who almost never use the old-fashioned word “media.”

Tish Shute: Oh, that’s interesting.  Yes.  So why do you think that has happened that way?

Bruce Sterling:  Well, it’s because they are trying to do a different thing than media does.  I mean, they are trying to “augment reality.”  They don’t want you to know that you are using a medium.  They don’t want you to realize that you’re watching computer animation overlaid on some video acquisition stream.  That would defeat the whole point of AR.  It’s entirely different from an analog medium like television, where you turn on the television and there’s a constant stream of station identification alerts.   That’s like: “Don’t touch that dial!  You’re on channel 13!  Stay with us!”  Then it’s like, “And now a few words from our friendly sponsors!”  That medium was engineered to keep your eyeballs locked to a single stream that they’re feeding you.

In AR, it’s much more participative, more geolocative.  I’m not particularly interested in station-identification branding from my AR provider. What I really want to see is the interactivity of the augments they’re bringing to me.  It’s like  FlickR, the photo sharing site. You don’t have any TV-style splash page for FlickR.  ”Hi! We’re FlickR! FlickR, bringing your photos to you!” No, FlickR is all about “you, you, you,” your photos, your tags, your friends, your activity around you.   It’s immediately trying to be very participative.

Tish Shute: Will Wright got to that point, didn’t he. He was trying to move us into an idea of blended reality. That the game is about the world, not about the dragons or the overlays per se.

Bruce Sterling: Right. I think that’s true. But see, the world isn’t a medium. A medium is something like this interview,  where I’m connecting to you and there’s a video Skype channel between us.  Whereas AR is more about spatial 3-D,  about 3-dimensional impositions.  Pieces of media: sound, vision, information visualization, tags, floating tags, air tags, icons, arrows, warning signs, warning sounds, tactility, whatever, being brought into the environment around us.

That’s why it’s properly called “augmented reality” instead of just augmented media.   If you call your work “augmented media,” you’re really in trouble. Because if it’s all about augmenting somebody else’s media, why doesn’t that medium just buy you, and augment their own selves?    If you think that way, instead of augmenting the world, you’ll just be a modest little plug-in for old-school media.

The World as the Platform

Blaise Aguera y Arcas, Microsoft, Santa Clara, are2010, photo from Chris Cameron’s Flickr stream

Tish Shute: Yes, which is why Blaise so generously gave the technical underpinning  for augmenting reality in his tech talk – about the trellis and the grapes,  he really explained how the world can become a platform for augmented reality.

Bruce Sterling: I wish I could have seen that. I did not see Blaise’s speech.

Tish Shute: We’re going to put the videos up in better quality.  People in the front row have put it up on the web already.  He really went into some of the challenges of mapping for augmented reality.

Bruce Sterling: His visual-mapping technique is important.  Registration is super important for AR.

Tish Shute: I think it was a really generous talk actually because he went step by step on how we will do this.

Bruce Sterling: I rather imagine that Microsoft has patented those steps.

Tish Shute: Oh, yes, I guess so!

Bruce Sterling: I could be wrong. Maybe they’ll open-source it. You never know.

Tish Shute: You never know. Because the world as a platform isn’t something one company can own, or go it on their own to exploit.

Bruce Sterling: I expect there to be a thorny path, but sometimes I’m surprised. Sometimes people really do try to fertilize the tech field in the hope of getting a good corn crop before they start fighting.

Tish Shute: We’ll I keep hearing that we may even see the unlikely marriage of Apple and Microsoft  – maybe wishful thinking, but there are motivations beyond AR for this unlikely match, and certainly between them these titans have what it takes to realize the grand visions of AR ? [laughs] But who knows…

Bruce Sterling: Well, yeah, it depends on where the thing catches fire.

Tish Shute: Yes. You mean whether AR catches fire in the form of  AR and mapping..

Bruce Sterling: It’s hard to say, but I’m convinced now that there’s more going on than I once thought. I thought that Bruno Uzzan made a very good speech for his company when he talked about how he worked on AR for eleven years.  Eleven years is no flash in the pan.   He has his long list of clients and successful applications. I thought he was right in his impatience with the press for not catching on. It’s gone on for quite awhile. The mere fact that you’re not aware of it, doesn’t mean it doesn’t exist.

The Illusive AR eyewear

My are2010co-chair, Ori Inbar, CEO and co-founder of the hottest new AR game development start-up, Ogmento, donning his goggles to open are2010 – picture from Chris Cameron’s Flickr stream

Tish Shute: Yes. So, the other theme you brought up in your opening keynote and I would be interested to know if anything you saw at are2010 changed your view is the illusive AR eyewear, and  if we actually got AR Goggles that worked they would bring AR’s gothic sister, VR, back from the grave right? [laughs]

Bruce Sterling: Right.

Tish Shute: It took quite a lot of work, but we pulled together a six-company HMD panel, right?

Bruce Sterling: Yeah. I was impressed to see so many of them there.  And I was chagrined to see how prototype-like all their gadgets were. But that doesn’t surprise me, because if any of those head-mounts were remotely working, they would be hyped out the wazoo. Everybody’s been waiting for them and hoping for the best. They’re obviously not ready for prime time. [laughs] Maybe in certain limited applications. Like maybe a diving mask. [laughs]

Tish Shute: No, I think what was nice though they got inspired and they all got together on the last day. I saw them having a meeting about standards. They got inspired to actually work together.

Bruce Sterling: Yeah, well, unless they’re going to invent mechanical eyeballs that those machines can fit onto, it’s going to be tough. OK, I’m a skeptic, but I’m prepared to be surprised. I’m also a skeptic in Artificial Intelligence, but as soon as they bring me an AI that can write a decent novel, I’m going to get it and review that book. [laughs]

Tish Shute: It’s interesting. Re AI, I’m totally in agreement with you. In terms of the way computers turned out, it wasn’t AI per se that they turned out to be good for, not in the way everyone had dreamed of it, rather it was the harvesting of human intelligence that turned out to be the big thing. But what is interesting is that despite all of that, AI or machine learning, as it is now called, permeates our whole society now from the stock market to how many businesses make many of their decisions.

Bruce Sterling: Well, there’s a lot of so-called collective intelligence.  But Marvin Minsky-style hard AI, no way. Alan Turing-style AI, forget about that.

Tish Shute: Yeah. So, that’s an interesting comparison with the HMDs.

Bruce Sterling: People stretch the definitions.  It’s like, well, my car engine is Artificial Intelligence. Yeah, so is your wall transistor. No, I don’t really think so.

And AR is a similarly big tent. I mean, Uzzan had to admit that he had denied that AR was AR, unless it was using his favorite technology. And he felt embarrassed to be rubbing shoulders with people who put AR into cell phones. And I can understand his feeling there, because, gee whiz, that’s certainly not what AR pioneers had in mind. But he had to admit he’d become more ecumenical about it. Obviously, they’re  there and doing business like gangbusters. You can’t very well ignore success, right?

I had a similar feeling about the goggles. Obviously, the goggles would be great, should they work. But if they did work, I rather think virtual reality would come very strongly to the fore.  You’d see people doing all kinds of elaborate immersive-style stuff.   A truly immersive technology doesn’t need to “augment” much of anything.

Tish Shute: Yeah, you’re right.

Social Augmented Experiences

Bruce Sterling: I think many of the most interesting AI aspects are not personal in the way goggles are.  They’re not about guys walking around with personal tech. They’re about big, communal, social-media experiences, like stage shows, and urban informatics, things where large numbers of people can interact with the same augmented reality. The projection mapping, which I go on and on about. Augmented public spectacles.

Tish Shute: Yeah, projection’s our best example of a social augmented experience right now because we are yet to have an easy way to do networked social augmented experiences easily – but that is of course the thrust of my interest in ARWave [see the slides for my presentation, AR Wave:  Federation, Game Dynamics, Semantic Search, Mobile Social Communications here].

Bruce Sterling: I think of Edison’s early days, when he wanted to sell movies to people for a nickel a clip.  You had to bend over and put your eyes on this visor and turn this crank. That coin-op device was easy for Edison to monetize, as opposed to getting a bunch of people to sit in theater seats. But people laugh at movies when they’re together in the seats.   Cinema is a more social, involving experience in a crowd situation.

Tish Shute: But it started with them, didn’t it, Hollywood – the movie biz? Basically Nickelodeons, right?

Bruce Sterling: That’s right. They were Nickelodeons. They were a lot like the goggles because they isolated the user.

Tish Shute: Yeah, that’s a really important point that the goggles are not Nirvana because of this question of whether they actually detract from the social augmented experience and blended realities, by drawing us into VR experiences?

Bruce Sterling: I’m tempted to claim that they’re more a VR technology than an AR technology.

Tish Shute: That’s a very interesting point because…

[thunder]

Tish Shute: Wow! What was that?

Bruce Sterling: Thunder storm.

Tish Shute: Oh, my God, how very Gothic! [laughs]

Bruce Sterling: It can get pretty loud up here in the mountains.

Tish Shute: Oh, you live in the mountains, better still!

Bruce Sterling:  Torino is in the foothills. This is Piemonte. So the Apennines are over there. The Alps are over here. We do get some rather spectacularly unstable weather.

Tish Shute: It sounded like a bomb to my NYC ears. [laughs]

Bruce Sterling: Yeah, it didn’t hit the building, but it was maybe half a kilometer away. I saw the flash.

Tish Shute: Oh, you did?   Well, I hope you don’t lose your power midstream here.    I was really happy to hear of that connection between Rudy Rucker and Layar  [Rudy was touched when Maarten Lens-FizgGerald from Layar said that he met the Layar co-founder at a Rudy Rucker lecture].

Bruce Sterling: That was very fun, yes.

Tish Shute: Wasn’t that wonderful? What was that experience like going around the conference with Rudy?

Bruce Sterling: Well, you know, Rudy’s very into graphics. He’s a mathematician, so he understands the underpinnings of this stuff. But he’s a skeptic. He thinks they’re kid toys. He’s not a gamer. He’s a good old-fashioned computer-science hacker. So he wanted to tell me all about his new eighth-order, fifth-dimensional fractals. He showed me a great many of them. They’re psychedelic. Rudy’s fractals are considerably trippier than most apps that help you find a barber or a train station. [laughs] Rudy really is a visionary. He’s into some very weird stuff.

Gamer Guys at are2010

Brad Foxhoven, Chief Marketing Officer, Co-Founder, Ogmento at are2010

Tish Shute: At are2010 there was a lot of discussion about how game dynamics and AR are going to intersect, right? Anything that you saw of interest there?

Bruce Sterling: Well, obviously, there are gamer guys there. Ori’s a gamer. The gamer guys are getting some money. The big buzz right now in gaming is, of course, social gaming.  Farmville has kicked everybody’s ass because it’s not even a game and yet it has more users than the entire gaming industry.

Tish Shute: I know, right! [laughs]

Bruce Sterling: Obviously that’s kind of humiliating. For a long time, I’ve seen people trying to do giant multiuser games on cell phones. It’s difficult to do because the interface on cell phones is crap, right? People aren’t going to run around responding to SMSs.

I can imagine people running around with little Wii-style bats that have audio and visuals on them. It makes a very large native AR game seem more plausible.

Tish Shute: Yes. that would be cool!

Bruce Sterling: Again, it’s not very gamelike to use those little fiduciary markers.

Tish Shute: No.

Bruce Sterling: Moving little cardboard chips, around like with card games…. It would be pretty easy to set up a little AR chess game.  Star Trek style hologram chess pieces,  and so forth. But it’s just cumbersome.

Tish Shute: And also, from what we’ve seen from things like Foursquare, the proximity based social gaming doesn’t have to offer very much [a crown badge, a mayorship] to get some mind share.. the social is the primary game dynamic…

Bruce Sterling:  I’ve seen a lot of different philosophies of gaming over the years. Who’s to say that Second Life doesn’t have the best idea? They built a little scene and then slammed their gate shut behind them.  But at least they’ve got a really nicely-paying little cult stuck in there. It’s different. And it’s manageable and it’s really theirs, theirs, theirs.  They don’t have to call in outside experts to try and run the monster.   They haven’t blown it up to the scale of Yahoo! where they’ve lost control of the enterprise, and gone into a tailspin of management overhead. Second Life has a very intense, almost a cultish atmosphere among the player-slash-developers.

Tish Shute: One thing that helped them was the thing they were always criticized, that the barrier of entry was so high. But once they got people they never left, right?

Bruce Sterling:  That’s not a bug, that’s a feature.

Tish Shute: One of the best features!

Bruce Sterling: Yeah, it’s like being in Mensa. Why don’t you lower your barriers to entry and get in some interesting stupid people?

Tish Shute: [laughs]

Bruce Sterling: In Mensa, we’d rather sit here making puns about neutrinos and fourth-order quadratic equations. [laughs] OK, that’s a business model, if that’s what you want.

The Man With the X-Ray Eyes!

Jesse Schell’s during his keynote, “Seeing,” at are2010, picture from Chris Cameron’s Flickr stream

Tish Shute: Ok!  Now to unpack the man with the x-ray eyes idea, Jesse Schell’s keynote theme.  This is a root metaphor for AR – making the invisible visible, seeing through walls. To me. I think you kind of wrote the book on this because all my ideas on what radical transparency might be come from you – your idea of Amazon.org is key to how I understand this..

Bruce Sterling: Oh, really? That’s funny.   I was touched that Jesse brought up that famous Corman film, because I was a judge in a fantasy film conference in Trieste earlier this year.  And Roger Corman was there.  He was the guest of honor.   ”X: the Man with the X-ray Eyes” was one of the films shown during the conference, and I saw it.  I even had dinner with Roger Corman.  I had never met him before, so that was quite amusing.  The difficulty with a film of that kind is that what we science fiction writers call a “House of Cards Ending.”  In that story structure,  you ramp the thing up until the protagonist sees God, and then he has to be destroyed by the falling pillars of the temple.  That’s a classic science fiction structure:  like Frankenstein.  For the sake of the drama, Corman evades the issue of what’s really going on. For instance, let’s just suppose “the Man with the X-ray eyes” is not in fact a psychopath.  Let’s say he gets a grant from the Department of Health and Human Services, and he acts like a real scientist, not a stock B-movie “mad scientist.” So he has, like, backup guys, and some placebos, and a large group of people to test it on, trusted colleagues, and so forth.  You wouldn’t get any of that movie’s wild activity out of that.  What you would get is like a 5% improvement to people’s vision.

Then, in a year, there would be a 10% improvement in people’s vision.  There would be a  classic industrial story.  A rising star, you know, a cash cow.   Real tech isn’t done by a single guy as a divine curse.  It’s created by classic  tech startup culture.  So a runaway technology really behaves in the way that personal computers do.

Tish Shute: The things that get me all Utopian and happy about this are the ideas like those you first outlined with the notion of Amazon.org.

Bruce Sterling:  It would be easy to do an entirely different kind of film than “Man with the X-ray Eyes.”  Something much less B movie,  much less pat.  I mean, at the end of the film,  he destroys his own hardware and blinds himself.  Why?  For what rational reason would he do that?  Why doesn’t anybody else know the big secret of what he’s doing?  Why aren’t there Koreans doing it?  Why aren’t there Austrians doing it?  Why aren’t there Italians doing it?  Why?  AR doesn’t behave like that.  It’s not one lone guy with magic eye drops.  It’s entire teams of people that have been working on stuff for 17 years.  They all approach it in different ways.

Now, they are going to get scandals in AR.  I can guarantee you that.  They are going to get into  hot water eventually.  At least some people will surely come out and accuse them of being Roger Corman B movie monsters.  But unless they accidentally discover atomic fission or destroy the Gulf of Mexico with an oil spill [laughs], I don’t think they’re going to be particularly badly off!   The trouble I imagine  for AR people is very typical new media trouble.  It’s like movies being accused of corrupting our morals, or comic books being accused of leading to violence, or Google being accused of making us stupid and warping our brains.

I’m not an alarmist in that sense, but at least I’m concerned about real threats.  Roger Corman is a B-movie director who’s trying to sew up his lost plot ends by destroying his hero and his hardware. That’s not very plausible. It’s a nice science fiction movie device, but technology isn’t a movie.

Tish Shute: Yes. Well, the other thing that you always remind us of with AR is not to be saying it’s going to be this glorious moment when it’s no longer gimmickey, no longer pop culture. You always emphasize that’s actually part of what’s good about it.

Bruce Sterling: It’s not an accident that practically everybody in that audience knew about Roger Corman.  Nobody looked surprised; not the Austrians, not the Koreans. They were all like: “Oh, yes! Roger Corman!  Love him!”

Tish Shute: There were so many Rudy Rucker fans. Were you watching Twitter? People like Eric Gradman were succumbing to fanboyz moments..

Bruce Sterling: “Yeah. Rudy Rucker, he’s the best.”

“Rudy Rucker gripping an Augmented Reality shoe” from brucesflickr

Tish Shute: [laughs]  I noticed you inspired him to join Twitter..

Bruce Sterling: Well, I’ve got 8,000 followers and, obviously, a lot of them are Rudy’s fans.  Of course he’s going to be gang-rushed on Twitter. That’s not really any more surprising than two motorcycle stunt guys at the same attraction. And I’m a big fan of his Rudy’s blog.   He’s always got interesting things to say.

Tish Shute: Yes. AR does seem to bring out some of the coolest smartest people!  This morning I had breakfast with Joshua Kauffman in Central Park.  He is an advisor and entrepreneur working on design in the public sphere.  I was feeling rather brain dead and jet lagged.  I told Joshua I was wondering how to get the cottonwool out of my brains for this interview and he suggested,  the All Souls College one-word question interview!  Have you ever heard of that? – although apparently they recently scrapped it.

Bruce Sterling: Well, I’ve heard of All Souls College there in Oxford. What was their interview question?

Tish Shute: They used to use only one word, so they would only give you one word. It’s not a question. Basically, they throw out the word and then you had to spin off from there.

Bruce Sterling: You’re supposed to free-associate on a single word?

Tish Shute: I guess so. I hadn’t heard about it, but Joshua suggested it.

Bruce Sterling:  Well, it’s possible..

Tish Shute: Joshua came up with some good words..

Bruce Sterling: Right.

Tish Shute: We were talking about these proximity-based social work networks like Foursquare and Gowalla and how they may influence the emergence of social augmented experiences.

So Joshua’s suggestion for the first word was “territorialization” e.g. how do these new mobile social experiences like Foursquare,  and the observation that actually rather than breaking down territorialization, which would be a good thing, tend to support territorialization…but perhaps new forms of territorialization?

Bruce Sterling: Yeah, they’re re-intensifying it in a very odd, electronic fashion.

Tish Shute: Yes.

Bruce Sterling: I have noticed that.  It’s not true of stuff like projection mapping or the webcam fiduciary display stuff. But with the handheld stuff, and especially the urban informatic stuff, it really can’t help but take on a local flavor. Layar is like “Augmented Dutch Reality.”

And TonchiDot really is “Augmented Japanese Reality.” It’s hard to imagine a Layar interface going gangbusters at Tokyo.  Whereas the TonchiDot interface, which is very clearly influenced by Anime and cartoon graphics…. Maybe it could find some niche of hipsters in Amsterdam hash bars…

…to be continued in Part 2

The Augmented Reality Event: Bruce Sterling’s keynote from Ori Inbar on Vimeo.

Bruce Sterling’s keynote (aka the prophet of the augmented reality industry) set the bar high on the opening day with a keynote address reminding us all of how awesome it is to be the “world’s first pure play experience designers,”  but also not to forget “whose reality needs to be augmented most,” and “to cut ourselves a space of our own.”

“It’s 9 am in the augmented reality industry…without vision people perish…it’s your chance to build a big pie before you start slicing it up…it’s time for you to get dressed…good luck to you, I’ll be watching you”

Wow! Augmented Reality Event was hotness.  I know, as one of the chairs, it is not like I am a neutral observer.  But, if you were there, and you didn’t pick up on the “warm, affirmative feeling,” as Bruce put it, and the optimism, fun, and cool uber geekery, of are2010, I guess I will have to hear from you in the comments ‘cos the feedback has been incredibly upbeat, so far.

Anyway,  I am still basking in that rare glow of a geekgasm at 9am – the augmented reality industry embracing lovers whose dreams are in still in full flight – no burst bubbles, betrayals, train wrecks, or tragedies in sight yet?  Although Roger Cormanesque horrors were threaded through  Jesse ‘the man with the x-ray specs” Schell’s  closing keynote, “Seeing.”  So keeping watching Ori’s Vimeo stream for Jesse’s keynote  -  it will be up soon and it’s quite a ride!

I leave for England tomorrow to celebrate my mother’s eightieth birthday, so this will be a brief post for now.  But I am looking forward to posting soon a long interview with Bruce Sterling on his experience of are2010.

Bruce Sterling hung out with with Rudy Rucker (see Rudy’s post and pics on are2010 here), attracting fans wherever they went, visiting the expo, going to sessions,  talking generously with all – their table always crowded.  Bruce said he hadn’t had so much fun in a while.  And, apparently, Rudy was touched when Maarten Lens-FizgGerald from Layar said that he met the Layar co-founder at a Rudy Rucker lecture. Awesome!

As Ori says here:

“are 2010 is over. It was a blast.  Many thanks to 400 AR enthusiasts who joined us for 2 days of AR goodness.

Special thanks to our 90 speakers from 40 augmented reality companies, our exhibitors, sponsors, and above all – for Qualcomm.”

People’s slides will be up on the are2010 web site soon. My presentation, AR Wave: Federation, Game Dynamics, and Mobile Social Communications, is already up on slideshare.

You can catch up on some of the highlights of are2010 in Rouli’s linkfest, and check out these pointers he gave to following the event for more.

• Chris Grayson’s twitter account
• Patched Reality’s Patrick O’Shaughnessey and the augmented citizen Dan Romescu are also doing their share on twitter.
• Sophia Parafina ( @spara ) live blogs the event using Google Wave.

I will have a mega post up soon.  But for now here are a few photos that will give you a taste of some of the #are2010 magic.

Bruce gets a t-shirt from tonchidot CEO, Iguchi Takahito ( @iguchi ), after checking out Helen Papagiannis’ AR art

If  an event with Bruce Sterling, Rudy Rucker, Jesse Schell and Blaise Aguera y Arcas there wasn’t already enough awesomeness, check out how Sophia Parafina, Locatively, and Patrick O’Shaughnessy of Patched reality got to hang out with Will Wright – hacking the Parrot AR Drone.  And then, when a dead power supply stopped that adventure in its tracks, Will played some of Patrick’s AR games and gave him feedback.  OMG!  Will’s only the most important game designer the whole world.  There is already some videos shot from the front row up of Jesse Schell’s, Blaise Aguera y Arcas’ and Will Wright’s keynotes up, but the hi res versions are coming soon!

First an incredibly big thank you to The Open Planning Project office (TOPP) – @TOPPLabs, and Sophia Parafina, @spara,  for organizing, hosting,  sponsoring and providing so much inspiration for this event.

There is an incomplete list of attendees below, and there were about 70 people at one point watching the Ustream (thank you Dimitri Darras and friend – sorry I missed getting your card!) for setting this up.

There were at least ten or more people participating in a live skype conference moderated by Sophia with great skill.

I am sorry I didn’t get everyone’s contact info.  But please feel free to add you name into the comments of this post if I have missed you out.

After a gearheady morning, we spent the afternoon and evening brain storming the “The Big ARNY” – “a collaborative game development project modeled after a Swarm of Angels.”

Some of the morning tech discussion highlights included:

*skype presentations on AR Wave from Thomas Wrobel, Joe Lamantia, Matthieu Pierce.

*the beginnings of an iphone client from the PyGoWave Crew.

*discussing Microvision, Augmented Reality eyewear – and trying out Nomad Unit courtesy of Noah Zerkin, @NoaZark

*an awesome deep dive into the code of the open Goblin XNA VR/AR platform – courtesy of Ohan Oda (pic below) video of presentation here.

Ori Inbar live blogged the morning sessions on Games Alfresco.

But, during the afternoon, Ori presented, and we all got so caught up in the brainstorming of  “The Big ARNY Game” that live blogging, skyping, and twittering ground to a near halt.   The “meat space” (perhaps the slide show captures some of  incredible coolness of the location) was alive with brilliant ideas that were matched by an incredibly high level of technical input – see the AR DevCamp attendees list below.

During the game session we really had a master class in augmented reality tech.   Steven Feiner’s awesome discussion of markers really opened my mind to exploring markers in a new way.  And the geolocated data discussion with Sophia Parafina, Kate Chapman, Philip Ashlock,  and Steve Feiner at dinner was very interesting.  The opportunity to break out into smaller in depth discussions during the day was one of the valuable opportunities of AR DevCamp, so I can’t possibly mention them all.  But thank you everybody!

We did have some fun with low tech AR too – courtesy of Thomas Wrobel & Bertine van Hövell, Lost Again (their business card is the coolest AR card I have seen to date).  In the pic on the right, I try out their business card/AR overlay on @comogard as he presents.  The lighting does not do the overlay justice in my photo (on right), but I think you get the idea at least.

Unfortunately we didn’t manage to hook up our afternoon live session with The Mountain View AR DevCamp, as we lost the streaming laptop.  But hopefully we will be able to catch up on each other’s activities with session notes on the AR DevCamp Wiki. There is also a public wave,  AR Dev Camp NYC Shared Notes.

Sean White set the afternoon off to a great start by collecting topics and organizing topics of interest on the board.  While we didn’t get time to cover everything, it was interesting how, by working on developing a collaborative game project, we had to tackle many of the topics suggested, and come up with workable approaches.

Next weekend TOPP will host the OpenNY Summit & Codeathon on Dec 11 & 12, an event “produced by open government practitioners and volunteers.”  This would be another great place to explore some of the citizen 2.0 mobile, social AR game ideas that came up at AR DevCampNYC. In addition, Ori Inbar has started an AR New York Meetup.

Below is an incomplete list of AR DevCampNYC attendees.

Sophia Parafina, OpenGeo, @spara, organizer

Marco Neumann, KONA, @neumarcx, interested in developing Semantic Web based Augmented Reality demos.

Tish Shute, Web,@tishshute, Open Distributed AR, Google Wave Protocol for AR, Imagining the Future of the Outernet

Dimitri Darras, @dimitridarras, Visual designer, web developer, and virtual worlds content creator. Interested in multimodal input and AR/Virtual Worlds integration.

Heidi Hysell, @heidihysell, Creative Technologist & Software Engineer; Interested in the application of AR for entertainment technology for print, web and video.

Joe Lamantia, Muku / ARWave, @mojoe Amsterdam, interested in creating open frameworks, social augmented experiences, emerging media – (attending via skype)

Kate Chapman, Web Developer, FortiusOne, @wonderchook

Matthieu Pierce, itinerant poet, @matthieupierce, Pittsburgh, PA. Interested in AR Use Cases and observation. Attending via Skype.

Ori Inbar, ogmento games alfresco Let’s get together to brainstorm on the “Big AR NY Game”: The first location-based, social, augmented reality game designed for New York by New Yorkers.

Noah Zerkin, [1] – AR software and hardware interfaces; Exploring the idea of an AROS.

Ohan Oda, webpage – Columbia University; NYC

Sean White, webpage – Augmented reality research at Columbia University and Smithsonian Institution.

Steve Henderson, Columbia University, webpage, @stevehenderson

Omer Gunes, [[2] webpage] – NLP, Speech Recognition, Mobile Software Development

Steve Feiner, Computer Graphics and User Interfaces Lab, Dept. of Computer Science, Columbia University, personal, lab – Augmented reality, mobile/wearable computing.

Jon Russek, NYC, website, @filmaddict – AR as applied to film/theater/art.

Daniel Leslie, Reflexions Data, LLC @dan_leslie, Principal at application consulting/development firm where we’re working on a mobile app for proximity-based real time social graph analysis.

Donald Schwartz, NYC, @Ishkahbibelvirtual worlds, social media, technology writer

David Oliver, Oliver+Coady, Inc. NYC, @davidmoliver mobile strategy, mobile product definition, mobile development.

Chris Grayson, NYC, Twitter: @chrisgrayson | Blog: GigantiCo | Contributor: H+ Magazine | Web developer and marketing consultant — Interests: Future of commercial mobile AR / Outernet (GeoSearch & OOH marketing convergence); Future AR Form Factors; AR/Virtual Worlds integration re: distance learning & collaboration.

Saul Devitt, NYC

Bert Picot, NYC via Skype probably around 10:30 am for a few hours. Very interested in learning the value chain for AR applications and the development of applications for Festivals and live entertainment.

MZ – startup to develop a platform to use semantic data to enable virtual worlds

Jon Russek – film production + law + internet. Interested in AR as artistic medium for creativity

Davide Byron – developed the game Spads and Fokkers and code

Philip Ashlock @philipashlock, The Open Planning Project

Michael Keating, The Open Planning Project

Yohan Baillot, @yohanBaillot future of commercial mobile AR, emerging AR standards

Images  from Mitsuo Iso’s Denno Coil (Click to enlarge), the game “Metroid Prime,” and Terminator.

Thomas Wrobel, Sophia Parafina, Joe Lamantia, Matthieu Pierce, and I will lead a  session tomorrow for AR DevCampNYC introducing the AR Wave Project.  Thomas, Joe and Matthieu will be participate via skype (10am to 11.30am EST), and Sophia Parafina and I will both be at AR DevCampNYC at the The Open Planning Project office (TOPP).  The PyGoWave crew will be introducing PyGoWave via LiveStream.

At 1.30pm EST to 2.30pm EST there will be a shared PyGoWave/AR Wave session with Mountain View (if bandwidth permits).

The skype conference will be at ardevcampnyc .  To participate in Wave,  please join the public Wave,  AR Wave: AR DevCamp Session.  There is also a AR Wave Wiki up now – see here.

Dimitri Darras (avatar Dimitri Illios) is working on streaming the AR DevCampNYC sessions into Second Life,  SLURL here.

Thomas has done a very nice introduction and FAQ below.  This should help people new to this project to get up to speed quickly.

There are already several Waves that show the history of this project including: AR Wave: Augmented Reality Framework Development,  AR Wave Use Cases, PyGoWave AR Tech Discussion,  AR Wave Augmented Reality Wave Development, AR Wave / Muku Organization and Admin.

Also I have several posts for people interested in more of the background, including: The Next Wave of AR: Mobile Social Interaction Right Here, Right Now!, AR Wave: Layers and Channels of Social Augmented Experiences, Total Immersion and the “Transfigured City:” Shared Augmented Realities, the “Web Squared Era,” and Google Wave.

Thomas uses the term Arn (augmented reality network) which is one of the candidate names for the project, Muku (crest of a Wave) is another suggestion.  Thomas’ intro and FAQ below can also be found here.

What is the AR Wave Project?

In simple terms its a protocol for storing geolocated data on Wave servers that’s currently being developed.

We believe this will help lay the foundations for an open, universally accessible, and decentralised system for shared augmented reality overlays which various clients can connect to and use.

This AR Network should spark a lot more rapid adoption of AR technologies, give existing browsers more functionality, and provide the network infrastructure, allowing many of the fictional depictions of AR to become a reality one day.

The AR Network.

When we speak of a future AR Network, we mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.

Where people can just point their phone, webcam, or pair of AR glasses anywhere where a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.

The Arn should be an inclusive and open platform where any number of devices can connect to, and anyone can make and host their own location-specific models or data.

It should allow people to communicate both publicly and privately, and not have their vision constantly cluttered with things they don’t want to see.

This is our vision, and we think a Wave protocol will help it become a reality.

Why Wave?

Wave allows the advantages of both real-time communication, as well as the advantages of persistent hosting of data. It is both like IRC, and like a Wiki. It allows anyone to create a Wave, and share it with anyone else. It allows Waves to be edited at the same time by many people, or used as a private reference for just one person.

These are all incredibly useful properties for any AR-experience, more so Wave is open. Anyone can make a server or client for Wave. Better yet, these servers will exchange data with each other, providing a seamless world for the user: a single login will let you browse the whole world of public waves, regardless of who’s providing or hosting the data. Wave is also quite scalable and secure: data is only exchanged when necessary, and will stay local to just one server if no one else needs to view it.

Wave allows bots to run on it and thus allowing blips in a waves to be automatically updated, created or destroyed based on any criteria the coders choose. Wave even allows the playback of all edits since the wave was created.

For all these reasons and a few more, Wave makes a great platform for AR.

How?

In basic terms, we will diverse a standard way to geolocate a bit of data and store it as a Blip within a wave.

This data could be a 3d mesh, a bit of text, or even a piece of audio.

Then various clients on various devices could logon, locate, interpret and display this data as they see fit.

Click on image above to enlarge.

A typical example of this might be holding up your phone and seeing messages written by your friends and family in the locations which they are relevant.

You could see an arrow hovering over the café your meeting a friend at, notes above their flat saying if they are in or out, or messages by shops telling you to pick up the particular brand of cereal they like.

This data would be personal to just yourself and whoever you invite to share that wave with.

Other forms of data could be public, like city-maps, online games, or historical landmarks being recreated. Custom views of the world with data for entertainment, commercial, environmental or informative purposes.

The possibilities with geolocated data are endless, as are the various ways to display and make use of them.

One of the things I’m most passionate about is people being able to see many different types of data, both public and private at the same time and from many different sources at once.

For instance, if your playing a AR game, why shouldn’t your chat window be viewable at the same time?

If you have skinned your environment with a custom view of the world, why shouldn’t you also see mapping or restaurant recommendations?

The ways to present these layers of data and toggle them on/off in the most intuitive and flexible ways would be a task for the client markers, and I’m sure we will see many innovations in those areas.

But by using Wave it at least provides the framework for having multiple information sources controlled by many different people yet accessible, and user-submittable, via the same protocol.

Who?

This idea first sprouted from a paper I route focusing on the potential for IRC to be used for AR;

http://www.lostagain.nl/testSite/projects/Arn/AR_paper.pdf

I suggested near the end Wave might be a better alternative (using Google Wave was an idea Tish Shute, Ugotrade, brought up in response to the Arn prototype design on IRC), and it quickly became apparent that Wave was a very suitable medium.

Since then, there was a lot of interest, and numerous people have offered to help.

In particular, recently, the PygoWave team is helping us out, as they have an existing server supporting c/s protocol, which is currently being actively developed.

Where?

You can join the general discussion here;
Augmented Reality Wave Development

Technical side here;
Augmented Reality Wave Framework Development

When?

There’s lots still to do, and we are at an early stage.

Our current targets: (last updated 11/12/2009)

• Getting reading/writing of prototype ARBlips to the PygoWave sever. (the PygoWave team have already made a standalone client and have the protocol for this sorted!)
• Establishing a minimal spec for ARBlips to be later expanded.
• Writing a very simple prototype online client showing how to store/retrieve the data.
• Expanding client to work for some use-cases.
• Establish a logo/branding for the project.

Other FAQs.

Where’s the catch?

While we believe Wave is highly suitable for development, it has the drawbacks of being a new system with just a few servers worldwide, which (at the time of writing this), have not yet been federated together yet.

Naturally, as a new technology, its likely to have some growing pains. And building a new technology on other new technology will multiply that somewhat. The first pain is the lack of a standard client / sever protocol. PygoWave have stepped in to the rescue a bit here, by being not just one of the most developed Wave server other then Google, but also leaping ahead with support for Json based c/s interaction. Google has stated they want community to take the lead on on a c/s protocol, so we are hoping they will adopt a Json variant, or a XMPP one and add it to the spec. We hope in much the same way as POP3/IMAP have been a standard for email server interaction, a similar one will develop for Wave.

In the meantime we plan to keep the code for writing ARBlips somewhat abstracted so as to make it easy to adapt in future.

As for the newness of Wave and other potential problems it will bring, we aren’t that worried as its built on XMPP, which has proved reliable already.

The other catch is we are unfunded, which slows development down considerable as we have to fit it around our other jobs.

I’m making my own AR Browser, and am slightly interested in maybe supporting you.

We are naturally very keen for support, and particularly for those with skills and visions to give feedback on the proposed protocol. Specifically: what do you want stored in a blip?

That’s what’s important at this stage.

We don’t see the Arn as a replacement for existing browser systems at the moment. We don’t want to restrict innovation or development in this fast developing market as we are very impressed at what’s been achieved so far. In many ways our task is small in comparison to what’s already accomplished.

However, we do believe the Arn will make a good addition to existing browser systems. It will allow users contribute data and have social features without having to worry about accounts or hosting.

It will still be quite some work to support; new GUIs will need to be developed to make it easy to submit data from the devices, as well as to login to waves.

However, we hope over time to build a set of example libs to make the read/writing of ARBlips as as easy as possible to implement in your software.

Perhaps a good way to think about it is existing AR Browsers are like word-processors, supporting the Arn will be like adding support for *.txt, but doesn’t limit what you can do with your own format.

Eventually we do hope ARBlips hosted on Wave will become the majority of AR data, and its functionality will be analogous to the internet is today. We truly believe in the long run a standard is essential.

But for now we think merely getting a baseline format established for how AR data can be communicated will increase user-ability, usefulness, and help the market grow.

Can I help?

Sure.

We particularly need people with technical skills in relevant fields. (both gwt/javascript web programming and c++(/qt)standalone programming help very welcome!).

But we also welcome people just with vision to help focus use-cases and to conceptualise what we want to be able to do with the system.

Please either join the relevant AR Waves or Wiki

We are especially interested in those with JSON and Comet experience. Specifically those with the abilities to make standalone applications to read/write to a sever using these methods.

What type of data will a AR Blip store?

This is still actively being decided, but essentially its a physical hyperlink.

A connection between a physical location (or object, see below) and a piece of data.

Specifically, we are thinking about the following fields;

Location in X,Y,Z,
Coordinate System used for the above,
Orientation,
MIMEType [the type of data stored]
DataItself [either a http link for 3d meshs and other larger data, or an inline text string if its just a comment]
DataUpdateTimestamp [so clients know if its necessary redownload]
Editors [the user/s that edited/created this blip]
ReferanceLink [data needed to tie the object at a non-fixed location, such as an image to align it to an object in realtime],
Metatags [to describe the data]

Are you purely tying stuff to fixed geolocations?

Certainly not
As part of of the spec we wish to be able for people to be able to link data to dynamically moving objects, trackable by image or other methods.

The idea being that one day someone could link a piece of text or 3d mesh to an image on a t-shirt they are wearing, or perhaps link a dynamically updating twitter feed, or perhaps provide information on a product (based on its logo).

There’s a large number of possibility’s for image-based linking alone, and that’s not even considering possibilities like linking RFIDs, or other forms of less precise but invisible binding data.

We need a lot of feedback from those companies already doing markless tracking. What types of images do you need, idly to link a mesh to an object? is one enough?

Summary of AR Wave Work to Date

Purpose: To provide an open, distributed, and universally accessible platform for augmented reality. To allow the creation of augmented reality content to be as simple as making an html page, or contributing to a wiki.

Specific Goal: To establish a method for geolocating digital data in physical space (or linking it to physical objects) using wave as a platform.

(For justification as to why we are using Wave see: our faq )

Wave as a platform

We are developing on the PyGoWave server at the moment but the goal is to be compatible with all Wave servers

PyGoWave has already achieved an important aspect in enabling the project in being a waveserver with a working and well documented server protocol. This allows both standalone and webbased clients to interface with it already.  See - The PyGoWave Qt-Based Desktop Client

This is one of the reasons why we have chosen to develop for the Pygo server at this stage.

However, the overall goal of AR Wave is to have a framework compatible with all servers using the Wave Federation Protocol. As more wave servers get c/s protocols then ARblips (the data needed to geolocate objects) could be posted and retrieved from various servers using the same client software. For this a standard should emerge. Just as websites don’t have to be hosted on specific servers, neither should AR data need to be hosted on specific wave servers.

In order to reach our goal, there are a few very achievable steps involved – see below.

Feedback

We are still actively seeking feedback, so feel free to join the Wave discussions, and see the history of how the specifications of the protocol evolved. You can also read the justification for some of the choices already made. Note a new discussion for AR DevCamp will be begin at AR Wave: AR DevCamp Session

This will, of course, only be the first draft of the specification, and it is sure to develop much in future.
The important thing now is to make working prototypes while maintaining flexibility.

So what do we need to do?

Steps :

* Establish the overall method – Done.

Each Wave will be a layer on reality which an individual or a group can create.  Each Blip in this Wave refers to either a small piece of inline data (like text) or a remote piece of larger data (like a 3D mesh) as well as the data needed to pin-point it in either relative or absolute real space.
We call these blips: ARblips. They are simply blips that stored the data necessary to augment a single object onto a specific bit reality.

It is up to the clients how they interpret and display the data. They could interpret it as a simple 2d list of nearby objects, or as an advanced 3D overlay, whereby multiple waves from different sources could to be viewed at once. What’s important is that there is a standard way to link the digital data to the real world space.

* Establishing the specification for the ARblip – In progress
We have a good idea of what’s needed to be stored in an ARblip, and we have hammered out a rough format.
The data might be stored as blip-annotations, but this has yet to be finalised.
A rough outline of the type of data stored can be seen in this c++/qt header for ARblip data can be seen at the end of this document.

* Storing and retrieving these pieces of ARblip data on the PyGo server – In progress.
The Pygowave team has made some excellent libraries that should make reading and writing data on the PyGoWave server very trivial for those with c++ skills.
This, however, is a real critical step, so more developers with C++ skills are very welcome!

* Making the above client mobile, and using a devices gps device to place the data. – Not started.
The next step would be to port the code to a mobile phone and use it’s gps-input  to post geolocated data and view what others have posted. This would be a fairly simple and not to useful app in itself. However, it would mark the first time anyone could post AR data and anyone could view it, all using open-source infrastructure.
As a bonus, because we are using wave infrastructure, the updates to any ARblip should appear in near realtime.

* To continue with the proof of concept, we would like to have simultaneous wave input from a PC
and mobile phone at the same time. – Not started.
For example, someone could post a pin on Google maps API and have that data posted to a ARBlip in a wave. Someone logged into that wave on their mobile device would then see the data posted appear.
More so we hope that when the Google map pin is dragged about, the mobile phone viewer, with just a few seconds lag, will see its location updated in real time.

We hope to make a modest yet practical app at this stage.

* After all this, we can go onto the interesting things:
3D data, camera-overlays, data fixed to objects and many more.  There’s plenty of existing software using these features (such as Wikitude, Layer) and some that are even open source software (like Gamaray and Flashkit). The open source code can give us a leg-up. However, we prefer to establish the protocol first. So naturally, these fancy features aren’t a priority for us. Rather we think our energy is better spent establishing the protocols and infrastructure so that other people can build more advanced bit of software easier.

However, once our primary goals are established, we will look to make a open source augmented reality browser ourself which will surely feature many of these features.

Overall, we hope once we have a simple proof of concept, there will be many groups, both existing and new, wanting to use this Wave system for their own apps, games and data.

Conclusion:
Really it’s now all about growing the community. We hope as soon as we show how great Wave can be for augmented reality, that lots of individuals and teams will start making their own clients to read/write geolocated data.
Overall we don’t think anything we make will be that impressive in itself. That’s not our goal.
We instead hope that our project will enable AR-content to be made as easily as web content. That games, information and apps will be able to be created without the creators having to worry
about the infrastructure behind it.

Technical information -

Current ARBlip header file

(below is a c++/qt header file for an ARBlip object that should illustrate the data being stored)

{

public:

arblip();

~arblip();

arblip(QString,QString,double,double,double,int,int,int,QString);

QString getDataAsString();

QString getEditors();

QString getRefID();

QString getXAsString();

QString getYAsString();

QString getZAsString(); bool isFaceingSprite(); 

private:

//ID reference. This would be a unique identifier for the blip. Presumably the same as Wave uses itself.

QString ReferanceID;

//Last editor(s)

QString Editors;

int PermissionFlags = 68356;  // default 664 octal = rw-rw-r–

//Location

double Xpos;   // left/right

double Ypos;   // up/down

double Zpos;  // front/back

//Orientation

// names, ranges and directions are taken from aeronautics.

// If no orientation is specified, it’s assumed to be a facing sprite.

// Roll: rotation around the front to back (z) axis. (Lean left or right.)

// range +/- 180 degrees with + values moving the objects right side down.

int Roll;

// Pitch: rotation around the left to right (x) axis. (tilt up or down)

// Range +/- 90 degrees with + values moving the objects front up. (looking up)

int Pitch;

// Yaw: rotation around the vertical (y) axis. (turn left or right.)

// range +/- 180 degrees with + values moving the objects face to its right.

int Yaw;

bool FacingSprite; //if no rotation specified, this should default to true

//if set to true when a rotation is set, then it keeps that rotation relative to the viewer

//not relative to the earth.

//Data format

QString DataMIME;

QString CordinateSystemUsed; //The co-ordinate system used. This should be a string representing a Open Geospatial Consortium standard. This could be earth-relative for gps co-ordinates, or in some cases relative to the viewer, for data to be displayed in a HUD like style.

//Data itself

QString Data;

QString DataUpdatedTimestamp; //Time the Data was updated changed

//Note; A seperate timestamp should be used for updates that dont effect the data itself.
//(such as if a 3d object moves, but its mesh isnt changed)

//Data metadata QMap<QString, QString> Metadata;

};

I first started talking to Robert Rice, CEO of Neogence Enterprises, Chairman of the AR Consortium, in 2008.  Robert was already actively working on creating the world’s first global augmented reality network.  But it took a few months before what Robert had said to me about impending explosion of  augmented reality into our lives really sunk in – “this is going to be much bigger than the Web!,” he extolled.

By January, 2009 I was convinced and I posted my first interview with Robert, “Is it OMG Finally for Augmented Reality?..” As I mentioned in the intro, I had recently tried out Wikitude and Nathan Freitas’s grafitti app on the streets of New York City and I was impressed.  Now, 7 months later, Augmented Reality has  not disappointed and there is an explosion of new applications, and the arrival of some of first commercial and practical toolsets, SDKs, and APIs for aspiring developers.

For more on this see my previous post, Augmented Reality’s Growth is Exponential: Ogmento – “Reality Reinvented,” talking with Ori Inbar, which is an introduction to my series of interviews with the key players in augmented reality and founding members of the ARConsortium – Int13, Metaio, Mobilizy, Neogence Enterprises, Ogmento, SPRXmobile, Tonchidot, and Total Immersion.

As I mentioned before, Maarten Lens-FitzGerald of SPRXmobile told me the other day that my first Interview with Robert Rice, in January of this year, was a key inspiration for SPRXmobile to get started on the development of Layar – a Mobile Augmented Reality Browser. Much more on Layar and Wikitude – world browser in my upcoming interviews with Maarten Lens-FitzGerald and Mark A. M. Kramer, respectively.

Recently, both Layar and Wikitude earned a mention in the white paper by Tim O’Reilly and John Battelle, Web Squared: Web 2.0 Five Years On. Web Squared is essential reading not only because it covers the underlying technological shifts of “Web Meets World,” which augmented reality is a vital part of;  but, crucially, Web Squared focuses on how there is a new opportunity for us all:

“The new direction for the Web, its collision course with the physical world, opens enormous new possibilities for business, and enormous new possibilities to make a difference on the world’s most pressing problems.”

I am currently working on a post on Green Tech AR which is one of the areas augmented reality can play an important role “in solving the world’s most pressing problems.” Augmented Reality has a lot to offer Green Tech development.  As Gavin Starks of AMEE said at Euro Foo in 2006, “climate change would be much easier to solve if you could see CO2.”

But really useful Green Tech AR requires still hard to do markerless object recognition (going beyond feature tracking and modified marker recognition), and a tight alignment of media/graphics with physical objects, in addition to a quite a high level of instrumentation of the physical world.  And for Green Tech AR to really shine, we are going to need innovators like Robert Rice who are working on, and solving, multiple really hard problems like:

“privacy, media persistence, spam, creating UI conventions, security, tagging and annotation standards, contextual search, intelligent agents, seamless integration and access of external sensors or data sources, telecom fragmentation, privilege and trust systems, and a variety of others.”

Recently Robert Rice presented at MoMo Amsterdam. Here is a drawing of him in action (picture below from wilgengebroed‘s Flickr Stream).

In his Twitter feed Robert Rice ( @RobertRice ) Robert reminds us: “By the way folks, what you see out there now as “augmented reality” is not what it is going to be in two years.”   Robert plans to show the first public demo of his “platform for platforms” at  ISMAR 2009.

Robert is writing up a series of White Papers currently.  I got a preview of the first, “The Future of Mobile – Ubiquitous Computing and Augmented Reality.”  Robert points out, “AR through the lens of the mobile industry and ubiquitous computing is almost overwhelming compared to AR as marker based marketing campaign.”

I asked Robert, “What are the key take-aways for investors interested in the augmented reality field at the moment:

“First, Mobile AR is going to be bigger than the web. Second, it is going to affect nearly every industry and aspect of life. Third, the emerging sector needs aggressive investment with long term returns. Get rich quick start ups in this space will blow through money and ultimately fail. We need smart VCs to jump in now and do it right. Fourth, AR has the potential to create a few hundred thousand jobs and entirely new professions. You want to kick start the economy or relive the golden days of 1990s innovation? Mobile AR is it.

Don’t be misguided by the gimmicky marketing applications now. Look ahead, and pay attention to what the visionaries are talking about right now. Find the right idea, help build the team, fund them, and then sit back and watch the world change. Also, AR has long term implications for smart cities, green tech, education, entertainment, and global industry. This is serious business, but it has to be done right. I’m more than happy to talk to any venture capitalist, angel investor, or company executive that wants to get a handle on what is out there, what is coming, and what the potential is. Understanding these is the first step to leveraging them for a competitive edge and building a new industry. Lastly, AR is not the same as last decade’s VR.”

Talking with Robert Rice

Picture of Robert Rice at MoMo from Guido van Nispen‘s Flickr Stream

Tish Shute: So perhaps we better start with an update on state of play with Neogence?

Robert Rice: Neogence is doing well actually. We don’t talk much about the fact that we are still a small startup and we face a lot of the usual obstacles related to that and being a small team. Fundraising has been extra difficult, mostly because people are just now beginning to see the potential in AR, but that is still colored by perceptions based on a lot of the gimmicky AR ad campaigns out there. Still, it is better than it was two years ago the idea of an AR startup was a bit of a joke to a lot of VCs we talked to. However, we do have an agreement from a new venture fund in Europe (which we can’t talk about yet) for our first round of funding, but we don’t expect to close that for several months.

If all goes well, we hope to debut our first public demo at ISMAR 2009 in Orlando to select individuals and a few press folks. We might release a few viral videos before then that are conceptual and about what we are building in the long run, but that depends on how things go over the next several weeks.

We are also very active in looking for and building strategic partnerships and relationships with other companies, and this is not restricted to the augmented reality or mobile sector. As I have said before, we are looking at this as a long term business venture and the industry as something that will be bigger than the web itself within ten years. We are doing typical contract work and custom AR solutions to keep the cash flow going and build up the corporate resume a bit. So, if you want something done, and better than the stuff you are seeing now with all of the generic “look at our brand in AR with markers and a webcam” you should definitely give us a call.

Tish Shute: Just to clarify because most of the recent press has been about browser type AR like Wikitude and Layar which are not in the purist sense AR ‘cos they do not have graphics tightly linked to physical world. Neogence, if I am correct, is focused on building a true AR platform in the sense I just described?

Robert Rice: Hrm, I have argued with a few others about the actual definition of AR. Some people prefer a narrow and limiting view (3D overlaid on video), but I think in terms of the market and the end-user, it is better to have a wider definition. In that sense, AR is purely the blend of real and virtual, with or without full 3D overlaid on video. If we go with that, then Wikitude, Layar, Sekai, NRU, and others all fit into the AR definition.

Anyway, you are correct. We are building a true platform for AR, and this is quite different from what others are marketing as AR browser “platforms.”

There are a few problems with the “AR Browsers” approach that no one seems to be noticing. One is that they are all trying to get people to build new applications for their browsers, when they should be trying to get people to create content that they can share and browse.

Second, someone using Layar is not going to see anything that is designed for Sekai or Wikitude.

Third the experiences are generally for one user. While I love all of these guys and think each of the teams has some real talent on it, the model is flawed until someone using Wikitude can see the same thing that someone using Layar or Sekai camera is seeing (provided they are in the same physical location).

While we are working on our own client side technologies that we hope will be useful and integrated with every mobile device and AR browser out there, our core focus is on connecting everything and everyone together, and facilitating the growth of the industry with the tools to create content, applications, and so forth. We want to solve the really difficult technical problems (some of which most people haven’t even considered yet, because of the perspective they are looking at the potential of AR with), and make it easy for everyone else to do the cool stuff. We want to be the facilitators.

If you really want an idea of where we are going or some of what has inspired us, you have GOT to read Dream Park, Rainbows End, and The Diamond Age. If you have heard me speak anywhere or read my blog, you know that I am continually suggesting these and others.

Anyway, short answer, yes, we are building a true platform for ubiquitous mobile augmented reality, and we are absolutely the first to be doing so. I hope to demo some of this in October at ISMAR, with a full commercial launch next year (10/10/10 at 1010am Hehe, seriously). We will probably launch a website soon for people to start signing up and building a community now (especially if you want in on the beta testing of the whole kibosh).

Tish: So just to clarify,  how will Neogence’s approach differ and fit into the  growing world of Augmented Reality tools that we have now, e.g.,  ARTookit, Imagination, Unifeye?

Robert: I guess you could say that we are trying to build the infrastructure for the global augmented reality network. This could be viewed as a service, or even a platform for platforms. If Neogence does its job right, anything you create using ARtoolkit, Unifeye, or Imagination would be applications you could ultimately link to, integrate with, or deploy on or through, what we are building, and not be tied to a specific set of hardware, browser, or walled garden.

Tish: You mention Neogence is going to provide a platform for platforms. Without knowing the details that sounds like a lot of centralization which prompts the inevitable question: “Who owns the data?” Do you think other AR applications or providers would resist a “Platform for Platforms?” I know the potential centralization power of Google Wave has already got people talking about these issues (one of the comments in my recent blog post was about how Google Wave protocol may be interesting for a least some parts of augmented reality communication).

Robert: It really depends on perception and how we end up building it. We aren’t talking about creating a closed system. As far as who owns the data, it depends on what data we are talking about. For the most part, I think that if the end-user creates something, they should own it and have control over it. They should also be able to do what they want with it, independent of everything else.

This is one thing that proponents of the smart cloud and the thin/dumb client don’t like to talk about. It sounds great on paper, but when you start thinking about it, all that does is strip away power from the end user. Case in point…Amazon recently wiped every copy of George Orwell’s 1984 from all Kindle devices. They claimed they didn’t have rights to distribute/publish it and it was available on accident. The scary thing though, is that they literally went into every kindle out there, found copies, and deleted them.

How would you like it if Microsoft suddenly decided to delete every copy of Microsoft Office? Or every file that had a .doc extension? That is a huge violation…we feel like we own what is on our computers. But with the whole cloud thing, your data is at the mercy of whoever is running the cloud servers. No privacy, no ownership, no control. And if the system breaks, all you will have is a pretty dumb device that can’t do much on its own. Now, that isn’t to say that the technical merits and benefits of a cloud model aren’t worth pursuing, they are.

But I think there needs to be some hybrid model. Don’t dumb down my computer or my smart phone, let’s keep pushing how much these devices can do. We should take full advantage of centralized and distributed systems, but in a hybrid mashup sense. That is what we are pursuing with our AR platform, while trying to protect ownership and intellectual property rights of the end user.

Tish: Earlier today I was telling you how impressed I was by Google Wave – it is quite mind blowing to experience massively multiplayer real time interaction on what will be an open internet wide platform – Wave is breaking new ground here and more than one person has mentioned its potential role in AR to me (see the comments to my recent post on Ogmento).

I know you are a strong advocate of this kind of real time shared experience being part of AR.  But we are only just beginning to see it emerge via Wave on the existing web – what will it take to have this kind of real time shared experience in AR!  We got briefly into the thick client, thin client, cloud versus P2P discussions – what is your approach to delivering a massively shared real time experience that is like Wave not confined to a walled garden?

Robert: I’m not a fan of any of those models as being stand alone or mutually exclusive. Again, the hybrid model with the best of both worlds is key. In the early stages of the emerging industry, you are likely to see some walled gardens (or perhaps a walled garden of walled gardens…).

No one knows how things are going to turn out in the next five to ten years and few people are thinking about it actively. For us though, I favor Alan Kay’s quote (pardon the paraphrasing): “To accurately predict the future, invent it”. That’s what we are doing. In the short term, there will be plenty of experimentation in the industry and a lot of model testing.

Tish: Do you think though Wave protocols might be useful as at least part of the picture for AR standards?  As you point out open standards and open protocols are going to be vital for shared experiences of AR.  Is it important to build off existing protocols to get the ball rolling and what do you see as being the important early protocols for AR?

Robert: I think for now, we will use a lot of existing protocols for communications and whatnot, as well as the usual standards for things like 3D models, animation, and so forth. This is only natural. However, as the industry and technology evolves, we will need entirely new ones. As far as I know there is no existing market standard for anything like the Holographic Doctor from Star Trek Voyager, and that type of thing is definitely in the pipeline for the future (sooner than you would think).

Tish: All the excitement at the arrival of the browser like mobile reality developments has been really great – I feel people are getting a taste for what it means to compute with anyone/anything, anywhere and and anytime.

Wikitude started the ball rolling. And with Wikitude.me it is the first to support user generated content. Now there is Layar, Sekai Camera also. But as you mentioned to me in an earlier chat, with Layar and Wikitude opening up “their are probably half dozen other apps coming out in short order with similar functionality (even the AR twitter thing has some similarities).”

What has been most exciting to you about these developments up to this point? What will these apps/platforms need to do to stand out in a crowd.  Up to now, these browser like AR experiences do nothing with close by objects. Do you see “world browsers” with near object recognition coming out in the near future. Could Wikitude do this with an integration of SRengine or Imagination?

Robert: Yes, Wikitude or Layar could do this (integrate with something else for “near” AR) and it would be a step in the right direction. Tagging things in the real world is the basic functionality that will grow from text tags to photos, videos, 3D objects, and all sorts of other types of data and meta data. This gets really fun when that data is generated by the object itself. First is just giving people the ability to tag something and share that tag with their friends, everything else grows from that. This sort of functionality is probably the most exciting in terms of near future advancement.

However, I think the idea of a stand-alone browser platform is a bit awkward…unless you also consider firefox a website browser platform. After all, you can create widgets (applications) for it. Anyway, the point is having access to the same data…if you put three people in a room, one for each browser, they should see and experience the same content, although the interface might be different (based on what browser and of course which hardware they are using). This means there needs to be some communication between whatever servers they are storing their data on (meaning, user tags) and some standard for how those tags are created.

Of course, if all they are doing is grabbing the GPS coordinates of the nearest subway station and telling you how far it is and in what direction, then they should all be able to see the same thing, regardless of the platform. But then, that isn’t really interesting is it? I could get the same info on a laptop with google maps.

This is part of the problem right now though…no one seems to be thinking about the bigger picture much. All of the effort is either on making the next cool ad campaign for a car or a movie, or creating a tool to tell you where the nearest thingamajig is, but in a really cool fashion on a mobile device.

No one is talking much about filtering data, privilege systems, standards, third party tools, interoperability, and so on. There is also little conversation about where hardware is going. Right now everyone is developing software based on what hardware is available. This needs to change where hardware is being developed to take advantage of new software coming out (this happened in the PC industry a while back and growth accelerated dramatically).

These are some of the reasons why I led the effort to start the AR Consortium. We brought CEOs from 8 different AR companies and startups together to start talking about these issues. We are still getting organized and have plans to expand the membership to other companies, but we want to do this right and we aren’t rushing things. The important thing is that we have started and there is at least a line of communication open now, where there wasn’t before.

I would expect to see the early movers expanding what they offer very soon, and they will probably lead the way in the short term. Definitely keep an eye on the companies involved in the AR Consortium. There are lots of very smart and motivated people there, and they are far ahead of all the experimental dabbling in AR we are beginning to see on youtube, twitter, and elsewhere.

Tish: When we had a discussion about what were the basics for an AR platform and an AR browser earlier, you talked about the difference between tools, a platform, and a AR browser – like Wikitude and Layar which should be about features/functionality e.g. to create treasure hunts AR geocaching, invisible AR yellow sticky notes you can leave at restaurants you don’t like, etc. Also you noted it should let you explore (browse) multiple formats, and open content content for AR – any data, information, or media that is linked to something in the real world and the visualization/interaction with the same.

Wikitude is a stepping stone to a true browser by your definition. But are we also seeing what you would define as an AR platform emerging – Unifeye, Wikitude (you can recap your definition if you like too)?

I think Wikitude hopes to provide the lego blocks for  augmented reality readers, browsers, applications, tools, and  platforms?

Robert: I expect some segmentation among the various AR companies that are out now, as they find their individual strengths and focus on them. Some will emphasize the client software (the browser), others will develop robust tools for creating content, SDKs/APIs will advance and facilitate rapid development of applications, etc. Neogence is ultimately working on the glue in the middle that ties everything together, makes it massively multiuser, persistent, and ubiquitous. Things like Unity3D have the potential to fill a need in the middleware space.

Tish: I know Blair McIntyre (see my interview with Blair here) and others are using Unity3D as an AR client, Could Unity3D become increasingly important?

Robert: It has the potential to become a favored middleware for providing the rendering layer. It already works nicely in regular browsers, and on several mobile platforms. Why code all the graphics rendering stuff from scratch when you can just license something and extend its features with AR functionality?

Tish: Now to ask your own question back to you! There seems to be a lot of reason to think that, eventually, there will be the kind of access to the iphone video API that augmented reality really requires and by that I mean more than we will get with OS 3.1 which is rumored to deliver only about half of what we really need for AR on the iphone – “not truly useful when you want to align video. with graphics.”  So:

“The iphone…future or failure? Seemingly anti-developer stance regarding augmented reality, and only a sliver of the global market share. Are we letting the short term glitz of Apple and the iPhone fad pull us in the wrong direction? Shouldnt we be focusing on symbian devices that have the lion’s share of the market? or should we be looking more at either other OSs (winmobile, android) or not at all and trying to create a new platform that is more MID and less smart phone with a hardware partner?”

Robert: Apple and the iphone are a bit problematic right now. There is no way I can go to a venture capitalist (at least in North America) and say hey we are building awesome AR applications for winmobile or symbian…they would either laugh or they simply wouldn’t get it. There is this false perception that the iphone is the ultimate mobile device, it is the sexiest, and the only thing that people want. Everyone wants a demo on the iphone, the media is mostly interested in iphone developments, and the apple fanatic market could give a fig about other devices. Other devices may have a larger market share or even better hardware, but we have to focus on the iphone right now at least in the demo stage to get any market attention and traction worth the time and effort.

In the future though, unless Apple changes its stance with their SDK and APIs, and starts adding hardware that is key for mobile AR (beyond what is there now), the market will move on without them. This is a really easy decision to make given Apple’s draconian policies and the fact that their percentage of the global market is miniscule. The smart companies are looking at the whole picture and not putting all of their eggs in the Apple basket.

Of course, once the wearable displays are commercially viable everything changes. Wearable computers with small screens or even no screens are going to be what everyone wants. The interface will go from handheld touch screens to virtual holographic interfaces that you interact with using your bare hands.

So for now, (the immediate short term), its all about the iphone. Taking mobile ubiquitous AR to the global market and building for the future will be based on something else. Hardware risks becoming a commodity or a closed platform. Do you really want to buy the Apple iGlasses and only see AR content that is compatible, where your best friend has a pair of WinGlasses and sees something entirely different? No. The hardware, and the client software (what people are calling the ar browser now) will become common and it won’t matter what brand you use, they will all be accessing the same content.

But at least for the forseeable future, we are building software for specific hardware, and the sexiest mobile on the block is the iphone. The second someone comes out with something much better and the paradigm shifts (software driving hardware instead of vice versa) everything changes.

Tish: How is the quest for sexy AR eyewear going.  I know we were checking out the Japanese eyewear with Adam Johnson from Genkii just now.  For the Neogence project – as you are going for a fully developed model of AR doesn’t this necessitate going beyond the iphone and getting the hardware companies moving on the eyewear?

Robert: The guys making wearable displays really need to get off the pot and stop paying lip service to mobile AR. If they don’t do something quick, I,  and others, are going to be scouring the planet looking for someone capable of building the lightweight stylish wearable displays with transparent lenses we are begging for. We aren’t going to be waiting around for hardware anymore. The AR Pandora’s box has been opened. I should note that many of us (AR Consortium members) have had less than pleasant experiences or communications with the half dozen companies or so that are making wearable displays. Either their visual design is terrible, the materials feel flimsy, the field of view is limited, or the companies are preoccupied with other business and government contracts. Any attention to the growing AR market is an afterthought and in a few cases condescending. AR is going to be a billion dollar industry in a very short time, and these guys are just leaving money on the table. If they were smart, they would be begging the CEOs from the AR Consortium to fly out to their offices and collaborate on building a pair of wicked sick glasses. The smart phone manufacturers should be doing the same thing, but I have to say that they at least seem to have some ambition and zeal to create better devices, so I can’t really complain too much there.

Anyway, to answer the rest of your question, we have to assume that the hardware guys, especially regarding the eyewear, is going to take a long time to develop and release the things we need for the ultimate AR experience. So, our goal is to start building things now for what is available. That means scaling things down and handicapping what AR can do, so it works on the “sexy” iphone. The important thing though is to start creating applications -now- so when the glasses are commercially available, there will be a wealth of content for people to access and use on day one.

As long as Apple isn’t playing nice, it is going to hurt everyone. Is it any surprise that they shut down Google Voice? There is a huge opportunity for someone to step up and leapfrog the rest of the industry. Give us the hardware and we will create amazing software for it. Don’t compete with the iphone, surpass it.

Tish: What is the state of play of current AR technology and toolkits?

Robert: The current crop of AR technology and toolkits is absolutely critical for this stage of the industry, and everyone should be leveraging it as much as possible. I talk down marker and image based tracking a lot, but I also like to point out that it is the necessary baseline that the industry is going to be built on. The problem is that there is only so much you can do with marker driven apps, and as creative people and marketing types start conceptualizing about all sorts of cool stuff for the future, they risk setting the expectations too high. It is one thing to show someone the future, it is another to say this is the future and its happening right now. This is why I cringe everytime I see a conceptual video presented as “our product DOES this” instead of “our product WILL DO this.” Something that simple can still cause the butterfly effect of raising expectations too high and contribute to overhyping.

Tish: One of the things that seems very exciting about the new Ogmento partnership is that experienced content producers  Brad Foxhoven and Brian Selzer from Blockade are now taking a leading role in AR.  What are the most exciting directions for content that you see emerging for AR in the next 12 months?

Robert: Virtual (well, augmented) pets, and multiuser mobile AR games (2-4 people) are probably going to lead in the next 12 months for content. Easy, accessible, engaging.

Tish: And are you at Neogence also involved in content partnerships?

Robert: Yes, we are in the process of finalizing some content partnerships with an eye for long term relationships. We are specifically looking for partners that want to find substantive ways to leverage AR technology, and not use it as a superficial gimmick or attraction that wears off after five minutes. I’m still cringing over the Proctor & Gamble Always campaign with AR.

Tish: So back to your observation about some of the tricky problems re creating a true global massively multiuser, ubiquitous, mobile AR platform – what are some of the main obstacles to this mission in our view? (aside from getting investment!)

Robert: Trying to explain it to people. The technical problems we can handle or have already solved. But trying to communicate what exactly we are doing is still tough. Not because it is overly complicated, but rather because it is so new and different. People are having a hard time grasping augmented reality beyond marker/webcam.

Tish: Which AR tools are most important right now?

Robert: Content is critical right now to show what the technology is capable of and to continue building the presence of augmented reality in the public mind the big benefit to integrated / unified platforms now is speed of development for content. I think that the flash artoolkit = papervision is rocking the planet right now. It is accessible, easy to learn, and lets people create something very quickly. More tools and middleware are coming out and this increases options for designers and developers.

Tish: What are your favorite papervision apps?

Robert: Hrm, I don’t have a favorite papervision app just yet, although I think the tech is solid. I expect to see a lot of stuff built on that platform in the near future. Especially as more ad agencies get on the bandwagon and start telling their IT guys to learn how to program flash so they can make something. Have you seen www.ronaldchevalier.com Not so much for the actual AR stuff, but because the whole thing is just brilliant. Its exactly like some cult figure spiritual guru would do with AR. I wish I had thought of it first actually. This is probably one of the best -seamless- implementations of AR in marketing where it fits…it isn’t just jammed in there for the sake of saying they used AR.

Tish: Do you think Apple is going open the iphone to the full potential of augmented reality anytime soon – a lot of expectations have been raised?

Robert: Apple is like that guy has a party at his house and owns this really awesome state of the art home theater in his basement, but makes everyone watch a movie in the living room on a regular TV with a VCR.

They need to get over themselves and quit being a wet blanket. Otherwise, we are taking the beer and pizza we brought, and going to someone else’s house. Sorry, the Apple thing is a bit of a sore point with me.

Tish: But will people leave all that candy and soda at the appstore?

Robert: I tell you what though, there is an opportunity for certain mobile phone manufacturers to give me a call and start talking to Neogence and the other members of the Consortium. We have some ideas and specs that could have a radical impact on the mobile market and stuff the IPhone in a box. Hint hint.

Tish: So what is your vision for the ARconsortium.  I know it kicked off with a letter to Apple about the video API.  What is the next step? There was a lot of hope that this year would be big for MIDs but this really hasn’t happened yet – do you think there is hope for a MID take off despite the lousy economy?)

Robert: MIDs? No, not yet. smart phones are too lucrative and too hot. It isn’t time yet for the MID to go mainstream. For that to happen, there needs to be a driving need (cough ubiquitous AR cough)

The AR consortium is mostly an informal affiliation. I expect that representatives from each member will probably meet at every significant conference to catch up over drinks. We are also going to be planning for our own members conference at least once a year. That will happen after we expand the membership though.

The main idea behind the consortium though was to open up a channel of communication between the CEOs so we could work together on standards, solving problems, collaborating, forming some partnerships, and using the collective to bang on the doors of companies like Apple and others. There is power in a group.

Tish: You mentioned there is a whole long conversation we can have about getting the eyewear.  As you point out true AR eyewear changes everything.  Can give a little road map of where this has to go?

Robert: There are essentially four or five main approaches, depending on whether or not you make the lenses special or if they are just plain. You would normally want them to be plain so people with prescription lenses wouldn’t have problems and would have the option to switch them out. Some types use a more prismatic approach for top down projection, or a corner piece mounts lasers and bounces them off the lens into the eye.  Another approach is embedding OLEDs or something else into the lenses themselves.

I really like the Lumus approach, but their product design isn’t quite there yet. If the wearables don’t look cool, people won’t use them. To be honest, if I had the money, I’d probably ask the Art Lebedev guys to design them based on someone else’s optical engineering. They designed the optimus maximus old keyboard…   brilliant industrial designers, loaded with engineers too. If these guys couldn’t build the glasses and make them look damn bad ass, I’d be shocked. Heck, I bet they could build the next gen MID while they were at it.

Tish: Getting the hardware innovation and software innovation feeding into each other would be really great.

Robert: Absolutely.

Tish: That would push the eyewear forward too wouldn’t it?

Robert: All it takes is one, and then the competitive landscape would fire right up.

Tish: What applications would the accurate gps enable?

Robert: Everything. for example, you know exactly where the phone is and where it is facing, that means you can put it on a table and hit a button, then move it somewhere else and do the same thing in a few minutes, you have a nearly accurate “mental” model of the whole place now you go back and start dropping virtual flower pots everywhere.

This is one area where I think the smart phone guys are missing the boat and taking the cheap route. It is possible to have very accurate GPS (down to a six inch area) with better chips and firmware, but it is cheaper to stick in old tech. Most apps today dont need that hyper accuracy, so they aren’t bothering. Mobile AR though, thats a different story.

With that level of accuracy, you would know exactly where the mobile device is, so all you would need to know is the direction it is facing (orientation), and you could solve one of the problems with registering exactly where 3D objects and augmented media is (it is more complicated than I am describing it, but we don’t need to get into that much detail here). You wouldn’t need markers anymore.

Tish: Isn’t Wikitude doing this with Wikitude.me their tagging app.?

Robert: Not really. That type of approach is on a very large scale using the accelerometers compass and GPS to determine where you are and what is in the distance. They (and others like Layar) don’t handle “near” AR. They effectively poll your GPS and then check a database to see what is nearby and what degree/distance it is and then they draw a representation on the screen. They don’t even need a mobile device’s camera at all.

Even if they did things up close, its still based on finding landmarks or on things that are broadcasting their location. For example, if they were standing near me, they might get “robert, 37 degrees, 15 meters away” but they wouldn’t be tracking me exactly as I walk around or have the ability to overlay graphics on ME.

Tish: I retweeted your #ar marketing using ARToolkit + flash (markers/webcams) = Photoshop pagecurl <six months. Bad design kills innovation. I know you like Dr Chevalier though!  What are some of the other AR marketing projects that you like. What would you like to see in terms of innovation in the next 6 months?

Robert: The marker/webcam approach is already becoming overused and cliche (tremendously fast). Older readers will remember the ubiquitous photoshop page curl that adorned nearly every website and graphic on the internet back in the day. It was horrible. Yes, the Dr. Chevalier stuff cracks me up.

I want to see some big companies or ad agencies really try to do something different with AR, preferably mobile. Take some risks, do something different. Don’t follow the crowd. Innovation? I want to see some wearable displays with transparent lenses, I want a mobile device specifically designed for ubiquitous AR, I want to see some experimenting with AR in the green tech sector, and I’d like to see someone get that GiFi wireless technology from that researcher in Australia and jam it into a smart mobile. I would also like my flying car and lunar vacation now, thank you. It is almost 2010 and no one has found that black obelisk yet.

Tish: So a few closing thoughts! What do you see as the next big thing? Hopes for the ar consortium?  Biggest bstacle for commercial AR?  And what is the coolest thing you have seen this year?!

Robert: The next big thing is what I’m working on hahaha. I hope the AR Consortium will grow and be the active catalyst in making AR mainstream, practical, and world changing.

The biggest obstacle is making sure that the right funding finds the right developers to develop the right technology and create kick ass applications.

The coolest thing I’ve seen this year would probably be the facade projection stuff (see below): Now, imagine that, but without the projector. Thats part of what I envision for AR in the future.

Recently, I talked to Ori Inbar (above), formerly senior vice- president at SAP.  Ori is on a mission to make augmented reality commercially successful not in 5, 10, or 15 years, but now. Ori is the founder of Pookatak Games – a video game company, “with a vision to upgrade the way people experience the world.” Ori will be participating May 20th, in O’Reilly’s Where 2.0 panel, “Mobile Reality” -  an event not to be missed IMO.

The taste for computing anywhere anytime has entered human culture via the iphone and is spreading like chocolate cake and pizza at a preschool party (see why the iPhone changed everything).  And while the full flowering of the next step is yet to come – computing anywhere, anytime by anyone and anything (“the internet of things”), our love for these first devices capable of being mediating artifacts for ubiquitous computing (Adam Greenfield) is a vital first step to free us from our tethers to computer screens, and fulfill the promise of augmented reality.

If you need more convincing on the pivotal role augmented reality will play as the web moves into the world, check out Tim O’Reilly’s recent comments in this video clip posted on Augmented Times and here early last year.

From another perspective, the gloomy specter of economic and environmental catastrophe  is driving a movement to “infuse intelligence into the way the world work’s.” But the challenge for a smart planet is not just about making environments smart, it is about using smart environments to enable people to act smarter (see my interview with Adam Greenfield).

We need a rapid upgrade in both the way the world works, and the way we experience the world.

((Note:  It is time to read (if you haven’t already) Bruce Sterling’s Caryatids (Cory Doctorow’s book of the year for 2009) “as a software design manual” (see Julian Bleeker) because Caryatids reveals the Gordian knots of human folly, greed, compassion and desire entwined in near future designs for technologies to save the world.))

Ori Inbar, worked with Shai Agassi (Shai is now leading the world changing Better Place ) driving Netweaver from a mere concept to a “major, major business for SAP.” So Ori has already been through the cycle of working in a very small startup and growing it into a billion dollar business.  He has both the experience and the passion to realize his vision for augmented reality.

At Pookatak, he explains :

“We design “reality experiences” that make users’ immediate environments more significant to them. We wish to free young and old from getting lost in front of the screen. By delivering the world’s information to people’s field of view, and by weaving real world objects into interactive narratives, we help people rediscover the real world.”

Pookatak will release their first game this summer. Currently it is under wraps. But Ori gives us some glimpses of what is to come in the interview below.

In addition to founding Pookatak, Ori is involved in a broader effort to move augmented reality forward. On his blog, Games Alfresco – he recently welcomed a new partner, Rouli Nir, Ori has focused his eye of wisdom on every significant recent advance in Augmented Reality (check out this essence of Ori’s thinking in a fast paced video presentation for WARM ‘09).

Also Ori is one of the organizers of the interactive media track at ISMAR 2009.  At ISMAR this year, Ori explained, “we are trying to bring in people that develop interactive experiences for consumers, beyond the traditional attendees coming from a research perspective.”

In the interview below, Ori explains much of his thinking on how augmented reality will become commercially successful.  Enjoy it, think about it, and share it. And most importantly, if you can, get involved with ISMAR 2009.

Ori  has inspired me to participate in ISMAR this year.  Ori pointed out:

The call for papers is on, and this year it targets well beyond the typical research papers audience and into interactive media and art folks.

There are plenty of opportunities such as:

Art Gallery

Demonstrations

Tutorial

Workshops

It’s a huge opportunity to shape the emergence of augmented reality.

Interview With Ori Inbar

Making Augmented Reality Commercially Successful

Tish Shute: You are considered a key trail blazer in AR and you have the go to blog for augmented reality!  What are the most important lessons you have learned researching, writing, and developing AR in the last couple of years?

Ori Inbar: You need to have a vision. You need to know where this is going to go in ten or fifteen or twenty years. But you’ve got to start with something really simple that makes use of the technology you have on hand. And do something that is practical, that people will like, and something they would actually want to buy. Its as simple as that. I’m currently looking at what we could do with existing technology. First of all, you have to put it in front of people. Right now most people have never heard about the term augmented reality. Go into the street, and ask 100 people about it, maybe 2 would know about it. So you need to put it in front of people because most people think it’s still science fiction or a special effect you see in movies, not something you can experience in real life.

Tish: It seems to me to that for augmented reality applications to become popular with existing technology the key breakthrough would be getting people to hold up their phones. What are the obstacles to getting people to use their mobile devices like this?

Ori: There’s a really nice cartoon by Tonchidot (below) – the Japanese company behind the Sekai Camera. It’s an illustration showing the evolution of man, from ape to man (holding a cell phone looking down), to the developed man holding a device like a camera – in front of its eyes.

Which is exactly what you’re talking about. People ask, “are people going to walk with this like that all day long?” Probably not. I mean you have to build it in a way that doesn’t require them to hold it like that all the time. People are used to this gesture with the ubiquitous digital cameras. I tested one of my prototypes on a two and a half year old girl. She had no problem holding it just like she holds a camera.

Tish: Blair MacIntyre mentioned, “The problem with the mobile phone as a AR device is a problem of awareness,” i.e., you have to have a way of letting people know when there’s something interesting wherever they are. One of the issues regarding this is if you get too many alerts, then you tune them out.

Ori: First of all Blair is one of the people in academia that get it. Because he looks at it from an experience perspective. Not just as an interesting technical problem to solve. Let’s start with getting people to enjoy this new experience. The AR demos so far were mostly eye candies, and mostly for advertising – the GE AR ad created a lot of buzz; but you look at it for 10 seconds and you forget about it.  You need to build something that people would want to experience over time and would be willing to pay for. I think that’s the big test, right?

Now in terms of having a ubiquitous experience where you’re continously connected, it doesn’t have to be an overwhelming experience. Just like some of the social media tools we’re using today, we decide when to connect, and we filter out the trash. You could get alerts only for things that really matter to you, not for everything that happens in your immediate environment.

There will be many layers of information, and it’ll be up to you to pick the ones you want to experience. The real benefit is that you get the information in your own field of view and in context of where you are or what you do.

Tish: So what are you working on these days?

Ori: We are working on a little app that targets a very different audience than what you’d expect: pre schoolers. We think we can encourage them to get away from a PC or TV screen and learn something while playing – in the real world. You’ll hear more about it as soon as this summer. Nuff said.

But, it is a small application that will run on the iPhone. People ask how many pre-schoolers own iPhones? Well, their parents do.

Tish: Yes there are certainly many New York kids with iPhones – my kid now has my old iphone.  He has pretty much switched from playing games on his DS to the iPhone. I noticed in your WARM video you place a big emphasis on AR as something that will get kids away from screens and engaged with reality.  This is something parents will approve of!

Ori: Yes I saw something really interesting at my kids’ party one day; they were all sitting around the room – looking down at their own DS screens.  You could play the DS anywhere, but kids would usually play it on the sofa, looking at the screen, isolated from the world. With an iPhone and a camera, and the application we’re producing, reality becomes part of the game. Yes that makes it all of a sudden much more interesting for parents. Because kids are spending so much time in front of the screen, all of a sudden they’re something that will encourage them to interact with real objects, real things. Every parent I’ve talked to loves that idea.

Tish: Yes that is what is cool about the work of Kati London – I think I saw someone say this on Twitter, “Kati puts the computer in the game not the game in the computer.”

Ori: Yes, kids are spending more time in front of games and the computer because it’s more interesting. It captivates them with “game pleasures ” that tap into their brain’s dopamine circuitry – constantly seeking reward and satisfaction. So you’re not going to be able to tell them to go back to playing in reality without these pleasures. We have to study these mechanics from games and bring them into reality. It’s about programming real life; and augmented reality helps you achieve that.

Here’s an example: cause and effect; in a game when you do something you always get an immediate effect. You’re good, you get a reward. You’re not good, you get a cue to improve. In real life you do things and you could wait 2 or 3 years until you actually get feedback (if you’re lucky). Augmented Reality allows you to bring these mechanics into the real world. I think that’s going to help kids rediscover reality, in a new sense, which is what every parent is dreaming about.

Tish: I don’t know how much you can say about your app. But in regard to doing augmented reality on the iPhone.. there’s no compass. Is this a limitation?

Ori: True, no compass yet. But the camera gives you a lot of information that you can interact with. When you run the application, you see the world in front of you, and if the app can recognize real life objects – it can put virtual elements on top of it.

Tish: But not with any accuracy unless you’re using markers. Are you using markers?

Ori: We’re using natural feature recognition. It doesn’t have to be an ugly looking marker. It can be any image.

Tish: So you’re using image recognition. Are you working with one of these image recognition startup companies (list here )?

Ori: We’re working with one of those. What’s unique about it is it runs very nicely on any cell phone, and on the iPhone it works the best. For this first app, it doesn’t really matter where you are physically; the geolocation is not part of the experience.

Tish: For a truly engaging AR experience we will need more of a backend than is currently available?

Ori: I call the backend the cloud, where you have all this information and ways to access it from anywhere. Actually I think it’s become pretty mature today. If you look at the different elements required to enable an augmented reality experience to work, you have – first – the user whose always in the center. Then you have the lens. The lens can be an iPhone, or glasses, even a projector. The lens allows you to watch, sense and track information in the real world: people, places, things. Then in the backend you have the cloud where you store and retrieve information.

So if you look at the maturity of these different elements, I think the cloud is in pretty good shape. Because there’s so much information we’re collecting and storing. Anything from Google, Wikipedia, Facebook, all that kind of stuff, it’s a lot of useful information you can access from anywhere using APIs. And a lot of it is also starting to include geolocation information. Take Loopt or Google’s friends service that allows you to see where your friends are and what they’re doing. There’s tons of information out there and it’s pretty easy to access it. Now what do you do with it is the question?

Wikitude is such a simple and brilliant application and nobody thought about doing it until this guy from Salzburg did. It doesn’t have any sophisticated visual tracking. It knows your position and it’s simply looking at the angle you’re pointing to. Based on these parameters it brings information from Wikipedia that pertains to your field of view. So most of it was already there. It’s just a matter of connecting the pieces in an experience that is valuable for people.

Tish: It is the uptake of even a very simple technology that puts the magic in it.

Ori:  Yes, take Twitter. If you go to its homepage it looks like a very simple boring app but it is something that is both enjoyable and very useful to people.

Why you should participate in ISMAR 2009

Tish: I know that you are involved in organizing  ISMAR (picture above from Ori’s post on “ISMAR 2009: The World’s Best Augmented Reality Event…,“) and there is a call out for papers and for volunteers, can you tell me more about it?

Ori: Yes, we hope to have the first ISMAR where we practice what we have just discussed: let’s build on all the research invested so far and instead of thinking only about 5-10 years from now, let’s see what we can do today. So we are bringing people in from other disciplines – artists, interactive media developers and people from the entertainment industry.  The goal is to use the technology to make something interesting for people – again, something that people would buy, and making it commercially successful.  Many people either don’t know about ISMAR because in the past it was a pure engineering-orientated event and people  from a commercial perspective of AR weren’t attracted to it.  The Chair of the Event this year is based in Florida and he is going to bring in a lot of people from the entertainment industry such as Disney. I think this will transform this event into something more like SIGGRAPH – more of an industry event.  As one of the organizers of the interactive media track we are trying to bring in people that want to build applications for consumers.

Tish: In terms of AR applications what are the flagships today?

Ori: There are very few because it’s just the beginning. There’s one tiny studio in France called Int 13 . They’ve created maybe the first commercial game running on a mobile device using AR technology. It’s called Kweekies. It was one of the contenders for the Nokia Mobile innovation awards. They were one of the ten finalists, but they didn’t win it. It’s looks really cool. It’s somethng that runs on your desk, with a marker. Many AR folks say markers are the past, markers are ugly. But it’s still a cool experience. I think people will go for it.

Tish: Yes I think we will have to look to small companies that are free to think creatively to lead the way.  It seems many games companies are tied up pulling off huge big budget projects and enterprise is still catching up on how to use social media!

Ori: Yes, last year I was in the game development conference (GDC); there was no mention of augmented reality – not on the exhibition floor, none of the sessions, nobody talked about it. I was stunned. Then this year, there was a little a change. There were like three demos on the exhibition floor, Metaio, Vuzix and a Dutch company called Beyond Reality.  And then there was Blair’s talk, which was very very cool. The room was packed with people. And after the talk there were dozens of people lining up to talk with him about the topic. There was definitely interest, but still on the very edge. The video game industry is still a hit driven business and publishers spend upward of 20-30 million dollar to create the best AAA game possible. They just can’t take the risk. So it’s going to come from smaller companies, from outsiders coming in with a vision and understanding on how to put the AR pieces together to create a totally new experience.

Tish: But the basic tool set is there isn’t it?

Ori: I talked to some folks at the games developer conference, many folks with MMO background, and they have great ideas about AR. It’s great to see different people with different views on what’s needed first. “Joe the Programmer” had this idea of creating a small piece of hardware that you can put in every house and provide accurate geospatial information in your home. That could  open up many opportunities for AR experiences in homes.

Tish: Don’t you think we have enormous resources in terms of image databases that provide a great basis for augmented reality.  I was talking to Aaron Cope at ETech about The Shape of Alpha – Flickr’s vernacular mapping project using all the geotagged photos in Flickr. That is such cool project. Aaron will be speaking at Where 2.0 also.

Ori: Think of Google Earth. Google Earth leveraged communities to basically map all the major cities around the world into 3D models. And that is an essential step to be able to do augmented reality outdoors. Because if you had to model everything from scratch, it wouldn’t be realistic.

Augmented Reality and Becoming Greener.

Tish: I am really interested in how AR interfaces might be useful to some of the emerging energy identity/metering projects like AMEE and WATTZON because I think it is very important that people have very intuitive, immediate, and enjoyable ways to relate to energy data so they can make greener choices.

Ori: Back in the day I had an idea to build an Augmented Reality application to become greener. You look at things around your home with the camera and it  recognizes its green gas footprint and makes recommendations to reduce it.  I guess it was a bit too early to do that based on visual recognition alone…you’d need  additional sensors that would provide related information about what you are looking at.

Tish: Well as there is more interest in Green technology do you think we may see VC interest in some green AR projects now?

Ori: I talked to some of the investment folks, Angels as well as VC’s about AR and they had no clue what it is. There’s a need for a whole lot of education. And there are no proof points (as in successful investments in this domain), and counter to popular belief – they don’t like risk so much…

Tish: And consumer adoption must lead the way, right?

Ori: Just like with every emerging technology in history, people never bought the technology, they bought the content, the apps, the benefits that came on top of the technology. Whether it was VHS winning over Beta Max, or BluRay winning over HD. It’s always because of more/better content. Look at the video game console war: Xbox, and Nintendo did better than Sony just because they had more and better games. Even Windows was a success thanks to its applications. People bought it for the applications not the OS. The content is the first to drive demand.

Tish: One of the challenges to giving people new ways to relate to their energy consumption is that you can just have them looking at graphs of how bad they have been in the past you – that may make them feel bad but that doesn’t necessarily give them ways or motivation to change. There perhaps needs to be more immediate relationship to the data to facilitate change. I think the mantra for optimization of anything from energy usage to supply chains is timely, actionable data?

Ori: There are a lot of ideas about measuring information and displaying it to people. For example, the Prius hybrid car, one of its interesting features – which is kind of game like – is a constant display of your current fuel consumption. That alone changes how people drive because they try to beat the “Score” and as a result conserve more fuel. That model can be applied to our homes…

Tish: Yes that is something I am very interested in. I have been following several projects in this area – one of my favorites is the Arduino, Current Cost/Tweetawatt, Pachube integrations I saw at Homecamp.

You joined a start up with Shai Agassi which was bought out by SAP right? He has a brilliant approach with Better Place.

Ori:  I think what’s really unique about Better Place’s approach is that he doesn’t require people to change their behavior. People are still going to have their own cars. They’ll be able to drive as far as they want, and for the same (or lower cost). Its not necessarily about a new technology, electric cars have been around for a long time but there was no way people were going to be limited by the 50 or 70 mile range and Better Place is solving that problem. With its infrastructure of charging spots and battery switching stations, drivers are going to be able to drive anywhere. And it’ll be similar to having to stop once in a while to refuel your car. The price maybe even lower than what you pay today for your transportation needs – and you’ll stop generating green gas. It’s a clever way of taking technology to a whole new level without changing the behavior of people.

Tish: Better Place is a classic example of things as a service isn’t it?  It is basically a utility company.

Ori: It is similar to a phone carrier model.  You pay for a membership that gives you access to the car (equivalent to the phone) and electricity (equivalent to the phone line) for the same price of fuel cost today. And as bonus you get to save the world.

How the iphone changed the game for AR – and the iphone versus Android

Picture from Ori’s post, “GDC 2009: Why the iphone changed everything”

Ori: And back to AR, you have to take the same approach, because nobody’s wants to don those huge head mounted displays or backpacks. You have to take advantage of people’s current behavior: they already carry their iPhones or similar devices.

Tish: As we discussed, you just have to get people raising up their phones and looking through them when that is a useful thing to do. Both Wikitude and Nathan Freitas’s graffiti app were enough to get me interested in the evolutionary step of raising my phone! Nathan’s graffiti app is nice. You leave a marker for your graffiti so other people can find view/add their own – a nice primal experience like pissing on the lamp post to let your pack know where you’ve been.  Also the graffiti app taps into a long history of  NYC street culture around tagging and graffiti art (see my interview, “Is it OMG finally for Augmented Reality?”).

Ori: The app store has fundamentally changed the mobile gaming industry. Last year they were in shambles. There was no growth. Everybody was complaining, “we can’t handle it, there’s a million phones, and you have to test it on each phone. And carriers suck, they don’t care about sharing and promoting your content. Everything was bad. This year mobile gaming is the hottest thing. And it’s all because of the iPhone. It changed the game.

Tish: How do you think Android is going to get traction against the iphone?

Ori: Well the number one thing is the form factor – the iPhone is just much cooler than the G1. Its OK but it doesn’t have the same feel. People thought it was going to be easy to clone the iPhone but none of the attempts succeeded so far.

Tish: How much does it matter for AR not being able to runs things persistently in the background on the iphone?

Ori: Actually they have add a such a capability in OS 3.  You can now make use of a background service.

Tish: OS 3 will open up new possibilities for AR?

Ori: The access to the video API is still not public.  But there is a new Microsoft application – Microsoft Tag that makes use of that API which means it is probably OK to use it.

Tish: (I ask Ori for his card and he shows me how to read it with my iphone.) Oh nice you have an AR card, of course!

In Search of Pong for Augmented Reality

Tish: So how will AR begin to, as Blair’s friend put’s it, “facilitate a killer existence,” particularly as we are probably looking at some new and perhaps pricey hardware?

Ori: You could take the Better Place approach. We’re going to give you a great experience and we’ll include the devices as part of that experience for the same price. Let’s say you subscribe to an AR experience  which offers access to multiuser, support, and all the information you need wherever you go – exactly according to the vision. You pay for a subscription on a monthly basis and included in that cost we give you a better device that offers a  better AR experience. It’s following the phone carrier approach, but in a good way.

But first of all we do need our Pong! I was sitting with a couple of AR game enthusiasts at the GDC and we were asking ourselves, “how do we create the first pong for AR?”

Was Pong a multiplayer game? Not necessarily! Did it connect to the network? No! We have to create the first dot in a long line of dots that will bring us to our destination.

Tish: You haven’t seen a Pong yet have you?

Ori: Not yet. I mean there’s maybe a handful of games and apps out there, but I don’t think any of them is a Pong yet. Still, it’s getting closer.

Tish: Kati London is doing some very interesting work on bringing games into reality, isn’t she?

Ori: Yes, she works with Frank Lanz at Area/Code. He teaches at NYU and has designed games for the “Come Out and Play” festival here in Manhattan. And a lot of these games are actually low tech.

Tish: Yes I have a big alternate reality game blog brewing that I haven’t had time to write yet!

Ori: The city is the gameboard is their slogan. It’s going to be a great playground for AR games. The city becomes a theme park. The city could become an even bigger touristic attraction. People will come to the city to be part of these games. So you’re having thousands of people running around the city playing all sorts of games from laser-tag style to history adventures, to treasure hunts.

Composing Reality

Tish: So why haven’t you focused on one of these kinds of games with your company?

Ori: We have a couple of scenarios along these lines that we’re planning for 2010-11. But first focus on what’s possible today.

Tish: And what’s stopping you from doing those kind of games today?

Ori: Many things. The devices are not there yet, location services are not accurate enough, ubiquitous sensors are not  there yet.

Tish: You think alternate reality gaming needs more “ubiquity” than is currently available?

Ori: Not necessarily. People are doing alternate reality games with no “ubiquity” at all. But my interest is to add the visual aspect. I believe humans are mostly driven visually.

Jane McGonigal said in a talk at GDC, that AR would allow us to program reality, which is exactly how I look at it. Once you can recognize things, some of it with WiFi and RFID and all sorts of sensors. But visual sensors is always going to be the ultimate way to recognize things. And once you recognize things and know what they are, and can pull information about those things (or people and places) from the internet, you can program it (visually). You could program it to be fictional, like in a video game, or it could be programmed as non-fictional, like a documentary. And that allows you to do things that before were unimaginable.

Tish: But you can’t forget the visual, it is primary the connection to peoples’ primary sensory relationships.

Ori: Yes, it’s like you go to a grocery store and you pick your vegetables, a lot of it is by sight and by touch. And what if you could also see just by looking at it that it’s from a local store, and that it’s organic?

Tish: It goes beyond overlays really?

Ori: By the way, I don’t like the term ‘overlay’. I know that’s how it looks: you either overlay or superimpose, but I’m still searching for a better term. A term I prefer to use is “composing reality”. Just like painters, they use brushstrokes and colors and compose a painting. We need to take the real element and the virtual element and compose them into something new. It’s not just about slapping one on top of the other.

Tish: yes I think the idea of dashboards is not so appealing.

Pookatak Games

Tish: Do you want to explain the evolution of your company? You have an interesting history of success with high end enterprise applications.

Ori: Since I was a kid I wanted to invent and create things. When I discovered software, that was a really cool way of actually creating things from nothing. From thin air; and you can do it very quickly. That’s what brought me into software. But I was always looking for the intersection between technology and art. Looking for ways to bring these things together. In the early nineties virtual reality was doing it. It had the appeal of cutting edge technology that can be combined with art. But then, as we all know, it crashed. So I joined Shai Agassi’s startup (who is now doing Better Place) back in the early nineties. I was one of the first employees in his startup which was developing multimedia products. I was leading the development of one of its flagship product. At some point we realized the technology could be great for an enterprise environment.

It was a really great experience. First going through this cycle from a very small startup and growing into this multi billion dollar business. I was responsible for defining and marketing SAP’s platform, which was called Netweaver. It was just an idea when we joined SAP and by the time I left it was a major, major business for SAP. I learned about the challenges of building a platform. No matter what purpose you’re building it for, it typically has similar rules. It’s definitely not just about the technology; the content that comes with it is really key to making a platform successful.

The third part of this platform trifecta is the community. If you don’t build a community, you won’t get the critical mass required for adoption. It may be your own platform but it’s not necessarily the people’s platform. That experience is very key to what we’re doing today. Now, a new industry is being born on the basis of a remarkable technology. But to drive adoption, first we’ll need good content. The content will be created using today’s technology with internal tools developed to simplify the process. Next step would be to make the tools used internally – available to other developers. Help scale the industry, enable innovation on a larger scale. That way we have a chance to create a platform. So it isn’t really just about my company. I’m so passionate about augmented reality, I want to it to become a healthy and successful industry for the next 5, 10, 15 years.

Tish: Yes I am so ready to be liberated from the sitting behind a computing screen! And I know that all this hardware is murdering the environment.

Ori: There’s ‘s the book by Rolf Hainich which is called “The End Of Hardware. ” It’s about hardware for augmented-reality. Once you use goggles or other AR interfaces you eliminate the need for screens, laptops, etc. It’s going to be great for the environment. You have read Rainbow’s End, right? According to the book in few years there will barely be any (visible) hardware. At least it’ll have a much smaller footprint for the environment. And it’ll touch every aspect of life, everything you do. It’ll change the way you interact with the world.

The Illusive Eyewear for Immersive AR.

Friend of Ori’s in San Francisco wearing retro AR goggles (from Games Alfresco, Ori’s roundup of GDC 2009)

Tish:OK lets talk about goggles.

Ori: Goggles are going to happen, we want to be hands free.

It’s going to happen because it’s just a more intuitive way to use this technology. But above all it has to look cool. Because if it’s not, if it’s a big headset, then maybe a small percent of the population might use it, but most people won’t. It has to look like an accessory, like new cool eyeglasses that you just must wear.

I recently talked to a friend, who runs an industrial design firm, and has experience in designing such glasses for companies like Microvision and Lumux. He says that when you try to bring the images so close to our eyes – there are some really hard problems to solve. Otherwise it can become really annoying and cause dizzyness.

But I’m optimistic. I believe it’s going to happen 3 to 5 years from now. It’s already starting now: Vuzix announced goggles that will be available this year. Some AR apps that are going to take advantage of next year. Initially only a fraction of the population will use it. And that’s going to help advance it and make it better and better. But it’s going to take time until it reaches the mass market.

Tish: In virtual worlds we have seen, I think, a lot of mistakes in terms of reinventing the wheel and producing too many proprietary versions of the same thing and not enough concerted effort on standards and open platforms that could create a vibrant ecosystem.  How can augmented reality not make the same mistakes?

Ori: There are some early AR open source efforts ARTookit, ARtag but it is not a movement yet.  One of the things we’re trying to do at ISMAR this year is to put together  discussions around key industry issues, such as standards. Some people say it’s too early, you have to have a defacto standard to start from. But pretty soon it’s going to be too late. Just like with virtual worlds, all of a sudden you have all these islands that don’t talk to each other. Why get to that point if we can plan to avoid it? Let’s start thinking about it right now. On the other front there are devices. There are pockets of people working on adapting devices for AR, second guessing the hardware companies. Why not get them together with the Intels and Nvidias of the world, and discuss what this device should be able to do. And then compete to make it happen.

Tish: How much luck are you having with this discussion part?

Ori: People are very interested in doing this. We proposed these panels for ISMAR. And I’ve got some key people already on board. They have tons of input, they want to get involved. We’ll see how much we can actually get out of it.

Tish: In virtual worlds it was a while before vibrant opensource communities developed.  OpenSim has I think been the breakthrough community in this regard.

Ori: You have to think about the elements up front. The dream job is to architect the industry. Say we agree on the required pieces. Then we could help the right companies succeed in delivering the pieces. Next, we have to collaborate so that these pieces talk to each other. And eventually these communication methods will become defacto standards and most developers will adopt it.

Tish: So I’m going to put you in the role. You’ve got your dream job. You’re going to architect this community. So what are the key pieces and where would you like to see the open source communities take hold first?

Ori: Open source will not be exclusive. It’s going to live side by side with proprietary technology.

The key pieces? You have the user at the center. And the user interacts with a lens. The lens includes both the hardware and the software. And then the lens senses and interacts with the world, which includes people, things and places. And these people-things-places emit information – about who they are, where they are, what they’re doing, etc  – which is then stored in the cloud.

And then you have the content providers, the people and companies, composers who weave AR experiences through the pieces we mentioned before. These composers need a platform that glues these pieces together. Pieces of the platform will be on the lens, and in the world, and in the cloud. If you manage to remove the frictions, and connect these pieces into an experience that people like – then you have a platform. What the platform does it reduces the overhead and accelerates innovation.

Tish: Another problem virtual worlds faced in their development was their isolation from the world wide web.  Will augmented reality avoid this plight?

Ori:  Yes, I believe the key, like you said before, is not to reinvent the wheel. The cloud is already there.  Take Wikitude for example, all Mobilizy had to do is build  a relatively simple client app, connected to wikipedia, and all of a sudden it offered a wealth of information in your field of view.

I think we can learn a lot from web 2.0. For example, in order to have a ubiquitous experience like Robert Rice and others are striving for, you’ll need to 3d map the world. Google earth like apps are going to help but it is not going to be sufficient. So let’s leverage people. Google became successful in part by making people work with them.  Each time you create a link from your blog to my blog their search engines learn from it.  So let’s find ways to make people create information that can be used for AR.

Ori Inbar directed Wiki Mouse – a WIKI Film co-created by a swarm of movie makers around the world.

Adam Greenfield’s new book, The City Is Here For You To Use, is coming soon (photo above by Pepe Makkonen is from Adam Greenfield’s Flickr stream). Adam told me:

“I’m aiming at a free v1.0 PDF release on 05 June 2009, with the book shipping as quickly thereafter as humanly possible. There will be a version zero or public alpha in about six weeks.”

I am not good at waiting for books I really want to read to arrive. But, on the upside, it brings out my already pretty highly developed investigative instinct. So when Adam very generously agreed to do an interview, impatience turned into delight in tasting what is to come. And Adam is encouraging this kind of engaged anticipation. He writes (see post) that The City Is Here For You To Use, is shaping up:

“as something of an unbook avant la lettre. It’s why we’ve [Nurri Kim and Adam Greenfield] always insisted on keeping you in the loop as to the book’s fitful progress, it’s why I take every opportunity to test its ideas here, it’s why I make explicit the fact that your response to those ideas is crucial to their evolution and expression. And it’s why, even though the process is inevitably going to result in a static, physical document as one of its manifestations – and hopefully a very nice one indeed – we’ve committed to offering a free and freely-downloadable Creative Commons-licensed PDF of every numbered version of The City, from zero onward.

You buy the book if you want the object. The ideas are free.”

I found the opportunity to ask Adam questions about some of his subtle renderings of technology, culture, and being in urban environments challenging and very illuminating.  Although I definitely get the feeling I am asleep at the wheel on some of the critical areas he is thinking and writing on.

Knowing the depth and range of Adam’s thought in his seminal book, Everyware, and his blog, Speedbird, before I began the conversation I asked Adam to point me to some of his posts that reflect key ideas he is working on at the moment (Adam has recently posted The City Is Here: Table of contents).  Adam directed me to these three posts.

Antisocial networking

More songs about context and mood

Messenger, space, messenger body, messenger mesh

I may ramble and diverge, as is my nature, but these posts inspired many of the questions I ask.

Adam is currently head of design direction for service and user-interface design at Nokia and living in Helsinki, so I did not have the opportunity to do the interview in person. But I have glimpsed Adam’s world through his Flickr stream and some of these images have found their way into this post. But I suggest you browse Adam’s photography for yourself. I cannot do justice to the thousands of nuanced perceptions of cities, networks and publics you will find there. In the meantime, here are three glyphs of Adam Greenfield that I liked a lot.

“My favorite shoes,” “My favorite chair” and photo by Adam Greenfield, “Favoriteplace”

A Conversation (in gdoc) with Adam Greenfield

Tish Shute: Could you explain a little about the evolution of your thoughts on urban environments, ubicomp and interaction design? What shifts in your thinking have taken place over the last few years re the dawning of the age of ubiquitous computing? It is a couple of years now since Everyware, what aspects of the uptake of Everyware have most surprised, disappointed or inspired you? Which of the many thesis you discuss in Everyware have become the most crucial for The City Is Here For You To Use?

Adam Greenfield: You know, there’s a little passage in the liner notes to the second Throbbing Gristle album that I always think of when I’m asked questions along these lines. As part of their stance, they’d adopted the dry tone of a corporate annual report, and the preamble began by saying, “Since our last report to you, many things have changed. Indeed, it would be foolish to assume that it could be otherwise.” And I think that’s just exactly right: the world keeps moving, and the positions we’d staked ourselves to not so long ago may no longer be correct, or even relevant, to the one we find ourselves inhabiting now.

So, first, I think it’s important to cop to all the places in Everyware where I just outright got things wrong. There’s a passage in Thesis 50, for example, where I unaccountably mock the idea that “the mobile phone…will do splendidly as a mediating artifact for the delivery of [ubiquitous] services.” OK, this was admittedly written in a pre-iPhone world – and was correct for that world – but you can really see my parochialism showing here. It took the iPhone to make the proposition as blazingly self-evident to me in North America as it had been for quite some time to folks in Europe and Asia.

Having said that, though, I think I’m justified in taking a little pride in what the book got right. The broader trends the book set out to discuss – the colonization of everyday life by information processing – well, take a good look around you. And so one of the points of departure for the new book is taking everything posited in Everyware as a given: the urban environment, and most everything in it as well, has been provisioned with the kind of abilities you mention. So what now?

How do you go about designing informatic systems so they don’t undermine the wonderful things about cities? How do you design cities so they can incorporate networked informatics to greatest advantage? How, especially, do you accomplish these things when the disciplinary communities involved barely speak the same language? And how do you keep everyone’s eyes on the prize, which is the ordinary human being asked to make sense of these new propositions? These are the questions The City Is Here For You To Use sets out to address.

Adam talking about the “Le Long Ici” in Paris (also see Adam’s post, “The long here and the big now”)

TS: You mention that the hardest parts of  producing The City Is Here For You To Use wasn’t “keeping on top of all the emergent manifestations of urban informatics, or even developing a satisfying spinal argument about their significance” but getting the voice right.  It seems that now is the perfect time for a book that would really speak to a wide audience.  But also it seems that the city that is here for you to use is manifesting quite differently in different parts of the world?  You seem to be somewhat of a nomad, Japan to NYC to Helsinki.  Can putting together different views of urban informatics give us more depth perception on the emergence of ubiquitous computing?

AG: There’s no question in my mind that the long-term experience of everyday life in Tokyo, New York, and now Helsinki has been an invaluable asset to me, as I imagine it would be to anybody interested in thinking or writing about the networked city. It’s given me a certain amount of parallax, you know? And that, in turn, throws a really interesting light onto how the selfsame technology can appear in substantially different guises in different social contexts.

But explaining those things – those complicated, delicate negotiations – getting them right, doing them justice, doing so in a way that doesn’t dumb anything down, and still remaining accessible? It’s a challenge, let me tell you. You want to remain approachable and humane, but you also want to explain things like different jurisprudential takes on property, or how advocates of RESTful architectures think that REST is the reason why Internet adoption spread as rapidly as it did. If you want to enjoy even one chance in a hundred of getting your message across, you’ve got to start with an understanding that those subjects are MEGO territory for most people – whether they hail from Shibuya, Shoreditch or San Pedro.

Everyware icons: Information processing dissolving into behavior (Icons inspired by Timo Arnall; design by Adam Greenfield and Nurri Kim).  [Adam notes on his Flickr page that he tweaked these icons as section headers for Everyware]

TS: Could you explain more about what you term “onto” and “ontome” and how this differs from spimes and spime wrangling?

AG: You know, I never did get to develop that idea as much as I would have liked. In my mind, at least, “ontome” referred to the totality – the global environment of addressable, queryable, scriptable objects. (An “onto,” then, would be any given such object.) I guess I was looking for words that would do two things: allow us to distinguish between the instantiation and the class, and leave us with a better word than “spime.”

TS: When you say better word than spime this is this because….

AG: Euphony, primarily. : . )

TS: When I first used the Android app,  Wikitude, on Broadway, NYC – a street I have traveled thousands and thousands of times, and it offered up new information about itself, it was definitely an “OMG this is big!” moment for me. Like the first time I clicked on a screen and Amazon sent out a book in the early nineties (something so ordinary now it seems impossible that it was exciting but I remember it was to me!). But if I understand your post here correctly, isn’t Android with compass the first easy-to-use context-aware mediator for wrangling onto, ontome and spimes?

AG: Wikitude sure looks pretty impressive, and maybe even useful. But I would never, ever call it “context-aware.”

To my mind, at least two more things would need to happen before we could comfortably think of it a “context-aware spime wrangler.” First, the buildings and other public objects around you would actually have to be spimy – they’d have to report something of their past and current state to the network. And then, some application running on your phone would somehow have to cross-reference that state information with some fact about your current state of being, and deliver you relevant information.

So, let’s take your Wikitude example. You’re walking down Broadway and you pass an unfamiliar building, and for whatever reason you want to know more about it. Your phone pings the building’s dynamic self-description, and it replies to the effect that Andy Warhol had his Factory there between 1973 and 1984. If Wikitude chooses to share this particular piece of information with you, and not some other potentially germane factoid from the building’s history, on the strength of the fact that “The Velvet Underground and Nico” was in your last.fm playlist? That would constitute some small measure of context-awareness.

But you see how hard we had to try just to come up with an example, how forced it is, how so-what. And I have to say that – short of some infinitely supple system that really could model your innermost desires ahead of real time, and present appropriate responses to them – most so-called “context-aware” applications and services are like this. They’re either trivial, or wildly overambitious.

Maybe we don’t need for things to be context-aware for them to be useful, anyway. Certainly a great many objects in the world are starting to report their own status, and many more will do so in the fullness of time. And for the most part, all you’ll need to avail yourself of them is a Web browser running on a device that knows where it is in the world. An iPhone or an Android device will work splendidly – I called the iPhone “the first real everyware device” the day it came out and I was able to play with it for the first time – and in that way, the answer to your question is “yes.” Not to be longwinded or anything. ; . )

This Object has imperceptible properties. [Adam notes on his Flickr page: "This is a custom RFID-enabled transit pass that Timo Arnall had made up for me here in Seoul. I've (clumsily) tagged it with the icon that Nurri and I developed to represent just such emergent situations as this in the everyware milieu - that there's no way for anyone to understand that this object has puissance beyond the obvious simply by examining it."]

TS: It seems that  we are just at the beginning of understanding how to create networks of spimes (e.g. Pachube). Gavin Starks of AMEE (”the world’s energy meter”) once suggested to me that AMEE could be described as a facilitator of networked spimes (everything will have an energy identity). I think you may be familiar with AMEE because you keynoted next to Gavin at Xtech 2007.

I would be interested to hear your thoughts on AMEE?

When you discussed onto and ontome in this post, you noted:

“The greater part of the places and things we find in the world will be provided with the ability to speak and account for themselves. That they’ll constitute a coherent environment, an ontome of self-describing networked objects, and that we’ll find having some means of handling the information flowing off of them very useful indeed.”

Is the idea of “energy identity” that AMEE proposes an ontome? 

AG: See below for a précis of my feelings regarding environmental/sustainability initiatives, AMEE included. Uh…is AMEE an ontome? No. There’s just one ontome, and it’s coextensive with what folks now call the Internet of Things. It sounds like individual AMEE sensors would be “ontos.”

But I think the difficulty we’re having is a pretty good indicator that the terminology is more trouble than it’s worth. Sometimes a coinage, as satisfying as it may be lexically, just doesn’t work for people. These days I’m trying to get out of the neologism trade.

TS: I know when Usman Haque talks about Pachube he talks about spimes and spime wrangling. I asked Usman for his thoughts on spimes and onto/ontome and he gave me some comments.

Usman Haque: I think I had somehow missed the conversation about onto and ontome but backtracked through blog posts to piece it together (unfortunately some posts at v-2 and Studies & Observations no longer exist!). There are a couple of things that have made me uncomfortable about the word ’spime’: (a) the fact that it might be too easy to confuse with an “object”. A ’spime’ should also encompass relationships between things, and not just the “thingness” itself. (b) the sound of it (as Adam noted above). But then I am reminded of that horrible gooey interface used to plug into people in eXistenZ – it somehow seems appropriate that it should be a horrible gooey word, and not something that can disappear politely… So I like onto/ontome because it speaks to my first concern about ’spime’; but my second concern, it turns out, is not the problem I thought it was, and so onto/ontome might be… ahem… too euphonic! On the question of this thing people are calling the “Internet of Things”, I’ve tried in lectures to reframe it as the “Ecosystem of Environments”. Further, Vlad Trifa makes a delicious point that just as ‘web’ is different from ‘internet’, so too should we consider the “Web of Things” rather than the “Internet of Things”, something I agree with.

TS: It seems like this point about the difference between “the web of things” and the “internet of things” is pretty important?

AG: The parallel distinction between Web and Internet sure is! They’re two completely different things, right? And http is far from the only protocol that runs over the Internet. Now, as to what Vlad means by extending this particular distinction to the domain of networked objects, I don’t yet know, I haven’t had time to check it out. But sure, in principle I’d totally be willing to go along with the idea that there’s a meaningful distinction between two environments named that way.

No information is collected here; network dead zone

TS: I was just going over Greenfield’s principles of ubiquitous computing.  I am not sure that I see any current manifestations of ubicomp that hold to these priniciples yet?

AG: Oh, sure there are. Look at the work Tom Coates has done on Yahoo!’s Fire Eagle; look at Dopplr. And look at some of the steps other, less compassionate developers (e.g. Facebook) have been forced to take by their own users.

Look, those principles are just codifications of common sense and basic neighborly virtues, expressed in language appropriate to the domain of application. The best, smartest and most ethical developers have never needed guidelines to do the right thing. But especially inside companies and other complex organizations, people who want to implement compassion in their design of a technical system may occasionally find it useful to have some color of authority to invoke in their struggles. That’s all those five principles are there for, and I’m well satisfied that people have been able to use them that way.

Boffi’s take on the smart home- photo by Adam Greenfield

TS: In your post, More Songs About Context And Mood, you suggest a direction for interaction design that you point out is not far from Yvonne Rogers’ ideas in “Moving on from Weiser” about a switch in goal of ubicomp from Weiser’s vision of calm living (”computers appearing when needed and disappearing when not”) to engaged living – ubicomp technologies not designed to to do things for people but to help people engage more actively in things that they do (ensembles, ecologies of resources).

You also suggest interaction designers should be:

“parsimonious about the interaction design challenges our organizations do take on, with an eye toward reducing the complications of context (and the attendant opportunities for default, misunderstanding, misfire, time-wasting, and humiliation) to some manageable minimum.”

As you have pointed out, “we don’t do “smart” very well yet.” But paradoxically smart grids, smart homes, smart products etc. etc. are ubiquitously coming to market right now.

Yvonne Rogers suggests interaction designers should be:

moving from a mindset that wants to make the environment smart and proactive to one that enables people, themselves, to be smarter and proactive in their everyday and working practices

What areas might interaction designers most productively direct their attention towards?

AG: You note that things called “smart homes” and “smart products” are coming onto the market, and that sure would seem to be the case. But as to whether or not these things are genuinely smart, we don’t have anything more to go on than the marketing department’s word. I think you can already see that I tend to take language very seriously, and I really don’t uses like the “smart” here, or the “aware” in “context-aware.” They overpromise, they cannot help to set us up for failure and disappointment.

You know what I’d really like to see interaction design wrestle with? I would love to see a rigorous, no-holds-barred examination of the complexities of the self and its performance in everyday life, and how these condition our use of public space (and personal media in public space). I would love to see the development of ostensibly “social” platforms informed by some kind of reckoning with issues like vulnerability, dishonesty, the fact of power dynamics. In other words, before we deign to go about “helping” people, wouldn’t it be lovely if we understood what they perceived themselves as needing help with, and why?

I’d also pay good money to see talented interaction designers turn their efforts toward tools for the support of deliberative democracy, for the navigation of complex multivariate decision spaces, and for conflict resolution.

Locative is a mood – photo by Adam Greenfield

TS: I know you said this would take too long to explain but I couldn’t help noticing that you seem to be, perhaps, skeptical about the role of everyware can play in sustainable living and yet, it seems at the moment, in the hacker and business communities at least, the role of everyware in reducing carbon footprint/energy management etc, is the great green hope?

Will everyware enable or hinder fundamental changes at the level of culture and identity necessary to support the urgent global need – “to consume less and redefine prosperity?”

AG: I’m not skeptical about the potential of ubiquitous systems to meter energy use, and maybe even incentivize some reduction in that use – not at all. I’m simply not convinced that anything we do will make any difference.

Look, I think we really, seriously screwed the pooch on this. We have fouled the nest so thoroughly and in so many ways that I would be absolutely shocked if humanity comes out the other end of this century with any level of organization above that of clans and villages. It’s not just carbon emissions and global warming, it’s depleted soil fertility, it’s synthetic estrogens bioaccumulating in the aquatic food chain, it’s our inability to stop using antibiotics in a way that gives rise to multi-drug-resistance in microbes.

Any one of these threats in isolation would pose a challenge to our ability to collectively identify and respond to it, as it’s clear anthropogenic global warming already does. Put all of these things together, assess the total threat they pose in the light of our societies’ willingness and/or capacity to reckon with them, and I think any moderately knowledgeable and intellectually honest person has to conclude that it’s more or less “game over, man” – that sometime in the next sixty years or so a convergence of Extremely Bad Circumstances is going to put an effective end to our ability to conduct highly ordered and highly energy-intensive civilization on this planet, for something on the order of thousands of years to come.

So (sorry again, Bruce) I just don’t buy the idea that we’re going to consume our way to Ecotopia. Nor is any symbolic act of abjection on my part going to postpone the inevitable by so much as a second, nor would such a sacrifice do anything meaningful to improve anybody else’s outcomes. I’d rather live comfortably – hopefully not obscenely so – in the years we have remaining to us, use my skills as they are most valuable to people, and cherish each moment for what it uniquely offers.

Maybe some people would find that prospect morbid, or nihilistic, but I find it kind of inspiring. It becomes even more crucial that we not waste the little time we do have on broken systems, broken ways of doing things. The primary question for the designers of urban informatics under such circumstances is to design systems that underwrite autonomy, that allow people to make the best and wisest and most resonant use of whatever time they have left on the planet. And who knows? That effort may bear fruit in ways we have no way of anticipating at the moment. As it says in the Qu’ran, gorgeously: “At the end of the world, plant a tree.”

Biowall! - photo by Adam Greenfield

TS: In your post “Antisocial Networking,” you make some telling comments on the sorry state of social networking systems.

But you do also state:

Which of your concerns is social networking more central to than you might wish and why?

AG: Well, you know I’m interested in social interaction, interpersonal behavior, and in how these things play out in networked environments. There’s virtually no way for me to avoid dealing with Facebook, as wretched as I think it is.

Facebook is pretty hegemonic, in that its reach and influence extend further than the universe of people who use it. I bump up against it constantly, in a few different ways. People send me links I can’t access, because I’m not on Facebook. People spend time and energy trying to convince me that I’m really missing out, because I’m not on Facebook. The last few months, there’s even been a few people who feel justified in expressing some kind of exasperation, that they’re really pissed off…because they can’t find me on Facebook. It’s become the sovereign interface to any kind of life in public, and as a result a great many people don’t question its modes, tropes and metaphors.

So when it comes time to build some kind of situated interpersonal mediation framework, some kind of intervention in the fabric of the city, those are the tropes they reach for: accounts, profiles, friend counts, friendings and unfriendings, nudges and pokes. And as a member of a team tasked with the design of such systems, as a potential user of them, and certainly as someone exposed to the social rhetoric flowing downstream from their use, you bet these tropes become central to my concerns.

But what if we admitted that Facebook and the whole paradigm it’s built on are broken? What would things look like if we started from a more sensitive understanding of the interaction between self and others? Say, the understanding Erving Goffman was offering us as far back as the late 1950s? Then you’d understand the need for provisions like a “backstage,” a place to swap out one mask for another, the ability to present oneself differently to different communities and networks. That’s what I’m interested in exploring.

TS: Social networking systems in their current form are crude and express a very narrow bandwidth of human relationship. But already people are connecting everyware’s networked social acts to existing social networking systems. At the ITP winter show there was kickbee – networked fetal communication (and tweetmobile which used twitter as an acctuator for an ambient display) and green everyware (energy monitoring) is showing up in a number forms on existing social networks. But rather than just hooking up everyware to these existing flawed social network systems, does everyware require a reimagining of networked social interactions and social networking systems?

AG: That’s a great question, and I think the answer is clearly “yes.” It’s one thing to confine the consequences of that brokenness to the Web, and entirely another to let it bleed out into the world.

Does that mean any such reimagining is going to happen, that people will somehow refrain from plugging real-world outputs into these terribly flawed frameworks? Not a chance in hell. It’s too late to put a fence on that particular cliff. But maybe there’s still time to park an ambulance in the valley below.

“A graphic representation of a portion of the Earth’s surface, as seen from above” – photo by Adam Greenfield

TS: I saw you tweet that you met Usman Haque from Pachube recently. What do you find most interesting about Pachube and EEML? Will you design a project for Pachube to push the conversation further?  Did Usman ask you to take a role in the future of Pachube. How does Pachube enable the vision of The City Is Here For You To Use? I could go on for ever with questions,  so please do tell!

AG: OK, I should probably reiterate that my fundamental interest is in people, and in what they choose to make and do with technology, not the technology itself. For the last few years, I’ve particularly been trying to understand how people interact with each other and with the urban environments around them when those environments have been provisioned with the ability to gather, process and take action on data. And this is how I come about my interest in what Usman is up to with Pachube, because those “gather,” “process” and “take action upon” functions are generally accomplished by different systems, designed by different groups of people, at different times and to different ends. What Pachube aims to do is make the difficult and not-particularly-glamorous work of connecting these pieces a whole lot easier.

Think of it as a step toward enabling the ontome, this so-called Internet of Things we’ve been talking about, the same way basic protocols like HTTP and HTML enabled the wildfire spread of the Internet we’re familiar with. What Pachube offers is a way – a relatively straightforward and self-explanatory way – to plug any given compatible input into a similarly compatible output. So if you’ve got an air-quality sensor or a soil-pH sensor or a personal biometric monitor, you can plug it into Pachube, and someone else can grab the data those things generate and use it to drive a visualization, or the state of a physical system like a window, or whatever else they can imagine. It’s as close as anyone’s yet come to providing a plug-and-play backbone for the creation of responsive environments.

And I think it’s absolutely brilliant that it’s designed to work with Arduino and Processing, two lightweight, open-source frameworks that hobbyists and researchers (and even one or two more serious developers) around the world are already using to build things. (Arduino’s a kit of parts for doing basic physical computing – using data to drive lights, motors, and other actuators that have effect out here in the world – while Processing is a very accessible language to do dynamic and interactive graphics for screen-based media). Given both its openness and modularity, and its willingness to build on top of the very popular frameworks that already exist, I’m very excited to see what people make of and with Pachube.

I have to be honest and admit that personally, I couldn’t really care less about the environmental angle, for reasons that I went into at embarrassing length above. What I’m engaged by in Usman’s work is the idea that Pachube is helping to create an open platform for people to share data more readily. And while, no, he hasn’t explicitly asked me to take any particular stake in things, I’m always happy to lend a hand in whatever way would be most useful. I think it’s a project worth supporting.

As to how Pachube enables some of the ideas in The City Is Here, the answer has to do with the book’s call for every “public object” – every lamppost, bus shelter, commercial façade, and so forth – to support an open API. Something’s got to string all those objects together, present them to people as resources to be taken up and used, and Usman’s offered us a critical first step in that direction.

TS: Usman suggested, it might be interesting to ask you about “the tension between ‘could’ and ’should.’

Usman Haque: There are a whole bunch of things that we “can” do, technologically speaking; how do we decide what we ’should’ do, as we find ourselves in an age where we can build almost anything we can imagine…? particularly with reference to technology/privacy/security triumvirate. e.g., leaving aside that the majority of the world is *not* in the technology ‘paradise’ that we’re in, here in the west, only a small fraction of people are currently producing the technology that the rest of us use; one aim is to get people more engaged in the productive process, but, in a sense that will also mean the whole wide ecosystem of technology will be even bigger, both “good” stuff and “bad” (that qualification firmly placed on how it’s used), as opposed to now when we can focus on quite specific things that government & industry are doing and saying “that shouldn’t be happening….”. part of this relates to something adam said on his blog  in the comments (see here).” 

AG: I think the first part of answering that question has to involve figuring out who “we” are in any given situation. A “we” composed of seven Helsinki-based Linux developers would most likely arrive at very different answers than the United States Air Force Materiel Command or Samsung’s board of directors, right? So clearly, a first challenge is getting to some kind of pragmatically useful alignment between those local and occasionally even painfully parochial perspectives with what’s best for the Big We. And this challenge is only going to become more vexing as the ability to imagine, design, build and deploy informatic componentry gets more and more widely distributed. In this respect the spread of simple, modular, low-barrier-to-entry tools only makes things worse!

The primary issue that I can see here is that the inherent clock speed of technical development is so very much faster than that of any meaningful deliberative process “we” might bring to bear on it. A concomitant concern is that the sources of technical innovation and production are now so widely distributed that you can be reasonably certain that somebody, somewhere will implement any given technically feasible idea, no matter how offensive, poorly thought-out, socially disruptive or frankly stupid. A public toilet you have to SMS to unlock and use? A “Friend Finder” visualization with high locational precision and no privacy features whatsoever? A first-person rape-simulation “game”? A clunky brown iPod knockoff? Somebody thought each one of these things was worth the time, expense and effort to actually go about making it. They exist.

But I’m pretty old-fashioned in some ways, in that I think the good old Habermasian idea of the public sphere still has some life left in it. And I think it should be self-evident by now that there’s no necessary contradiction between even the newest (cough) “social media” and the formation of such a sphere. So you’ve provided a forum, and in it I get to express my belief that these things are stupid and pointless and probably should not have been built. And if somebody gets all het up about that, they can argue right back at me in comments. And eventually one or another of these positions begins to tell, in terms of regulation, legislation, and other tools of the juridical order, in terms of protest campaigns or organized boycotts or litigation…in terms of nonexistent sales!

There’s nothing new in any of this, of course, though indubitably some of the dynamics are amplified or accelerated by e-mail, Twitter and YouTube. My main contention is that informatic technology now has such deeply pervasive implications, and for things like presentation of self that previous waves of technical development barely touched, that “we” as societies need to be very much more conscious of the consequences before committing to any one course of action.

I should also point out that I do not, at all, believe that we’re “in an age where we can build almost anything we can imagine,” though I might buy “…two or three of almost anything we can imagine.” On the contrary, as I implied above, I think the global constraints on our ability to operate freely are already becoming quite evident, and will continue to grow teeth over the next few decades.

TS: Also Usman  added …

Usman Haque: ..where Adam said: in this regard, I very much *do* have a problem with “just showing up.” — something I feel that as well. but i always wonder: What happens when one appears to be mandating participation…?

AG: Look, I happen to have a strong – maybe some would say obnoxious or hyperactive or overdeveloped – sense of personal responsibility and accountability. I think one is basically committed to some measure of responsibility for the commonweal simply by surviving to the age of majority. The choice of how, particularly, to discharge that responsibility can only be yours and yours alone, but it can’t be ducked or gotten around without severe and entirely predictable consequences. So to Usman I’d respectfully suggest that I’m not the one mandating participation. Life is.

TS: It seems we have grown accustomed to striking a Faustian bargain on the internet today -  in order to share and distribute parts of our identity we are expected to give up key information to one site to store and disperse our data. I took part in a discussion with David Levine, IBM and Eben Moglen on privacy last year.  And Eben Moglen gave a succinct description of the elements of privacy and how they have been treated in the American Constitution that is, I think, relevant to unpacking some of the challenges of ubiquitious computing. Here are some extracts from that conversation where, Eben notes:

there are three elements that are mixed up in privacy and we tend not to notice which one we are talking about at any given moment.

There is secrecy – that is the data should not be readable by or understandable by anybody except me or people I designate. There is anonymity which is the data can be seen by anybody but about whom it is should be knowable only by me or people that I designate. And there is autonomy which isn’t about either secrecy or anonymity but which is about my right to live under circumstances which reinforce my sense that I am in control of my own fate. And this form of privacy is actually the one we talk about in the constitutional structure when we talk about the right to get an abortion or use birth control.

“Anonymity” is a condition that is a deep structuring characteristic of the internet as you, Lessig and others have commented on.  And frequently we are promised (questionably) “secrecy” or anonymity as privacy protection by services handling our data on the internet.  But Eben (one of the US’s great constitutional lawyers) points out that “autonomy” is a key form of privacy in the  US constitutional structure that is often compromised in situations where our digital selves may constrain our non-digital selves.

The real issue here is about the forcing of choices on us…digital aspects of identity can quickly acquire an inflexibilty that constrains our non-digital selves.

I see again and again the ways in which people now find themselves unable to make certain life choices easily because there digital self has acquired an inflexibility that constrains their non-digital self.

As we go beyond the end to end internet and we lose the structuring characteristic that has privileged anonymity: How do you see these three elements of privacy, anonymity, secrecy and most importantly autonomy, being worked out in a networked world beyond the end to end internet?

Are there any new structuring characteristics that could privilege autonomy? (which Eben indicates is linked to having a flexible identity).

AG: If we accept for the moment a definition of autonomy as a feeling of being master of one’s own fate, then absolutely yes. One thing I talk about a good deal is using ambient situational awareness to lower decision costs – that is, to lower the information costs associated with arriving at a choice presented to you, and at the same time mitigate the opportunity costs of having committed yourself to a course of action. When given some kind of real-time overview of all of the options available to you in a given time, place and context – and especially if that comes wrapped up in some kind of visualization that makes anomaly detection and edge-case analysis instantaneous gestalts, to be grasped in a single glance – your personal autonomy is tremendously enhanced. Tremendously enhanced.

But as to how this local autonomy could be deployed in Moglen’s more general terms, I don’t know, and I’m not sure anyone does. Because he’s absolutely right: Bernard Stiegler reminds us that the network constitutes a global mnemotechnics, a persistent memory store for planet Earth, and yet we’ve structured our systems of jurisprudence and our life practices and even our psyches around the idea that information about us eventually expires and leaves the world. Its failure to do so in the context of Facebook and Flickr and Twitter is clearly one of the ways in which the elaboration of our digital selves constrains our real-world behavior. Let just one picture of you grabbing a cardboard cutout’s breast or taking a bong hit leak onto the network, and see how the career options available to you shift in response.

This is what’s behind Anne Galloway’s calls for a “forgetting machine.” An everyware that did that – that massively spoofed our traces in the world, that threw up enormous clouds of winnow and chaff to give us plausible deniability about our whereabouts and so on – might give us a fighting chance.

TS: The concept of autonomy is signaled clearly in the title you have chosen for your next book, The City Is Here For You To Use, and is a theme of all your writing!  While you talk about many of the possible constraints to presentation of self and potential threats to a flexible identity that ubicomp poses, your next book signals optimism. What are your key grounds for optimism?

AG: It’s not optimism so much as hope. Whether it’s well-founded or not is not for me to decide. I guess I just trust people to make reasonably good choices, when they’re both aware of the stakes and have been presented with sound, accurate decision-support material.

Putting a fine point on it: I believe that most people don’t actually want to be dicks. We may have differing conceptions of the good, our choices may impinge on one another’s autonomy. But I think most of us, if confronted with the humanity of the Other and offered the ability to do so, would want to find some arrangement that lets everyone find some satisfaction in the world. And in its ability to assist us in signalling our needs and desires, in its potential to mediate the mutual fulfillment of same, in its promise to reduce the fear people face when confronted with the immediate necessity to make a decision on radically imperfect information, a properly-designed networked informatics could underwrite the most transformative expansions of people’s ability to determine the circumstances of their own lives.

Now that’s epochal. If that isn’t cause for hope, then I don’t know what is.

Newson Obamanook – photo by Adam Greenfield, “The fact that it was one of the happiest days of my adult life may have colored my appreciation of this space. A bit, anyway.”

TS: In your writing you seem to imply that we will not find answers to our new relationship with Everyware by transposing the internet onto things for convenience’s sake but rather like the bike messengers -  we must explore the rich and complex terrain of the city that is ours to use in a give an take relationship.  Through our own exertions we find- how “anything reasonably smooth and approximately horizontal can become a thoroughfare,”  rather than be served up the city as something for us to consume.

You seem to be suggesting our city becomes ours to use because of the way we use it in our personal journeys -like “the messenger subconsciously maps the contours of an economic geography – known sources and sinks of courier assignments, or “tags” – and a threat landscape, this latter comprised of blind corners, cable-car and metro tracks, and traffic lanes.

But bike messengers are the lone ranger of our big cities. Others surf the city in tribes that ride the roiling tides of highly networked information together. How are the “natural” gestures of these tribes, e.g. day traders, who yoked to the tracings of a hive mind, part of the city that is here for us to use?  I thought the comment @ginsudo made shortly after joining Twitter and setting up TweetDeck particularly poignant:

“watching Tweetdeck is like watching stock market of your personality ebb and flow. needs analytics to maximize inherent self-involvement.”

But, for many of us our work has more in common with the day trader than the bike messenger, and are we pretty hooked on the ever growing possibilities for “contact” and identity sharing/construction, social media has produced  (with all the”Here Comes Everybody,” C. Shirky, benefits and risks).  Early theorizing of a “calm,” invisible” ubicomp seems out of synch with the excitable, active, engaged, contact driven, “users” that are watching stock market of their personality (or personal brand) ebb and flow.

How will these excitable/exciting processes of contact and identity sharing that have captured of a pretty large segment of popular imagination (not confined to the West -services like Gupshup does much of the same curating, linking and distributing of identity that web based social media does in SMS) be/ or not be part of The City Is Here For You To Use?

AG: Let’s remember that ubicomp itself, as a discipline, has largely moved on from the Weiserian discourse of “calm technology”; Yvonne Rogers, for example, now speaks of “proactive systems for proactive people.” You can look at this as a necessary accommodation with the reality principle, which it is, or as kind of a shame – which it also happens to be, at least in my opinion. Either way, though, I don’t think anybody can credibly argue any longer that just because informatic systems pervade our lives, designers will be compelled to craft encalming interfaces to them. That notion of Mark Weiser’s was never particularly convincing, and as far as I’m concerned it’s been thoroughly refuted by the unfolding actuality of post-PC informatics.

All the available evidence, on the contrary, supports the idea that we will have to actively fight for moments of calm and reflection, as individuals and as collectivities. And not only that, as it happens, but for spaces in which we’re able to engage with the Other on neutral turf, as it were, since the logic of “social media” seems to be producing Big Sort-like effects and echo chambers. We already “maximize inherent self-involvement,” analytics or no, and the result is that the tools allowing us to become involved with anything but the self, or selves that strongly resemble it, are atrophying.

So when people complain about K-Mart and Starbucks and American Eagle Outfitters coming to Manhattan, and how it means the suburbanization of the city, I have to laugh. Because the real suburbanization is the smoothening-out of our social interaction until it only encompasses the congenial. A gated community where everyone looks and acts the same? That’s the suburbs, wherever and however it instantiates, and I don’t care how precious and edgy your tastes may be. Richard Sennett argued that what makes urbanity is precisely the quality of necessary, daily, cheek-by-jowl confrontation with a panoply of the different, and as far as I can tell he’s spot on.

We have to devise platforms that accommodate and yet buffer that confrontation. We have to create the safe(r) spaces that allow us to negotiate that difference. The alternative to doing so is creating a world of ten million autistic, utterly atomic and mutually incomprehensible tribelets, each reinforced in the illusion of its own impeccable correctness: duller than dull, except at the flashpoints between. And those become murderous. Nope. Unacceptable outcome.

Uncanny Valleys - Adam comments,”Our apartment in NYC as rendered in Google Earth, with realtime traffic, weather, daylight and shadow as well as geodetic, street grid and service overlays. Camera view is South; that’s First Avenue just left of center-screen.”

TS: Smart phone’s are now drawing everyware data into the system and the net is reaching into who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc..

Nathan Freitas says Android: “seems to be the platform most likely to socialize the idea that sensor data could be a piece of every application.” (Android APIs for a wide range of sensor data.)

What in your view will be the most likely platform, Android or what?, to socialize the idea that sensor data could be a piece of every application?

AG: An open platform. A platform with lots of hooks and ways to plug things into it, a strong developer community, a shallow learning curve and/or an easy-to-use, high-level development environment.

I don’t have a dog in this race, mind you. I couldn’t care less who gets there first.

TS: New location based services, e.g., Xtify and ViaPlace, are offering us ways to share location data across lots of different applications (eg Xtify and a dating application like MeetMoi ). In return for services that allow us to share information, we must give up key information up to one site to store and disperse (although there are many differences in approach to our data, from the Twitter stance “show but don’t own” as opposed to Facebook’s stance – “in order to show we must have rights to it”). But the basic model of Twitter – to provide a white noise platform for people to build service on top off seems to be being transposed to location based services. Obvious questions arise like what happens to our data in a start up like MeetMoi if they go belly up?  Apparently in the dot.com bust data was the first thing to go on the auction block in bankrupcy cases.

Also, I suppose it is hardly surprising (if disappointing to me) that some of the early location based services are trying to get mindshare by picking up on the glue celebrities give to mass culture. At the last New York Tech Meetup, OMGICU demoed a rather terrifying new pre-launch location based “participatory celebrity gossip application” which seems to combine all the worst features of social media with celebrity stalking, plus a narrative to change the notion of celebrity itself by “turning D listers into A listers.”

Hopefully location based applications  will not get stuck on “stalker, stalker, stalker” apps like OMGICU .

David Oliver, Oliver Coady gave me a good question: “How does timeliness and location-independence change our ideas of social media?

And how can we design new architectures that can reinforce the sense that I am in control of my own fate?

AG: But we’ve already come so far in terms of turning D-listers into A-listers! On a daily basis, I’m exposed to almost as many cues insisting I attend to nonentities and dullards like Robert Scoble as those insisting I attend to nonentities like Madonna or Thomas Friedman. It’s gotten ridiculous.

Now, how does timeliness and location change our ideas of social media? It makes them dangerous!

Look, even a proud Z-lister like myself – I’m a public person only in the most debased and degraded meaning of that word – I’ve had experiences that shook me up, like having someone approach me while I was quietly hanging out in the back of St. Mark’s Books, and wanting to strike up a conversation based on some talk they’d seen me give a year or so previously. Now part of learning to deal with this kind of thing is shrugging it off, being grateful and flattered that someone thinks you’re interesting enough to single out for that kind of attention, or chalking this up to Sennett’s observation about the constitution of urbanity. Or doing all three at once.

But let’s remember that at the end of the day, a “social network” is nothing but a group of arbitrarily distributed human beings joined by a communications channel, and those people have eyes and ears. The degree to which they recognize some shared interest gives them significance filters. If social capital accrues to those in the network who are able to claim some connection with a “celebrity,” no matter how fleeting, then such connections are going to be mobilized, made explicit. And now say the network has been provided with the tools allowing it to plot the appearances of those putative celebrities in space and time, and what do you get? You get a circumstance in which it is very, very difficult to maintain any membrane between the private self and the world, for anyone who’s even remotely a public figure, whether they particularly want to be a public figure or not. You get network effects that amplify those locational traces, and further undermine any possibility of anonymity, even anonymity-by-suspension-of-interrogative-awareness (which is a clumsy way of referring to that blasé matter-of-factness around famous people that most big-city folks eventually develop).

Am I letting myself off the hook? Not in the slightest. I passed Terence Stamp on the street not so long ago, and you bet I Twittered it. My only excuse was that I Twittered it to a closed loop of no more than a few dozen people. But then, who knows what those few dozen people will turn around and do with that fact, on the open networks to which they in turn belong? And that, too, is my responsibility.

I’m not sure there’s anything to be done about any of this but cultivate our own urbanity, learn to say “so what” when we happen to find ourselves next to Philip Seymour Hoffman in the line at Whole Foods.

TS: Zittrain in The Future of the Internet: And How To Stop It, foregrounds “generativity” and a generative devices (as opposed to appliances) as the most fortuitous starting point for: “tools to bring about social systems to match the power of the technical one.”

Are appliances a threat to the city that is here for you to use? How can generativity ensure The City Is Here For You To Use as Zittrain argues it has ensured, even if imperfectly, that the internet has been here for us to use?

AG: You know, I haven’t read the book, I’ve only heard him give the talk, so it’s certainly possible there’s a subtlety to the argument that I’m missing. But I’m not sure Jonathan isn’t simply wrong about this notion of generativity. Not that the concern is misplaced, but that he’s insufficiently trustful in human agency. Is a car “generative,” by his definition? Certainly not. And yet look at all the cultural production that goes on around “the car,” look at all the assemblages people make with cars, from Beach Boys songs to ghost riding the whip, from J.G. Ballard novels and Herbie the Love Bug to Tokyo Drift.

Or probably more to his point: look at the Japanese mobile-phone market – seemingly one of the most locked-down and unpropitious circumstances imaginable for the production of culture, in technical terms and Zittrain’s both. And yet fully 50% of the bestselling books in Japan last year were written on mobile phones. Not read, which would already be impressive enough (if “impressive” is indeed the word): written. What does that imply for his argument?

So, yes, I think there are grounds for concern in that we don’t allow technologies and frameworks to appear that unduly limit the scope of human creativity. Code is still law. But I also think people are quite amply able to reach into what would appear to be the least propitious technologies and tell their own stories with same.

TS: One aspect of Everyware that seems in need of some visionary yoga is the how we will relate to pixels anywhere.

In Urban Computing and its Discontents you mention how our technological trajectories often make it seem as if we seem to get fixated on particular scenes in movies, e.g., Minority Report. You point out that so many ambient informatics projects seem simply “to expand the reach of signage and advertising in dense urban spaces….as if we’ve become transfixed by the scene from Minority Report where heterosexual cop John Anderton is on the run from his colleagues.”

Ideas from the Minority Report continue to hold sway in designs as we saw in the recent MIT demo of SixthSense at TED.

But visions of augmented reality were pretty high profile in this years Super Bowl commercials this year (including a highly anthropomorphic imagining of ubicomp that was a kind of WoW mashup with a Pixar movie).

What recent movies/commercials have produced scenes mostly likely to be are new fixation fodder for ubicomp and why?

AG: I don’t think I’m qualified to answer that, actually. We don’t have a TV, so I don’t see much in the way of commercials, and most of the films I wind up seeing are the kind that play at Anthology Film Archives. What I can say is that science fiction is currently suffering in toto from an inability or disinclination to posit future scenarios that are any weirder or more visionary than those emerging from other sectors of the culture. And that would be fine, except sf has traditionally been the place where we wrestled with the imaginary.

We need that set of tools, badly. If for no other reason than something I glean from personal experience: essentially my entire professional career has simply been the leveraging of ideas and concepts I originally wrestled with in the encounter with William Gibson and Bruce Sterling when I was 16. Today’s visionary sf means tomorrow’s halfway-competent generalist.

Nurri Kim in the waiting zone – photo by Adam Greenfield

TS: My AR friend, Robert Rice, who is working on a markerless AR platform, notes that data visualization is one of the critical elements of AR in terms of “make or break.” Robert says, “even with the ultimate in ubiquitious data from everything, without good data vis it will all be useless”

Also something Cory Doctorow said to me last year has really stuck in my mind. When I asked him what happens when Cyberspace everts, he talked about a reverse surveillance society:

Cory: Well this is like Spook Country the new Gibson novel – What happens when cyber space everts – hmmm? I’m not sure I have anything very pithy to say on that EXCEPT………
Apart from all the traditional kind of overlay reality stuff, if there is one thing I am actually interested seeing from a virtual world migrating to the real world its instrumentation.
I think lot of things that are characteristic of very successful internet based business is that they are extremely finally instrumented so like Amazon knows in aggregate on a second by second basis how their site is being used by people and they can twiddle the dials in real time.

As users of the world we have very little access to that kind of instrumentation. We don’t even know how the tube is running. The tube knows how the tube is running and we kinda of don’t. I would be really interested in seeing that. You’ve seen Joi Ito’s WoW interface right. Have you seen it …

Joi Ito’s WoW interface seems a long way from the calm, invisible imaginings for ubicomp by early ubicomp visionaries?

AG: Well, he’s got a particular kind of neural wiring. And there’s not a thing that’s wrong with that, except that I’d never, ever want to assert that what’s appropriate for Joi Ito necessarily is or should be understood to be appropriate for anybody else. The point of calling for open systems and frameworks is to allow us maximum scope of diversity in the ways we choose to interface with the world’s richness and complexity.

TS: What new imaginings/possibilites do you see when pixels anywhere are linked to everyware?

AG: Product placement. Commercial insertions and injections, mostly.

Beyond that: one of the places where Mark Weiser logic breaks down is in thinking that the platforms we use now disappear from the world just because ubiquitous computing’s arrived. We’ve still got radio, for example – OK, now it’s satellite radio and streaming Internet feeds, but the interaction metaphor isn’t any different. By the same token, we’re still going to be using reasonably conventional-looking laptops and desktop keyboard/display combos for awhile yet. The form factor is pretty well optimized for the delivery of a certain class of services, it’s a convenient and well-assimilated interaction vocabulary, none of that’s going away just yet. And the same goes for billboards and “TV” screens.

But all of those things become entirely different propositions in everyware world: more open, more modular, ever more conceived of as network resources with particular input and output affordances. We already see some signs of this with Microsoft’s recent “Social Desktop” prototype – which, mind you, is a very bad idea as it currently stands, especially as implemented on something with the kind of security record that Windows enjoys – and we’ll be seeing many more.

If every display in the world has an IP address and a self-descriptor indicating what kind of protocols it’s capable of handling, then you begin to get into some really interesting and thorny territory. The first things to go away, off the top of my head, are screens for a certain class of mobile device – why power a screen off your battery when you can push the data to a nearby display that’s much bigger, much brighter, much more social? – and conventional projectors.

Then we get into some very interesting issues around large, public interactive displays – who “drives” the display, and so forth. But here again, we’ll have to fight to keep these things sane. It’s past time for a public debate around these issues, because they’re unquestionably going to condition the everyday experience of walking down the street in most of our cities. And that’s difficult to do when times are hard and people have more pressing concerns on their mind.

Citywarecrash – photo by Adam Greenfield, “An occupational hazard for urban screens.”

TS: I know in Everyware you mentioned that architects have play an important visionary role in imagining ubicomp and I know you work closely with your wife, artist Nurri Kim.  Robert Rice asked me the following question – which I will in turn ask you: “In terms of augmented reality do you think virtual worlds and virtual reality experts / leaders / are good pioneers for thought and guidance on AR? Or, should we look for new leaders, or where are new leaders emerging? Is the tech similar enough for the old crowd to be useful or is it different enough to be a disadvantage coming from the old models?.”

AG: I should make it clear that I have absolutely no interest in virtual worlds or virtual reality. The so-called virtual worlds I’ve experienced seem sad and really rather tatty – eversions of the most predictable adolescent fantasies of unlimited power, reinscriptions of all the usual politics – and completely lacking in just about everything that makes life resonant, meaningful and awe-inspiring. And anyway, to paraphrase J.G. Ballard, ordinary, everyday life is now far more vividly and fantastically weird than anything you’ll see in Second Life. I mean, Garry Kasparov was heckled by a radio-control dildocopter, Joe the Plumber’s off to Gaza as a war correspondent, a sea of dust-covered BMWs waits in the long-term parking lot at Dubai International for owners who are never, ever coming back.

Look to virtual worlds for insight into the hard work of negotiating the actual, with its physics, its entropy, its suffering, with all its constraints? Oh my goodness gracious, no.
And look to leaders? Never. Leaders are for followers, and who wants to be that? I don’t mean you can’t take inspiration and insight from the work of others – not at all – but use your own imagination, take some personal risk, do your own damn work.

Now, having said that. This opposition of virtual and physical worlds strikes me as increasingly a false one, as it does many people. The hard-and-fast distinction between “the real world” and virtual environments make less and less sense, as righteously satisfying as making it can sometimes seem. There may be attributes of this physical environment that are impossible to see or make use of without access to the networked overlay, and those attributes may in time come to constitute the primary wellsprings of a given place’s meaning. And if you’re offering me some insight that I think could be of utility in resolving the challenge of making this overlay accessible to all, equally, I’ll gladly accept it, no matter what domain or disciplinary background you claim as your own.

Am I aware of any such insight coming out of virtual worlds? No. As Bryan Boyer notes, “If you want to start talking about some serious cross-disciplinary pollination then you better take both sides of that disciplinary divide seriously. When your ubi- runs into my building with its boring HVAC, mundane load paths, typical finished floors, plain old foundations, etc., the transformative powers of comp are bracketed pretty seriously by the realities of the physical world.”

The Cloud Gate has landed – photo by Adam Greenfield, “Tell me this doesn’t look *just* like the descriptions of “stasis fields” in 70s SF. In fact, the picture looks practically CGId to me.”

TS: Some people thought the whole world would have been plastered with RFID by now.  But before that has happened markerless AR seems to be in our sights.

If I understand it correctly marker versus markerless AR has quite different implications for how the cyberspace of ubicomp evolves?  I asked Robert Rice (he is developing a markerless AR platform) to explain some of the differences.  He said:

Robert also pointed out to me that markerless AR may prove even more problematic for privacy:

Do you have any thoughts about the different roles that markerless versus marker techinologies will play in AR and Ubicomp?

AG: I need to admit that I’ve never until this moment heard the phrase “markerless AR,” although I’d think it’s more or less self-explanatory to anyone who’s been following this stuff. Let me make the distinction explicit, shall I, for anyone who hasn’t been? And you or Robert can correct me if I’ve gotten it wrong.

Augmented reality means that I have some mediating artifact that provides me with a visual overlay on the world. This could be a phone, it could be a windshield, it could be a pair of glasses or contact lenses, doesn’t matter. And you’re going to use that overlay to superimpose some order of information about the world and the objects in it onto the things that enter my field of vision – onto what I see. So far, so good: that’s AR 101.

Now where does that information come from?

What you’re calling marker-based AR implies that there’s some reasonably strong relationship between the information superimposed over a given object, and the object itself. That object is an onto, a spime, it’s been provided with a passive RFID tag or an active transmitter. And it’s radiating information about itself that I’m grabbing, perhaps cross-referencing against other sources of information, and superimposing over the field of vision. Fine and dandy.

But there’s another way of achieving the same end, right? Instead of looking at a suit jacket on a rack and having its onboard tag tell you directly that it’s a Helmut Lang, style number such-and-such from men’s Spring/Summer collection 2011, Size 42 Regular in Color Gunmetal, produced at Joint Venture Factory #4 in Cholon City, Vietnam, and packed for shipment on September 3, 2010, you’re going to run some kind of pattern-matching query on it. And without the necessity of that object being tagged physically in any way, you’re going to have access to information about it. But this set of information isn’t, necessarily, what the object itself, or its creators or merchandisers, want you to know about it; it could be derived from online discussion fora or review sites, or blog posts, or whatever. All there needs to be is a lookup table, essentially, that tells you where to find information about any object in the field of vision whose identity can be established.

Do I have that right? And if I do, then as I understand it, the distinction is primarily a pragmatic one: it’s just easier to get to an augmented world, by far, if we don’t actually have to go to all the trouble of tagging everything in the world with its own dedicated RF transponder. Easier, and cheaper, and quicker, and more environmentally sound besides, because the relevant traffic is in bits not atoms.

Unless I’ve missed something, you don’t, then, get the distinction between classes of objects and instances of same. Sometimes, when there’s a 1:1 correlation between the two, that’s not going to matter: I’m walking down the street in Madrid, and my glasses or whatever can easily recognize that this building is the Caixa Forum. There’s only one of it, and I can get a positive ID via pattern recognition. But for some edge cases – twins and lookalikes, mostly – the same thing is generally true of people.

But other times it will matter. Is this specific watch a real, $10,000 Panerai or a $50 Kowloon fakery? How has this black 1998 Honda Civic over here differ from this other one in terms of its use and maintenance history? Does this O-ring gasket need to be replaced? I don’t see how you extract data from specific instances of things without the necessary sensor instrumentation, transmitter, etc., being coextensive with the object in question or very closely colocated with it over time – in the terminology you’re using, a “marker.”

So using these terms, I’d say that “markerless” AR comes first, is relatively easy to deploy, and generates not-insignificant value. But – again, unless I’m missing something – there are some things that it won’t ever be able to do, and for those things you need some provision for self-identification and self-location.

Ultimately I think it’s a distinction without a difference, from the user’s point of view. People will care much more about the source of whatever information shows up on their overlay than the precise technical means used to get it there.

The surrender to cynicism – photo by Adam Greenfield

TS: Much early thinking around ubicomp seems to have come from visionary architects and engineers but recently I was at the O’Reilly Tools of Change for Publishing Conference (publishing in the Digital Age) and I met several book futurists.  It struck me how ubicomp from the perspective of the book created some interesting questions for how particular material cultures will shape and be shaped by Ubicomp differently.

I noted, Google seemed well down the path to holy grail “converting images to original intent XML.” And Peter Brantley talked about machine parsed books.

At TOC there were many suggestions about how books might manifest as everyware. (Although it did not seem that many people felt books had a special relationship to time and history and would not vanish as one of the great metaphors of calm and solitary enjoyment in our culture soon).  Books as everyware will, it seems, include, amongst other things:

books that read books

books that read context

context that reads books

books that read me

books linked to mobility – timeliness and location independence

books that are not books

books becoming babble

books bubbling up from the babble

There is an Institute of the Future of the Book. Will all former material cultures require their own institutes of the future to guide their cultures into everyware?  Do you think books transition into everyware is especially significant and why?

AG: But all objects have a relationship to time and history, no?

TS: Yes! What I meant to convey really was the idea that many people expressed at TOC that books had a privileged relationship to knowledge in our culture that was valuable and related to some aspects of their current form, and that books as everyware, e.g. machine parsed books, and more sociallly generated forms would not replace that entirely.

AG: Gotcha. Well, I certainly agree that books constitute an interesting category unto themselves – I’ve held onto my physical books, and in fact still spend a fortune buying new ones, where I stopped buying music on discs a long, long time ago. But I don’t think this state of affairs can or should obtain forever.

Lately there’s been a good amount of thought around the notion of “unbooks,” which I regard as a container for long-form ideas appropriate to an internetworked age. By building on some of the tropes of software development, mostly having to do with version control, open-endedness and an explicit role for the “user” community, unbooks can usefully harness the dynamic and responsive nature of discourse on the Web. At the same time, you preserve the things books are really good at: coherence, authorial voice and intent.

The important part is in acknowledging two points which have usually been understood as contradictory, but which are actually nothing of the sort: firstly, that the expression of ideas in written form has something to learn from the practices that have evolved around the collaborative creation of dynamic, digital documents over the half-century-long history of software; and secondly, that certain ideas require elaboration in the reasonably strongly-bounded form we know as a “book,” and cannot meaningfully be shared otherwise. A third point, concomitant to the second, is that despite recent technical advances, screen-based media still cannot, and may not ever fully be able to, deliver the extratextual cues and phenomenological traces that support, inform and extend the meaning of written documents.

The unbook lets you have your cake and eat it too. So, for example, when we publish The City Is Here, one of its manifestations will be a static, physical document – and hopefully, if we do our jobs well, a very nice one indeed. But even before that, you’ll be able to download a Creative Commons-licensed PDF of every numbered version of the manuscript, from zero onward. Bottom line: you buy the book if, and only if, you want the object. The ideas are free.

TS: David Brin sees two futures:1) the government watches everybody, and 2) everybody watches everybody (the latter he calls “sousveillance”).  My friend Ben Goertzel says “hooking AI up to a massive datastore fed by ubicomp is the first step toward sousveillance?” What do you think the role of AI in ubicomp will be?  Is it worth thinking about what is the first important “AI meets AR” app is?

AG: I don’t believe that artificial intelligence as the term is generally understood – which is to say, a self-aware, general-purpose intelligence of human capacity or greater – is likely to appear within my lifetime, or for a comfortably long time thereafter.

Having said that, your friend Ben seems to be making the titanic (and enormously difficult to justify) assumption that a self-aware artificial intelligence would share any perspectives, goals, priorities or values whatsoever with the human species, let alone with that fraction of the human species that could use a little help in countering watchfulness from above. “Hooking [an] AI up to a massive datastore fed by ubicomp” sounds to me more like the first step toward enslavement…if not outright digestion.

Sousveillance - the term is Steve Mann’s, originally – doesn’t imply “everybody watching everybody” to me, anyway, so much as a consciously political act of turning infrastructures of observation and control back on those specific institutions most used to employing same toward their own prerogatives. Think Rodney King, think Oscar Grant.

TS: I have one last question from Usman Haque.

Usman Haque: insofar as a lot of what adam describes as desirable could be said to constitute pretty radical socio-political change (or perhaps… “adjustment”) i would be really interested to know how his current work @ nokia is or isn’t able to gel with the themes of his writing. in some senses there’s quite an undercurrent strongly challenging corporate practices, in other senses it could be seen as gentle nudges. how does adam see it? and how about the nokia behemoth? does he have success nudging nokia towards the kind of world he would like to see (i imagine the answer is ‘yes’ otherwise he wouldn’t be doing it…) but i’d love to know more about the limits/challenges.

AG: I am told that Henry Kissinger, on his first trip to China in 1971, asked Zhou Enlai whether he thought the French Revolution had or had not advanced the cause of human freedom.
Zhou thought for a moment, pursed his lips, and replied, “It is too soon to tell.”