We are in the age of unearthing and uncovering data, and only just at the beginning of the age of processing data and dealing with it (see my interview with Anselm Hook, Part 2 upcoming).  O’Reilly’s Strata Confernence 2011, will explore, “the change brought to technology and business by data science, pervasive computing, and new interfaces.” It is, perhaps, one of the most important events of 2011.

Data is driving a revolution much as coal, oil, and steel powered the industrial revolution.  And the world changing insight from Karl Marx that “the industrial revolution polarized the world into two groups: those who own the means of production and those who work on them,” is taking on on new life, as  Alistair Croll, co-chair of Strata 2011, points out in his post,  “Who Owns Your Data?”

“The important question isn’t who owns the data. Ultimately, we all do. A better question is, who owns the means of analysis? Because that’s how, as Brand suggests, you get the right information in the right place. The digital divide isn’t about who owns data — it’s about who can put that data to work.”

Strata is where a vanguard will be meet, not only to discuss this revolution’s futures, but to define how to create, handle, and build the platforms and experiences that will harness the data. My flight is booked!  (Also check out BigDataCamp which takes place the night before Strata.)

The picture opening this post is from Michael EdgeCumbe’s  Fall 2010: ITP Winter Show Project.  A project exploring ways to intuitively get the feel of what it going on with big data sets using “the gestural manipulation and stereoscopic visualization of complex data to create a meditative state for data analysis.”  Michael project will be part of the Science Fair at Strata.  For more on Michael’s work see Noise Derived. I also have a number of the  interesting new interface sessions at Strata in my schedule.

The daily Strata Gems on O’Reilly Radar are great place to get a gestalt of some of the Strata themes, and this post by Edd Dumbill, program chair for Strata, Three key data trends for 2011, looks at the year ahead.  This week, I got the chance to ask Edd a few of the questions that I will have on mind at Strata – see his responses below.

If you have been reading Ugotrade, you will know I am interested in our mobile social augmented futures and there is no question in my mind that these will be unleashed by our new capacities to work with data (see my post here).

Data is the how.

The pic above is from “Secrets of BackType’s Data Engineers.” This post on ReadWriteHack by Pete Warden, an ex-Apple engineer, and founder of OpenHeatMap, really lives up to its title.  Check it out if you want to know how  “three guys (the BackType team ) with only seed funding process a hundred million messages a day?”

I asked on Quora, “What would be the most important developments for Augmented Reality in 2011,” Michal Avny, Strategist & Real Time search expert, wrote:

“AR strongly relies on localized personalized real time information.

Having a stream of tweets based on keyword search, location or circle of friends doesn’t really make the AR experience; it is the processed real time relevant information that will make AR useful and intensify the experience.”

In 2011 Real Time search and Social Search will drastically change to provide the infrastructure required.”

I followed up on Michal’s Quora answer with some more questions – see below in this post.

Also note the response from David Molnar, here is an excerpt:

“2. A wave of actionable, important data APIs opened up, enabling useful non-gimmicky AR apps for the first time. Think geoloqi.com , or the work Max Ogden has done with Portland civic data. Plus of course face.com , email providers and calendar providers, etc.”

Amber Case, one of the founders of Geoloqi, is on the programming committee of Strata and will be speaking. Be sure to catch her session! Posthumans, Big Data and New Interfaces, and if you haven’t already seen it, Amber’s TED talk is a must see.

Geographic proximity is a powerful filter, as is route, and time. But clearly social proximity, social relevance, and shared tastes are also key dimensions for location based experiences, (see my convo with Schuyler of Simple Geo, upcoming).

While the whole business of location based search and curation of augmented mobile social experiences is still, for the most part, uncharted terrain, the danger of key points of control being only really accessible to elite players looms large. I asked Sophia Parafina, a pioneer in the open geo space for some thoughts on real-time local /geosearch and geomessaging, and the future of openess & big data (see Sophia’s response below).

Is the market ready yet for P2P cloud computing?

This is another question I’m following,  Is the market ready yet for P2P cloud computing? It is one of those questions that we seem to have been asking in various forms for a very long while now, but without a  major shift in sight.  The pic above is from, The Cloud Made Open Source “Invisible” This Year.  But, perhaps, we are at the point when open p2p clouds will find a place in the market because of their potential importance in real time social search and discovery. Borislav Agapiev, Search Entrepreneur and founder of Vast.com, writes on Quora:

“I believe a P2P cloud is ideally suited for social & real-time search and discovery.

Consider MapReduce, a very interesting and popular paradigm for distributed computing. MapReduce is very much about bringing computation to data i.e. doing computation at nodes (map) and then aggregating results through network (reduce).

It is very clear now that user attention data (what they click on) is very valuable for search and discovery, yet a centralized model relies upon uploading all that to a single location and then doing a supposed local MapReduce. Clearly, MapReduce could be done across the network, without any centralized uploads.

In addition to the efficiency argument raised here, it is even more important to consider privacy issues. Uploading massive amounts of user attention data to a centralized location is not something that is going to make users warm and fuzzy as we are increasingly seeing.

In a P2P cloud, there is no big brother watching over anyone, all computation and data storage is done in the cloud, fragmented in many, many small encrypted pieces ala BitTorrent.”

Picture above from Brynn Marie Evans, “It takes two to tango: review of my social search panel“

The Delta of Now – Transforming Search into a Social Democratic Act

Picture of Maneki Neko “beckoning” cats from Journeyetc

New ecologies of human and machine intelligence are beginning to change basic social structures – see the Future of Work (Biewald and Chirayath Janah 2010). And projects like Swift River, using search and machine mining to filter out streams on topics of interest that can then be subsequently curated by human beings. This may be extended to the curation of real-time data streams and employment of machine learning algorithms based upon the explicit relationships.

Augmented mobile social experiences are a new frontier in which ideas and practices from a number of fields collide, including: ambient findability (Morville 2005), urban psychogeography, narrative structures, ambient games and devices, 4d (time-space), explorations of place and memory, enchanted objects and people (Kuniavsky 2010), and designed animism (Laurel 2010), to mention just a few.

Mobile local interaction presents an opportunity to invert the search pyramid and to transform search into a social, democratic act (see my interview with Anselm Hook upcoming).  Up until now search has been predicated around a very narrow revenue model. Google has an implicit model of a B2C – business to consumer brokerage. We are only just beginning to get a glimpse of the disruptive potential of C2C – consumer to consumer brokerages. Mobile local C2C brokerages that allow us to transact in a trustworthy way over our local geography in close to real time (Hook 2010) have the potential to enable new forms of social organization. Bruce Sterling’s short story about a networked gift economy, Maneko Neki, is a brilliant glimpse at the disruptive potential of such re-imaginings.

Augmented experiences that shift or change a person’s situated geolocal experience of social reality, and change our relationship to the people and the place by augmenting engagement in, and reputation through, socially driven consumer tie ins and game dynamics, like Four Square, & Gowalla are beginning to emerge, as Kati London pointed out in her excellent keynote at Web 2.0 Expo. And, while the integration of mobile local interaction and an augmented view that shifts our geolocal experience visually will involve creative solutions to some well churned mobile, tracking, mapping and registration challenges, the exploration and development of new dimensions through which we can filter and create trusted and meaningful augmented mobile social experiences is vital, whether you are considering a mobile screen, map, camera view, or futuristic HUDs and gestural interfaces.

Talking with Edd Dumbill

Picture from O’Reilly Community.

Tish Shute: First congratulations on Strata! On the Strata homepage there is a quote from Jason Hoffman:

“My gut feeling is that we’re going to look back at the upcoming Strata Conference like we do at the Web 2.0 Conference in 2004/2005.”
—Jason Hoffman, CTO/Founder, Joyent, Inc.

Why do you think Jason’s comparison might be prescient?

Edd Dumbill: Web 2.0 is a development that ran through every brand that has a web presence and radically changed the way business is done for many companies and brands.

Strata will have a similar impact: every business has data, every business collects an increasing amount of data. This data is the new oil – a valuable raw material that when refined or combined creates value and opportunity.

Tish Shute: The rise of real time was one of your three key data trends for 2011. Hadoop is bringing the capacity to work with big data to more than just a few elite players. But the challenge is still real time. You mention we will be seeing a hybrid approach to real time and batch MapReduce processing. Will we hear more about these approaches to real time at Strata? And, what do you see as the most important conversations on real time data analytics emerging at Strata?

You point out “open source projects and cloud infrastructure means developers can evaluate and learn to love technologies without requiring support or approval from above.” What are the most exciting developments on the horizon for open source tools?

Edd Dumbill: Here are some projects worth watching, in the key areas of real time, cluster management and Hadoop.

* Cassandra and MongoDB — NoSQL databases that will prove vital for anybody with real time big data needs

* Mesos — a compute cluster management tool, modeled after that which powers Google

* Hadoop ecosystem’s continuing maturation, especially HBase and Hive.

Tish Shute: Do you think the market is ready for p2p cloud computing?

Edd Dumbill: The market is emerging for decentralized and distributed cloud computing, and P2P technologies are one way of achieving that. They key trends will be moving computation nearer the data sets or nearer the point of user consumption of the result.

P2P is a difficult model for anybody wanting to commercialize a service, so I think it will tend to form part of a hybrid solution.

Tish Shute: We have seen enormous strides in our ability to work with giant unstructured databases recently. Do you think, perhaps, that the dream of a web of linked data – “a web of data that can be processed directly and indirectly by machines,” will be attained through brute force – i.e. through our ability to harness the power of massively parallel processing, as much as by Semantic Web approaches focused on machine readable metadata? [Also see my question on Quora, “Is this a good approach (www.dist-systems.bbn.com/people/…) to use Hadoop to build a scalable, distributed triple store?”]

Edd Dumbill: I’ve been an observer of the SW for over a decade and I tend to believe that on the web, data means to you whatever meaning you give it as the consumer. With that model, the links are made by the consumer rather than sitting out there explicitly. Some links become de facto standards, and some very few become web standards.

I think the actuality will be a mix of both explicitly stated metadata and that which is inferred. The Semantic Web is a great framework for certain operations, especially interoperable exchange of metadata. A great many more private meanings, never intended to be shared, will be created by consuming software.

There’s no question that machines will learn how to process most of the Web. Furthermore, machines will learn how to process most of the physical world we’re in. And that by the end of this decade.

Talking with Sophia Parafina

Picture of Sophia at Where 2.0

Tish Shute: Sophia you have worked in the trenches for a long time now to support the growth of open geo data. What do you hope to see emerge in 2011 in the field of geo-data?

Sophia Parafina: Better support for displaying and handling location data across multiple apps. Fred Wilson recently blogged about content-shifting, he talks about overcoming content silos across devices. We’ve worked very hard to reduce data silos via formats, but devices are creating their own silos. I would like to see a standard method for sending geo data and geo information to mobile devices.

Producing content for mobile is different from producing content for a computer browser. Web 2.0 produced a lot of infrastructure for browser based interfaces, but in mobile devices that gap has been filled with apps which is fragmenting how data is handled by various devices. What is even more interesting in the mobile space is that devices can push data back that contains location, user updates, photos and even sensor data.  If mobile data standardizes, it could lead to browser based applications and stem the continued fragmentation of the mobile application market.

Tish Shute: Simple Geo and Factual are startups emerging in the geodata space. What do you see on the horizon in terms of both the growth of business opportunities and an open geo data community?

Sophia Parafina: In the near future think we’ll see startups providing curated data + API and in response we will also see companies that provide a single interface across multiple data providers. We saw this when everyone released a mapping API and companies such as Mapufacture provided a single interface across multiple APIs.

We will see a resurgence in data providers repackaging the the 2010 US Census data in different ways to respond to market segments, some of this will be open data but all of it will be provided through an API instead of file. Additionally, we’ll see more data from outside the US.

Tish Shute: What are the biggest obstacles to having the open geodata sets available that we need to enable mobile local interactions and social augmented experiences?

Sophia Parafina: Licensing for both crowd sourced data and private curated open data will become an issue. We recently seen VLC, the open source video player, pulled from the Apple app store because of licensing issues. Also, licensing of content by geography will be problematic, limiting searches by geographical location. In addition, how will licensing of data that is updated by crowd sourcing work?

Multiple APIs for accessing data sources. The current trend for each provider to create an API for their data sets will result in data silos – there needs to be a single sign-on equivalent for requesting data.

Size of data on the wire, the current models for delivering data is based on broadband connections. However, as mobiles increasingly become the way people use the web, the data needs to be sized accordingly. This also goes for mobile interfaces. Have you tried to shop on a mobile device, or buy a train or plane ticket? It’s frustrating and error prone. There is a large untapped market of people who only use the Internet on mobile devices.

Tish Shute: You pointed me to this link in Strata Gems re “an interesting and pertinent (also a competitor to GeoLoqi),” – the Android Tasker app. What do these emerging services bring to the table in terms of the next generation of location based services?

Sophia Parafina: This app let’s your device interact with the environment. I think that this is a great way of using the sensors on existing platforms to increase interaction and to implement ambient findability. The basic premise of Tasker is that some action happens in response to an event in an application, time, date, location, event, or gesture. Tasker has defined 180 actions that can occur based and number or combination of events. This can provide a basic vocabulary for interaction between the user and the device and more importantly between users. Tasker also can use Android script plugins, which lowers the bar to creating your own ambient application.

Programs such as Tasker can provide a way for people to interact with social networks beyond sending messages. People can use their mobile devices to interact with their surroundings with out having to interact with the device.

Tish Shute: We have had many conversations about emerging ideas of geo-search, geo-messaging and geo-fencing. What are the most interesting developments in these areas and what do you see on the horizon for 2011?

Sophia Parafina: The map will fade into the background and become less important. Display of information will be context aware, that includes location. For example, let’s say I make a grocery list, when I’m at the grocery story, the list will just pop-up without the need for me to find the app that has the list. Or reminders or offers pop-up when you are near a place at a certain time, let’s say you need to buy a present for a birthday party for a child, you could send out a request that you are looking for an item and retailers could offer “on the spot” discounts if you are in the area.

Geo-search, geo-messaging, and geo-fencing are geared to towards mobile devices, so I expect to see them soon as part of apps. Building generic applications that implement geo* will fail because that sort of information is useful only within a context. Geo* apps are solutions looking for an problem. The killer mobile app will use these functions transparently to reduce the cognitive load of the user who is busy moving around in the world.

User data gathered from multiple web applications will become consolidated profiles that will used for context aware applications. For example, there could be a service which matches prices of items that you have shopped for on the web, so for example the service would have access to your cookies, know your favorite retailers, things you have shopped for, your location and activity patters (when you are at home, work, restaurant). When you are in the vicinity of a brick and mortar retailer with the same or similar items, the service can send you alert to match the price of the item you found on line. So your digital life will become more closely linked with your day to day activities.

Talking with Michal Avny

Tish Shute: At Web 2.0 Summit, one of the highlights for me was the, Q&A:The New Search Insurgents lunch where Charlie Cheever of Quora, IMO, stole the show. I tweeted:

“One of my takeaways from #w2s is that #quora points to future of augmented mobile social experiences – a search filter for experience! #AR”

In your view what are the biggest challenges for location Q&A to emerge as a search filter for location based experiences?

Michal Avny: The biggest location Q&A challenges yet to be conquered are immediacy (real time dynamic data), relevancy (strong personalized filters) and user experience (simplified interface).

Location Q&A enables different use cases. The most prominent are Follow (follow places, topics and friends to learn about a location), Interact (meet new people based on common interests), Plan ahead (plan a trip, night out or a shopping day by asking and searching for local information) and On-site (check for recommendations, friends, deals, events and traffic nearby).

Unlike Follow, Interact and Plan ahead that can be added to existing Q&A platforms (such as Quora) by attending location specifics as they share similar characteristics, the on-site mode introduces a completely different experience, first and foremost it requires immediate attention. It is real time based and the nature of the data is dynamic. Traffic updates, current events, nearby friends, all that changes constantly. Posting a location question on-site implies the response should be in real time (e.g. best kid friendly restaurant), the normal Q&A response latency wouldn’t work.

Strong relevancy filters are required to accommodate for the overwhelming flood of information. Moreover, some of the data should be filtered by user behavior and preferences, check in notifications (type of relation), restaurant recommendations (type of food, price level, etc), shopping deals (commercial categories) and more.

Mobile experience requires ease of use and simplicity. A new Q&A interface and query language that allows for posting questions should be defined as well as coherent summarized response interface. User on the go should not have to post lengthy questions, browse through tens of results or search for the right service, but instead use a simple intuitive tool.

Tish Shute: Real- time location based search is in its infancy. Real time questions can be answered using different services such as Yelp, TripAdvisor, Waze, Foursquare, IMDb and more. But what are the challenges to moving forward with aggregating these sources and then into “locals” that are able to process and deal with vast amounts of information?

Michal Avny: Using some of the leading location services to answer question is sufficient to start with.

In order to provide broad coverage (worldwide) and reliable information, aggregation of the different services is required for instance to normalize product and service rank, aggregate classified, and more. This is quite challenging as there is no one standard available.

When location Q&A user base is big enough, I foresee a tendency to rely more on ‘locals’ input as the base of information. As the platform grows, communities will be formed with different cultures, relationships and trust levels, making the information more valuable and customizable. Some of the challenges I already mentioned are implementing filters, query language and interfaces to enable using the vast amounts of real time data in a mobile environment. More of the challenges lying ahead are integrating the ‘locals’ data with location based services as they are integral components of the Q&A ecosystem. Merging trust levels and relationships while adhering to different privacy guidelines is a challenge yet to be explored. (This should be discussed in more detail under the protocols topic).

It is quite evident that Quora is now facing growing pains and is struggling to maintain its character. Same as with Quora, it will also be a challenge to support and maintain the ecosystem while allowing for massive scale-up.

Tish Shute: I have been very interested in exploring protocols that will be enablers to micro local interaction and mobile social interaction for AR – particularly the XMPP extensions and operational transform work of Google Wave (now Apache Wave), and PubSub protocols like PubHubSubbub and Erlang based RabbitMQ. We are beginning to see protocols emerging that could enable new real time local services. What do you think are some of the most valuable use cases for “locals” that this new generation of real time protocols can enable?

Michal Avny: AR is about interacting with digital information; the AR ecosystem is composed of layers and components such as devices, platforms, browsers, applications and content. For the different components to interact new protocols, security guidelines, and privacy policies must be in place. A standard will enable local vendors and service providers to publish specials, deals, updates and events for any application to broadcast, identify people and places by proximity (without having to use the same application or device), local recommendations will be shared by services, devices will be able to interact, location based platforms, such as Q&A, will have access to vast breadth of information, geo aware devices will provide consistent experience globally, and much more.

Tish Shute: What do you think are the biggest challenges to going mainstream for this emerging field of real time social discovery?

Michal Avny: The biggest challenge is building towards real time, geo-aware, localized, personalized ambient data. Discovery is in its infancy, location social based Best, Top, and Trending lists with some basic filtering options are available, and this is great as people are getting accustomed to information surrounding them. To some degree it can intensify the AR experience, for instance suggest the most popular dish in a restaurant, or map the best coffee shops nearby, but it is customized at best by friend recommendations and depends on the coverage and broadness of the specific discovery service.

There is a need for the next generation of discovery, customized geo social aware discovery that filters the vast amount of real time data by learning user preferences and behavior (built on top of the much needed local social real time open protocol)

Tish Shute: Who are your favorite startups/upstarts in the the field of real time search and why?

Micha Avny: My6Sense - My6sense provides a sharper and better way to experience your information from feeds you subscribe to (Social Networks, News, RSS feeds, etc.). It’s personal – Content is ranked based on what’s relevant to you. It learns what’s valuable to you by translating your consumption behavior into a personalized ranking function.
My6Sense – because it is a personalized prediction filter, a critical foundation for AR

Topsy – Topsy is realtime search powered by the social web that finds the most relevant conversations happening online. The site’s underlying technology examines popular links as well as the influence of each person citing a link. Topsy augments traditional search engines by finding information that people are talking about.
Topsy – because its ranking is based on retweets and influencers, a great social experience

Collecta – Collecta is a real-time search engine for the social web. It monitors the update streams of popular realtime blogs and sites like Twitter, WordPress, and Flickr, and shows results as they happen. Results can be filtered by status updates, comments, stories, or photos. The entire engine is built around the XMPP standard, which pushes out data on a continual basis, so that for every search you end up watching a stream that keeps updating itself.
Collecta – because it is built around XMPP, a real time experience

The Points of Control map is interactive, so please click here or on the image above for the full experience.

Today at 4pm EST, 1pm PDT John Battelle and Tim O’Reilly will discuss the Points of Control map and The Battle for the Internet Economy in a Free Webcast:

“More than any time in the history of the Web, incumbents in the network economy are consolidating their power and staking new claims to key points of control. It’s clear that the internet industry has moved into a battle to dominate the Internet Economy.

John Battelle and Tim O’Reilly will debate and discuss these shifting points of control as the board becomes increasingly crowded. They’ll map critical inflection points and identify key players who are clashing to control services and infrastructure as they attempt to expand their territories. They’ll also explore the effect these chokepoints could have on people, government, and the future of technology innovation.”

I’ve been wanting to start a discussion on the  Points of Control map in the Augmented Reality community for a while now, and Chris’ recent post on the latest edition of the Gartner Hype Cycle, “Is AR Ready for the Trough of Disillusionment?” and this post by Mac Slocum, “How Augmented Reality Apps Can Catch On,” and the conversation in the comments between Mac, Raimo (one of the founders of Layar), and Chris, all prompted me to get a conversation started…(see below for all that followed!).  Chris put me on the hot seat back in June when he did this very generous interview with me on Boing Boing, so it was time to turn the tables.

Tim O’Reilly, in his  keynote for Web 2.0 Expo, pointed out there is both a fun and a dark side to the Points of Control map.  There are companies on this map, he noted, that rather than “growing the pie,” are trying to divide up the pie, and they are forgetting to think about creating a sustainable ecosystem. I expect the conversation between Tim O’Reilly and John Battelle to dig deep into this Battle for the Internet Economy.  If, like me, you have another engagement at the time of the webcast, you can register on the site to receive the recording.

AR is still too young to figure in the battles of the giants, but there will be a lot to be learned from this conversation.  And, The Points of Control map is good to think with from the POV of AR in many ways.  As Chris Arkenberg observed:

“When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?”   You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development.  And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.”

Chris Arkenberg is a technology professional with a focus on product strategy & development, specializing in 3D, augmented reality, ubicomp and the social web. He uses research, scenario planning, and foresight methodologies to help organizations anticipate change and adopt a resilient and forward-looking posture in the face of unprecedented uncertainty. His personal work is collected at urbeingrecorded, and his professional profile is here.

He is also one of the founder/organizers of AR DevCamp which is currently scheduled for Dec. 4th (somewhere in SF or The Valley!)  Chris said, “No further details atm (still trying to find a venue and get sponsors) but please direct people to http://ardevcamp.org for upcoming information.”

Talking with Chris Arkenberg

Tish Shute: I know some people thought the positioning of AR by Gartner near the peak of the hype cycle was misguided, and based on a very narrow understanding of AR as used in marketing apps. But reading your post I thought you made a lot of good points.

Chris Arkenberg: It’s tracking hype, right? It’s not necessarily tracking the growth of the technologies or their maturation so much as it’s tracking the general attention level. And what’s interesting to me is that tends to affect the amount of money that goes into those technologies.

Tish Shute: I was particularly interested in your post because I have been writing a post about two recent O’Reilly events in NYC, Maker Faire, Web 2.0 Expo, and then Hadoop World, where Tim gave a very interesting 45 minute keynote.   AR was pretty low profile at all three events.   But the NVidia augmented reality demo attracted a lot of attention at the sponsors expo, and Usman Haque, Founder of Pachube announced in his presentation, they are working on an augmented reality interface for Pachube called Porthole, its designed for facilities management and, “as a consumer-oriented application that extends the universe of Pachube data into the context of AR – a ‘porthole’ into Pachube’s data environments.. “  Usman also mentioned, when I talked to him, that he is contributing to the AR standards discussion and on the program committee now for the W3C group on augmented reality.  For more on this standards discussion and the Pachube AR interface, see Chris Burman’s paper for the W3C, Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual.”

I think pioneers in the augmented reality commmunity should pay attention to these wider conversations about the Battle for the Internet Economy, and the exploration of the  “Platforms for Growth” theme at Web 2.0 Expo is very important- this is a course also a nudge to read my upcoming post on these O’Reilly events!

Also I have another project I have been chewing on that I would like to talk to you about.   I want to start an AR conversation about the wonderful Points of Control map produce for Web 2.0 summit by John Battelle. [ Note there will be, "Battle for the Internet Economy" free Web2Summit webcast w/ @johnbattelle & @timoreilly Wed 10/27 at 1pm PT http://bit.ly/b46cmb #w2s]

Up to this point, understandably given the immaturity of the technology, AR has little role in the “Battle for the Internet Economy.”  But this doesn’t mean that the map isn’t good for AR visionaries, enthusiasts, entrepreneurs, and developers to think with.   And both you and Tim have pointed out the potential for AR to leverage the giant data subsystems in the sky.   I have to say the positioning of Cloud Computing on the brink of heading down into the trough of disillusionment in this recent rendition of the Gartner Hype Cycle seems ridiculous!

Cloud Computing is already ubiquitous hardly seems credible that it is headed for a trough of disillusionment!

Chris Arkenberg: Yeah, it’s ubiquitous so why even talk about it when it’s your fundamental infrastructure?

Tish Shute: Yeah and I seriously doubt it is imminently headed for a trough of disillusionment….and this brings me back to the Points of Control Map which as John Batelle points out, “aims to identify key players who are battling to control the services and infrastructure of a websquared world” in which the “Web and the world intertwine through mobile and sensor platforms.”  This instrumented world, of course, creates a great deal of opportunity for augmented reality. Have you seen that, that points of control map?

Chris Arkenberg: I think I have, actually.

Tish Shute: There has been much debate about how this intertwining of the web and the world will play out in augmented reality.   Chris Burman points out in his position paper for W3C,  Portholes and Plumbing: how AR erases boundaries between “physical” and “virtual”, that “trying to draw parallels between a browser based web and the possibilities of AR may solve issues of information distribution in the short-term,”  but it must not have a limiting effect in the long-term.   But now we at least have one web standards-based browser for AR thanks to the work of Blair MacIntyre and the Georgia Tech team.  But I think the discussion in the comments of Mac Slocum’s recent post, “How Augmented Reality Apps Can Catch On” is an interesting starting point from which to think about platforms of growth for AR.  I am not sure if I am stretching his meaning but I think Raimo, Layar, is suggesting that what the Point of Control map call the the Plains of Media content is very important to the growth of the fledgling AR industry right now. And I would agree with this, and add that the neighboring terrain of gaming levels will be pretty key as one of my other favorite AR start ups Ogmento hopes to reveal in the near future! But what do you think was most important in this brief but pithy dialogue between you Raimo and Mac?

[The screenshot above is a teaser video the Gary Hayes from MUVEdesign for his upcoming (2011 release date), game called Time Treasure.  See Gary’s blog for more and Gary’s  post from over a year ago on AR Business models.  Thomas K. Carpenter, on Games Alfresco notes, “I think this is a terrific idea and I find it interesting he’s planning this on a tablet rather than a smartphone.”

Chris Arkenberg: The way I took it…And to give a little bit of context, I came from sort of this apprehension of augmented reality as an expression of the existing Internet. So as sort of a visualization layer that allows you to kind of draw out data, and then, with all the affordances of being able to anchor it to real world things.

And my own sort of path has led me to want to really try to understand that and refine it, particularly with respect to the sort of Internet of things and the smarter planet idea of just having embedded systems everywhere. And specifically, what is the value-add for augmented reality as a visualization layer of an instrumented world?

And so that’s caused me to be a bit biased towards that side of AR. And the way I took Raimo’s comment was that he was saying that, “You know, really what we’re interested in is media.” That he was effectively saying that AR for them is really just about that space between the screen and the the world, or between your eyes and the world, and what you can do there.

Certainly I had considered it in the past, but I hadn’t really focused on it or assumed that it was a priority as a business model. And so he kind of reminded me that, actually, there’s a lot of entertainment applications. There’s a lot of, obviously, advertising and marketing applications.
And so I felt that I was being a little narrow in my focus…

Tish Shute: Yes this comes to the heart of what I am interested in about the role AR can play in opening up new relationships to the world of data that we live, not just making it more accessible and useful to us when and where we need it, but AR as a road to reimagining  it..

Have you seen any interesting work yet to explore these great data economies in the cloud through AR. I mean can you think of any others – there is planefinder.net but others?

Chris Arkenberg: I’ve seen a few just sort of skunk works type applications that people have been playing around with, again, to try and reveal things. One of them was similar to the aircraft, but it was more for military use and being able to identify things of interest in the sky. I’ve seen a couple other for navigation, so being able to identify mountain peaks on a visual plane, for example, but this isn’t so much about revealing an instrumented world.

Tish Shute: Yeah, I think that was from the Imagination right? I know that’s an interesting one. Usman at Web 2.0 Expo, in his presentation, mentioned the work Pachube is doing on an Augmented Reality interface. I interviewed Usman again as my last long interview with him was nearly 18 months ago now and Pachube is well on the way to becoming the Facebook of Data or the analogy that Usman prefers – the Twitter of sensors!

Chris Arkenberg: Hmm, interesting.

Tish Shute: And to go back to your comments on Augmented Reality not getting caught in some of the traps that have made virtual worlds lose relevancy I think that is vital that AR developers understand the strategic possibilities of key points of control in the internet economy because the isolation and Balkanization of virtual worlds were certainly a factor in their rapid slide into the trough of disillusionment – although many would argue that a fundamental flaw in the kind of virtual experience that Second Life and other virtual worlds constructed was really the fatal flaw (see James Turner’s interview with Kevin Slavin Reality has a gaming layer).

But Second Life’s isolation from the other great network economies of the internet was certainly a limiting factor.

Chris Arkenberg: And that’s been exactly my sense, and I’ve, over the years, tried to encourage development in that direction for virtual worlds. I did work, through Adobe, to help develop Atmosphere 3D back in the the early 2000’s. And we did a lot of work to try and understand the marketplace and the specific value-add of doing things in 3D over 2D.

And this is kind of why I keep referring back to VR and VW’s with respect to augmented reality, is that with immersive worlds, there was this idea…there was this big rush. Everybody was so excited about it. It was obviously the next cool thing. And everybody wanted to try to do everything in it. You could do your shopping in virtual worlds. You could have meetings in virtual worlds.

Tish Shute: and shopping, yes ..that didn’t work out so well!

Chris Arkenberg: And everybody was very excited in developing these things. And what it really came down it is, “Yeah, you can, but it’s actually a lot better to do those things on a flat plane or in person.” Meeting Place, WebEx, TelePresence – those tools generally do a much better job at facilitating TelePresence meetings than a virtual world does. The same with TelePresent Education. There are only very specific things that both VR and AR are really good at.

And that’s where I find myself with augmented reality right now, trying to really pick through that and critically look at which uses are really appropriate for an AR overlay. And again, I think that’s why the hype cycle is important, because it reflects back this desire that AR is going to be the next big thing – the be-all, end-all of interacting with data in the cloud – and forces us all to take a critical look at why we should do things in AR instead of on a screen.

AR is not going to work well for most things but it’s going to be very good for certain uses. Right now I’m very keen at trying to understand what those things might be.

Tish Shute: I had this wonderful conversation (more in an upcoming post) with Kevin Slavin one of the founders of Area/Code at Web 2.0 Expo and I think some of what he describes about the data brokerages of High Frequency trading have some interesting implications for AR’s role, say, in ubiquitous computing. The trading markets are now pretty much dominated by machine to machine intelligence; machine to machine brokerages. They are basically game economies on the scale that we can barely wrap our heads around where the speed that bots and algo traders can access the network is the key. We really have no clue what is going on until we lose our house…

Kevin was also interviewed by James Turner on O’Reilly Radar. He talked about how much of the interesting work in location based mobile social apps is defined in opposition to the model of Second Life. He also talked to me about how we are seeing “first life” take on the qualities of “second life.” What goes on the trading floor is largely a performance secondary to a more important world of machine intelligence with giant co-located servers and bots fighting for trading advantages measured in fractions of seconds.

He pointed out how we draw on all these tropes from sci-fi movies, these HUDs based on ideas of machine intelligence where the robot talks to the other robot in English through an English HUD!  Many of our current visual tropes for AR are perhaps just as inadequate for the kind of data driven world we live in.

Of course, when you are thinking of having fun with dinosaurs, or illustrated books, or whatever, this is not, perhaps, an issue.  But if you are thinking of augmented reality interfaces as being important in a battle for network economy, and platforms for growth,  how this new interface helps us live better in a world of data is an important issue.

Chris Arkenberg: Now, does that indicate that the UI just needs more overhaul and innovation, or more that the visual interface for those experiences shouldn’t really leave the screen? It shouldn’t move on to the view plane?

Tish Shute: Yes we have a few concept videos that try and explore this ..

Chris Arkenberg: Well, and I think this will happen at the level of human-computer interface. I mean that’s always been its role, in making coherent the sort of machine mind, for lack of a better term, making it coherent to the human mind. So I mean there is a lot of this sort of machine intelligence, the semantic Web 3.0 revolution, where it really is about enabling machines, and agents, and bots to understand the content that we’re feeding them.

But at the end of the day, they, for now, need to be providing value to us human operators. So there’s always going to be a role for human-computer interface and user experience design to make this stuff meaningful.

I mean, if you look at the revolution in visualization & data viz, this is of incredible value because it takes a tremendous amount of data and collates it into a glanceable graphic that you can look at and immediately comprehend massive amounts of data because it’s delivered in a handy, visual way.

So I see that as a fascinating design challenge, how the user experience of the data world can be translated into meaningful human interaction.

Tish Shute: Yeah. And when we see Stamen Design pursuing a big idea in AR, that’s when we might start to rock and roll, right?

Chris Arkenberg: Yeah. In my article, I sort of jokingly suggested that Apple will create the iShades. But, they’ve got the track record of being way ahead of the curve and delivering the future in very bold forms.

Tish Shute: A key part for the battle for the network economy is to bring the complexity of data into the human realm in a way that increases human agency. Kevin suggests that the giant robot casinos of markets should actually lift off into total abstractions as theses machine-driven trades get back into the human realm in ways that are so damaging to our lives – a lost house or job! The notion of a counterveillence society where people have more agency over the important aspects of their lives, health, housing, job (which I discussed with Kevin – interview upcoming) has gotten pretty tricky!

But I think we will begin to see AR eyewear for specific applications (gaming and industrial) get more common fairly soon – possibly as smart phone accessories.

And it is clear that AR is going to be, increasingly,  a part of our entertainment smorgesborg in coming months. Itouch has a camera (although lower resolution), Nintendo’s are AR-ready and many aspects of the AR vision of hands-free spatial interfaces will go mainstream through Natal.

But we are yet to see an app/platform emerge for mobile. Social AR games that turn every bar and cafe and ultimately the whole city into a gaming venue  -although I think Ogmento and MUVE aim to lead the way here! Will an AR company achieve Zynga level success by using the Foursquare, for example?

My feeling is that the lesson of Zynga is pretty important for mobile social AR games. Could Flash social gaming have taken off without Facebook?

Chris Arkenberg: And that’s the real driver. And again, as you mentioned with Second Life, and this was exactly my own sense, is that they stuck to the closed garden model and didn’t get the power of social and collaboration. They attempted to add some of those affordances within the world, but, you know, ultimately most people aren’t in virtual worlds, and most people aren’t using augmented reality. So leveraging the really predominate platforms like Twitter and Facebook and Foursquare, being able to leverage those affordances, that connectivity, into a platform like augmented reality, I think, is really critical. Because again, you get nothing unless you have the masses, unless you have people present.

Tish Shute: In AR research there is a long history of the notion of powerful AR-dedicated devices, but smart phones and tablets are good enough,  and can launch augmented reality into the heart of the internet economy. I think  the elusive AR eyewear will come to us initially as a smart phone accessory for specific apps.  But, for the moment, most AR apps make little attempt to play in the wider internet economy.

Chris Arkenberg: And I think it’s actually much lower hanging fruit, really, to do gaming, marketing, transmedia. Because then you don’t really care about the cloud, or maybe you only really care about a little part of it that your gaming property is addressing. Then it becomes much more about entertainment, and much more about persuasion, and sensationalism. And if you’ve got dancing dinosaurs on your street, great! It’s entertaining, it’s cool, it’s new. That stuff is fairly straightforward.

I keep coming back to this idea of, you know, the instrumented city. What sort of data trails do you get out of a fully instrumented city? So maybe you get traffic patterns, maybe you get geo-local movements of masses, maybe you get energy usage, that sort of thing, all the, sort of heat maps you can generate from a city. But then what good does it do to be able to have that on an augmented reality layer versus just looking at it on a mobile device or looking at it on your laptop?

Tish Shute: Of course the use cases for “magic lens” AR are different from the kind of hands free, 360 view with tightly registered media, that a full vision of AR has always promised. The 360 view is quite a different metaphor from the web and mobile rectangular screens.

Chris Arkenberg: Yes, yes.

Tish Shute: Did you see that great parody of Michael Jackson’s “Beat It” with the iPads versus the iPhones, right?

Chris Arkenberg: Oh, really?

Tish Shute: I tweeted it cos i thought it was quite funny and a little close to the bone!
[laughter]

“ur wanna an ipatch 2 b the new fad?” #AR gets cameo in Twitter, iPads & iPhone’s Michael Jackson-Inspired Parody via @mashable

It is hard to get away from the importance of eyewear when discussing AR!

Chris Arkenberg: Yes, so the hardware, to me, is a big stumbling point right now, or it’s a large gating factor, I think, for realizing what an augmented reality vision could really be like. That it really does need to be heads up. This holding the phone up in front of you is fun to demonstrate that it’s possible, and it’s valuable in some ways…

Tish Shute: And it’s particularly nice in some applications like the planes app, the Acrossair subway app where you hold the phone down and get the arrow, right?

Chris Arkenberg: Yeah, the way-finding stuff I think is really valuable…

Tish Shute: Sixth Sense really caught people’s imagination because it managed to deliver the gesture interface with cheap hardware, even if projection has limited uses (no brightly lit spaces or privacy for example!).

The other important and as yet unrealized part of the AR dream is real-time communications. Many interesting uses cases would require this. As you know that is my chief excitement, along with federation, in the Google Wave Servers for (which should soon be released at Wave in a Box) for ARWave.

Chris Arkenberg: Well my sense of Wave is that it was a ChromeOS protocol that they instantiated, or that they exhibited in the public deployment of Google Wave. That that was a proof of their sort of low level architectural solution. Because, you know, they’ve been rumored to be working on this cloud OS for some time. And so my sense is that Wave is actually one of their core components of that cloud OS, and that it just happened to incarnate for the public in a test run as Google Wave.

Tish Shute: I do hope that Wave In the Box will lower the barriers to entry to people experimenting with this technology. The FedOne server was just way too hard for most people to take the time to set up. Of course, it is the brilliance of the Wave Operational Transform work that also poses problems in terms of ease of use. But Wave Federation Protocol is pretty innovative. And could even play an important role in a real time communications for AR eyewear connected to smartphones. The challenges that Wave takes on re real-time communications, federation, permissions and filters are pretty important ones for AR…

Chris Arkenberg: Especially when you’re trying to federate a lot of permissions and filter a lot of data, which all of that gets even more important when you have a visual layer between you and the real world.

Tish Shute: You got it. Yeah!

Chris Arkenberg: I think that’s really valuable real estate, both for third parties that want to get access to your eyes, as well as for you, as the user, who still needs to navigate through the phenomenal world and not be occluded by massive amounts of overhead data.

Tish Shute: Yes, I am sure Google has big plans for the next level of cloud computing and Wave looks at some key challenges. I suppose federation poses some key business problems. I think it was Michael Jones who said to me that it was a bit like socialism in that you have to be willing to give something up for the greater good.

Perhaps federation does not present enough appeal because of its challenges re business models?

Chris Arkenberg: Well, I wonder. I mean there’s got to be some value for their ad platform as ads are moving more towards this personalized experience. Advertising is becoming less of a shotgun blast and more of a very precise, surgical strike. So being able to track user data to such a fine degree to mobilize the appropriate ads around them wherever they are, on any platform, is certainly very valuable to Google and their ad ecology.

Tish Shute: Many people have high hopes that HTML 5 by lowering the barrier of entry for  browser style AR could also pave the way for some interesting AR work..

Chris Arkenberg: Well, as much as I would hope that all the different players are going to come together and establish some shared set of standards, really, what’s happening is it’s a rush to the finish line to be the first…to get the most penetration in the marketplace so that Layar, for example, can say, “It’s official. We’re the platform.” And then the consolidation that will follow, where the Googles and the other big players like Qualcomm say, “OK, it’s mature enough. We’ll start buying up all the smaller companies.”

And that’s where the real challenge is right now is that there are no standards. It’s such an immature technology that you have a lot of different players trying to establish the ground rules. And again, this is one of the challenges that faced public virtual worlds, is that you had a lot of different virtual worlds that weren’t talking to each other in any particular way, and that they each had their own development platform. And so you end up with a very fractured ecosystem or set of competing ecosystems, which is kind of what’s happening with AR right now, where a developer has to choose between a number of different new platforms or hedge by deploying across multiple platforms. Basically, the web browser wars are set to be recapitulated by the AR browsers.

Among them, Layar and Metaio seem to be getting the most traction. But there’s still not a really strong case for a unified development ecosystem to emerge.

Tish Shute: So a discussion of ecosystem development brings us back to the Points of Control Map I think. So what do you see as key points of interest for AR developers to watch in the Points of Control Map? And where do you want to sort of put your bets, right? We are still really waiting for mobile social AR to emerge into the mainstream.

Chris Arkenberg: Yes. And that’s primarily the shortcoming of the hardware itself, but also of the accuracy of current GPS technology. That’s another kind of gating factor, because again, AR wants to be able to express the data within a distinct place or object.

So in a lot of ways, other than kind of what we’ve allowed for the broader entertainment purposes, for AR to really work, there needs to be more resolution in GPS location. So for it to be truly locative…because it’s OK to tell Foursquare that you’re in Bar X. But if you want to be able to draw data directly on a wall within that bar, or do advertising over the marquee on the front, you need more factors to accurately register those images on a discreet location. So that’s another, sort of, aspect of the immaturity of AR, is that it’s still very hard to register things on discreet locations without employing a number of diverse triangulation methods.

Tish Shute: Right. The mobile AR games we see at the moment are really just faking a relationship to the physical world unless they rely on markers or some limited form of natural feature recognition which is really just a more sophisticated form of markers. But the Qualcomm SDK does offer some opportunities to tie AR media to the world more tightly as does the Metaio SDK. But in terms of a mobile social AR game that could be like the Cape of Zynga to FourSquare in Location Basin [see the Points of Control map]… We haven’t seen anything close yet.

AR should be able to bring the check-in mode to any object in our environment.

Chris Arkenberg: Yes, yes. And that’s actually one of the early interests I had in the notion of social augmented reality. I wanted a way to tag my community with invisible annotations that only certain people could read, and found pretty quickly that that’s very difficult to do. I mean you can kind of do some regional tagging, like on a beach, for example, but if you wanted to tag the bench that was on the cliff above the beach, it’s very difficult to do that using strictly locative reckoning.

There’s all sorts of really cool social engagement that can be revealed when people are allowed to attach things to the world around them, to the streets they normally pass through, or the points of interest that they normally engage in. To be able to author on the fly on the streets and attach it discreetly to an object effectively.

Tish Shute: And yes we do have all kinds of markers and QR codes. But Erick Schonfeld of Tech Crunch made a good point that QR codes: “Until QR code scanners become a default feature of most smartphones and they start to become actually useful enough for people to go through the trouble to scan them, they will remain a gee-whiz feature nobody uses.”

Chris Arkenberg: So again, this gets back to competing standards and who gets access to the phone stack, the bundle. Who gets the OEM deal…?

Tish Shute: Yes, the battles for the networks on the Handset Plains are pretty important for AR!
[laughter] I think Layar have made some smart moves on The Handset Plains.

And there are a lot of acquisitions of nearfield technology to look at.  If I remember rightly Ebay bought the Red Laser tech from Occipital – now there’s any interesting company. Their panorama stuff rocks!

Chris Arkenberg: Right. There’s a lot of nearfield stuff that’s supposed to hit all of the major mobile platforms in the next year or so.

I mean I think where this is heading, in my mind, is basically smart motes. You know, little nearfield wide-range RFID’s that are the size of a small, tiny square that you could attach to just about anything and then program it to be a representative of your establishment or of an object, that then you can start to tag just about anything. I mean you can’t rely on geo to do it, but if you have a Nearfield chip there that costs maybe like two cents to buy in bulk, and you can flash program it, then you can start to attach data to just about anything.

Tish Shute: Yes ‘cos some things still remain very difficult for near field image recognition technologies like Google Goggles.

Chris Arkenberg: Well, if your phone can interrogate for Nearfield devices, and it detects a chip in its near field, it can then interrogate that chip. The chip may contain flash data on itself, or it may contain the local server in the establishment, or it may go to the cloud and get that data back.

Tish Shute: Yes there is moverment from the top and open source hardware like Arduino has created an opportunity for all sorts of creativity with instrumented environments.  And the handheld sensors in our pockets – our smart phones create a lot of opportunity for bottom up innovation too.

Chris Arkenberg: I mean that’s my guess. If you look at what IBM is doing with their Smarter Planet initiative, they’re partnering with a lot of municipalities, and obviously with a lot of businesses and their global supply chains.

But they’re basically working with municipalities and all these stakeholders to instrument their territory, their business, or their city, as it were. So they’re working to provide embedded sensors and the software necessary to read them out and run reports & viz. And presumably that software can extend to include some sort of mobile device to interrogate the sensors and read the data.

That’s kind of a top-down approach of a very large global company working with top-down governance bodies to do this. Simultaneously you have the maker crowd experimenting with Arduino and such to build from the grassroots, the bottom up approach.

And that’s primarily gated by the amount of learning it takes to be able to program these devices, to be able to hack them. Typically, the grassroots creators who make these devices don’t have the luxury of very large budgets to make things highly usable and Wizywig.

So the bottom up community is a sandbox to create tremendous amounts of innovation, because they are unconstrained by the very real financial needs of the top down innovators. And so you get a lot of fascinating innovation, a very rich ecology from the bottom-up approach, but you don’t get a lot of wide distribution. But that does filter up to and inform the top down approach that has a lot more money to put into this stuff. And it ultimately has to respond to the needs of the marketplace.

I mean if there’s an answer to the question of whether something like AR will succeed through the bottom-up grassroots approach or the top-down industry approach, I would say it would be both. That handsets will be hacked to read the bottom up innovations of the maker community, and handsets will be preprogrammed to read the top down efforts of the IBMs of the world.

Tish Shute: Yes but i have to say it is very time-consuming hacking phones (I have just seen a few days suck up in this myself so that I could upgrade my G1 to try out the new ARWave client!). I mean Android has obviously been the platform of choice because of openness but the business model of iPhone and its market share in the US sure make it important for developers.  It’s like you don’t exist if you don’t have an iphone app for what you are doing.

Chris Arkenberg: Yeah, and that’s the challenge, because at the end of the day developers prefer not to work for free and a solid, reliable mechanism to monetize their efforts becomes very appealing.

When I look at this map, the points of control map, it’s really interesting to me, because what it says to me with respect to AR is each of these little regions that they have drawn out would be a great research project. So every single one of these should be instructive to AR.

In other words, we should be able to look at social networks, the land of search, or kingdom of ecommerce, and apply some very rigorous critical thinking to say, “How would AR add to this engagement, this experience of gaming, or ecommerce, or content?”

Looking at each of these individually and really meticulously saying, “OK, well yes, it can do this but how is that different from the current screen media experience, the current web experience that we have of all these types of things?” You know, how can augmented reality really add a new layer of value and experience to these? And I think that process would really trim a lot of the fat from the hopes and dreams of AR and anchor it down into some very pragmatic avenues for development. And then you could start looking at, “Well, OK, what happens when we start combining these?” When we take gaming levels and plug that into the location basin, as you suggested.

Tish Shute: Some of the important platforms for AR don’t appear to have spots on the map like Google Street View and other mapping technologies that hold out so much hope for AR, or am I missing something?

Chris Arkenberg: You mean on the map?

Tish Shute: Yes for the full vision of AR we need sensor integration, computer vision and cool mapping technologies to come together. Do you see where Google Maps and Google Street View… Where would they be?

Chris Arkenberg: Yeah, I mean it’s certainly content, it’s location…

Are you familiar with Earthmine?

Tish Shute: Yes, yes I am, definitely. Earth Mine, Simple Geo, Google Street View, user generated internet photo sets like Flickr all of these could be very important to AR, potentially.

Chris Arkenberg: Well, and the interesting thing about Earthmine is that they’re effectively trying to do an extremely precise pixel to pixel location mapping. So they’re taking pictures of cities just like Street View, except they’re using the Z axis to interrogate depth and then using very precise geolocation to attach a GPS signature to each pixel that they’re registering in their images. Effectively, you get a one-to-one data set between pixels and locations. And so you can look at something like Google Street View, and if you point to the side of a building, in theory, it should know exactly where that is.

They’re rolling this out with the idea of being able to tag augmented reality objects in layers directly to surfaces in the real world. So that’s another approach to trying to get accurate registration and to try and create what are essentially mirror worlds. Then your Google Street View becomes a canvas for authoring the blended world, because if you plop a 3D object into Street View on your desktop, and then you go out to that location with your AR headset, you’ll see that 3D object on the actual street.

Tish Shute: There was some experimental work with Google Earth as a platform for a kind of simulated AR but I suppose Google Earth doesn’t figure in the battle for the network economy as it never got developed as a platform.

Chris Arkenberg: It hasn’t tried to become a platform, to my knowledge. I mean I know some people are doing stuff with it, but as far as I know, Google owns it, they did it the best because they have the best maps, and there’s not a huge ecosystem of development that’s based around it other than content layers.

And my sense of everything else on the Points of Control map is they’re looking more at these sort of platform technologies that…

Tish Shute: Yes, re platforms for growth for AR. Gaming consoles will probably emerge as a significant platform for AR this year.

Chris Arkenberg: There will be much more of a blended reality experience in the living room for sure, and with interactive billboards. Digital mirrors are another area. So I mean if we kind of extend AR to include just blended reality in general, you know, this is moving into our culture through a number of different points. As you mentioned, it will be in the living room, it will be in our department stores where you can preview different outfits in their mirror. We’re already seeing these giant interactive digital billboards in Times Square and other areas.

It’s funny. I mean for me, the sort of blended reality aside, the augmented reality, to me, is actually a very simple proposition in some respects. When I look at this map, augmented reality is just an interface layer to this map in my mind, just as it’s an interface layer to the cloud and it’s an interface layer to the instrumented world. It’s a way to get information out of our devices and onto the world.

Tish Shute: The importance of leveraging existing platforms has become pretty clear but it is interesting Facebook definitely gave Zynga the opportunity but would Facebook be so big without Zinga’s social gaming boost?

Chris Arkenberg: I feel that Zynga has definitely helped its growth…But I think Zynga has benefited a lot more from Facebook than Facebook has from Zynga.

Tish Shute: Zynga certainly proved you could build a profitable business on Facebook’s API!

Chris Arkenberg: They did. And they also really validated the Facebook ecosystem and the platform. They really extended it… Zynga benefited from the massive social affordances that Facebook had already architected and developed. They brought gaming directly into Facebook, and particularly, this emerging brand of lightweight social gaming that when you sit it on top of a massive global social network like Facebook, it suddenly lights up.

Tish Shute: AR pioneers should quite carefully go through this map. There is so much to think about here. I’m a kind of fanatic about Streams of Activity in AR. Real time brokerages and their potential for AR is something I am fascinated by. That is one reason I love the ARWave project.

Anselm Hook, to me, is one of the great thinkers in this area of real time brokerages – with his project Angel, and the work of Ushahidi, which is now the platform for augmented foraging (see here). Anselm is now working on AR at PARC which is exciting.

Chris Arkenberg: Well, there are some challenges working with data streams. Presentation and filtering I think is a big challenge with any sort of stream. Because obviously, you have a lot of potential data to manage, to parse, and to make valuable and comprehensible. So I think this is bound very closely to being able to personalize experiences, or having very discreet valuable experiences. Disaster relief, for example, I think is an interesting idea that ties into the Pachube type of work. Where, if you had the headset and you were a relief worker, and you had immediate lightweight, non-intrusive, heads up alpha channel overlay, waypoint markers showing you all of the disaster locations or points of need, AR becomes extremely valuable, because it’s a primarily hands-free environment. This is why the military stuff is so interesting.

Tish Shute: Ha! We are running into the eye patch/shades/goggles/sexy specs thing again. But filtering and making streams of activity relevant will be very interesting for AR.  Again that it why I love the Wave Federation Protocol work because what they have built into their XMPP extensions. You can have your real-time personal data streams, or community streams, or broadcast publicly – the permissions are built.

And Thomas Wrobel’s original vision of these layers and channels is only fully expressed if you have the eyewear.

Chris Arkenberg: Well, and it becomes redundant if it’s on a mobile. To use a very basic example, Twitter, obviously there’s an app you can view those streams of activity on the camera stream. But you can view that real time data on the screen. Why do you need to see it heads up?

The reason I really pay attention to what the military is investing in, one, because they have a ton of money, but also because they tend to represent the core bio survival needs of the species…So, when I look at computing, I see this very obvious trend of computers getting smaller and smaller and closer and closer to us because they’re so valuable to our success. They give us so much valuable information for engaging our world on a moment by moment basis. So, of course now we have these tiny little handheld devices that give us access to the global knowledge depositories of human history, because it’s so useful to have that stuff right at hand.

The only impediment now is that it takes one of our hands, if not both of them, to access it. So if you are in the natural world, which we are all always in the natural world, ultimately, you want your hands free in order to engage with the world on a physical level.

I see computation, or rather, our access to computation is just going to get thinner and thinner, and we’ll very soon move into eyewear, and inevitably, we’ll move into brain computer interface in some capacity.

So when you’re the disaster worker, or a deployed soldier, or the extreme mountain biker, or the heli-skier, or just an adventurer, there are a lot of very practical reasons to have access to information on a heads-up plane. I see AR as being so profound and so valuable, but we’re getting a glimpse of it in its infancy, and it’s got a ways to go to be able to really contain what it is we’re reaching for.

Tish Shute: I agree.

Chris Arkenberg: And that’s been a big criticism I’ve had with all the existing AR implementations that I’ve seen, is that the UI really needs a revolution. It’s very heavy handed. It is not dynamic, even though it’s supposed to be. It does not take advantage of transparencies. It treats the screen like a screen. It doesn’t treat the screen like a window onto the real world. When you’re looking on the real world, you don’t want a lot of occlusion. You want very soft-touch indicators of a data shadow behind something that you can then address and then have it call out the information that’s important to you.

Tish Shute: Now, that’s a very nice kind of image you’ve conjured for me there. Do you see that more could be done on the smartphone than is being done within that? Or are we like waiting for the old ishades?

Chris Arkenberg: I think there’s definitely a lot of room for improvement on the smartphone UI. Nobody’s really played around with it much. And again, I think that’s in part that there hasn’t been a really established platform with enough money to fund interesting UI work. We see it in some of the concept demos that float around every now and then.

I guess it’s both a blessing and curse that I’m always five steps ahead of where I’m trying to get to.

Tish Shute: Yeah, I am familiar with that feeling!

Chris Arkenberg: So I’m always trying to reach for the vision even though it’s a bit distant. I think there’s going to be a lot of development on the handsets. But again, I think we need a lot of refinement. We need a lot of real critical analysis of why this is a good thing.

To get back to the original point of Raimo’s comment, it struck me. And I knew it, but I just had set it aside as gimmickry. But he’s right. Content is a huge driver for this. Just stuff that’s engaging, and fun, and cool, and shows off the technology so they can get enough money to make it through whatever Trough of Disappointment may be waiting.

Tish Shute: Yeah, don’t underestimate the Planes of Content!  They are a great place to get interest and money to keep AR technology moving on, right?

Chris Arkenberg: Yeah, yeah. Because, you know, there’s a lot of freedom there. And you can piggyback on all the rest of the content that’s out there and jump on memes and marketing objectives, etc…

And there’s a lot of stuff…I’m blanking on some of the names, but some of these historical recreations of city streets. There’s a street in London where they overlaid historical photos in a really compelling experience. [Museum of London - http://www.museumoflondon.org.uk/] Again, I’m completely forgetting the attributions, but hose are the type of things that can really be pursued on the existing platforms. There is stuff that’s really compelling and really cool.

I heard of another interesting use case – and I should say that I can’t find attributions to this anywhere on the web and I may be paraphrasing or mis-representing the actual work, but I think the concept is worth exploring anyway. But the idea was that you could take the locations of border checkpoints and conflict sites in Palestine and Israel and visually overlay them on an AR layer in San Francisco. And it would do some sort of transposition where you could virtually view these things in San Francisco with the same locational mapping superimposed. So you could see where the checkpoints where. You could see where the wall was. You could see where suicide bombings were and where there had been conflicts. [I cannot find any citations for this!]

Tish Shute: But with an AR view? But why would you use an AR view if you are in San Francisco, then?

Chris Arkenberg: Because it superimposes two realities, translating the Gaza conflict into San Francisco as you are walking around. You can interrogate the world. There’s a discoverability aspect where you’re using the headset to reveal things, or the handset rather, to reveal things that you could not see otherwise in your city. It was done as an art piece, but as a provocative, obviously political art piece.

Tish Shute: Very interesting. I’d love to see that. Because that’s interesting to get away from this idea that you actually have to sort of have this one to one relationship between the data and the world is kinda nice, isn’t it? Well, not one to one, but a very literal…getting away from that literalness is kind of good.

Chris Arkenberg: And that’s a possibility of virtual reality and augmented reality merging, that maybe virtual reality is actually going to do best by coming out of the box and writing itself over our reality, so that as you are walking around, you are no longer seeing San Francisco, but you are seeing part of Everquest or World of Warcraft.

Tish Shute: Well this is where Bruce Sterling gets to that point he made in his keynote for are2010, that if we actually have viable AR eyewear, then you get the gothic stepsister of AR, VR rising from the grave!  He asks whether the very charm of augmented reality, is in fact that it adds rather than subtracts from your engagement with the world and that getting get sucked back into the black hole of VR might not be so great.

Chris Arkenberg: And then you get all sorts of interesting challenges to social cohesion if you have a lot of different people experiencing very different worlds, effectively. That if there is no real consensual reality and a majority of your local populous is, in fact, experiencing very different and unique versions of the world, what does that do to social cohesion? How does that reinforce tribalism, for example, when only you and certain others get to opt in to a particular layer view of the world?

Tish Shute: Yes Jamais Cascio wrote an interesting piece on that issue on AR and social cohesion a while back.

An eye patch is a more logical vision than the goggles in many ways but I suppose the loss is stereo vision?

Chris Arkenberg: And actually, there were developments in military helicopter technology many years ago that used a single pane square of glass over the eye mounted to the helmets of pilots. And then they drew various bits of heads-up information on it. So that ensures that you’re having a real strong engagement with the real world, which, obviously, when you’re a helicopter pilot is quite important. But you still have access to the data layer of the invisible world.

Tish Shute: I just went to Hadoop World and I have to say, I was awestruck about how big that’s got. I mean Hadoop has gone from like zero to huge in just a few years. I mean it’s just like now everyone has the power of the Google big table at their fingertips.

What’s the play for AR in the land of search?

I could imagine Hadoop being very powerful tool for AR analytics?

Have you got any thoughts on the land of search and AR? Of course visual search is proceeding at a fast pace and there is a lot of promise for integrations with AR in the future but the latency for visual search is still pretty high?

Chris Arkenberg: In the near term, not a lot. In the medium term, there’s a larger trend towards virtual agents that you can program or teach to keep watch over things for you as an effort to scale down the data overload. So search is something that’s going to become more personalized and more active. There’s a movement to make it so people can essentially deputize these agents to be always searching for them; to be out there looking for the things that they have told these agents are important to them.

So active search for AR I think presents some challenges, obviously because you need to do text input, typically, or voice input. Voice input, I think, is much more achievable than text input for AR. But I can certainly imagine an AR layer that is being serviced by these agents that we have roaming around the web for us reconciling their visual view of the world with our personalizations. AR apps are contextually aware so it knows that if you’re downtown, it’s not going to be giving you a ton of information about Software as a Service infrastructure, or what have you. But that, instead, it’s going to be handing you little tidbits about a particular clothing brand you’ve opted in to follow and information about music venues & schedules, for example. Or perhaps you’ll be on the lookout for other users that have opted in to publicly tag themselves as a member of this or that affinity.

I keep coming back to this idea of AR as really just a simple visualization layer that all of these other technologies can potentially feed into. So in that sense, search becomes a passive thing that AR is just simply presenting to you in a heads-up, hands-free, or potentially hands-free environment.

Tish Shute: Yes, the big challenge is the stepping stones to that point! Small steps that keep interest going into developing the underlying technology (and not just in research labs!) that will bring us that interface.  We have seen some movement already with Qualcomm.

Chris Arkenberg: And there are bandwidth issues as well, as we can see with the Google Goggles, which is a great idea of visual search. But you have to take a picture and send it to the cloud and wait for your results. It’s not a real-time dynamic interrogation of the world.

Tish Shute: Yes we are really only at the very beginning of AR being ready for prime time.. it would be interesting to ask AR developers how many of them use AR on a daily basis.

Chris Arkenberg: I think a lot of us, we’re just informed by the sci-fi myths and fascinated with the potential now that’s it’s starting to become real. But I think we all kinda get that it’s still extraordinarily young. I mean the web is extraordinarily young. And AR is itself far younger in a lot of ways in its implementations.

Everybody has a lot of excitement about all of the great potentials that are being unleashed by this great wave of the Internet and the web and ubiquitous mobile computing. So that’s why, you know, you look at that map and we talk about AR and you can’t talk about any of the stuff without talking about all of it, in a lot of ways, particularly with something like AR where it’s so ultimately agnostic and could be completely pervasive across all of these layers.

So my fascination is with the future, and I measure our progress towards it by the young nascent offerings from the platform players and the developers. And yeah, a lot of it is…it’s akin to getting that first triangle on the screen in 3D. You know, when the renderer finally works and you get a triangle on the screen, and you go, “Oh my God, it renders.” And then you can start to really build polygons and build objects, and start doing boolian operations, and get light and rendering in there, and textures, and on, and on, and on.
So I’m fascinated by the Layars and the Metaio’s…
[laughter]

Tish Shute: Yes and hats off to all the players in the emerging industry, Layar, Metaio, Ogmento, Total Immsersion, and all the others who are finding clever ways to bring fun aspects of AR into the mainstream, and fuel interest to take the technology to the next level.

Chris Arkenberg: Absolutely. And the hype cycle is very valuable. It has really helped launch the AR industry. It’s brought a lot of eyes, and it’s brought a lot of money into the industry. And it’s forcing people like us to have these conversations to understand how to refine its growth and really focus on the potential in all these different venues, whether it’s trying to save lives, or better understand your city, or have really compelling entertainment experiences.

Everybody’s excited, and everybody’s sharing, and everybody’s trying to move it forward in a way that’s the most productive.

Where 2.0 is going to be epic this year (see my interview with Brady Forrest here), and it is so exciting to be part of it.  Location technologies and augmented reality are annointed rulers now.  Time Magazine recognized augmented reality as one of its 10 Tech Trends for 2010 (for more see ReadWriteWeb).

The photo above is by Jeremy and Lisa Hight.  Jeremy Hight is an information designer, theorist and artist working in Augmented Reality and Locative Media.   His essay “Narrative Archaeology” was named one of the 4 primary texts in Locative Media.

Jeremy Hight will be part of our panel: The Next Wave of AR: Exploring Social Augmented Experiences, with Anselm Hook, Joe Lamantia, Sophia Parafina and myself. We will debut the video of the ARWave project demo that brings together augmented reality, geolocation, and wave federation (more details later in this post).  And, Jeremy will bring to our presentation some augmentations on his recent brilliant work and paper, “Writing Within the Map.”

Greg J. Smith’s points out in his in depth look at Jeremy’s work that it, “dovetails with some of the main points in Bruce Sterling’s recent atemporality keynote at Transmediale” – fortunately there is a transcription of Bruce’s keynote here.  What is so awesome about this dovetailing is that you can get a feel for the fun part of living in an, “atemporal network culture.”  And, if you want to really understand just how much locative media and augmented reality have changed us, you  might want to dig into these texts.

Bruce Sterling and Jeremy Hight, and members of the ARWave team, and a superb cast of augmented reality movers and shakers – including Will Wright and Jesse Schell, will be speaking at Augmented Reality Event in Santa Clara, June 2nd and 3rd.

But, this week, the AR community’s attention will be on the events at Where 2.0.   The keynote speakers will be streamed live, so if you are not fortunate enough to be there, tune in!

The Next Wave of AR: Exploring Social Augmented Experiences

On our panel, Jeremy Hight, Anselm Hook, Sophia Parafina, Joe Lamantia and I will cover some of the key social, cultural, technical and interactional questions for exploring social augmented experiences. There will be five lightning presentations, and an opportunity for questions from the audience, and a world premier of the ARWave demo!

1) “Augmenting the map as interface: AR and Locative Narratives” - Jeremy Hight

*Map augmentation of the historic route 66 can house an essay contest and publication globally but as embedded within that map augmentation instead of books or even web sites.

* A place on a map can be a graphic index and database to save and collect
the writing of that place with a graphic or textual search index.

*One can pop immersive visualizations of abandoned or lost buildings from map location in shared software and collectively augment (imagine channels within the lost core of detroit where one is memories and accounts tagged within parts in the immersive visualization while another is of poems and stories written by people moved by the place and its semiotics and story).

*The news stand is to be the map.

*New forms of literature will be born of mapping, spaces,augmentation and
new tools

The concept drawings below (click to enlarge) are a collaboration between Jeremy Hight and Paul Wehby, Senior Designer at LA County Museum of Art.

2) Anselm Hook will look at, “10 reasons why AR isn’t a flash in the pan,” and how, “AR can help us see the world we would like to have exist.”

Anselm notes, “So much of what we do is so fickle and I’m looking for ways to connect digital media work to deep values.”

3) Sophia Parafina will present on, “Social AR and Crisis Response”

“Augmented reality as a multi-party conversation.  Rather than being passive viewers of AR with a limited ability to  checkin to places and make annotations, current devices can broadcast sensor information that can be fused into an interactive stream. AR users can send and receive information, location, and sensor data from their mobile device.  The streams can be federated into a unique AR view composed by the user.

Entertainment and gaming are obvious applications, but it can also be applied to crisis situations such as the search and rescue operations in Haiti.  Efforts such as Mission 4636, the SMS translation service, could benefit from AR views.  The collaboration among the Mission 4636 volunteers was the key element  in their success for providing location and rapid translation to responders on the ground.

With an AR view, responders can send back their sensor information from their mobiles to provide contextual information to remote volunteers.  This extends the conversation between remote volunteers and on the ground responders and fosters collaboration which was a key element for the success of Mission 4636″

4) Joe Lamantia, an experience design and strategy consultant helping to define the interaction framework and scenarios behind ARWave, will discuss, “Design Principles For Social Augmented Experiences:”

“With the exotic mixed realities envisioned by futurists and science fiction writers seemingly around the corner, it is time to move beyond questions of technical feasibility to consider the value and impact of turning reality inside out for everyday social settings and experiences. Thanks to the inherently social nature of augmented reality, we can be sure the value and impact of many augmented experiences depends in large part on how effectively they integrate with the social dimensions of real-world settings, in real time.”

Joe will share, “eight guiding principles for designing experiences that engage naturally with the social dimension, and increase the value of augmented experiences.”

5) “ARWave – A demo and state of play,” from Tish Shute

I will have the awesome privilege, on our Where 2.0 panel, of showcasing ARWave.   We will  premier the ARWave demo which shows how ARWave has accomplished the basics of geolocating data on Wave Federation Protocol (and real time collaboration on this geolocated data).  If you’re interested in the ARWave project join the Mailing list, FAQ are here, and have a peek at the current state of development at Google Code, and the specification for an AR Blip.  We also have Waves for the project hosted on Google Wave.  You can join the general discussion here, and the technical side here.

The picture below is a screen shot from the demo video produced by core AR Wave developer and concept designer, Thomas Wrobel.

Click on the image to enlarge, and note: “The pink thing is from Dennou Coil. Its an anti-virus program (that literally chase’s down bugs and glitches and removes them).”

ARWave

In ARWave, stories or art are tied to place. And as Jeremy Hight writes:

“The possibility exists to take a part of an area and overlay a dystopia, a utopia, multiples of each of these, or even recreations of previous incarnations in the past. Writing and publication thus cannot only be of place, and form(s), but of selected augmentations of icons, streets, buildings and related texts on top of the map. These spaces can be built in real time and can be turned on and off as channels of augmentation that over time illustrate many faces of place in its present, past, possible futures,etc. with texts within these alternate spaces as commentary, as fused aesthetic analysis, or simply creative writing relevant to these charged and hybrid spaces.”

As Thomas notes, Jeremy Hight’s,  “idea of channels ties into the concept of waves = a layer, and people can have many layers on at once.”

This is different from the Layar concept of a layer or rather “layar.”

“We are not talking about layers in the classical map layer way of thinking, where you have a layer of all restaurants or a layer of all mountain peaks, etc.,” notes ARWave developer Markus Strickler.

Currently all geo location apps like Layar have to use their own servers, so users have to use different clients with different log ins to see data from different sources.  But because ARWave uses federation, we don’t depend on centralized infrastructure where the client of one company can only connect to the server of that company.  This opens up many exciting new possibilities for how people can decide to view and publish geolocated data.

With AR Wave, via one login, people can access the whole distributed network of servers (see diagrams below), and any content will be accessible to them. ARWave will make it easy for individuals, not just developers, to layer their environment – allowing the creation of augmented reality content to be as simple as contributing to a Wave.

“ARWave will enable individuals to publish easily to everyone….or just a few people,” Thomas notes:

“To ‘publish’ is also self publication and distribution in communities or like minded groups without the hard read of publication or rejection.” = publishing on a Wave. No one approves it, anyone can publish to communities, or their friends and family. Or even just personal publishing it for their own reference.”

But ARWave does not compete with existing AR Browsers.   On the contrary, AR browsers like Layar, Wikitude and others, could implement ARWave and use it to enhance their applications.

“Layar has a killer browser already,  ARWave would add social features. They can keep their “walled garden” of data and still join the federation of open data too ” (Thomas Wrobel)

Yup, that is the cool part of federation – you can have your cake and eat it too!

Sophia Parafina and I will be organizing a discussion session on ARWave and Federation at WhereCamp, right after Where 2.0, April 3rd and 4th, and Dan Peterson who is in leading the federation effort for Google Wave will join us.

The diagrams below illustrate how ARWave and federation can revolutionize the way we share our augmented realities.

Real Time Social Augmented Experiences

Another key aspect of ARWave is it’s near to real time update capabilities.  As Jeff Pulver pointed out in, “SXSW 2010: The days twitter became less relevant:”

“At South By Southwest 2010 (SXSW), a strange thing happened on the way to Austin. A community of twitter faithful shifted from sharing everything about everything on only twitter (and maybe Facebook) and changed their habits to rely on learning about what was happening and where things were happening by using foursquare and Gowalla instead. I’m sure there were other products and platforms being used including Loopt and GySPii but foursquare and Gowalla were the dominant platforms.”

Later Jeff wrote:

“There were times where I could feel the ebbs and the flows of the people move as different people checked into various locations. While most of this was felt locally in the place I was in, it also became apparent on the platforms when hundreds of people would rush to check in to a location. There were also times when it felt like I was chasing ghosts; These were the times I would go to a spot because a friend had checked into that spot only to discover they were no longer there.”

ARWave’s realtime collaborative capabilities are going to introduce some fascinating dynamics to “chasing ghosts,” as the ARWave framework gets integrated into services like foursquare – a project we have already begun to look at.

Augmented Reality Search

As I mention in my previous post, ARWave presents some fascinating possibilities for AR Search.  For example, one might do advanced searching within waves using SPARQL, which could then display in the form of a personal blip in your viewpoint (which in turn could be shared with others).  Linked data will be massively important in filtering and delivering useful info for augmented views (see my conversation with Will Wright about the problem of augmented reality overriding our very smart instincts and not being useless or worse as a result).

Anselm Hook, who I interviewed in depth recently about, Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook, has some very interesting thoughts on real time stuff, trading brokerages, and  the view within a single city block, which he elaborated on in the second half to this interview which is upcoming on Ugotrade soon!

The ARWave Developers

There are three people who unfortunately can’t join us at Where 2.0 –  the costs of travelling from Europe being an obstacle.  But as they have been developing the code for ARWave that will rock our augmented world, I asked them, in a Wave conversation, to give me a few comments about their interest in working on ARWave, and a pic and a short bio.   Also I should mention the work of the PyGoWave team whose incredibly fast work creating PyGoWave has given ARWave a rocket launch pad.  Also many thanks to the Wave community, see the Wave Federation Protocol documentation, Google’s Wave Server, RubyOnSails (Ruby On Rails based Wave server).

Davide Carnovale @need2revolt

“Imho, the coolest geolocated related thing is that we’re making a world where the info does not necessarily comes from an explicit search from the user, but comes also from the actual locaton you’re in. For instance, you can have special offers in stores like fourquare does, or your friends can leave geolocated notes for you that are triggered when you walk by.  We can have games based on the treasure hunt schema requiring you to actually go to specific location.

Other than this I can think about self-guided tours of the city, maybe user generated too, or for museums.

Naturally these are long term goals with some rl use cases.

As for my bio, there isn’t much to say… I got a first level degree in computer science and I’m taking the second (and last) level. I’ve developed with mobile agents, osgart/artoolkit, brain computer interfaces, linux kernel and that’s pretty much all…”

Thomas Wrobel

“If you are looking for specific advantages of using Wave I’d say;

* Federated – Letting creators tap into bigger userbase. Each new app or data layer will add to the “incentive” for users to join in. Google had some good stats a few months back as to how much a simple login screen can put people off using stuff.  By breaking that barrier it should make AR userbase’s grow.

* It deals with user accounts, permissions, and real-time updating without creators needing to make a new server standard themselves. It lowers barriers to development.

* As the clients, servers, and data can be made separately by different parties, its easier for developers to concentrate on just providing what they want. You want to just make content? No problem! You dont need to worry about doing anything else but that. It would become as easy as making a webpage (or easier!).

* Bots will allow the development of interactive AR games very easily. Just like modern version of IRC bots, the infrastructure does the heavy lifting, and interesting things can be done with just simple scripting.

* The idea is anyone will be able to make a layer onto the world, and people can mix, match and share their layers as they wish. Its not just the data that becomes interesting to see augmenting our world, but the combinations of data! For example, perhaps you could see the profits generated by different companies above their buildings, but also see how environmentally friendly they are at the same time. Or maybe see pollution levels against health-statistics.  Seeing combinations of geolocated data from different sources at the same time has many interesting possibilities both for scientific as well as casual (game/ map/ chat) use.

hmz..I could go on forever listing stuff here really…..

I guess if we are supposed to be forming a roadmap of significant/interesting things for ARWave?

* Example clients letting people make their own layers (waves) and add points to them.

* Letting people log in to different servers

* Servers federated together. (not our responsibility, but essential part of the roadmap).

* Anyone logged into any server can see data from anyone else that’s shared with them, regardless of where they are logged into

* 3D support, demonstrating various sorts of geolocated data.?

* Use of bots for example games?
—-
My Bio’s quite simple.
Studied 3D Animation in Portsmouth, UK.
Moved to the Netherlands, have since been working in creating ARG games, in the last year founded Lostagain (Lostagain.nl).”

Markus Strickler @kusako

“I think the main point behind ARWave is to go beyond simply displaying existing placemarks on top of a live camera view, towards a highly personalized, augmented world where everybody can edit and share localized information collaboratively and in real time. Wave provides the means to do this through its model of persistent real time conversations and adds even more by providing a way for personal agents (robots) to participate in these conversations.

As for my Bio: I’ve been developing Web applications for the last 15 years, hold a degree in Image Sciences and am currently working as a Java developer in Cologne, Germany.”

The video above, The Imawik commercial, is a collaboration between In The Can Productions and Paige Saez for Makerlab

“The Imawik (ImageWiki) is a visual search tool for mobile devices. It allows for the ability to turn images into physical hyperlinks, conflating visual culture with a community-editable universal namespace for images.”

Paige Saez is an artist, designer and researcher.  In 2007 she founded Makerlab with Anselm Hook, an arts and technology incubator focused on civic and environmental projects.

Paige and Anselm (see my interview with Anselm Hook here, Visual Search, Augmented Reality and a Social Commons for the Physical World Platform: Interview with Anselm Hook) have been asking a very important question:

“Who Will Own Our Augmented Future?”

But most importantly, they have been actually developing applications (again see my interview with Anselm for more background on this), to allow people to play with, hack and explore and create with the physical world platform, and to imagine new possibilities for physical hyperlinking and augmented realities.  This is pretty important stuff, and kudos to Paige and Anselm for beginning this work before the big players – Google Goggles, Point and Find, and SnapTell came hurtling into the field of visual search and physical hyperlinking  – see this demonstration of translation and optical character recognition in Google Goggle’s.  Also check out Jamey Graham’s (Ricoh Research) Ignite presentation at Tools of Change, 2010 – Visual Search: Connecting Newspapers, Magazines and Books to Digital Information without Barcodes, for more see ricohinnovations.com/betalabs/visualsearch.

We are only just beginning to get a glimpse of how contested the social commons of the physical world platform is going to be – see the Yelp controversy.

As Paige points out:

“The lens that you are actually looking through was as important as what you were looking at. And democratizing that lens became the most important thing that we could possibly do.”

I am in total agreement.  One reason I have so much enthusiasm for ARWave (note: if you are interested in following the developer conversations there are several public Waves) is I see this open framework playing an important role in the democratization of our augmented views, by creating an open, distributed, and universally accessible platform for augmented reality that will allow the creation of augmented reality content and games to be as simple as making an html page, or contributing to a wiki.

Federation, real time collaboration, linked data – ARBlips that contain metadata that is usable for semantic searches, and modified wave servers that can listen to and respond to  SPARQL HTTP requests properly (see Jason Kolb’s many interesting posts on XMPP and Wave).  These are just some of the reasons why ARWave could revolutionize augmented reality searches and more! (see my presentation at MoMo13 – video here)

For more on real time social augmented experiences see our panel, The Next Wave of AR: Exploring Social Augmented Experiences at Where2.0 2010, and don’t miss the Where2.0 conference which has been the crucible for the emergence of location technologies.

Augmented realities, proximity- based social networks, mapping & location aware technologies, sensors everywhere, linked data, and human psychology are on a collision course in what Jesse Schell calls the “Gamepocalypse”   See Jesse Schell’s Dice 2010 talk here, and check out his Gamepocalypse Now blog.  As Bruce Sterling’s notes in his post here:

*Another precious half hour out of your life.  However: if you’re into interaction design, ubiquity, social networking, and trendspotting, in the gaming biz or out of it, you’re gonna wanna do yourself a favor and listen to this.

And don’t forget to register now for Augmented Reality Event (ARE2010 in 2-3 June, 2010 – Santa Clara, CA).

Bruce Sterling, Will Wright, and Jesse Schell will be keynoting, and there is a totally awesome line up of AR innovators and industry leaders, including Paige and Anselm!

And:

You are in luck!

Here is a discount code for the first 100 folks to register to the event (before the end of March). Go to the registration page, type in code AR245 and “you’ll be asked to pay only $245 for 2 full days of AR goodness.”

“Watching AR prophet Bruce Sterling, and gaming legend Will Wright, visionary game designer Jesse Schell deliver keynotes for this price – is a magnificent steal.  And on top, participating in more than 30 talks by AR industry leaders will turn these $254 into your best investment of the year,” as Ori  put is so well on Games Alfresco!

If you want a preview of just how exciting it is to be involved in augmented reality right now check out Ori Inbar’s great round up on our latest monthly Augmented Reality Meetup NY (or as, Ori notes, we fondly like to call it ARNY.)  There is lots of video up now (much thanks to Chris Grayson, who  live streamed it).  Augmented Reality Magician, Marco Tempest, is an absolutely must see.  (developers note this is an awesome use of open Frameworks and OpenCV).   The video of the show includes a rare explanation of how it all works  – see here.

Talking with Paige Saez – “Software is candy now!”

Tish Shute: What interests me about ImageWiki is that you have thought about physical hyperlinking beyond the obvious of where to get your next good hamburger and beer, right?

Paige Saez: Right. It was interesting for me in just thinking about the two things. How do you design a tool to work in a way that people are getting value from it? And also, how do you make it work in a way where people can explore and hack it? I think the most interesting technologies, and this is probably something somebody else said sometime, are the ones that disappear, that we don’t see, instead we see through. They become just the intermediaries.  They don’t interfere with what we are trying to do.

It’s a struggle whenever you are developing a new way for people to get information or make something happen, because you are playing with magic a little bit. And you have to make it vanish the way a good magic trick makes an experience a magical one. But at the same time you also need to reveal just enough that you let people in and they can see how to change it and make it their own. That is the interesting tension for this space right now, the idea of augmented reality begins to lead the idea of a social commons for physical things. The Imagewiki project was a locus of just this tension. Tish you and I have previously discussed how difficult it was to even get people to understand the two concepts independently.

Tish Shute: Right, until recently most people hadn’t even heard the term augmented reality and I am not sure that a particularly high percentage of people would recognize it now despite the recent interest in smart phone apps.

Paige Saez: It’s very difficult to get people to understand the two concepts, and now you are adding in the third level of participation as well. So I don’t think it is impossible, but I do think it requires narrative. It is interesting that you were talking about the stories you heard this morning from the creatives at the event [Tish mentioned David Curcurito, Creative Director, Esquire gave an excellent presentation at Sobel Media event NYC] because it’s narrative and the attention to telling a story that help you walk through all of the ways you can understand how completely expansive this area is right now.

So I think we have to play with it, play with the space and the tools. I think we need to have an idea of what we want people to use the tool for, and we need to not only introduce them to the tool and the technology, but also introduce them to the concepts as well. So I see it as a three part process.

I’m really excited to be there with people, helping them do that. I think we need to do this face to face. I don’t think this can be only through a social network. The ImageWiki website is like one quarter of the entire picture, you know? The website is the resource center and the place where you can see people adding images, but what value is it to you to see an added image? It is more valuable for you to be interacting with the image or interacting with the object in the real world.

Designing for the experience of using the ImageWiki got very complicated very fast. I was trying to figure out the main thrust of the design for the UI for the ImageWiki and at a certain point I had to take a step back and say “Okay, this has to be good enough for now because we can lay it out and prototype as long as we want on the Web or mobile UI. What we need to be doing is going outside and actually aggregating and putting images into the database in order to see what exactly happens when we are adding.”  It’s not just like you are taking a picture of something and adding it to Flickr. Using the tool is very context specific and the information is context specific, and you can’t necessarily make that all happen at the exact same time. I think these are really fascinating spaces to be struggling in and I’m so glad to be working in this space.

Images by Chris Blow of unthinkingly.com

Tish Shute: Could you explain why we need ImageWiki? I mean I think I have ideas on this, but perhaps you can explain to me from you point of view why we need an ImageWiki, as opposed, to say, extending the image space of Wikimedia or something added on to Flickr.  I mean maybe something leveraging the geotagged photos sets and APIs we already have?

Paige Saez: Yes, definitely. It’s a really good question, I mean it really is. Like, do you need an entirely new place to be holding images outside of the places that we are already holding images? That’s a huge question; enormous. Especially when you take a look at the problems around that. Its’ exhausting for an end user. Who the heck wants to go and reload everything into yet another place, right?

Tish Shute: Right.

Paige Saez: Moreover, who is going to really bother? Another problem would be what happens to the existing datasets that people have already committed to? And then of course there is the problem of authority and explanations why….Gaining interest and authority in a space when nobody even understands why that space should exist in the first place. And those are just three, you know, off the top of my head problems with that idea.

And yet at the same time, I don’t actually know how else to go about thinking about the ImageWiki unless I think about it as it’s own thing. Then you start thinking about models of large independant image databases that exist already, examples of this from a product standpoint- references to consider. The Getty Foundation comes to mind. There are many other historical centers that have huge resources and images that are licensed out and used. So here we have a working example of people already doing this. But succesfully? I don’t know. We do have a ton of intellectual property rights and copyright issues and ownership and use issues with images currently. As a working artist these issues for me were a major red flag to consider. Working on the social commons for augmented reality starts paralleling issues found in digital rights management and intellectual property.

Tish Shute: But one good thing about Wikimedia, why I focused on Wikimedia, is Flickr and Wikimedia already use a creative commons licensing, right?

Paige Saez: Creative commons, you know they have their own resource center, too. But you know they haven’t been successful as great databases for images so far.

Tish Shute: What would you like to see that they don’t have? Like say maybe start with Wikimedia, right?

Paige Saez: There’s just still a lot of issues with how to encourage people to want to contribute. It’s hard to show the value to someone who doesn’t already understand the value for some reason. At least for me personally this is something I have run into frequently. I don’t know if it is necessarily what Wikimedia doesn’t have, I think it is a lack of understanding of what creative commons really means. And there is still a very strong sense of ownership and concern about creative property rights. Being paid to be creative is a tremendously difficult thing to do. People fear losing their livelihoods. They think this is possible. Is it? I dunno.

For example : Look at me, I take a photograph of something, I can sell that. And there’s a question about whether or not, as an artist, I want to have my photographs in a pool of images that is open and accessible when I could be making money on it instead. Now that is just an example. Me personally, I can see the value. But that is a common concern. The gist of the question being, ‘what value does it bring to give something away versus holding on to it?’ A hugely popular discussion right now.

This is the same crux of the problem we are dealing with when we talk about thinking about images in the social commons for the real world. It’s a conversation about ownership. It’s about, who does this belong to really? If I take a photograph of a Levi’s billboard, does that photograph belong to me or does it belong to Levi’s? We know the boundaries of that. But when the image becomes a living image, an image capable of transmutation; an image that provokes an action or hyperlinks to a product, experience, information….where are the boundaries in that?

Tish Shute: But how is ImageWiki handling that differently from Wikimedia, I suppose is my question.

Paige Saez: We haven’t solved the problem.

Tish Shute: Yes, I suppose it is not like we have fully solve the problem of a creative commons for images on the internet let alone the issues of a social commons for the real world! So neither one has solved the problem, right?

Paige Saez: Exactly. To be honest, it made my head spin. I realized we were building a web application and a mobile tool doing augmented reality, real time feedback on the world and suddenly we weren’t. Suddenly we were dealing with DNS and talking about physical hyperlinks and ownership and property. And basically at that point you just have to sit and really start looking at catching up on IP issues and figuring out how to deal with that space in a much more wholistic way. It became so important that we had to take a step back and go

“Oh my god I think we have really uncovered a real problem here.”

At the point when we were building out the tools we realized something was really going on with our project. Here we were thinking that this was just a beautiful experience of learning about the world around us. We really…Anselm and I both just really wanted this tool to exist. It was something that we both just really wanted to happen in the world, something that we felt really just thrilled to make. And we looked at and used it and realized that instead of it just being a beautiful experience, it was a fundamental shift in how we understood everything. That it impacted our world in the same way the Internet impacted our world. It was a fundamental shift in understanding. A sea-change.

So I put down the prototype and went back to researching, read a ton of books on IP and went and presented to friends, family, schoolmates and co-workers trying to explain the project and then the larger conceptual framework that had emerged from the project. I began using the metaphor of thinking about Magritte’s “Ceci n’est pas une pipe.” Thinking about a pipe that isn’t actually a pipe.

Tish Shute: Oh, yes!

Paige Saez: ..to try to help explain to people that the image that you see is actually not, you know, it’s not an image of a thing. It’s an image. And that image has a tone and that image has a voice, and that image was chosen. And there were decisions that were made through the interface of the camera, specific decisions that defined the view of what you were looking at. And that that wasn’t being acknowledged and that that was a fundamental part of what the ImageWiki was aiming to do. The lens that you are actually looking through was as important as what you were looking at. And democratizing that lens became the most important thing that we could possibly do.

Tish Shute: So the emphasis for you on ImageWiki was in fact the lens, even though you found obstacles to creating the interface, right?

Paige Saez: Yes. Definitely. That’s what I fell in love with first. I really wanted to be able to use my phone to learn about what kind of tree this was or to buy tickets for the band on the poster I just saw, or see a hidden secret. For me it was very much a story, a narrative experience that I just thought was magical. And that is how I fell in love with it, which is not where I ended up. Where I ended up was realizing it was a fundamental shift in not only my own understanding of how to use the world around me, but in our understanding of looking at the world.

Tish Shute: It would be pretty scary if an image DNS was basically in the hands of either one or very few people, right? I mean even ImageWiki would be stuck with this problem, that if you set up a bunch of servers, you are going to be holding a very, very large image database. I mean, whatever your motivation, right? I think at the minute that is why I am very into seeing everything through the lens of federation, I see that unless we have federation, these giant central, databases are inevitable aren’t they?

Paige Saez: Essentially, yes. I mean I wasn’t able to walk through it as quickly as that. It kind of just overwhelmed me. Looking back on it, it seems perfectly obvious. I was just like “Oh my god, what have we done? Like what is going on?” Particularly for me because so much of my life has been spent in art, it was really easy to immediately understand the connection between the view, the viewer, and what’s being viewed as all just different layers of ownership and understanding that it is a gaze. Right? We know that we are never able to look at something without passing judgment on it, but to see that become a part of the interface in a real-time fashion just blew my mind.

Tish Shute: Yes.

Paige Saez: I think you are right. Getty Images, Flickr images, no matter what you are always holding on to something and you have to be responsible for it. Right? So how do you deal with the responsibility but don’t take on too much ownership? Where is the boundary with that?

Tish Shute: And for me, the simple answer to that is loosely connected small parts, distributed systems and federation. Because there is only one way to be able to utilize these things is to have them distributed so that no one holds all the cards. Right?

Paige Saez: Definitely and I personally agree with you wholeheartedly. However, the idea of distributed power is a concept that most people just don’t know how to deal with.

Tish Shute: And it’s easier said than done because actually the root problems that you are talking about aren’t got rid through federation, because if someone really holds the, sort of, all the good image databases just because they have the potential to be federated, they may not choose to open them up on many levels.

Paige Saez: And even then you have to think about, sort of, like the next level of it, which is we want it to be all open and accessible, but everything is owned by somebody. Like, what really is public anymore, in general?

Tish Shute: And what is interesting though, regardless of what we speculate conceptually on this, we already set off down the road. I mean we have already several large…they are all in beta I suppose, Google Goggles, Point and Find, right? But we have applications that are beginning to implement this. They are beginning to implement search on it, and it is geo-located even if it’s not in an augmented view, right? So it is proximity based.

Paige Saez: Right, right. I mean maybe the solution is that if we follow that line of thinking then Flickr will be partnering with Google Goggles. And then my images would stay under my ownership through the authority of Flickr. And I would use Flickr as my place to add images and they would just be responsive via my devices via AR.

Tish Shute: That’s very interesting.

Paige Saez: Definitely I think so. It is also the shortest distance between things.

Tish Shute: Yes, and as Anselm kept pointing out, basically it is going to happen in the simplest way possible, really, regardless of the implications of that. But OK, getting back to ImageWiki. As you say neither Wikimedia nor Flickr were really designed to take this role, right?

Paige Saez: Right.

Tish Shute: With ImageWiki, you’ve had these ideas and a concern with the social implications of physical hyperlinking in your mind since it’s inception. Are there any design ideas you’ve come up with that you know, as opposed to sort of, as you say, connecting Flickr to Point and Find, or who knows, Google Goggles. How is ImageWiki going to be different, do you think? Is that a hard question at this point?

Paige Saez: It is, and it’s a great question, and it’s a question I really love to think about. I think we have to introduce the politics with the tools. It has to be acknowledged that it’s not just a place to hold information, that’s what I feel in my heart.

At the same time, is that too much for people to really grasp at one time? In my experience it really has been, so the design of the experience needs to allow for an understanding of the power of the tool and the level of authority that the tool offers, while not getting in the way of it; just using it. Because ultimately, at the end of the day, nobody will use anything if it isn’t valuable to them. And so I could talk for miles and miles and miles about how important it is that corporations don’t own all of the rights to all of the visual things in my life, right? For the rest of my life I could talk about that. The idea that advertising is dominating all of our views of anything in the world around us is horrifying. It doesn’t matter unless I can show somebody why it matters to them or how it affects them. It’s just that that is a tremendously difficult thing to explain through a user interface.

And I actually think that it’s great that tools like Google Goggles and Nokia Point and Find are here to do a lot of the hard work of showing people how it works. Recently somebody explained to me their experience of using Google Goggles. They went through this process of saying how the Google Goggles took a picture and then did this really complicated visual scanning thing over the image and it took a full minute.

And I said, “Well of course they did it that way.” And they said, “Well what do you mean?” I said, “Well, what they are really doing there when they are doing all these fancy graphics, is they are showing you how it works.” And even if it isn’t actually related at all to how it functionally works, algorithmically, that’s not the point. The point is that this gesture of the time taken to make it look like it’s scanning an image and going back and forth with pretty colors is giving people the time to process that as an experience. That’s a metaphor for what’s really happening. And these kinds of metaphors are crucial with user experience design. We have lots and lots of examples of them and how they work, and many of them aren’t necessary. Like you know, for example, the bar that shows you the time it’s taking for something to process.There is no relationship between that and reality. But it is really important.

Tish Shute: Yes those bars often have no relationship between the actual time..

Paige Saez: And that’s the thing. Like the idea of time versus our perceived understanding of time. Right? The length of time it takes for your Firefox browser to open and load your last 30 tabs, versus the reality of what’s actually happening. When you are doing that sort of research you are actually accessing millions and millions of places and points of interest all over the world, so we need more of that. We need more of the process shown. Anselm and I worked with a film maker named Karl Lind from In the Can Productions here in Portland to try and make a video about the ImageWiki. We made this little video and I can try to show it to you or send it to you if you want.

Tish Shute: One of the issues with this kind of visual search is that it is inherently dependent on large databases, regardless of where they are federated, are going to be very large. Right? I mean someone is going to have something big, and aggregated there. I suppose someone will figure out the challenges of federated search eventually but that is quite a big challenge!

So I suppose I am still trying to understand what ImageWiki can offer that we can’t get with any other existing service? How will their be a social commons and even a social contract for the world as a platform for computing and physical hyperlinks?

Eben Moglen brought up something when I talked to him about virtual worlds, he said we need code angels to let us know what was going on in the virtual space – who was gathering data and how, for example.

Paige Saez: Tell me more about that, I want to hear more about that.

Tish Shute: Eben suggested this metaphor for when I was asking him about privacy in virtual worlds. The fact that people just didn’t know that when they were pushing avatars around virtual worlds what metrics were being gathered on their behavior. And he basically said that what we need is code angels when we enter these spaces because having the rules of the game buried in a TOC was ridiculous.

Paige Saez: That is a really interesting idea.

Tish Shute: Maybe ImageWiki needs to be our code angel to navigate the augmented world. I mean that’s what I want to see it as. And when I hear you talk, what I hear is you talking in broad categories about what a code angel might be in the space of images and image links to the physical world. I mean that is what I hear from you.

Paige Saez: Yeah. No, I definitely agree with that. It is interesting. In that sense, it is kind of a protection layer. Is that what you are thinking?

Tish Shute: Yes, I suppose because we can’t be navigating a lot of complicated opt-ins and opt-outs just to get around our neighborhood safely (in terms of privacy (also see Eben Moglen’s definition of privacy here…) We will need a code angel that is sort of keeping up with you in real time!

Paige Saez: Right, right. I wonder how that would work in regards to images, though. That is a really interesting thing to try and put on an image. I guess why I am having such a hard time being specific about it, is I am just trying to work it in my head, thinking of a specific use case, like what would be an example of that?

Tish Shute: Well I suppose the example, and this is a crude one, is when you point your Google Goggles to the book jacket, the code angel, this is very crude, would say “You are right now drawing images from the Amazon database – they are collecting data such and such data from your search.

And then of course the ability to have crowd sourced tagging and corrections..

There was a wonderful book that came out last year on how we can have commercial intelligence -Dan Goleman’s new book: “Ecological Intelligence: How Knowing the Hidden Impacts of What We Buy Can Change Everything”…

how corporations various different stakeholders, including their customers will drive corporations to do the morally right thing because they will lose the commercial support of customers who won’t support them unless they are more green, fairer, do the things we would like them to do whatever that happens to be – physical hyperlinking and tagging I guess would be a big part of this.

Paige Saez: Sort of a transparency issue. And that almost becomes a page rank algorithm in and of itself. I mean now we are really talking about search more than anything, and what tool becomes the dominant search tool. Anselm and I talked a lot about one platform… I mean eventually we will have a unified platform. It will…No matter what, for the Internet and for physical objects and visual objects in the real world. It will just be a matter of, literally, who can find the best and most valuable, most relevant information on a thing. Currently we just have it very proprietary.

Tish Shute: Yes.

Paige Saez: That definitely won’t last. It just can’t, because of the exact problem that you are raising. And we already know too much about resources and information as they pertain to products for us to ever go back to a time where we are not considering other ways of getting information about it anyway. Right?

Like I have the same concerns nowadays when I look at fruit. I look at a piece of fruit in the store. I would never just assume that the person who put the sticker on that fruit, anymore, is the ultimate authority necessarily. I would always assume at this point I could go online and go find out more information about a company. Issues about like eco-footprint or how much toxicity, or pesticides or whatnot are now totally accessible already.

So I am thinking when you look at that piece of fruit and that sticker for Google, say what you are describing, do we just go immediately to the company’s website, or is it even more specific? Do we know that the sticker on that piece of fruit is going to tell us specific information about that? Or are we just getting back the nutritional resources, or are we getting a listing of all of the different options out of a page rank algorithm that shows us, “Well this is the website for the fruit. Here is the nutritional information. Here are the last 15 comments on it.” It’s basically just a basic search.

Have you heard of Good Search?

Tish Shute: you mean http://en.wikipedia.org/wiki/GoodSearch

Paige Saez: Right.

Tish Shute: A code angel interface would have to give you options, wouldn’t it on possible views available?

Paige Saez: Yes. You are then talking about filtering your view. Then it really gets really interesting, of course. I don’t even know if we have a choice in that. I think we are really kind of hitting a wall with who owns the space and the platform. Is it just a basic search because we are already familiar with search? If you had an option to choose, say, “I want to look at this apple sticker and I only want to get…programmatically only looking at my friend’s opinions of this company.”

Or I have a safety valve on it that only shows me certain information based on what the code angel knows about me, my preferences, my age, things like that. Then that gets really, really interesting, because we are trying to do all that work right now just with social media and the Internet. We are already overwhelmed with too much information. It is already past the point of comprehension. So to think that we would actually drill down even more specifics is very interesting.

Tish Shute: That was a point Anselm made about the fact that once you are into this mobile, just in time, one view kind of situation, it is quite different than the Internet where you can bring up all these different screens and go to another website.

Paige Saez: Well yes, mobile is a different level of engagement. Very contextual. Much less information. Much more about timeliness. I don’t want to look an apple and get back a Google search. Oh my God no. That’s the last thing I want. I would love to be able to look at an apple and my phone already knows exactly what I want, information-wise, to get back from that apple. But I don’t know. It’s all contextual and personal. So I think the code angle concept you are talking about is really interesting because you still need to think about who is the person that is adding or creating those level filters- is it you, a filtered friend network, an algorithm? How much work is too much work? Where do we draw the line? How much of this are we willing to let the machine do for us?

Tish Shute: Right.

Paige Saez: And then of course once you have those filters in place, you need control over them. You will need to dial them up and dial them down, be able to choose and add new ones, so on and so forth. It becomes very modal at that point. For example, I want to change my view: To walk into a grocery store and instead of finding out information, I’d want to see where the hidden Easter egg puzzles were that my friends left last week because we’re playing a game.

I’m still really attracted to the creative opportunities with the ImageWiki. I’m really attracted to changing this experience from being a one-to-one relationship (from Corporation to Consumer) to an open-ended relationship (From Person to Person). If I look at a book jacket, sure I can find out where to buy the book, but that’s boring. Who cares? I’d like to find out a link to a story or an adventure or a movie or something unthought-of before.

How do we build that in? How do we encourage serendipity? Mystery? I think the ImageWiki is the space for building that in, actually. Not how, that would be the one place, right? That’s my really big fear is that this relationship just stays one-to-one. Click an image of consumable object, get back objects retail value. How completely dull. We have to do better than this.

Additionally, what if I want to take a photograph of a book, an apple, or something and I don’t want to pull back data. Instead, I want to pull back music, or I want to pull back a video, or I want to pull back a song, or lyrics, or a story, or another image. It’s just a hyperlink at the end of the day, you know? That’s all we’re really doing. Hyperlinks can pull back so many different things.

Tish Shute: And that’s one of the reasons I’m into mobile social interaction utility building, because without that, if we don’t have that way to do that in mobile technology…that’s very available on the Internet, as we’ve seen, with Twitter. These applications are very easy to do on the Internet. They’re not easy to do natively in a mobile application..

hey, I’m just promoting AR Wave again. I should shut up.

Paige Saez: Oh, no. I think it’s a fascinating concept, I really do. I totally agree. As we’ve talked about it before, it’s amazing that marketing and advertising are helping push forward AR, and it’s great. It’s fantastic.

But it’s also the worst possible thing that could ever happen because it is such a singular way of looking at an overall ubiquitous computing experience. There are other ways.

The best experience I ever had was trying to explain to people about physical hyperlinks. I had to walk them through it. Good interactive isn’t something you present or show, it’s something you do. Nothing beats just walking around and showing people with a device or a tool or something else.

I mean, God forbid it always stays in our computers and our phones. I really hope we don’t have to be stuck living our entire lives with these horrible interfaces. But for the time being, we will. Having an AR app show you a puzzle, or a mystery, or a game, or an adventure is a magnificent experience, totally overwhelming, and people get it right away. There’s no question; they totally understand.

Tish Shute: Yes, I agree.

Paige Saez: You walk them through the experience with a physical hyperlink and then you say, “Here, I could use this device and I could show you where to buy this thing, or I could use this device and we could start playing a game.” Then everybody gets it.

Tish Shute: So then I have a question, because one of the things Anselm said to me when he wanted me to refer back to you is that he feels that the direction for ImageWiki should be perhaps to focus less on the technology and more on just the actual, I suppose, gathering of the images, how they’re going to be annotated, the metadata, right? But my question to him was, the problem if you do that, without the platform, there’s no experience or motivation for people to do that. Right? Is there?

Paige Saez: Yeah, I agree with you on that one. I’m curious what his…I think the reason why he wants to do that is he wants to be able to show people examples via the resources. Like to be able to show someone a library, essentially, which I think makes sense with some people. I definitely think that some audiences would really relate to that. For me, it doesn’t make sense because I’m just very experiential. I need to do it and I need to show other people how to do it and I need to grow that way. I think that at the end of the day, those are great ways to go about doing it. It’s just it’s a huge thing to do in either direction.

What Anselm’s really thinking on, I believe, is more about exemplifying how we read and understand images culturally. Then you’re really getting into Visual Studies and Critical Theory which is what I did for my Masters at PNCA. I worked on the ImageWiki while I was in grad school, it was something I was doing for fun. Independently of my studies, the project lead to issues on democracy and objects and property and I ended up right smack in the middle of what I was studying; the nature and cultural analysis of images Questions like, ‘what exactly do we get out of images?’ and how all these different things are happening in an image, and people get tons of totally different things out of an image depending on many factors.

The questions I began to ask myself got very philosophical. Questions like “Is this apple red? Is this apple red-orange? Is this a small apple? What’s my understanding of small versus your understanding of small?”

Because you supposed that you needed a text backup to the search, how would I be able to search for an apple? Because what if my understanding of apple is red and your understanding of apple is green. And so if I’m looking for a green apple, am I looking for the same green apple as you? It’s all semantics, sure. But at the same time, it gets bigger and bigger, and it’s fascinating.

Tish Shute: Google Goggles seem to work best on book jackets, basically.

Paige Saez: But book jackets are actually perfect for this. Book jackets are perfect for this problem, because book jackets are specifically designed art. So at the end of the day, we are still talking about creative works, artistic works, that have been designed as a communication tool. But that is not something that people can own. Creative works that are designed are a communication tool, with varying levels of skill to be sure, but still something anybody can do. What we need to do is we need to be using that language. We don’t need to be trying to reach as far as facial recognition. We need to develop our own logos, our own brand, our own…I mean not brand. Brand is a bad way of saying it. Another way of saying it would be like, just use it. Develop a visual language that we can use that is as effective and as well utilized as book jackets or the movie posters or something.

Tish Shute: What are some of the use cases for ImageWiki you would like to develop first?

Paige Saez: My dream…I have like four or five use cases that I want to see happen. One of them is I walk down the street and there is a new poster for my favorite band. And I can just go up to the poster and I use my device, whatever it looks like, and I download the latest album. It’s transactional. I am able to just plug in my headset and walk down the street and the transaction is done. I saw something I wanted. It was beautiful. I was able to get it and I was able to move on in my life. And that is totally possible.

Another one would be I walk down the street and there is a piece of graffiti. And I am able to use my device to find out who the artist was that made it and to give them props, and to point my other friends to the fact that the piece is there and it will most likely be there only for a short period of time- information retrieval and socialization.

Or, use my device to find an Easter egg, to find a narrative puzzle that ends up going on for weeks, and everybody is involved, and we are all playing this game together. Adventure-based, non-linear experiences. I want playfulness, not just purchases.

Tish Shute: Did you think of piggybacking on the Flickr API for geo-tagged photos as a way to work with those databases or not?

Paige Saez: Yeah, we definitely thought about that.

Tish Shute: And why did you decide not to, for any reason or…?

Paige Saez: Ultimately, we just…we were such a small group, we just had to tackle certain things at a certain time.

Tish Shute: Right. And you were so prescient, you were working slightly before we had the mediating devices, weren’t you? You were just before the mobile devices really got adequate for this.

Paige Saez: Yeah. We started on it…I believe it was January…No. December 2007. Basically, the iPhone had just launched like maybe six months prior or something like that.

Tish Shute: But not 3G and not 3GS, right?

Paige Saez: Not 3GS. It was the first generation iPhone. We built the ImageWiki before the App Store existed.

We knew that the App Store was coming out. And we knew that the App Store was going to be the biggest thing in the whole world. I remember getting into multiple fights with friends about how revolutionary the iPhone and the App Store were going to be and people thinking I was totally crazy; people just thinking I was absolutely nuts for being so excited about it.

It sucks that it is a closed proprietary system, but the App Store has done something for software that nothing has ever done in the whole world. Software is candy now. It’s candy. It is like when you are waiting at the grocery store at the checkout line and you are stuck behind somebody, and you have got all these little tchotchka’s, candy bars, magazines, nail-clippers and things. That is the equivalent of software now. It’s become an impulse buy, which is amazing. Nobody would ever have thought…that is actually revolutionary. That’s huge.

Tish Shute: Steven Feiner, who is one of the founding fathers of augmented reality said to me during a conversations at the ARNY meetup that one reason that augmented reality, despite the hype, is manifesting very differently from how virtual reality burst onto the tech scene is that it is about affordable apps on affordable readily available hardware.

It is now nearly two weeks since the Google Wave preview launch and I am happy to say we have some AR Wave news. The diagram above shows Thomas Wrobel’s basic concept for a distributed, multi-user, open augmented reality framework based on the Google Wave Federation Protocol and servers (click on the image to see the dynamic annotated sketch or here).

Even in the short time we have had to explore Wave, some very exciting possibilities are becoming clear. Thomas puts some of the virtues of Wave as an AR enabler succinctly when he writes:

“Wave allows the advantages of both real-time communication, as well as the advantages of persistent hosting of data. It is both like IRC, and like a Wiki. It allows anyone to create a Wave, and share it with anyone else. It allows Waves to be edited at the same time by many people, or used as a private reference for just one person.

These are all incredibly useful properties for any AR experience, more so Wave is open. Anyone can make a server or client for Wave. Better yet, these servers will exchange data with each other, providing a seamless world for the user…..a single login will let you browse the whole world of public waves, regardless of who’s providing or hosting the data. Wave is also quite scalable and secure…data is only exchanged when necessary, and will stay local if no one else needs to view it.

Wave allows bots to run on it…allowing blips in a waves to be automatically updated, created or destroyed based on any criteria the coders choose. Wave even allows the playback of all edits since the wave was created.

For all these reasons and more, Wave makes a great platform for AR.”

There will be much more coming soon on Wave enabled AR because the Google Wave invites have begun to flow out to a wider community now. This week, many of our small ad-hoc group looking at the development challenges and implications of Google Wave for AR actually got into Wave for the first time.

Many thanks to all the people who have contributed to this discussion so far including: Thomas Wrobel, Thomas K. Carpenter, Jeremy Hight, Joe Lamantia, Clayton Lilly, Gene Becker and many others.

We will be setting up some public AR Framework Development Waves this week.  If you have any trouble finding them, or adding yourself to it, please add Thomas and I to your contact list.  I am tishshute@googlewave.com  Thomas is darkflame@googlewave.com  The first two are currently called:

AR Wave: Augmented Reality Wave Framework Development (developer forum)

AR Wave: Augmented Reality Wave Development (for general discussion)

The discussion so far has been in two areas. On the one hand, it is gear-heady and focused on the Google Wave Federation Protocol, code, development challenges, and interfacing to mobile, while on the other hand people have been looking at use cases and questions of user experience.

Distributed, “shared augmented realities,” or “social augmented experiences” – that not only allow mashups, & multisource data flows, but dynamic overlays (not limited to 3d), created by users, linked to location/place/time, and distributed to other users who wish to engage with the experience by viewing and co-creating elements for their own goals and benefit – are something very new for us to think about.

As, Joe Lamantia, puts it, now:

“there’s a feedback loop between which interactions are made easy by any given combo of device;/ hardware / software / connectivity, and the ways that people really work in real life (without any mediation / permeation by tech).”

Joe Lamantia whose term, “social augmented experiences” I borrow for this post title, has done some thinking about “concepts and models for understanding and contributing to shared augmented experiences, such as the social scales for interaction, and the challenges attendant to designing such interactions.” Check out Joe Lamantia’s blog for more on this later this week.

It is very helpful, as Joe points out, to shift the focus  back and forth between the experience and the medium.

It is super exciting to have clear evidence that shared augmented realities are no longer merely possible, but highly probable and actually do-able now.

I should  be absolutely clear about what Google Wave does to enable AR because obviously Wave plays no role in solving image recognition and tracking/registrations issues.  But, for example, Wave protocols and servers do provide a means to exchange, edit, and read data, and that enables distributed, social augmented realities.

Thomas explains how the newly named “AR Blip” works as:

“An AR Blip is simply a Blip in wave containing AR data. Typically this would be the positional and url data telling a AR browser to position a 3d object at a location in space.

In more generic terms, an AR Blip allows data of various forms (meshes,text,sound) to be given a real-world position.”

I have mentioned in other posts (here and here) that Wave can be used for AR as precise or as loose as the current generation devices can handle. And as the hardware and software for the kind of AR that can put media out in the world to truly immerse you in a mixed space, the framework  should  be able to handle this too.

(a note on the Wave playback feature – this opens up a whole new world of possibilities.  Check out this post on some of the implications of playback for writing!)

The use cases we have been coming up with are too numerous to go into in detail this post.  The open nature of an AR framework/Wave standard will lead to many new applications we have barely begun to imagine.  As Thomas points out, different client software can be made for browsing, potentially allowing for various specialist browsers, as well as more generic ones for typical  use. The multitudes of different kinds of data in/output that could be integrated in an open AR framework as it evolves are mind boggling.

But, for now, some  obvious use cases do come to mind:
eg.

- Historical environmental overlays showing how a city used to be/and how this vision may be constructed differently by different communities

- Proposed building work showing future changes to a structure/and the negotiations of this future (both the public and professionals could submit their own comments to the plans in context), seeing pipes, cables and other invisible elements that can help builders and engineers collaborate and do their work.

- Skinning the world with interactive fantasies

I asked Thomas to help people understand how Wave enables new interactions to data by explaining how Wave could enable citi sensing and citizen sensing projects (e.g. this one being pioneered by Griswold):

“Sensors, both mobile and static could contribute environmental data into city overlays;

Having these invisible aspects of the world made visible would create ways to improve sustainability, social equity, urban management, energy efficiency, public health, and allow communities to understand and become active participants in the ecosystems and infrastructure of their neighborhoods.

The key is reflecting this  kind of data back to people “making it not back story but fore story,” right where we are, right where it happens, as well as having it available for analysis.

As well as  creating new opportunities to interact/respond to/and enhance data, making visible the invisible as Natalie Jeremijenko’s work on Amphibious Architecture and Usman Haque’s project Natural Fuse shows, can also create new connections/understandings between humans and the non human’s that share our world, e.g. fish, plants, waterways.

At a more prosaic level  potential buyers of property could see more clearly what they are buying, city planners could see better what needs to be worked on, and environmental researchers could see more clearly the impact people are having on an area.

Also Wave can provide some of the framework necessary to begin to begin to address tricky problems of privacy. Sensitive data can be stored on private waves, e.g. medical data for doctors and researchers, but the analysis of the  data could still be of benefit to all, e.g., if it’s tied disease occurrences to locations and  relationships between the environmental data and health were…quite literally…made visible.

“The publication of energy consumption and making it visible as overlays, could help influence the public into supporting more energy efficiency companies and businesses. It could also help citizens to try to keep their own energy usage down, to try to keep their street in “the green.”

Thomas notes:

“With all of the above, it becomes fairly trivial to write persistent Wave-bots that automatically send notice when certain criteria are met (pollutants over a certain level, for example). On publicly readable waves, anyone can use the data in their local computers, process it, and contribute results back on a new wave. Alternatively, persistent remote severs could run Cron jobs, or other automated processing, using services such as App Engine to run wave robots.

All these possibilities become “free” when using Wave as a platform for geographically tied data.”

But of course this is just the beginning!

Recently, I talked at length with Jeremy Hight who has been thinking about, designing and creating shared augmented realities, that anticipate the kind of dynamic, real time, large scale architecture we now have available through Wave,  for quite some time now.   This is exciting stuff.

Modulated Mapping: Talking with Jeremy Hight about Layers, Channels and  Social Augmented Experiences

image from Volume Magazine (Hight/Wehby)

Tish Shute: I know you have been involved in locative media from its early days. Perhaps we can talk about how AR continues the locative media journey?

Blair MacIntyre gave me this distinction, recently: “AR is about systems that put media out in the world, and immerse you in a mixed space.  Even the current “not really registered” mobile phone AR systems are still “sort of” AR (e.g., Layar, etc).

Locative media/ubicomp/etc are very different, in that they tend to display media on a device (phone screen) that is relevant to your context, but does not attempt to merge it with the world.
The difference is significant, and making it clear helps people think about what they do and what they want to do, with their work. The locative media space though points toward future AR systems (when the technology catches up!).”

Jeremy Hight: The need is to finish the arc that locative media and early AR have started and to now truly return to the map itself, but as an internet of data, interactivity, channels of data , end user options like analog machines once were but in high end tools, a smart AI-ish ability for it to cull data for the user, and to allow social networking to be in real world places on the map both in building augmentation and in using and appreciating it..not hacks..which have their place…but a rhizome, a branched system with shared root,end user adjustable and variable..this is the key.

This takes AR and mapping and makes a possible world of channels in space and this eventually can be a kind of net we see in our field of vision with a selected percentage of visual field and placement so a geo-spatial net, a local to world wide fusion of lm into a tool and educational tool

VR[virtual reality] has greatly advanced, but in nodes as it has limitations…LM [locative media] is the same…AR [augmented reality] is the way.. it now has locative elements and aspects of VR integrated into its functionality and nodes…it is the best option with all of these elements, greater hybridity and data level potential a well as end user and community sourcing potential

I wrote an essay for Archis’ Volume, the architecture magazine on a near future sense of some of this….a visual net on the lens like ar but with smart objects and social networking and dissent.

I also wrote of these things for immersive graphic design, spatially aware museum  augmentation, education through ar and lm and nod to the base interface of eye to cerebral cortex in layered and malleable augmentation in my essay “Immersive Sight” a few years back

image [above] is simple illustration of a possible example on a screen or in front of eye where in a mondrian show..the graphic design of information actually builds as one moves

(key is calibrated spatial intervals and related layers of further augmentation which is logical due to location and proximity)

from immersive sight on immersive graphic design: “The design can work with this in a way that creates an interactive supplemental set of information that is malleable, shifts based on location, builds and peels away as one moves closer to a work and plays with the forms of the works and the elements of the space itself. The sequence can contain many different elements and their interplay (both in the field of vision and in terms of context and layers of information). This is the model of sections of augmentation turning on and off at key points as individual spatial and concepts moments and nodes.

Another interesting possibility is that individual points of augmentation don’t turn off, but instead are designed to build as one moves in a direction toward a specific part of the exhibit. The design can work in a sequence both content wise and visually in terms of a delay powered compositional development and style in which each discreet layer of text and image does not fade out, but builds on each other into a final composition. This can form paintings similar to Mondrian perhaps if it is a show of similar works of that era or it can form something much more metaphorical and open interpretation of the space and content but utilizing a sense of emergence spatially in terms of the composition (pieces laid bare until final approach for effect).

Each section will be well designed, but they build in layers as one moves until finally forming the final composition both visually and in terms of scope of information or building immediacy. The effect can be akin to taking a painting and slicing it into onion skin layers laid out in the air at intervals, each the same dimensions, but only one section compositionally of the greater whole. This has many semiotic applications beyond its potential aesthetically and as spatialized information possessing a sense of inter-relationship as one moves.“

Tish Shute: One of the things I found very inspiring when I read your papers was that your ideas are not all dependent on a model of AR that would necessarily require goggles, back packs and lots of CPU/GPU – not that that wouldn’t be nice, but that even using “magic lens” AR of the kind smart phones has enabled in an open distributed framework would open up a lot of new possibilities for what you call modulated mapping wouldn’t it?  What kind of social augmented realities might be enabled by a distributed infrastructure like this [AR Wave]?

Jeremy Hight: right….I see that as wayyy down the road…most important is the one you talk about as it is more immediate and thus more essential and needed. Eventually the goggles will be like a contact lens and a deep immersive ar version of  this will come, that to me is certain, but a ways down the road.  An incredible amount is possible now, and this is a more pragmatic move as opposed to the more theoretical of what is a few steps from here. Thus it is more important and essential now. Tools like Google Wave are taking what even 2 years ago was more theoretical discussions of what may be and instead introducing key elements to a more immediate, powerful, flexible level of augmentation. What have been hacks and isolated elements are to be integrated and social networking, task completion, shared tools and graphics building and geo-location.

Tish Shute: I think some people question what augmented reality has to bring to the continuum of location based experiences that other forms of interface/mapping do not?

Jeremy Hight: right….and the schism between its commercial flat self and tests with physics etc and in between …there are a lot of unfortunate assumptions it seems as to where ar and lm cross and how ar can be many things beyond deep immersion or the opposite pole of a hockey puck having a magic purple line etc….like lm is seen as either car directions or situationist experiments with deep data…..the progression to me is deeply organic….and now augmentation can be more malleable, variable and end user controlled.

Tish Shute: Yes, it is really exciting time for AR.  Historically AR research has gone after the hard problems of image recognition, tracking and registration because we have had available to us these dynamic, real time, large scale architectures like Wave available (until now!),  so less work has been done on exploring the possibilities for distributed AR fully integrated with the internet and WWW hasn’t it?

A distributed augmented reality framework such as we have envisaged on Wave would  allow people to see many layers from many different people at the same time. ‬And this kind of model has been part of your thinking and fundamental to your work for a while, hasn’t it? But it is a very new idea to most people to think about collaboratively editing layers on the world, and to be able to view  augmented space through channels and networked communities?  Could you explain some of the ways you have explored these ideas and how they could be explored further now to create meaningful experiences for people?

Jeremy Hight: right..exactly…modulated mapping to me can be an amazing tool for students…back end searching data visualizations and augmentations based on their needs…while they do something else on their computer or iphone…that can be amazing..and not deep immersive..The map can be active, malleable, open source fed, and even, in a sense, intelligent and able to adapt. The possibility also exists for this map to have a function that based on key words will search databases on-line to find maps, animations, histories and stories etc to place within it for your study and engagement. The map is thus a platform and yet is active. Community is possible as people can communicate graphically in works placed on the map and in building mode in the tool. All the tropes of locative media are to be in a mapping system of channels of augmentation and a spatial net. The software by design will allow development on the map and communication like programs such as second life but in mapping itself.

image from Parsons Journal of Information Mapping Volume 2 (Hight/Wehby)

I wrote an essay a few years ago for the Sarai reader questioning the traditional map and its semiotics and need to reconsider – then did work looking into it and what those dynamics were and they got into 2 group shows in museums in Russia…so it actually was my arc toward modulated mapping…an interesting way to it! But yes the map itself..this is a huge area of potential and non screen based alone navigation etc. I see now that my 2 dozen or so essays in lm,ar, interface design and augmentation have all also been leading in this direction for about 10 years now

Tish Shute: I  love immersive visualization but can we “return to the map – the internet of data” as you mentioned earlier and produce interesting augmentation experiences that go beyond locative media’s device display mode without having the goggles, for example, through the magic lens of or smart phones?

Jeremy Hight: yes, absolutely.  the map in the older paradigm is an artifice born often of war and border dispute and not of the earth itself and its processes…the new mapping like google maps is malleable, can be open source, can read spaces and can be layers of info in the related space not plucked from it as in the past..this is amazing. the old map also was born of false semiotics/semantics like “discovery of new lands” or ” pioneer”  while the places were there already and names often were of empire…now this is no longer the case

Tish Shute: So geoAR is an a better way to express a new social relationship to mapping? And how does this fit into the evolving arc of locative media that evolves into augmented reality?

Jeremy Hight:…early lm was mostly geocaching and drawing with gps..it took new paradigms to invigorate the field  a lot of folks focus on tools and what already is, cross pollination can ground ideas that are more radical…a metaphor in a sense to place what can be in a familiar context.

Tish Shute: one of the great disappointments in VR has been its isolation from networked computing and also, up to now, augmented reality – to achieve an immersive experience with  tight registration of media/graphics have to create separate system isolated from the internet and power of the web.

Jeremy Hight: yes….this will change. vr is to me an island but ar takes a part of it and shifts the paradigm and new things open this way. Do you know the project “life clipper”? friends of mine..doing interesting things..they are a clear bridge betwen lm and ar….and from vr

in ar augmentation and what is being augmented become fused or in collision or in complex interactions as a means to a larger contextualization and exploration of what is being augmented..this is true in immersive or non ar….huge potential

vr is a space, now can be surgery which is amazing. but not layered interaction, thus an island and graphic iconography on a location can use symbolic icons which opens up even more layers (graphic designer/information designer in me talking there I suppose..)

Tish Shute: Yes !  talk to me more about layers and channels I think this is one of the most interesting questions for me  in augmented reality at the moment – what can we do with layers and channels and the new possibilities on connections between people and environments that these can create?

The ability for anyone to post something is critical to the distributed idea but one of the reasons I am so excited by Google Wave is I am fascinated by the playback function. How do you think this will enable new forms of collaborative locative narratives (nice post on Wave playback here ).

Jeremy Hight: We are in an age of cartographic awareness unseen in hundreds of years. When was the last time that new mapping tools were sold in chain stores and installed in most vehicles? When was the last time that also the augmentation of maps was done by millions (Google map hacks, etc)? The ubiquitous gps maps run in automobiles while people post pictures and graphic pins to denote specific places on on-line maps.

The need is for a tool that combines all of these new elements into an open source, intuitive layered and rhizomatic map that is porous (like pumice, organic in form yet with “breathing room” ),ventilated (i.e: adjustable, a flow in and out), and open (open source,open access,open spatialized dialog).

I wrote of this in my essay “Revising the Map: Modulated Mapping and the Spatial Interface .”( http://piim.newschool.edu/journal/issues/2009/02/pdfs/ParsonsJournalForInformationMapping_Hight-Jeremy.pdf )

image from Parsons Journal of Information Mapping (Hight/Wehby)

Tish Shute: One mapping project I really like is Mannahatta.  How could distributed AR contribute to a project like Mannahatta?

Jeremy Hight: that is a good example..imagine taking manhattan and having channels of options to overlay, that being an excellent option, and imagine being able to even run a few at once with deliniating icons..you can augment a space with history, data, erasure, narrative, scientific analysis, time line of architecture, infrastructure, archaeological record etc….endless possibilities, and this agitates place and place on a map into an active field of information with end user control…and open options for new layers

Tish Shute: and do you think we could do interesting things with AR on a project like Mannahatta even with the current mediating devices we have available – i.e. our smart phones as obviously the rich pc experience of Mannhatta has built for it’s web interface would not be available as AR at this point?

Jeremy Hight: yes….k.i.s.s right?   these projects do not have to only be immersive and graphic intensive……take how people upload photos onto google maps….just make that on a menu of options, there are some pretty cool hacks already..
…options is key, a space can have a community as well, building on it in software, and others navigating it, i see it near future and down the road..always have with ar really

image from Volume Magazine (Hight/Wehby)

Jeremy Hight: and yes, a lot of people focus on ar as its limitations and processing power needs as a major road block

Tish Shute: so do you see AR on smart phones adding any value to a project like Mannahatta?

Jeremy Hight: yes…that it can be integrated into other similar works and even disparate but cloud linked ones…so a place can be “read” in diff ways on the iphone….beyond its map location, and more can be possible if you are there…others away, so it becomes channels of augmentation

Tish Shute: AR like locative media puts who you are, where you are, what you are doing, what is around you center stage in online experience but it also “puts media out in the world” – people I think understand this well as a single user experience but we are only just beginning to think about how this will manifest as a social experience – could explain more about modulated mapping as an experience of social augmentation?

Jeremy Hight: Modulated Mapping is a tool that will allow channels to be run along the map itself. This will allow one to view different icons and augmentations both as systems on the map and in deeper layers of information (photos, videos, animations,  visualizations, etc) that can be turned on and off as desired. The different layers of icons and data may be history, dissent, artworks, spatialized narratives, and annotations developed that are communally based on shared interests, placed spatially and far beyond. The use of chat functionality in text or audio will be open in building mode and in mapping navigation/usage as desired. This also allows a community to develop or augment in the spaces on the earth. These nodes can be larger and open or small and set by groups in their channel. The end result is an open source sense of mapping that will also have a needed sense of user control as one can select which layers of augmentation they wish to see and interact with at any time. It also will incorporate all the functionality of locative media in mapping software and mapping. In building mode and in map mode, icons will be coded to represent within channels (remember that the person using it has selected channels of augmentation from many based on their current interests and needs). Icons will be coded as active to show work in progress in cities and the globe to both invite participation and to further agitate the map from the sense of the static as action is visible even with its icons as people are working and community is formed in common interest/need .

locative media got a buzz for “reading” places…when I helped create locative narrative that was what blew me away back in 2001…that we could give places a voice by placing data from research and icons on a map……this meant lost history or augmentation was possible as kind of voices of a place and its layers…….I called it “narrative archaeology.” We now have tools that can push these ideas and concepts farther..much farther…and with a range beyond what was before, and then the map was just a tool….but now we are returning to the map itself…..and this as place as much as marker..this is where ar takes the ball to use a bad metaphor

also that project could only work if you came to our spot of a 4 block augmentation and with us there to lend you our gear…we are far beyond that now but it had its place

Tish Shute: How do you see “in context” AR and something we might call “context aware” cloud computing models interacting?

Jeremy Hight: sure…and I must add that I have issues with cloud computing as much as it is a good idea...

Tish Shute: because of loss of autonomy?

Jeremy Hight: tivo is simply a hard drive…but it keyword reads and gives  suggestions..that is the is cro magnon link to what can be

Tish Shute: The nice thing about Wave is because of the Federation model, the cloud model and local store ur own data models should work together.

Jeremy Hight: yes..that is better…..loss of autonomy also opens up the arbitrary which is the flaw of search engines as we know it…even Bing fails to me in that sense

Tish Shute: how do you mean, could you explain?

Jeremy Hight: spiders  cull from words  but cull like trawlers at sea …. tested Bing with very specific requests.. it spat out the same mass of mostly off topic results….
I wonder if there is a way to cull from key words and topics from a user…not Orwellian back end of course…but from their preferences, their searches etc..

Tish Shute: did you see the discussion on search in the AR Framework doc? AR search will be a massively important thing that will take a lot of intelligence and all sorts of algorithm development won’t it?

Jeremy Hight:It also has one area of key functionality that moves into more intuitive software. Upon continued usage, the mapping software will “learn” and search based on key words used and spheres of interest the user is mapping or observing as mapped and will integrate deeper data and types of animations, etc. into the map or will have them waiting to be integrated upon user approval as desired. Over time the level of sophistication of additions and of search intuition will increase dramatically. The search can also, if the user wishes, run in the back end while working in the mapping program, or in off time as selected while doing other tasks. It also can never be used if one is not interested. One of the key elements of this mapping is that it is not composed of a closed set or needs user hacks to augment, but instead is to evolve and deepen by user controls and desired as designed. Pre-existing data,visualizations and augmentations can be integrated with relative ease.

Tish Shute: One of the things that Joe Lamantia points out about social augmented experiences is that they will operate across a number of different scales – conversation > product design & build team > neighborhood / town fixing potholes > global community for causes. How do designs for channels and layers change across these different social scales?

Jeremy Hight: quote myself …”The “frontier” is often defined as the space just ahead of the known edge and limit, and where it may be pushed out deeper into the previously unknown. The frontier in the world of ideas is not the warm comfort of what has been long assimilated; and the frontier in the landscape is not of maps, but of places beyond and before them⁄

The border along what has been claimed is not only that of maps – it is of concepts, functions, inventions and related emergent industries. Ideas and innovations are like the cloud shape that briefly forms around a jet breaking the sound barrier, tangible yet not fully mapped into measure. It is when things are nailed down into specific entities, calibrated and assessed, that the dangers may inflict themselves – greed, competition, imitation, anger, jealously, a provincial sense of ownership either possessed or demanded”. (from essay in Sarai reader). Otherwise channels and augmentation do not have to be socio-economically stratifying or defined by them. We built 34n  for almost nothing on older tools.

image from 34north 118west  (Spellman/Hight/Knowlton)

The ar that is not deep immersion can be more readily available and channels can be what end users need like the diversity of chat rooms or range of Facebook users among us.

I had two moments yesterday that totally fit what we talked about.  I went to west hollywood book fair and traditional directions off of mapping for driving directions were wrong and we got lost…our friend could only get a wireless signal to map on itouch and we had to roam neighborhoods then we called a friend who google mapped it and we found we were a block away….so a fast geomapping overlay with an icon for the book fair on some optional grid service or community would have made it immediate.  Then at the book fair talked to a small press publisher who is trying to map works about los angeles by los angeles authors on a map..she was stunned when I told her it could be a kind of google map feature option

it also has great potential to publish and place writing and art in places..both for commentary and access. imagine reading joyce in chapters where it was written about and then another similar experience but with writers who published on a service into their city.

Tish Shute: The challenge of shared augmented realities is not just a matter of shipping bits around, but also of how it we will use channels and layars – to create and negotiate different, distributed perspectives, understand a shared common core/or expressions of dissent (this came up in an email conversation with Simon St Laurent).

Jeremy Hight: well my example earlier could have been communal in a way too..a tribe sort of augmentation channeling ….like subscribing to list servs back in the day but of augmentation communities/channels, and for folks to build and use in shared live form, coordinating too

Tish Shute: one good thing though about building an open AR Framework is that as bandwidth/CPU/hardware gets better shared high def immersive experiences could be supported by the same framework..

Jeremy Hight: excellent

Tish Shute: were you thinking of the image recognition and tracking with this example?

Jeremy Hight: yeah….like scanning across a multi channeled google map augmentation with diff icons and their connected data…and poss social networking and fle sharing even in that mode…and rastering etc….could be cool with google wave - on the map..then zooming in a la powers of ten..(eames film).

-I have pictured variations of this for a few years now in my head like the example of my friends and I yesterday…we could have correlated a destination by icons in diff channels..one being lit events within lit channel in l.a map…maybe things streaming on it too…remote info and video etc… that would be awesome

Tish Shute: So many of the ideas in you paper on modulated mapping (see here) are brilliant use cases for shared augmented realities. Perhaps you could talk more your ideas about locative narrative because this is something I think is at the core of the kinds of experiences that a distributed AR Framework would make possible?

Jeremy Hight: on the project “34 north 118 west” we mapped out a 4 block area for augmentation of sound files triggered by latitude and longitude on the gps grid and map and the map on the screen had pink rectangles that were the “hot spots” where the augmentation had been placed.

image of interactive map with map based augmentation connected to audio augmentation on site for 34north 118west (Spellman/Hight/Knowlton)

We researched the history of the area and placed moments in time of what had been there at specific locations ….I called this “narrative archaeology” as it allowed places to be “read” by their augmentations…info that was of the place beyond the immediate experience (diff types of info) that otherwise would be lost or only found in books or web sites elsewhere. there now are locative narratives around the world but they need to be linked.  from humble origins “narrative archaeology” went on to be recently named of the 4 primary texts in locative media which is pretty amazing to me…but it is growing

- the limitations then were what I called the “bowling alley connundrum” – the specifc data had to reset like pins…..and was isolated….this led me to think about ar back then and up to now.  How these could lead to much more from that point, data that would be more layered, variable , fluid..yet still augmented place and sense of place and social networking within data and software

lifeclipper to me is a bridge

Tish Shute: But Life Clipper is isolated from the internet currently is it?

Jeremy Hight: yes…ours was too.. that is what google wave makes possible.. our project only ran on our gear..in 4 blocks…with additional auxiliary info online, and not malleable..but hey 2001 and all..

Tish Shute: so the sites for 34 north 118 west are still active though?

Jeremy Hight: oh yeah!

Tish Shute: nice I really like sound augmentation – have you seen Soundwalk?

Jeremy Hight: yes, very cool.. we chose sound only as it fought the power of image..instead caused a person to be in a sense of two places and times at once

Tish Shute: and in 2001 that was definitely a visionary project!

You must be very excited that finally the pieces are coming together to make this stuff scale!

Jeremy Hight: I can’t even tell you!! it is funny..i have known that this would come..just waited and waited…

..knew it needed the right people and tools..

..so the bowling alley connundrum led me to develop my project shortlisted for the iss (international space station)  as I thought a lot about how points and works are not to be isolated…but connected  and should be flowing in diff parts of a map….to open up perspective and connected augmentations , but also to think about the map again…not as a base only. then moved into my work with new ways to visualize time and it all really began to gell.  The ideas first were published as an essay (http://www.fylkingen.se/hz/n8/hight.html) and later my project blog (http://floatingpointsspace.blogspot.com/)

Tish Shute: One thing I noticed when I was reading your paper is how you have been exploring non-euclidian geometries.  Could you explain how this is part of your idea of modulated mapping?

Jeremy Hight: Yes, this first came to me when my wife was reading to me from a book on the Poincare Conjecture and I was hit with a new way to measure events in time and after months of sketches, schematics and research came to see how it could also be connected to a geo-spatial web of projects and augmentations.  It was published in the inaugural issue of Parsons School of Design’s Journal of Information Mapping which was an exciting fit. I call it “Immersive Event Time”(http://piim.newschool.edu/journal/issues/2009/01/pdfs/ParsonsJournalForInformationMapping_Hight-Jeremy.pdf)

so the last 3 years I have been working on how it could all work as channels of augmentation, and building and navigation as open and community in a sense as well as ai capability that was the time work especially. how time as experienced within an event is not a time “line”  but points on and within a form….and how this model is better for visualizing events in time and documenting them. it actually sprang form reading a book on the poincare conjecture, popped a bunch of other stuff together so one could visualize an event in time as like being in the belly of a whale..with time as the ribs..and our measure of time as the skin…and moving within it….hoping this will be used as educational tool

and this also can be tied to ar and map again…how documentation of important events can be kept within icons on a google map..then download varying visualizations based on bandwidth and desired format

Tish Shute: I have been thinking about is the new forms of social interaction/agency that these kinds of augmentations of space/place/time will create.  it seems there are two poles – one is the area Natalie Jeremijenko explores of shifting social relations from institutions/statistics to real time/location based/interactions and new forms of social agency.  The other pole completely is more like the cloud based AI and perhaps crowd sourced machine learning.

Your ideas explore the possibilities of both these poles.  And certainly one of the big deals of distributed AR integrated with would be the possibilities it opened up both for new forms of networked social relationships and for new ways to draw on network effects.

Jeremy Hight: and cross pollinations within …that is what my mind goes to

Tish Shute: The other night I met Assaf Biderman, MIT, from the Trash Track team. Trash Track doesn’t utilize AR but I could see that there are possibilites there.
What do you think?

Jeremy Hight: yes, absolutely, there can sort of skins on locations that user end selection can yield …like channels of place….and can range from pragmatic core to art and play and places between….how this recalibrates the semiotics of map…more than just augmentation as seen as a kind of piggy back on map..map becomes interface and defanged platform if you wil, interestingly my more poetic/philosophic writing led me here too

Tish Shute: I know they are at very different poles of the system but I do wonder how AR can bring some of the level of social agency/interaction that Natalie Jeremijenko works on into a productive interaction with the kind of innovations in Machine learning that Dolores Lab style machine learning!!and others are pioneering?

Jeremy Hight: Natalie’s genius to me is in practical functional tech that also opens deeper questions and even new openings of what is needed..amazing layers in her work that way.. succint yet deep..very deep

Tish Shute: Yes – I a just writing a post about her work – I find it deeply moving the way she has delved into the possibilities to using technology to open us up to our world.  One of the reasons I find distributed AR so interesting is because it will make it possible for all kinds of people to create and use augmentation in their lives and communities.

So to return to how a distributed AR framework could contribute to a project like Trash Track?

Jeremy Hight: what about using it for community, dissent and awareness raising then?  like Natalie’s work but building like a communal work of multiple points, like the old adage of the elephant and the blind men  sorry..metaphor – like one of my points in immersive sight was how one could take augmentation as multiple works sort of turning the faces of a thing or place…and how this would make a larger work even in such a flow so people moving in a space could also build..

what of ar traces left as people move calibrated to user traffic and trash as estimated in an urban space…like it goes back to chris burden in the 70′s making you know that as you turn the turnstyle you are drilling into the foundation and may be the one that collapses the building?

so their movements leave trash. Natalie is all about raising awareness to cause and effect and data , space and ecology. love that.  so maybe …
a feedback loop , artifact and user end responsibility can leave traces …trash…

.. cybernetics vs ecology and human waste

Tish Shute: could you elaborate?

Jeremy Hight: brain fart…that the mass of trash people leave is a piece at a tiime….and how like the space shuttle mission when it was argued first true cybernaut occured….one cord to air for astronaut..one for computer on their back to fix broken bay arm…if there is a way to build on that and in relation to the topic…..how this can go further, that machines do not waste as much…as ar is a means to cybernetic raise awareness..eh..In a sense it is like  the space shuttle mission when arguably the first true cybernaut occurred….one cord to air for astronaut..one for computer on their back to fix broken bay arm…if there is a way to build on that and in relation to the topic…..how this can go further, that machines do not waste as much…as ar is a means to cybernetic raise awareness..eh.. hmmm... sensors etc…wearables too – could be eco awareness with data and machine and human

what about a cloud computing system with a slight ai in the sense of intuitive word cloud and interest scans…..so as one moves through say new york they can be offered new ai data and services as they move ? could also be of eco interests? concerns about urban farming, eco waste, air pollution etc….perhaps with (jeremijenko element here)  sensors placed in locations and these also giving data reads in public areas  with no input but hard data itself……hmm..could be interesting

it can also give info of the carbon footprints (estimated prob unless data is public record somehow) of chain businesses  and data on which are more eco friendly as well as an iconography color coded and icon coded to the best places to go to support greening and eco friendly business?  and the companies could promote themselves on this service to attract eco aware customers who would be seeing them as kindred spirits and helping the
larger effort?

kind of eco mapping..and ar on mobile app

what about sensors that read air pollution levels, levels of solar radiation (to aid with skin protection in shifting light values in a city space..ie put on some skin cream now…), light sensors that detect density and over density in public spaces…to use the old trope in art of reading crowds in a space..but instead could indicate overcrowding, failing infrastructure in public spaces (which is a congestion that leads to greater pollution levels as well as flaws in city planning over time..), and perhaps a tie in to wearables……worn sensors  on smart clothes….this could form a node network of people in the crowds ….and also send data within moving in a space…

here is a kooky thought… what of taking the computing power and data of people moving in a space..and not only get eco data and make available to them levels of
data..but make possibly a roving super computer…crunching the deeper data of people open to this……a hive crunching deeper analysis of the space, scan properties from sensors, and even a game theory esque algorithm of meta data if say 40 people out of 50 hit on a certain spike or reading…and even their input…..I worked in game theory for paleontology in this manner for a time as a teen….a private project……   the reading can lead to a sort of meta read by what hits most consistently..as well as in their input..text of what they experienced, observed,postulated,analyzed even…. this could be really interesting…even if just the last part from collected data and not from any complex branching of servers..

I thought at 19 or so that the flaw in paleontology was in how so many larger theories were shifting exhibitions and larger senses of things like were there pre-historic birds that were mistaken for amphibean and then back again….so why not make a computer program and feed all the papers published into it and see what hits were counted in terms of an emerging meta theory…and landscape of key points being agreed upon…this data would be in a sense both algorithmic and a sort of unspoken dialogue …came from a lot of study of game theory one summer…

hope this makes some sense…I forgot to mention that I originally planned to be a research meteorologist and my plan in middle school or so was to get a phd and develop new software to have a global map and then run models of hypothetical storms across it in real time animations of cloud forms, radar and wind analysis/fields, barometric pressure spaghetti charts etc….and to also do 3d cut away models of storm architectures…so been into visualizations of complex data and mapping for a long time!

Tish Shute: Wow let me think about this one!

Picture on the left Mirrorshades, picture on the right a Metroid Hud.

“Augmented Reality is like a Philip K Dick novel torn off its paperback rack and blasted out of iPhones,” Bruce Sterling in Beyond the Beyond

“a techno visionary dream come true – those are rare, really rare, you have to be patient,  it’s super cyberpunk”… Bruce Sterling, “At the Dawn of the Augmented Reality Industry.”

The Dawn of the Augmented Reality Industry continues to brighten, and now we have two augmented reality companies, Total Immersion and Ogmento, firmly established in Hollywood – the dream mother of so many of our augmented realities.

Ogmento is the most recent of these two pioneering augmented reality companies to set up shop in LA.  Total Immersion’s CEO Bruno Uzzan moved to LA from France two years ago, although he still has a fifty person RandD team in France.   Total Immersion began 10 years ago in the quiet, lonely, hours before the dawn of an AR industry.  But Total Immersion’s AR toys for Mattel, and augmented reality for Topps baseball cards, fired CNet writer Daniel Terdiman up enough to say, “I have seen the future of toys, and it is augmented reality” (see full post here on CNet).

Recently, I talked with  Ori Inbar, one of the founders of Ogmento and  the premier augmented reality blog Games Alfresco about his new venture in Hollywood. Bruce Sterling, @bruces, had some fun with my invention of brand new augmented reality trade jargon here!  Ori pointed out Ogmento brings two important new facets to the rapidly growing augmented reality field: firstly they are bringing leadership from veterans of the entertainment industry into augmented reality development. Brad Foxhoven and Brian Selzer from Blockade have partnered with Ori on Ogmento.  And, in an another important step forward for a young industry, Ogmento announced they will be acting as publishers for a fast growing cohort of augmented reality application developers and helping AR development teams out there bring their concepts to the market.

So I was very happy also to have the opportunity to talk with Brian Selzer.  Bruce Sterling pointed out in his seminal sermon from the flatlands at the Layar Developer Conference, AR is kind of a “Hollywood scene.” We have seen the web early adopter/developer/blogger community  embrace augmented reality browser experiences in recent weeks in an awesome wave of enthusiasm. Are Hollywood creatives equally smitten? For the answers see the full interview with Brian Selzer below.

Brian Selzer (www.brianselzer.com and twitter – brianse7en ) has an extensive involvement with emerging platforms:

“from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC [user generated content] projects (mashade.com, instafilms.com).”

I began my own career in Hollywood doing motion control photography and creating software that bridged the language of robotics and servo motors with the visions of  film directors. Eventually our little company, NPlus1, moved on to 3D vision systems and image recognition stuff.  So yes, I have been really, really patient waiting for this particular techno visionary dream.  And, while I have been waiting for augmented reality to manifest, I have grown to love the internet.  But now, how awesome, It is OMG finally for mobile AR!

Augmented reality is busting out all over – through our laptops, our phones, on the streets, toys, baseball cards, art installations, sticky light calligraphy and more.

Many of my questions to Brian were directed at at how and when we will see augmented realities with near field object recognition, image recognition and tracking and, of course, the illusive eyewear.  As Bruce Sterling points out we are just at the very, very beginning – the dawn of an industry.  I created the photomontage below on the right to compliment Tonchidot’s illustration suggesting the evolutionary inevitability of holding our phones up (below on the left).  The Evolutionary Reality of AR will not end there.  It is just a step into eyewear, hummingbirds or Nano Air Vehicles, and more…….

The Evolutionary Reality of AR

Cartoon on the left by Tonchidot on the right a collage of a stock photo and the Pentagon’s Robot Humming Birds – “Nano Air Vehicles.”

While we finally we have, an affordable mediating device with the horse power, mindshare and business model to bring AR mainstream with the iphone.  The much anticipated Apple 3.1 Beta SDK to be released in September will not, I am sure, open up the Video API at the levels that augmented realities with near field object recognition and tracking require (I would love to be proved wrong though). But the magic wand to deliver even tightly registered AR graphics/media (that require a lot of CPU and GPU) to a wide audience is in our hands, so full access to may not be far off. And others, of course, can/will/might knock the iphone off its current pedestal.  AR made it’s mobile phone debut on the Android after all.

Like everyone else who loves AR, I wish that Apple would open up faster (and I wish Android would manifest on some rocking hardware). But we will see enough of the iphone Video API open for the next generation of mobile augmented reality games and applications to emerge in the coming months.

One of these will be Ogmento’s. Although Ogmento is in stealth mode, they have released a teaser for their first game, “Put A Spell,” developed by ARBalloon – screenshot below.  Ori did reveal to me in this interview that they are doing image recognition and using the Imagination AR engine.

As Brian notes, Hollywood has had the AR bug for a long time. AR has been everywhere in Science Fiction Movies and video games. Nintendo’s SPD3 head Kensuke Tanabe, “effectively the man in charge of overseeing all the Metroid franchise underneath original co-creator Yoshio Sakamoto,” explains the story of Metroid to Brandon Boyer of Offworld here (an image of a Metroid Hud on the right opening this post) :

“the idea of the different visors you use in the Prime games to interact with the world: the scan visor, for instance, set the game apart from other first person shooters in that the player was using it to proactively collect information from the world, rather than having the story come to them passively, in the form of cut-scenes or narration. “Prime could have adventure elements with the introduction of this visor,” says Tanabe, “That’s how we came up with the genre — first person adventure, instead of shooter.”

But as Brian points out:

“the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear).”

Talking with Brian Selzer

Tish Shute: Bruce Sterling’s sermon at the Layar Developer conference, “At the Dawn of the Augmented Reality Industry,” was absolutely awesome. He spread the future feast/orgy of augmented reality before us  – and described many of the dishes we will tasting both delectable and diabolical.  One of the many things he points out is, AR is kind of a “Hollywood scene.” And, as Ogmento is one of only two augmented reality companies in Hollywood, I am interested to hear how it looks from your neck of the woods. We have seen the web early adopter/developer/blogger community  embrace augmented reality browser in recent weeks in an awesome wave of enthusiam – are Hollywood creatives catching the buzz?

Brian Selzer:  It was a thrill to hear Bruce Sterling mention Ogmento. I devoured all of his Cyberpunk books back in the 80′s, along with writers like Gibson, Rucker, Shirley… To me, sci-fi writers are the visionaries who define and influence our technological paths into the future. They make science and tech sexy enough to want to manifest those experiences in the real world. Clearly Bruce sees the AR industry as being sexy. I love that he called it “a techno-visionary dream come true… and super-cyberpunk.”  And yes, kind of a Hollywood scene.

Hollywood creatives caught the AR bug before they knew what AR was.  Look at science fiction movies and video games to see AR everywhere. Terminator, The Matrix, Minority Report, Iron Man.. the list goes on.  Look at any video game with an integrated heads-up display.  It’s clear Hollywood loves AR.  It’s only been in the past few months though that the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear). So yes, the buzz is here and it’s strong.  With that, we all have to be prepared for the good, the bad and the ugly as AR goes mainstream.

It certainly goes to show how young this industry is when Ogmento and Total Immersion are currently the only AR companies based in Los Angeles. It’s very exciting to be the only company right now demonstrating a natural feature tracking (markerless) iPhone experience in Hollywood. We are in talks to bring some very big brand and properties to the mobile AR space. The goal is to deliver experiences that create added engagement and value to the consumer.

Tish Shute: Also in his landmark sermon Bruce Sterling noted that augmented reality has been around for 17 yrs and now at last we are seeing the dawning of  an augmented reality industry. What inspired you to take up the challenge of launching an augmented reality company in Hollywood?  Oh congrats that Bruce Sterling name checked Ogmento in his list of companies that prove that this really is the dawn of an industry!

Brian Selzer: I’ve always been involved in emerging platforms… from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC projects (mashade.com, instafilms.com). Working with all these great CG game assets, I continued to think about what’s next, and that’s when I started to follow AR very closely and started engaging with those who were pioneering in the space.

I remember swapping instant messages with Robert Rice (@robertrice) right after the 2008 Super Bowl.  We were not chatting about the football game, but rather about some of the commercials that aired during the event as a sign that AR was making its way into the mainstream.  A lot of people became aware of AR for the first time when the GE SmartGrid commercial aired.  There were all these YouTube videos popping up of people blowing on holographic wind turbines.

The commercial that really got me excited though was the Coke Avatar commercial.  In that commercial people in the city were sporadically being portrayed as their digital persona’s, avatars, gaming characters, etc..  For me that spot did a great job showing how many of us already have these ‘alter egos” that live in cyberspace, and how the line between these worlds can sometimes be blurred. I remember watching that commercial and thinking that is exactly the type of experience I’d like to create with mobile AR.  I want to overlap the virtual world into our every-day reality. Why cant I bring my World of Warcraft or Second Life persona with me into the real world?

I am big on the notion of “Games and Goals.” I believe that games have the power to motivate people in a very powerful way. By challenging ourselves while playing a game we can climb mountains.  Augmented Reality is the perfect platform to bring gaming into the real world.  By mixing the virtual world with the physical world, this added layer of perception provides a very powerful experience for something like a role-playing game.

One of my earlier social-gaming projects was a website called Superdudes.  This was a “Be Your Own Superhero” concept that celebrated and motivated kids to create superhero avatar/persona’s online, and we gave members all sorts of games, challenges, and rewards, some of which carried into the real world. The site recognized members for teamwork, creativity, volunteer work and things like that. So the Superdudes were often involved in charity events and benefits to help children.  Everybody called each other by their Superhero names, and the line between fantasy and reality were being blurred.  This project really got me thinking about what happens when you take positive role-playing like this and mix it into the real world.  I started to work on a plan for location-based activist missions for points and rewards, but never got to complete that. So I have some unfinished business here.

I think it would be fantastic to be able to show up to some type of fun event with friends, and everybody could see each others alter ego personas standing before them. When you can turn the world into a playground, and use the power of gaming to make a positive impact on the planet… well, I don’t think there is anything better than that.  These are the types of projects that drive me, and I think AR is the best platform to support these types of social gaming experiences.

Tish: Does Ogmento have any RPGs under development?  I noticed in the Google Wave on RPG someone has been working on doing something with the Dungeons&Dragons API.  I am interested in exploring the web of protocols underlying Wave as a transport mechanism for multi-person, mobile, AR experiences (not requiring downloads), on an open global outdoor AR network. If not Wave, what do you see as the potential infrastrucure and protocols we could harness for an open augmented reality network?

Brian:  Ogmento has a deep background in video games and we interact regularly with most of the major game publishers. As a company we are not so much developing our own RPGs right now, but rather exploring what mobile AR extensions make sense for existing brands.  There are many limitations to location-based gaming, but a global AR network is exactly along the lines we are thinking.  Lots of discussions are taking place on protocols, platforms, API’s, and there are numerous ways to approach this.  We need to be able to use what’s available now and continue to refine and customize for AR’s specific needs and issues as we progress.

In general though, Ogmento is focused on what types of experiences can be had today and over the next couple of years. I still think we are several years out from a truly open augmented reality network.  We are certainly looking at launching our own “Ogmented Network” which would support some fun treasure hunt type experiences, or add an entertainment layer on top of traditional outdoor marketing campaigns.

Tish: I don’t know whether you have read Thomas Wrobel’s ideas for an open augmented reality network that I just published here on Ugotrade.  The principals he talks about are very important for augmented reality to become a major part of our lives – .  Considering the difficulty open networks can pose for emerging business models how can we fund the development of an open framework for augmented reality?

“a future AR Network, I mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.
Where people can just point their phone, webcam, or pair of AR glasses anywhere were a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.”

Brian: I think funding for these types of projects will definitely come from Venture Capital groups in the near future.  It’s early in AR, but the VC’s are watching and deciding which horses to bet on.  Until that time, it’s about service work, and developing AR experiences for others with what is possible today. That work will help fund internal development of original AR products, and platform development.

Tish: How did you get started with Ogmento?

Brian: My first conversation with Ori was actually about my interest in Location Based RPG concepts.   We had a long conversation about the possibilities with AR, and it was clear that we shared similar interests, but were coming from different complimentary backgrounds. The idea of collaboration was exciting, so we just kept talking until the timing felt right. Now, with Ogmento we bring a unique blend of AR development experience with a deep backgrounds in AR technology, animation, video games, entertainment, social media, etc.   I think this is a powerful mix that will allow us to do some great things.

It’s still so early, and things are just getting started in AR. There are only so many webcam magic tricks you can enjoy before you are ready for something else.  The location-based apps have the most potential in my opinion, which is why we are really focused on mobile AR.   We have some board-game type projects, which do not instantly scream location-based gaming, but if you look at something like the ARhrrr board game, you can see how much more compelling it can be when the game invites the player to be actively moving around during the experience.

Tish: I am interested in your perspective on how we can create the kind AR experiences that really embody what has always been so exciting about AR – the tight alignment of graphics and media with real world objects and ultimately a rich immersive 3D experience, so I am going to hit you with a bunch of those, “Is this really eyewear or vaporware?” questions.  The real deal eyewear changes everything!

While eyeware is a big challenge technically and aesthetically,  I am pretty sure that there are several outfits out there that can pull off the optics and projection. 
Will the entertainment industry get excited enough to put a major push into delivering the eyewear in short order instead of the 5 to 10 year project that some people still think it is?    The business development challenge is bigger perhaps than the technical obstacles perhaps? What is your view on this?

And, perhaps, the eyewear is a clear example of a need for partnerships. For example, we have seen efforts from companies like Vuzix and Lumus, and recently a Japanese Company, Masunaga.

I have no reports from people who have tried the Maunaga eyewear yet.  But,  limited by small field of view, and tethered, currently eyewear offerings, available at a reasonable price point, are not workable solutions for augmented reality experiences. But the problems are not insurmountable. What will facilitate the real deal?  


It seems that it is critical to start creating hardware relationships now. The industry is costly and slow moving and as Robert Rice put it to me in a recent conversation, “once the software cat is out of the bag, its going to go wild and if the hardware isnt there, its going to stutter.”

As Ori notes some of the hardware companies like Intel and others don’t seem to be paying enough attention to AR.  Ori points out they don’t see the demand yet.  But in order to create an awesome AR experience and demand from a mass audience, don’t we need to work in conjunction with hardware designers?

Brian: It’s fun to think about who will eventually deliver a great hardware solution for AR glasses. It will happen. It would be cool to see somebody like an Oakley or Nike partnered up with a company like Vuzix to deliver something people actually might wear in public.  Perhaps a hardware manufacturer like Apple or Nokia will bring us something like the iSight or the NGaze down the line.  I’d love to see a set of glasses designed by Ideo.  Microsoft or Sony are already playing with technologies like Project Natale and the EyeToy, so I think its only a matter of time before they deliver an eyewear solution. I would even look to the toy companies to eventually make an investment here.

Gamers will be the early adopters, and in a few years we may start to see people running around in the park wearing glasses with headsets, but it will be acceptable because it’s clear they are using them for a game.  It’s going to take a very sexy and stylish piece of hardware for everyday people to be willing to wear AR glasses in public while going about their everyday business.   It’s like the recent cover of Wired magazine where Brad Pitt is wearing a mobile headset in his ear, and the editors point out that even he can’t pull that look off, so why do you think you can.  When AR glasses come in designer frames, and you can’t tell them from non-AR glasses, to me that’s when things get really interesting from a mass-adoption perspective.   Compare how many people were carrying around a mobile phone in the 80s to now.  I think it will be the same thing with glasses.

I was in an AR pitch meeting the other week at a very significant media company, and brought up the point that today’s handheld Smartphones will eventually evolve into tomorrows Smartglasses. My comment was quickly shrugged off as sort of a sci-fi notion that was irrelevant to the business at hand.  Probably true, but I think it is important to understand where digital media and entertainment is going, so you can adapt quickly, and evolve into those spaces more naturally.  The more we see people walking around with their Smartphones in front of their face (like a camera), the sooner it will be that we make the jump to eyeglasses as a key hardware device for AR experiences.

At Ogmento, we definitely are working on AR experiences with the hardware and software available today.  We will get some product out this year, and 2010 will be a banner year for markerless mobile AR in general.  I think the entire AR community is looking forward to bringing this technology to the mainstream in the form of games, marketing campaigns, virtual docent apps, and much more.  It might not be the full experience we are all dreaming about for some time, but we can see the path and the true potential, and it’s pretty spectacular.

You mention the tight alignment of graphics and media with real world objects.  That is really our focus. A lot of well-deserved attention is going to the browser overlay “post-it” approach right now, which uses compass and GPS.  We are focused on markerless natural feature tracking, so once you identify something that is AR enhanced in your environment, you can interact with that integrated experience.  On an iPhone that can be as simple as using your touch screen to interact.  When you are wearing glasses, it becomes more about visual tracking. There are lots of smart people thinking through these issues. Many of which you have interviewed. It is my hope that there are exciting collaborative efforts to be had in the coming months to get us all there together and faster.

Tish: Bruce touched on some of the hard problems that have to be solved for augmented reality – and he noted for instance security needs to be tackled in the early stages. Robert made a nice list, “privacy, media persistence, spam, creating UI conventions, security, tagging and annotation standards, contextual search, intelligent agents, seamless integration and access of external sensors or data sources, telecom fragmentation, privilege and trust systems, and a variety of others.” Will Ogmento be leading the way in solving some of these hard problems?

And, won’t trying to solve these hard problems for networked AR in walled garden scenarios one company at a time lead to a lot of reinventing the wheel wasted energy?

Brian: These are all important issues, and again there are a lot of smart people thinking about solutions to these problems on a daily basis.  Ogmento is interested in partnering with developers and supporting their efforts as a publisher of mobile AR experiences.  While we intend to roll up our sleeves in these areas, we are currently more focused on taking AR mainstream with the hardware and software available today.  As the industry evolves, so will Ogmento. As the opportunities evolve, our ability to make a greater impact tackling these issues will be realized.

Tish: Another area of development that could really kick AR into high gear might be creating augmented reality hotspots  where we use can deliver the kind of location accuracy/instrumentation necessary to create interesting AR experiences (partnership with Starbucks, perhaps ?!).  Augmented reality hots spots, could deliver the kind of high quality AR experience that isn’t possible ubiquitously at the moment, and may be a real way to get people really exploring the potential of AR now, rather than later?

Brian:  Agreed. I see a great opportunity here with this approach.

Tish: Although there are many obstacles to Green AR – the energy hogging servers at the backend for starters! Last week I had a conversation with Gavin Starks, AMEE, and Robert Rice and Jim Purbrick about how to work with AMEE and the technology available and encourage Green Tech AR development (see an early exploration of green tech AR from Pachube here).

We came up with the idea of holding a competition perhaps centered around a targeted instrumented space. But I would really love to hear your thoughts on the topic of Green Tech AR (the energy hogging servers at the back end being the first cloud on the horizon!.)  Cool GreenTech AR imaginings, social gaming ideas, RPGs, not even necessarily even tied to the immediately practical, would be like rain in a drought!

Brian:  I go back to “Games and Goals”… If you make environmental and other activist efforts fun and rewarding, more are likely to be motivated and participate.  Can you imagine having a personal “carbon footprint stat” floating over your self at all times? Or over your home or factory?  How would that change your behavior?  We all love stats. Look at how the Nike+ campaign has used technology and gaming to motivate people to run.  I think there is a lot that can be done to make being green fun. It starts with the individual, and spreads from there.  Keep me posted on that one!

Tish: I would also like to explore further the RRW suggestion that ambient intelligence is both the Holy Grail of AR and possibly snake oil:

“The holy grail of the mobile AR industry is to find a way to deliver the right information to a user before the user needs it, and without the user having to search for it. This holy grail is likely in a ditch somewhere beside a well-traveled road in the district of the semantic Web, ambient intelligence and the Internet of things. Be wary of any hyped-up invitation to invest in a company that claims to have gotten the opportunity right. What we’ve seen in the commercial industry to date is a rather complex version of a keyboard, mouse, and monitor.”

So Holy Grail, Snake Oil, or a ditch somewhere….?

Brian:  I instantly think of Minority Report, where Tom Cruise’s character is being bombarded with holographic ads personalized with his name and to his current situation.  In the future, Spam is a nightmare, especially when it knows who you are.  I think the key thing here is delivering “the right information”, and we still dont have that down. I do see a day where we can truly customize what comes to us, how we want it, when we want it.  My future vision of ambient intelligence is the ability to “turn everything off” if I want to… block out the stimuli and replace it with images of nature, or natural surroundings, etc.  Where I live in Los Angeles, we have those digital billboards everywhere, so it’s like advertising overload wherever you look (hints of Blade Runner).  I personally don’t mind them, but I know there is great debate on there being simply too many billboards everywhere. So AR would only add to the noise of life by adding yet another digital overlay of information, right?

Perhaps the holy grail is to use technology to filter things out. AR might become a solution to leading a simpler life, or a perfectly customized life if you want that. Ultimately the control needs to be with the individual.  I guess I am talking about something like TiVo taken to the extreme.

Tish: And then that other biggy – augmented reality search! I am asking this next question of  Wikitude and Sekai camera too and now I must also ask Acrossair and several others I guess! Obviously a huge area of opportunity in this broader landscape that uses location-awareness, barcode scanners, image recognition and augmented reality is to harness the collective intelligence – a whole new field of search. There is the beginning of a discussion on this in the comments here.

What will it take, in your view, to become a leader in augmented reality search?

Brian:  I’m more of a content guy, so I tend to focus on things like UI, quality of creative, etc..  From that perspective, I am looking forward to evolving beyond the “post-it” text overlay user-experience we see now in AR search. I was impressed with the TAT Augmented ID concept and hope we start seeing more smart design solutions like that emerging in the space.  There are some great new design approaches coming out of the location-aware space that should be applied to AR search. I’ve been studying the heads-up display designs being used in video games, and re-watching movies like Iron Man for ideas. This is another example where Hollywood has painted a polished picture of what AR can and should look like, and the masses have already accepted these design approaches.  So from that perspective, from my view the leaders in search will be delivering sexy, smart and simple solutions. It’s all about the S’s.