Images  from Mitsuo Iso’s Denno Coil (Click to enlarge), the game “Metroid Prime,” and Terminator.

Thomas Wrobel, Sophia Parafina, Joe Lamantia, Matthieu Pierce, and I will lead a  session tomorrow for AR DevCampNYC introducing the AR Wave Project.  Thomas, Joe and Matthieu will be participate via skype (10am to 11.30am EST), and Sophia Parafina and I will both be at AR DevCampNYC at the The Open Planning Project office (TOPP).  The PyGoWave crew will be introducing PyGoWave via LiveStream.

At 1.30pm EST to 2.30pm EST there will be a shared PyGoWave/AR Wave session with Mountain View (if bandwidth permits).

The skype conference will be at ardevcampnyc .  To participate in Wave,  please join the public Wave,  AR Wave: AR DevCamp Session.  There is also a AR Wave Wiki up now – see here.

Dimitri Darras (avatar Dimitri Illios) is working on streaming the AR DevCampNYC sessions into Second Life,  SLURL here.

Thomas has done a very nice introduction and FAQ below.  This should help people new to this project to get up to speed quickly.

There are already several Waves that show the history of this project including: AR Wave: Augmented Reality Framework Development,  AR Wave Use Cases, PyGoWave AR Tech Discussion,  AR Wave Augmented Reality Wave Development, AR Wave / Muku Organization and Admin.

Also I have several posts for people interested in more of the background, including: The Next Wave of AR: Mobile Social Interaction Right Here, Right Now!, AR Wave: Layers and Channels of Social Augmented Experiences, Total Immersion and the “Transfigured City:” Shared Augmented Realities, the “Web Squared Era,” and Google Wave.

Thomas uses the term Arn (augmented reality network) which is one of the candidate names for the project, Muku (crest of a Wave) is another suggestion.  Thomas’ intro and FAQ below can also be found here.

What is the AR Wave Project?

In simple terms its a protocol for storing geolocated data on Wave servers that’s currently being developed.

We believe this will help lay the foundations for an open, universally accessible, and decentralised system for shared augmented reality overlays which various clients can connect to and use.

This AR Network should spark a lot more rapid adoption of AR technologies, give existing browsers more functionality, and provide the network infrastructure, allowing many of the fictional depictions of AR to become a reality one day.

The AR Network.

When we speak of a future AR Network, we mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.

Where people can just point their phone, webcam, or pair of AR glasses anywhere where a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.

The Arn should be an inclusive and open platform where any number of devices can connect to, and anyone can make and host their own location-specific models or data.

It should allow people to communicate both publicly and privately, and not have their vision constantly cluttered with things they don’t want to see.

This is our vision, and we think a Wave protocol will help it become a reality.

Why Wave?

Wave allows the advantages of both real-time communication, as well as the advantages of persistent hosting of data. It is both like IRC, and like a Wiki. It allows anyone to create a Wave, and share it with anyone else. It allows Waves to be edited at the same time by many people, or used as a private reference for just one person.

These are all incredibly useful properties for any AR-experience, more so Wave is open. Anyone can make a server or client for Wave. Better yet, these servers will exchange data with each other, providing a seamless world for the user: a single login will let you browse the whole world of public waves, regardless of who’s providing or hosting the data. Wave is also quite scalable and secure: data is only exchanged when necessary, and will stay local to just one server if no one else needs to view it.

Wave allows bots to run on it and thus allowing blips in a waves to be automatically updated, created or destroyed based on any criteria the coders choose. Wave even allows the playback of all edits since the wave was created.

For all these reasons and a few more, Wave makes a great platform for AR.

How?

In basic terms, we will diverse a standard way to geolocate a bit of data and store it as a Blip within a wave.

This data could be a 3d mesh, a bit of text, or even a piece of audio.

Then various clients on various devices could logon, locate, interpret and display this data as they see fit.

Click on image above to enlarge.

A typical example of this might be holding up your phone and seeing messages written by your friends and family in the locations which they are relevant.

You could see an arrow hovering over the café your meeting a friend at, notes above their flat saying if they are in or out, or messages by shops telling you to pick up the particular brand of cereal they like.

This data would be personal to just yourself and whoever you invite to share that wave with.

Other forms of data could be public, like city-maps, online games, or historical landmarks being recreated. Custom views of the world with data for entertainment, commercial, environmental or informative purposes.

The possibilities with geolocated data are endless, as are the various ways to display and make use of them.

One of the things I’m most passionate about is people being able to see many different types of data, both public and private at the same time and from many different sources at once.

For instance, if your playing a AR game, why shouldn’t your chat window be viewable at the same time?

If you have skinned your environment with a custom view of the world, why shouldn’t you also see mapping or restaurant recommendations?

The ways to present these layers of data and toggle them on/off in the most intuitive and flexible ways would be a task for the client markers, and I’m sure we will see many innovations in those areas.

But by using Wave it at least provides the framework for having multiple information sources controlled by many different people yet accessible, and user-submittable, via the same protocol.

Who?

This idea first sprouted from a paper I route focusing on the potential for IRC to be used for AR;

http://www.lostagain.nl/testSite/projects/Arn/AR_paper.pdf

I suggested near the end Wave might be a better alternative (using Google Wave was an idea Tish Shute, Ugotrade, brought up in response to the Arn prototype design on IRC), and it quickly became apparent that Wave was a very suitable medium.

Since then, there was a lot of interest, and numerous people have offered to help.

In particular, recently, the PygoWave team is helping us out, as they have an existing server supporting c/s protocol, which is currently being actively developed.

Where?

You can join the general discussion here;
Augmented Reality Wave Development

Technical side here;
Augmented Reality Wave Framework Development

When?

There’s lots still to do, and we are at an early stage.

Our current targets: (last updated 11/12/2009)

• Getting reading/writing of prototype ARBlips to the PygoWave sever. (the PygoWave team have already made a standalone client and have the protocol for this sorted!)
• Establishing a minimal spec for ARBlips to be later expanded.
• Writing a very simple prototype online client showing how to store/retrieve the data.
• Expanding client to work for some use-cases.
• Establish a logo/branding for the project.

Other FAQs.

Where’s the catch?

While we believe Wave is highly suitable for development, it has the drawbacks of being a new system with just a few servers worldwide, which (at the time of writing this), have not yet been federated together yet.

Naturally, as a new technology, its likely to have some growing pains. And building a new technology on other new technology will multiply that somewhat. The first pain is the lack of a standard client / sever protocol. PygoWave have stepped in to the rescue a bit here, by being not just one of the most developed Wave server other then Google, but also leaping ahead with support for Json based c/s interaction. Google has stated they want community to take the lead on on a c/s protocol, so we are hoping they will adopt a Json variant, or a XMPP one and add it to the spec. We hope in much the same way as POP3/IMAP have been a standard for email server interaction, a similar one will develop for Wave.

In the meantime we plan to keep the code for writing ARBlips somewhat abstracted so as to make it easy to adapt in future.

As for the newness of Wave and other potential problems it will bring, we aren’t that worried as its built on XMPP, which has proved reliable already.

The other catch is we are unfunded, which slows development down considerable as we have to fit it around our other jobs.

I’m making my own AR Browser, and am slightly interested in maybe supporting you.

We are naturally very keen for support, and particularly for those with skills and visions to give feedback on the proposed protocol. Specifically: what do you want stored in a blip?

That’s what’s important at this stage.

We don’t see the Arn as a replacement for existing browser systems at the moment. We don’t want to restrict innovation or development in this fast developing market as we are very impressed at what’s been achieved so far. In many ways our task is small in comparison to what’s already accomplished.

However, we do believe the Arn will make a good addition to existing browser systems. It will allow users contribute data and have social features without having to worry about accounts or hosting.

It will still be quite some work to support; new GUIs will need to be developed to make it easy to submit data from the devices, as well as to login to waves.

However, we hope over time to build a set of example libs to make the read/writing of ARBlips as as easy as possible to implement in your software.

Perhaps a good way to think about it is existing AR Browsers are like word-processors, supporting the Arn will be like adding support for *.txt, but doesn’t limit what you can do with your own format.

Eventually we do hope ARBlips hosted on Wave will become the majority of AR data, and its functionality will be analogous to the internet is today. We truly believe in the long run a standard is essential.

But for now we think merely getting a baseline format established for how AR data can be communicated will increase user-ability, usefulness, and help the market grow.

Can I help?

Sure.

We particularly need people with technical skills in relevant fields. (both gwt/javascript web programming and c++(/qt)standalone programming help very welcome!).

But we also welcome people just with vision to help focus use-cases and to conceptualise what we want to be able to do with the system.

Please either join the relevant AR Waves or Wiki

We are especially interested in those with JSON and Comet experience. Specifically those with the abilities to make standalone applications to read/write to a sever using these methods.

What type of data will a AR Blip store?

This is still actively being decided, but essentially its a physical hyperlink.

A connection between a physical location (or object, see below) and a piece of data.

Specifically, we are thinking about the following fields;

Location in X,Y,Z,
Coordinate System used for the above,
Orientation,
MIMEType [the type of data stored]
DataItself [either a http link for 3d meshs and other larger data, or an inline text string if its just a comment]
DataUpdateTimestamp [so clients know if its necessary redownload]
Editors [the user/s that edited/created this blip]
ReferanceLink [data needed to tie the object at a non-fixed location, such as an image to align it to an object in realtime],
Metatags [to describe the data]

Are you purely tying stuff to fixed geolocations?

Certainly not
As part of of the spec we wish to be able for people to be able to link data to dynamically moving objects, trackable by image or other methods.

The idea being that one day someone could link a piece of text or 3d mesh to an image on a t-shirt they are wearing, or perhaps link a dynamically updating twitter feed, or perhaps provide information on a product (based on its logo).

There’s a large number of possibility’s for image-based linking alone, and that’s not even considering possibilities like linking RFIDs, or other forms of less precise but invisible binding data.

We need a lot of feedback from those companies already doing markless tracking. What types of images do you need, idly to link a mesh to an object? is one enough?

Summary of AR Wave Work to Date

Purpose: To provide an open, distributed, and universally accessible platform for augmented reality. To allow the creation of augmented reality content to be as simple as making an html page, or contributing to a wiki.

Specific Goal: To establish a method for geolocating digital data in physical space (or linking it to physical objects) using wave as a platform.

(For justification as to why we are using Wave see: our faq )

Wave as a platform

We are developing on the PyGoWave server at the moment but the goal is to be compatible with all Wave servers

PyGoWave has already achieved an important aspect in enabling the project in being a waveserver with a working and well documented server protocol. This allows both standalone and webbased clients to interface with it already.  See - The PyGoWave Qt-Based Desktop Client

This is one of the reasons why we have chosen to develop for the Pygo server at this stage.

However, the overall goal of AR Wave is to have a framework compatible with all servers using the Wave Federation Protocol. As more wave servers get c/s protocols then ARblips (the data needed to geolocate objects) could be posted and retrieved from various servers using the same client software. For this a standard should emerge. Just as websites don’t have to be hosted on specific servers, neither should AR data need to be hosted on specific wave servers.

In order to reach our goal, there are a few very achievable steps involved – see below.

Feedback

We are still actively seeking feedback, so feel free to join the Wave discussions, and see the history of how the specifications of the protocol evolved. You can also read the justification for some of the choices already made. Note a new discussion for AR DevCamp will be begin at AR Wave: AR DevCamp Session

This will, of course, only be the first draft of the specification, and it is sure to develop much in future.
The important thing now is to make working prototypes while maintaining flexibility.

So what do we need to do?

Steps :

* Establish the overall method – Done.

Each Wave will be a layer on reality which an individual or a group can create.  Each Blip in this Wave refers to either a small piece of inline data (like text) or a remote piece of larger data (like a 3D mesh) as well as the data needed to pin-point it in either relative or absolute real space.
We call these blips: ARblips. They are simply blips that stored the data necessary to augment a single object onto a specific bit reality.

It is up to the clients how they interpret and display the data. They could interpret it as a simple 2d list of nearby objects, or as an advanced 3D overlay, whereby multiple waves from different sources could to be viewed at once. What’s important is that there is a standard way to link the digital data to the real world space.

* Establishing the specification for the ARblip – In progress
We have a good idea of what’s needed to be stored in an ARblip, and we have hammered out a rough format.
The data might be stored as blip-annotations, but this has yet to be finalised.
A rough outline of the type of data stored can be seen in this c++/qt header for ARblip data can be seen at the end of this document.

* Storing and retrieving these pieces of ARblip data on the PyGo server – In progress.
The Pygowave team has made some excellent libraries that should make reading and writing data on the PyGoWave server very trivial for those with c++ skills.
This, however, is a real critical step, so more developers with C++ skills are very welcome!

* Making the above client mobile, and using a devices gps device to place the data. – Not started.
The next step would be to port the code to a mobile phone and use it’s gps-input  to post geolocated data and view what others have posted. This would be a fairly simple and not to useful app in itself. However, it would mark the first time anyone could post AR data and anyone could view it, all using open-source infrastructure.
As a bonus, because we are using wave infrastructure, the updates to any ARblip should appear in near realtime.

* To continue with the proof of concept, we would like to have simultaneous wave input from a PC
and mobile phone at the same time. – Not started.
For example, someone could post a pin on Google maps API and have that data posted to a ARBlip in a wave. Someone logged into that wave on their mobile device would then see the data posted appear.
More so we hope that when the Google map pin is dragged about, the mobile phone viewer, with just a few seconds lag, will see its location updated in real time.

We hope to make a modest yet practical app at this stage.

* After all this, we can go onto the interesting things:
3D data, camera-overlays, data fixed to objects and many more.  There’s plenty of existing software using these features (such as Wikitude, Layer) and some that are even open source software (like Gamaray and Flashkit). The open source code can give us a leg-up. However, we prefer to establish the protocol first. So naturally, these fancy features aren’t a priority for us. Rather we think our energy is better spent establishing the protocols and infrastructure so that other people can build more advanced bit of software easier.

However, once our primary goals are established, we will look to make a open source augmented reality browser ourself which will surely feature many of these features.

Overall, we hope once we have a simple proof of concept, there will be many groups, both existing and new, wanting to use this Wave system for their own apps, games and data.

Conclusion:
Really it’s now all about growing the community. We hope as soon as we show how great Wave can be for augmented reality, that lots of individuals and teams will start making their own clients to read/write geolocated data.
Overall we don’t think anything we make will be that impressive in itself. That’s not our goal.
We instead hope that our project will enable AR-content to be made as easily as web content. That games, information and apps will be able to be created without the creators having to worry
about the infrastructure behind it.

Technical information -

Current ARBlip header file

(below is a c++/qt header file for an ARBlip object that should illustrate the data being stored)

{

public:

arblip();

~arblip();

arblip(QString,QString,double,double,double,int,int,int,QString);

QString getDataAsString();

QString getEditors();

QString getRefID();

QString getXAsString();

QString getYAsString();

QString getZAsString(); bool isFaceingSprite(); 

private:

//ID reference. This would be a unique identifier for the blip. Presumably the same as Wave uses itself.

QString ReferanceID;

//Last editor(s)

QString Editors;

int PermissionFlags = 68356;  // default 664 octal = rw-rw-r–

//Location

double Xpos;   // left/right

double Ypos;   // up/down

double Zpos;  // front/back

//Orientation

// names, ranges and directions are taken from aeronautics.

// If no orientation is specified, it’s assumed to be a facing sprite.

// Roll: rotation around the front to back (z) axis. (Lean left or right.)

// range +/- 180 degrees with + values moving the objects right side down.

int Roll;

// Pitch: rotation around the left to right (x) axis. (tilt up or down)

// Range +/- 90 degrees with + values moving the objects front up. (looking up)

int Pitch;

// Yaw: rotation around the vertical (y) axis. (turn left or right.)

// range +/- 180 degrees with + values moving the objects face to its right.

int Yaw;

bool FacingSprite; //if no rotation specified, this should default to true

//if set to true when a rotation is set, then it keeps that rotation relative to the viewer

//not relative to the earth.

//Data format

QString DataMIME;

QString CordinateSystemUsed; //The co-ordinate system used. This should be a string representing a Open Geospatial Consortium standard. This could be earth-relative for gps co-ordinates, or in some cases relative to the viewer, for data to be displayed in a HUD like style.

//Data itself

QString Data;

QString DataUpdatedTimestamp; //Time the Data was updated changed

//Note; A seperate timestamp should be used for updates that dont effect the data itself.
//(such as if a 3d object moves, but its mesh isnt changed)

//Data metadata QMap<QString, QString> Metadata;

};

Above is an image above  from Total Immersion’s augmented reality experience developed for the “Networked City” exhibition in South Korea, – “a fun scenario created for a u-City’s infrastructure and city management service”

“To the naked eye, the exhibit looks like a bare bones model of a city. But when visitors put on the special AR goggles a whole new world unfolds – as graphics overlaid on the city model.” (Games Alfresco)

“The Networked City,” is a large scale augmented virtuality of a scenario for a networked city. But my guess, reading the     Korea IT Times, is the plan is to move from an augmented virtuality to an augmented reality as Incheon Free Economic Zone  (IFEZ) realizes its vision to become a leading u-City – where reality is turned “inside out” (see Inside Out: Interaction Design for Augmented Reality ).  If you are not familiar with South Korea’s u-Cities, check out this post for a short primer (and note Google Trends search on Augmented Reality shows  South Korea leaving everyone else in the dust).

Ubiquitous computing and augmented reality are like adenine and thymine – a DNA base pair.

A sky view of Incheon Free Economic Zone (from Korean IT Times). For more on the IFEZ vision to become a leading u-City see here.

Korea IT Times writes about the u-city concept:

“Korea began using the term u-City after accepting the concept of ubiquitous computing, a post-desktop model of human-computer interaction created by Mark Weiser, the chief technologist of the Xerox Palo Alto Research Center in California, in 1998. There have been a lot of research in this field since 2002. As a result, many local governments in Korea have applied this concept to various development projects since 2005 based on a practical approach to it.”

The back story to many of my recent posts, including this one, is an understanding of a relationship between ubiquitous computing and augmented reality that emerged, for me, in a February conversation with Adam Greenfield, Towards a Newer Urbanism: Talking Cities, Networks, and Publics with Adam Greenfield.  In cased you missed it, here is the link again because I think it holds up very well considering the rapid developments of recent months.  Also, importantly for this post, it includes a discussion of  moving on from Weiserian visions.

Adam Greenfield’s Speedbird is one of my key sources for understanding “networked urbanism,” and the list he makes of the elements of networked urbanism here (also see the comments) – is my mantra for thinking about the DNA base pair relationship of augmented reality and ubiquitous computing.

Adam Greenfield’s, “summary of what those of us who are thinking, writing and speaking about networked urbanism seem to be seeing” is:

1. From latent to explicit; 2. From browse to search; 3. From held to shared; 4. From expiring to persistent; 5. From deferred to real-time; 6. From passive to interactive; 7. From component to resource; 8. From constant to variable; 9. From wayfinding to wayshowing; 10. From object to service; 11. From vehicle to mobility; 12. From community to social network; 13. From ownership to use; 14. From consumer to constituent.

Augmented Reality – Making Visible the Invisible

The screenshot above is one of  the coolest “making visible the invisible” AR applications. It was developed at Columbia University Graphics and User Interface Lab where Steven Feiner is Director (see the deep list of projects from the lab here).  This app “shows carbon monoxide levels projected over New York City. The height of each ball reflects concentrations of the pollutant.” Credit: Sean White and Steven Feiner  (via Technology Review).

The recent emergence of “magic lens” augmented reality apps for our smart phones – Wikitude, Layar, Acrossair, Sekai Camera, and many others now, have given us a new window into our cities. But we are yet to realize the full potential of the AR/ubicomp base pair that can “make visible the invisible” and give us new opportunities to relate to the invisible data ecosystems of our cities, not merely as a spectator experience,  but as an interactive, in context, real time opportunity to reimagine social relations.

Mark Shepard says in his curatorial statement for, “Toward the Sentient City:” (Much more soon on this very significant exhibit which runs from Sept. 17th to Nov. 7th, 2009.)

“In place of natural weather systems, however, today we find the dataclouds of 21st century urban space increasingly shaping our experience of this city and the choices we make there.”

Augmented Reality, as Joe Lamantia points out, is becoming the great “ambassador of ubiqitous computing.” AR is. “…mak[ing] it possible to experience the new world of ubiquitous computing by reifying the digital layer that permeates our inside-out world,” and we are only just glimpsing the razor thin end of the wedge in this regard.

I am still working on my Gov 2.0 Summit write up  and, amongst other things, I will talk about how an emerging new social contract around open data, here in the US,  will put augmented reality  apps center stage  – “doing stuff that matters.” At Gov 2.0 Expo Showcase Tim O’Reilly tweeted:

Tim O’Reilly Really enjoyed @capttaco (Digital Arch Design) @ #gov20e: “Augmented Reality could be a new public infrastructure” http://bit.ly/18iCx

Also see Tim O’Reilly and Jennifer Pahlka on Forbes.com discuss the The “Web Squared” Era -  “the Web Squared era is an era of augmented reality arriving (like the sensor revolution) stealthily, in more pedestrian clothes than we expected.… …our world will have “information shadows.” Augmented reality amounts to information shadows made visible.”

Again there is back story to how I came to think about Information Shadows in relation to augmented reality.  So in case your missed it the first time, here is the link to a conversation that began in a hallway meeting between Tim O’Reilly, Mike Kuniavsky, ThingM, Usman Haque, Pachube, and Gavin Starks, AMEE, at ETech earlier this year,  “Dematerializing the World, Shadows, Subscriptions and Things as Services: Talking With Mike Kuniavsky at ETech 2009.”

Slide from Augmented City Lab @ Picnic ’09

So What’s Next for Mobile Augmented Reality?

These videos from Daniel Wagner’s team from Graz University of Technology showing Realtime Panorama Mapping and Tracking on Mobile Phones and Creating an Indoor Panorama in Realtime, as Rouli from Games Alfresco points out,  indicate that there is a lot in store for us at ISMAR09.

We may not be so impressed by directory style/”post it” AR anymore, as these applications have become common place so quickly!  But while these early mobile AR apps may be disappointing in relation to some futurist visions of AR – merely AR/ubicomp appetizers,  there are still good implementations of this model coming out (see new comers to the app store Bionic Eye and RobotVision). And Layar, always on the ball, has upped the ante for the new cohort of AR Browsers with Layar 3D.

But as Bruce Sterling notes here:

*In AR, everybody wants to be the platform and the browser, and nobody wants to be the boring old geolocative database. Look how Tim [creator of RobotVision] here, who is like one guy working on his weekends, can boldly fold-in the multi-billion dollar, multi-million user empires of Apple iPhone, Microsoft Bing, Flickr, and Twitter, all under his right thumb

(watch video here)

But if  you looking for something more from AR, you probably won’t have to wait too long.  The two pioneering companies in AR, Total Immersion – founded in 1999, and Metaio – founded in 2003 are both coming out with “mobile augmented reality platforms” in a matter of weeks (see press releases here and here).  And both companies, it seems, will deploy much more sophisticated AR rendering and tracking than we have seen to date.

I approached Bruno Uzzan, founder and CEO of Total Immersion, for an interview as part of my look at the new industry of augmented reality through the eyes of the founding members of the AR Consortium. These consortium members are some of the first commercial augmented reality companies.

The interview below with Bruno began early this summer and then we both went on vacation and it picks up after the announcement of the partnership between Total Immersion and Int13.

The significance of this announcement is that Total Immersion is now positioned to take the augmented reality experiences they have developed for a number of top brands onto multiple mobile platforms with, “Int13′s very clever embedded solution that allows our [Total Immersion's] solutions to work across many [mobile] platforms,” while Int13 gets to extend their reach.

Total Immersion has a 50 person R&D team and their two main focuses have been, firstly getting:

“Augmented Reality to work with as many platforms as possible – PC, Mac, Mobile, Game Consoles, all those are the platforms that we are targeting. We are currently doing lot of work in the R & D team in cross platform compatibility….”

and, secondly:

“Our R&D guys are working on the real world interacting more with the virtual world.  And I have started seeing some results which are pretty much crazy and this will be ready for next year.”

Pandora’s Box – Shared Augmented Realities

Spes or “Hope”; engraving by Sebald Beham, German c1540 (see Wikipedia article on Pandora’s Box)

There are many weaknesses to the mobile smart phone AR experiences we have now, and the lack of near field object recognition (to date), and difficulties with accurate positioning aren’t the only ones.  Note re solving positioning problems in mobile AR, we are yet to see AR  leverage public libraries for analyzing scenes like Flickr’s geo tagged photos, see Aaron Straup Copes’s work on “The Shape of Alpha.” And for more on this my post here.

But, as Joe Lamantia points out:

“One of the weakest aspects of the existing interaction patterns for augmented reality is their reliance on single-person, socially disconnected user experiences.”

In my view, The Pandora’s Box of Augmented Realities is an open, distributed, multiuser augmented reality framework, fully integrated with the internet and world wide web.

As Yochai Benkler has pointed out many times, and argues again in, Capital, Power, and the Next Step in Decentralization, it is “open, collaborative, distributed practices that have been at the core of what made the Internet.”  We have to try to make sure that open, collaborative, distributed practices are at the core of mobile augmented reality.

Can Google Wave be the basis for an Open, Distributed, Multiuser Augmented Reality Framework?

I have been exploring the idea of using Google Wave protocol as the basis for a distributed, multiuser open augmented reality framework with a small group of AR enthusiasts and developers. And I am happy to say the proposal is beginning to get fleshed out a little.  New collaborators are welcome both for “gear heady” input and use case suggestions (but re the latter, you can’t just say everything you see in Denno Coil..!).

This effort started with Thomas Wrobel’s  proposal for an Open AR Framework prototyped on IRC – see here, and click to enlarge the image above of, “Sky Writer: Basic Concept for an Open Multi-source AR Framework.”

But recently we began looking at the Wave Federation Protocol.  And, if you check out this post, and this post, you may get a glimpse of why Google Wave protocol might be a good basis for an open, distributed, AR Framework.  You will notice, if you study what Google Wave has done with the XMPP protocol, that many of  the elements of networked urbanism that Adam Greenfield describes resonate strongly with what is being attempted in Wave.

But enough said for now!  Regardless of the details of implementation,  Google Wave or an AR protocol built from scratch (phew! the latter does seem like a lot of work) -  an open, distributed, multiuser AR framework integrated with the internet and web would explode the potential of AR, creating new possibilities for data flows, mashups ,and shared augmented realities.

And we are excited by Google Wave because, as Thomas puts it:

“The really great thing wave does ….(aside from being an open standard backed by a major player…hopefully leading to thousands of worldwide servers )….is that it allows anyone to create any number of waves, set precisely who can view or edit them, and for them to be able to be updated quickly and continuously (and even simultaneously!) Better yet, changes will (if necessary) propagate to all the other servers sharing that wave. It does all this right now. From my eyes this does a lot of the work of an AR infrastructure already.

I cant see any other protocol actually doing anything like this at the moment, although correct me if I’m wrong, as alternatives are always welcome :)”

Also, Thomas notes, “even the playback system (that is, the ability to playback the changes made to a wave since its creation) …this could give us automatically some of the ideas Jeremy Hight has mentioned in his visionary work here,  and here on “the geo spatial web, interlinked locations and data, immersive augmentation and open source geo augmentation.”

One of the many reasons why an Open, distributed AR Framework would be so cool is it would open up all kinds of possibilities for GeoAR by providing the over-arching standard protocol for communication of updates necessary for the substandards that will facilitate GeoAR.

Also important to note is the  Wave Federation Protocol allows anyone:

“to run wave servers and become wave providers, for themselves, or as services for their users, and to “federate” waves, that is, to share waves with each other and with Google Wave. – “the federation gateway and a federation proxy and is based on open extension to XMPP core [RFC3920] protocol to allow near real-time communication between two wave servers.” See Reuven Cohen’s blog for more here and here, “HTTP is Dead, Long Live the Real Time Cloud.”

Still some people have expressed concern that an AR Framework using Google Wave protocol would give Google disproportionate influence.   Will Google-specific functionality be an issue?  How much stuff is Google specific just because no one else is using it (yet)? And how much is Google specific because it holds no value to anyone else but Google? These are some of the questions that have come up.

You are going to see a variety of suggestions for standards and specs for open AR coming out out in the next few months which as, Robert Rice of the AR Consortium points out is: “a good thing, we need that competition early on to settle down on best case.” Recently,  Mobilizy have offered up an ARML (“an augmented reality mark-up language specification based on the OpenGIS® KML Encoding Standard (OGC KML) with extensions”) for consideration see here.

So it is, perhaps, also important to note, that an Open AR Framework should be neutral/transparent to techniques of  “reality recognition,”  and methodologies of registration/tracking, allowing various ones to work on the system as new techniques evolve, and to support as many evolving standards as possible.

Augmented Reality developers, like Total Immersion and others with powerful rendering/tracking AR software, should be able use an Open AR Framework to exchange the data which their tracking will use. And the tracking/rendering problems they and other researchers have solved are much harder than figuring out data exchange on on a standard infrastructure or protocol!

So I pricked up my ears when I heard Bruno Uzzan, CEO of Total Immersion -  the first and currently the largest augmented reality company, with a 50 person R&D team in France and offices in LA, where Bruno himself is now based, say: “Total Immersion is  only months away from launching shared mobile augmented reality experiences using near field object recognition/tracking across multiple platforms” (for more details read my conversation with Bruno Uzzan below).

I was happy when I asked Bruno about the possibilities for developing an open, distributed, multiuser augmented reality framework fully integrated with the internet and world wide web (possibly using Google Wave protocols), and he replied:

“I think this is feasible. I think that’s doable, that’s just  in my opinion. I mean some people might have another kind of opinion but I think that that’s definitely doable.”

Total Immersion – working with the “symbiosis between augmented reality and brands”

Total Immersion has created many of the best known and most ambitious augmented reality experiences for major brands to date, including Mattel’s new AR toys to be released in conjunction with the James Cameron film Avatar, and AR baseball cards for Topps, video here (or click screenshot above), and the UK’s first augmented reality books.

Bruno founded Total Immersion 10 years ago when he was just 27. And the kind of conviction it took to survive as an augmented reality business in the decade before augmented reality captured the world’s attention is remarkable.

AR’s first steps out into the world after 17 years as predominantly a lab science maybe “wobbly” (what new technology isn’t), and sometimes gloriously kitsch – check out this riotus video of the 3D Interactive Live Show Total Immersion produced in Korea (also see the Total Immersion Augmented Reality Blog for more on the TI’s turn key  Interactive 3D Live Show Solution).

As Lamantia points out here, ” projecting mixed realities into public, common, or social spaces makes them social by default.”

However, the potential for shared location based augmented reality experiences is as yet untapped.  So I see the entry of the most experienced commercial augmented reality company into mobile as pretty interesting.   While  smart phone AR still has significant limitations, and it certainly does differ from some of the futurist dreams of AR (see Mok Oh’s post here on his disappointment in this regard), it is significant that Total Immersion is committing to becoming a leader in mobile AR.

Our smart phones, the powerful networked sensor devices that so many people carry in their pockets, have proved themselves a “good enough for now”  mediating device for early manifestations of the ubiquitous computing and augmented reality base pair.  And now AR and ubicomp is mixed in the  rich, messy soup of everyday life, commerce, business, marketing, art, entertainment, and government, we should get ready to see these technologies grow up fast, and unfold in some surprising ways that lab science didn’t necessarily predict.

And, perhaps, the new dialogue between scientists and entrepreneurs may spur both communities to outdo themselves.

Particularly, as Joe Ludwig notes: “It seems to me that the biggest disconnect between the academics and the entrepreneurs is that they disagree on how far we are from the finish line.”

See the comments’s on Ori Inbar’s post, Augmented Reality Entrepreneurship: Natural Evolution or Intelligent Design?, for  a courteous but spirited discussion on the potential benefits and frictions of the newly expanded AR community of  researchers and  entrepreneurs.

As Blair MacIntyre (see my long conversation with Blair here) notes:

“not all academics and researchers are only interested in the traditional models of impact. Case in point: I wouldn’t be building unpublishable games, nor investing so much time talking to the press, entrepreneurs and VCs if I did not believe strongly in the value of the impact I am having by doing that — and I know others with the same attitude.”

In this vein, check out the Marble Game (video here) developed by Steve Feiner and his team at Columbia U. It’s enabled by Goblin XNA, an open source AR framework built on top of Microsoft’s XNA, which powers XBox live games, Zune games, and some Windows games. For more about Goblin XNA and AR from Columbia U see here.  (Hat tip to Brian Jepson for this link)

While we are still waiting for the kind of sexy AR specs – nothing totally game changing in Gigantico’s AR eyewear rounup (maybe note this Apple patent), that might get wide adoption. But at least researchers are not afraid to explore the possibilities of AR Goggles.

But how far are we now, with or without sexy goggles,  from a fuller expression of the base pair DNA of ubiquitous computing and augmented reality?

We may have a LAN of things before we have an Internet of Things

The picture above is a workshop I attended at Conflux last weekend – Fish ‘n microChips, with Natalie Jeremijenko. We are at the site of the Amphibious Architecture project (a commissioned work for Toward the Sentient City) and “a collaborative project with xClinic, The Living and other intelligent creatures.”

We are probably as far off some grand futurist visions of ubiquitious computing as we are some of the futurist visions of augmented reality. But as it turns out that may not be a bad thing! Recently, @mikekuniavsky noted in a tweet:

“Another argument for the LAN of Things before the Internet of Things: http://tinyurl.com/lgp9uq”

Bert Moore, in the article Mike linked to points out, the grand vision of an “internet of things” with everything connected to everything  can “distract people from thinking about the benefits of RFID in smaller, more easily implemented and cost-justified applications.”  The same argument I think applies to sensor networks and augmented reality.

In New York City, a series of commissioned works for the Architectural League of New York’s exhibit, “Toward the Sentient City” are giving us the opportunity to dip our toes into the ocean of a “networked urbanism.”   For only a small budget, two of the five commissioned works, Amphibeous Architecture and Natural Fuse demonstrate how sensor networks can allow us to explore new kinds of communities – connecting people to environments in interesting ways to create new forms of social agency.

“Amphibeous Architecture” -  from The Living Architecture Lab at Columbia University Graduate School of Architecture, Planning and Preservation (Directors David Benjamin and Soo-in Yang) and Natalie Jeremijenko, Environmental Health Clinic at New York University, uses a skillfully built (electronics and water are notoriously hard to mix) array of partially submerged sensors to pierce the blinding, reflective surfaces of the rivers  surrounding Manhattan and to create a new two way relationship with the ecosystem below – the water, our neighbors the fish and even a beaver that lives in the water surrounding Manhattan.

Image from Toward the Sentient City

In a similar spirit, “Natural Fuse” – Usman Haque, creative director, Nitipak ‘Dot’ Samsen, designer, Ai Hasegawa, designer, Cesar Harada, designer, Barbara Jasinowicz, producer, creates a network of people and electronically assisted plants to explore what it takes to work together on energy consumption and to experience the consequences of “selfish” and “unselfish” behavior interactively before it is too late to modify our actions.

The “Greedy Switch“ from Natural Fuse on the left. On the right “The System” – click to enlarge.

Much more to come in another post on these works, and “Toward the Sentient City.”  Also an update on how Pachube – an important part of both these projects and a very important contribution to ubiquitous computing because it creates the opportunity to connect environments and create mashups from diverse sensor data feeds – has matured since my interview with Pachube founder, Usman Haque, “Pachube, Patching the Planet,” in January this year.

In the picture above Natalie Jeremijenko, and Jonathan Laventhol give the Amphibious Architecture sensor array a last look over, as it will soon be lowered into the East River. Jonathan is on a busman’s holiday to help out at the pre launch of Amphibious Architecture, nr Manhattan Bridge, NYC.

I was very happy to get  a chance to talk to Jonathan Laventhol - more on our conversation in another post. Jonathan Laventhol is CTO of Imagination – one of the world’s leading design, events, and branding agencies. We talked about the importance of  Pachube, which Jonathan called the “The Facebook of Data,”  and  how the symbiosis between brands and augmented reality, and healthcare applications, would  be key to augmented reality emerging into the mainstream.

Natalie Jeremijenko’s workshop at Conflux on the social negotiation of technology and how “everyware” can give us the chance to experience new forms of agency and connection was a totally inspiring.  And I will cover this too in another post.  I have so much awesome stuff  to write about at the moment!

None of the projects in, “Toward the Sentient City,” included a mobile augmented reality, or “magic lens” component, but they all pointed to why “enchanted windows into our newly inside-out reality” are going to be so important. And why the DNA base pair of ubicomp and augmented reality can really do stuff that matters.

Shangri- La – “Transfigured City”

Screenshot from Shangri-La episode Transfigured City

In my AR Consortium founder member interview series, I have found that, understandably, the visionary founders of these first augmented reality companies are a little reticent about sharing their full vision.  They are basically on stealth mode in this regard.  So as you will not, from my interview with Total Immersion founder and CEO, Bruno Uzzan, get a fully drawn scenario of his vision for a next generation of shared augmented reality experiences, here’s a really interesting anime episode from the anime Shangri La called, Transfigured City, to mull over instead.

As you can tell from this rather long and circuitous intro to my my conversation with Bruno Uzzan, I  have been investigating shared augmented realities pretty intensively recently. And Mike Kuniavsky pointed me to Shangri-La, and Transfigured City, in a conversation with Mark Shepard, after Mark’s presentation at Conflux, Sentient City Survival Kit.

Mike Kuniavsky with Tod E. Kurt is founder of ThingM, a ubiquitous computing device studio. Also Mike Kuniavsky researches, designs and writes about people’s experiences at the intersection of technology and everyday life – see Mikes blog Orange Cone.  And I interviewed Mike at Etech- see here.

In Transfigured City, the “Metal Age” group has to figure out how to share and communicate in a city transfigured by augmented realities/virtualities, where no-one sees the same place in the same way.  Only one character can figure out from her previous experience of the city the relationship between the transfigured city and how it used to be.

The conversation I had with Mike Kuniavsky on The Transfigured City continued at a picnic in Washington Square Park the next day with Elizabeth Goodman, who I met at Etech when she gave a brilliant presentation, Designing for Urban Green Space.  We covered so many areas at the picnic related to ubiquitous computing and augmented realities that this conversation probably deserves a post of its own (my writing to do list is growing longer!).

The Plot Synopsis for Shangri La:

“In the mid-21st century, the international committee decided to forcefully reduce CO2 emission levels to mitigate the global warming crisis. As a result, the economic market was transferred mainly into the trade of carbon. A great earthquake destroys much of Japan, yet the carbon tax placed on the country is not lifted, so Tokyo is turned into the world’s largest “jungle-polis” that absorbs carbon dioxide. Project Atlas is commenced to plan the rebuilding of Tokyo and oversee the government organization, which the Metal Age group opposes due to its oppressive nature. However, Atlas is only built with enough room for 3,500,000 people and most people are not allowed to migrate into the city. The disparity between the elite within Atlas and the refugees living in the jungles outside of its walls set up the background of the story.”

Talking With Bruno Uzzan

Tish Shute: We won’t have fully opened the Pandora’s Box of Augmented Realities until we have ubiquitous, shared augmented realities, will we?

Bruno Uzzan: Yes. The most important for augmented reality is the experience we want to share. Now we are working on the cell phone, we can potentially do some marketing components that we already have developed now on cell phone. Done. It’s working.

But the most interesting part of it is how these new components [cell phone AR] will be used for marketing campaigns by brands. And we are also pretty much well positioned to transform some of the AR that we currently have working on Mac and PC and to transform these to applications working on mobile devices.

Tish Shute: We haven’t really experienced yet what it means to actually share mobile AR experiences?

Bruno Uzzan: It’s hard — we did a Facebook app. It’s a first try, it has a way to go.  But to go more and more into social, is the way forward for us – to share and expand AR experiences. But yes, I mean what you’re seeing is how two people on two different applications can share that same expanse.  For sure we are going in that direction. We are currently working on those kind of solutions. How people can share and experience together at the same time. That’s how we start creating excitement in augmented reality, and it’s coming up.

It’s a new market and there’s so much more in store for augmented reality. You know, some people are telling me, don’t you believe that augmented reality is a gimmick? It will be a trend for a few weeks or a few months and then gone? I say, you’re kidding me. This is only the beginning. I mean I can assure you that the applications that are on the market today are one percent of what we will have five years from now.

Tish Shute: I agree.

Bruno Uzzan: And I’m sure that augmented reality will be a part of a lot of components that we are currently using today – GPS, web browser, glasses, I mean there are so many applications that will come up shortly. This is only the beginning. I’m completely convinced that augmented reality will be in three years from now what virtual reality is today, which is a billion dollar market.  I know that it’s not just a gimmick of a few weeks or a few months, because so many brands are jumping into it, spending money, exploring solutions.  I know that it’s not just short term -what they are willing to do and we are willing to do, but also middle and long term. And that’s what makes this adventure pretty much unique and what makes creating a cutting edge technology, very, very much exciting for us.

Tish Shute: First could you explain more to me about your partnership with Int13. I am not sure I understand what is in the arrangement from Total Immersion’s POV. I mean what happens re your own mobile software development? Haven’t you only been licensed the Int13 SDK for a limited period of time and have limited access to all it’s power? Stephane from Int13 said to Ori on Games Alfresco, here, “we have licensed the SDK4 for two years,” and then Ori asks, “but you have basically kept the power to yourselves, right?” So if they are the only ones that can enhance it and develop the software, where will  TI be in two years in mobile if you haven’t really had the chance to develop your own software .

Bruno Uzzan: Actually it’s a real win-win situation. Int13 is a very small company and they have so many requests they can’t possibly fulfill them all. So  this is a way for both of us to be, as quickly as possible, the first mobile provider for all the requests we have. Also they give us exclusivity so nobody else can use INT13 SDK for such applications.  I think that it is a good partnership,

And concerning our own mobile application… First of all we have currently some mobile applications working. But with Int13 we have a mobile solution that can work on many different devices. That’s a fact and that’s working. And, believe me you will hear from us a lot more about this soon. We are fully independent on our mobile development. The reason we closed the partnership with Int 13 is  to be able to deploy mobile in a broad way.

I mean you know that the difficulty with AR mobile is that each separate device needs some customization. Working on the iPhone is different from working on the Nokia, different from working on the Palm; it’s different from working on the Samsung. Each of them have their own operating system inside and so we were interested in Int13′s very clever embedded solution that allows our solutions to work across many platforms.

The reason we are working with Int13 is that we are able to work on so many mobile devices, thanks to Int13. And in the mobile AR race that we are currently in, the next two years will be extremely important to us…

Tish Shute: OK, that definitely clarifies it a lot. So Int13 has done an embedded solution to allow TI developed AR solutions to work easily across many devices?

Bruno Uzzan: Yes  they have kind of an embedded solution, a way to address extremely quickly new cell phone… But, currently on our side, we are in discussions with a mobile company… and that only refers to some very specific mobile devices.  And what they have is also a way to embed deeper our technology into mobile, so that we can have quicker… applications that work on a large number of cell phones.

Tish Shute: So, basically it means you don’t have to go through some complicated negotiations with each of the cell phone companies, is what you are saying?

Bruno Uzzan: Not only negotiations, but also hard development. You know? Working on the Windows mobile is completely different from working on the Palm OS. You know, that’s different! Its a big work, to have a mobile application working on many other devices. So, INt13,  provides us a way for us to save some time and some development cost too.

Tish Shute: And Int13 doesn’t have powerful AR development tools like D’fusion right?

Bruno Uzzan: Right! That’s right. That’s why we say it’s a true win-win solution. They can benefit from our work too. And we can benefit from their work, in order to deploy quicker and faster mobile solutions.

Tish Shute: Now, the second thing is… there is a lot of debate and disagreement about how far mobile augmented reality is from delivering something more that the “post it” approach that has been much publicized in recent months, via all the AR browser apps.

But from my understanding from the conversation we had earlier this summer (see below), Total Immersion is targeting a much higher level of mobile augmented reality than we’ve seen to date?

Bruno: Yes the browser apps we have seen are a kind of augmented reality, but not exactly the way we see it. Let me explain you why. With this kind of application it’s true that you can overlay 3D-information and video. That’s a fact. So, in a sense, that’s augmented reality. But the way that they are working on the position of the 3D on that video is that they are using compass and GPS-information.. so it means that this AR solution will work only on some building and some physical objects that are FIXED. In a fixed and known position.

So you want to go to a theater?

The theater is here, for sure it will not move, so you know the position of the theater, and that’s a fact that you can superimpose an object on the theater. That’s what can be done currently. What we are achieving and what we are doing on mobile is more than that. We want to be able to port our solution with trading cards, with brands, into a smart phone.

I’m assuming that you want a can, a drink can, to be able to trigger an experience. The only way you can do it is to be able to understand what the can, it is. And the current solutions that are out there can’t do that, it’s impossible.

Tish Shute: Right, yes. There’s no near-field object at all in these early browser apps.

Bruno Uzzan: And the solution we have is that we can recognize a can and then — in a very, very precise way and that activates geo-location, so we can superimpose 3D. I mean in that case, it opens up all the applications that we currently have, so they could work on mobile.

Tish Shute: So for example, if you’re working with a soft drink company, people can trigger that experience wherever they see that can?

Bruno Uzzan: Correct.

Tish Shute: Yes. Yes, I assumed that was what you’re doing

Bruno Uzzan: We believe — and maybe that’s not the case, but we believe that our marker-less tracking technology is pretty much unique on the mobile devices.

I haven’t seen yet, from anyone, a full augmented reality mobile solution working.

I really see AR being part of the Web 3.0 next generation. I mean the vision I have is that, you know — today, when you want to have information, you go on a website and then you find your information. AR — and the future is that I think it will be part of the opposite. You want to have information about a product, you just show it to your computer and the information will automatically pop up. I see here a new way to market some key messages, a new way to get information is that some physical product by themselves could be a way to get information, and you don’t have to search anymore for them, it’s coming out to you.

AR is definitely for me, one of these components. Another thing that AR is a solution, another thing that AR itself will create these kind of results in how information is being displayed. But I’m seeing  here a way that could be part of a new way to have access to information. And that’s part of the vision I have. Whatever, if it is through mobile phone or web or PC, Mac, whatever, I really believe that now this kind of new generation of receiving information will come shortly and could be a kind of a new — could be part of the new 3.0 generation of the web.

Tish Shute: My friend Gene Becker did an interesting post recently on some of the current limitations of mobile AR where he pointed out the problem of:

“Simplistic, non-standard data formats – POIs, the geo-annotated data that many of these apps display, are mostly very simple one-dimensional points of lat/long coordinates, plus a few bytes of metadata. Despite their simplicity there has been no real standardization of POI formats; so far, data providers and AR app developers are only giving lip service to open interoperability. Furthermore, they are not looking ahead to future capabilities that will require more sophisticated data representations. At the same time, there is a large community of GIS, mapping and Geoweb experts who have defined open formats such as GeoRSS, GeoJSON and KML that may be suitable for mobile AR use and standardization.”

Bruno Uzzan: That’s interesting. I mean — I know exactly what his is referring to. He is mainly referring to a localization and how you can have a quick, accurate localization.  If you look at current solutions, and you look at this 3-D superimposing on the video, the 3-D is shaking a lot. I don’t know if you see that in some of these early efforts.

It’s hard to use because the 3-D, you know, is  part of the magic of augmented reality, that is when the 3-D is being inserted in a very easy way and smooth way in your solution. Here, when you see this overlay, 2-D or 3-D overlaid on the video, it’s shaking a lot. One reason for this is that the GPS compass is not accurate enough to coordinate the perfect location of the user. And here, what Gene says is interesting. I think we are addressing this localization issue in a pretty smart way.

But to be frank with you, I don’t believe mobile augmented reality in the extremely short term — I’m talking about three weeks, one, two months is mature enough for good AR applications.  It will be shortly.  But for now it is more proof of concept than a true and easy application to use.

But we are starting to see a lot of new application coming out, but I really believe that marketing and entertainment are the two key markets for AR right now.

I’ve been working ten years in augmented reality. And, eight years ago, when I was talking about augmented reality, I was E.T., you know? Nobody understood what I said, and I thought it was crazy. And now, today, yes it’s completely different.

Tish Shute: The Pandora’s Box of Augmented Realities, in my view, is an open, universal and standard, distributed, multiuser, augmented reality framework fully integrated with the internet and world wide web. I have been looking into Google Wave protocols as a basis for this would you be interested in this? Do you think it is feasable?

Bruno Uzzan: I think this is feasible. I think that’s doable, that’s just  in my opinion. I mean some people might have another kind of opinion but I think that that’s definitely doable.

Tish Shute: Yes I suppose an open AR Framework involves cooperation and collaboration, it is more about business and politics than technological problems.

Bruno Uzzan: Yes!  Actually the Web is politics. Business is politics.

Tish Shute: I would be interested if anyone in your R&D team would be interested in looking at some of the ideas that are emerging in our little discussion of Google Wave and an Open AR Framework  to offer feedback. it is an interesting time now to input on the Wave Federation Protocol docs because nothing is set it stone right now.

Bruno Uzzan: Just shoot me an email, I’ll try to put you in touch with the right person and, and a team member that can input on this.

Tish Shute: For mobile augmented reality the best thing we’ve got now is the phone, right?

Bruno Uzzan: Right.

Tish Shute: And the only way we can use the phone is by holding it up, right?  Isn’t this a bit of an an obstacle as you introduce better object recognition and tracking?  People are going to have to stop moving to use their phone. What do you feel about that experience? Isn’t AR eyewear and essential part of a tightly registered AR experience?

Bruno Uzzan: We don’t do hardware and we don’t have the current solution for eyewear that would do all we need for a good mobile AR experience, so I guess we don’t have the current answer for that.  But we are beginning to see the next generation of this — of these glasses.

Tish Shute: But you’re happy enough with the mobile experience of augmented reality on smart phones that you’re investing in this next generation of software for this.

Bruno Uzzan: Yes, I know. We know that some application will not work on the iPhone. And yes, whatever you do, you still need to hold the iPhone, so it means that you can’t play with your hands anymore. So we know that partially, some AR solutions  we have on other platforms will lose the magical effectivities on just the iPhone.

But I’m starting to see on the market some glasses that could perhaps be not too expensive — that’s a challenge!  And easy to use — that’s another big challenge. And, that could fit on anybody’s faces and head — there’s another big challenge. So yes, I’m starting to see that, but so far AR glasses are only applicable for some very, very specific application, like design or theme park or, you know, some specific location where it makes sense to move forward with glasses.

I don’t believe that kids will use glasses for — in our toys and for games in the next months or maybe othe next one or two years. But maybe something will come out shortly and that could be a big breakthrough, and enable us to think another way. But  from what we have seen so far and from what we know in this hardware market, I don’t believe that currently there is a workable solution.

Note: The following section of the interview took place earlier in the Summer.

Tish Shute: You are the first commercial AR company  – you started in 1999 right?

Bruno Uzzan: Yes you are right. We started the extremely early in this augmented reality market. We were the first company worldwide to start doing augmented reality and to start promoting augmented reality. So it’s true, we are pretty old players although the market has been getting bigger and bigger for the last year and a half. So for a long time we were only in the market, and the market was not really there.

But for the past 8 months, the company has been growing really fast.

Tish Shute: Yes I’m sure. Congratulations for hanging in there long enough to get the pay off!

Bruno Uzzan: You know, my background is Financial. So I have been driving the company for many years in a very cash efficient way. So we have been waiting for the markets to reach maturity before starting make some investments. So that’s the reason we are still here, and that’s the reason I think we managed pretty smartly the cash that we raised for the company.

Tish Shute: Yes there is a saying that when a market takes off you can tell a pioneers because they are the ones with the arrows in their backs. But I am glad you are dodging the arrows!

Bruno Uzzan: You know, I’ve always driven the company with revenue. And because revenue was not there at the beginning I was extremely cautious about the cash. So now that the company is getting some revenue, for sure we are making more and more investments, and taking advantage of our situation as a worldwide leader of augmented reality.

This situation is not easy as it appears today but it’s now getting better, as you can see, AR, Augmented Reality, has very good momentum and we are benefiting a lot from all this momentum for augmented reality right now.

Tish Shute: You’ve been very involved in researching developing augmented reality tools. Are you still as active in the research area, or are you too busy keeping up with work for hire now, to be working on research and building new technology for Augmented Reality?

Bruno Uzzan: Both. First of all, we are part of lot of projects either directly with clients like Mattel or with some partners that are using our technology to promote and develop other AR projects. From what we he have seen, many, many, many, projects augmented projects have been done currently with our solutions.

To continue with your previous question. So we are being perceived as this leader in that space, and we  have some pretty heavy demand for our services. But we are coming up with new technology, of course, still connected to Augmented Reality.  But, our R & D is working in two different directions, which of course also bind together.

The first one is platform developments. So we want Augmented Reality to work with as many platforms as possible – PC, Mac, Mobile, Game Consoles, all those are the platforms that we are targeting. We are currently doing lot of work in the R & D team in cross platform compatibility.

Tish Shute: Robert Rice said recently, “markers and webcams equal Photoshop page curls…”

Bruno Uzzan: Yes. There are so many concerns with markers. The quality is extremely bad. As soon as you hide a part of the marker, a slight part of the marker, you’re dead. You can’t track any more of the object. So compared to our solution where I want to say play with cards or where you are going to play with a Mattel toy, even if you hide a part of the toy, it’s still working.

Tish Shute: But you haven’t offered the public an SDK to your engine right? Basically the way people get access to your tools is working in a partnership with Total Immersion right?

Bruno Uzzan: Correct.

Tish Shute: Do you think in the future you might open your SDK? Are you considering that?

Bruno Uzzan: Yes, it would be interesting.

Tish Shute: So that is something we can see coming soon?

Bruno Uzzan: Maybe, because it’s true that Total Immersion is starting to be mature enough for these kind of tools. The only thing is that we have to respect good timing for that.  It’s a big decision. You know what I mean?  It is a big, big decision. We would then compete with others using our technology.

Tish Shute: Oh I know, it is a big decision when you have so much skin in the game! But it would be nice to have your SDK being THE platform for AR, wouldn’t it?

Bruno Uzzan: It is a really big decision that we can’t just take like that, you know.  There are a lot of friends who told me you have to be extremely careful about timing. This timing is pretty much connected to the maturity of the market. For sure, we see the market being more and more mature. But, there are a lot of low hanging fruits we still want to address

To get the best value possible for all the publicity we have and all the clients we have now.

Tish Shute: Yes, I know. You’ve been in this game so long. Now, there is an interesting question here though about tools and platforms because you know, A.R., augmented reality has already expanded  beyond its kind of original purist definition. And when I talk to people  about augmented reality, there are actually lot of different ideas and priorities of where the tools should go right now. You know, obviously we have these kind of browser-like applications, but these browser like applications are not dealing with recognizing near field objects yet.  What are your priorities for tool development and what are your priorities for AR development in the future? What areas are you going to focus on? Oh dear that is a rambling question!

Bruno Uzzan: [laughter]  So, one of our first priorities is we need to create our software with one development, one installer, one software that can be spread on different platforms. The same application, the same software can be used either on a PC, Mac, phone or console. So that’s a lot of work, because that means that our platform has to address many many different devices and that’s a big priority for us because we received this request from our clients. We want to be able to use one application on many different platforms and devices. So, that’s the first one.

And the second one is to add more and more interactivity between the real and the virtual world. So, we are working on some improvements to add some real components that will interact with virtual, and that also part of our big strategy and direction and these two worlds can more and more be bridged together, linked together so they can interact  one with the other.

Our R&D guys are working on the real world interacting more with the virtual world.  And I have started seeing some results which are pretty much crazy and this will be ready for next year.

There are so many different directions for interaction between the real world and virtual world to develop.  I’m sure ten years from now you’re going to have AR applications everywhere.  Its not just temporary fashion stuff or a gimmick for few months. I mean we are getting there, its getting stronger and stronger and we are getting a good adoption rate from our consumers. They like it, they test it, they play with it and brands wants more, people want more and its getting bigger and bigger.

Tish Shute: Yea and I totally agree, its not a gimmick because the interaction between “virtual” and “real” enhances the magic of both. Another question about you RandD operation. Is your R&D still in France or have you moved totally out to LA.

Bruno Uzzan: We are 50 people in France and I started this LA office two years ago and I moved permanently two years to LA. So I’m now permanently located in the US to take care of the US office, knowing that revenues are really getting bigger and bigger in the US. So it means that we are getting a lot of traction, working with large company and now I’m currently located in the US.

Tish Shute: My sister lives in Paris. Could I visit your R&D lab at some point? I’d love to visit!

Bruno Uzzan: Yeah sure sure sure. I mean if you want to go. You won’t have access to all the research. But if you want to go out and meet all the team please do.

Tish Shute: I’d love to.

Bruno Uzzan: No problem. Shoot me an Email you and I will introduce you to Eric Gehl, COO, he is the COO of the French team. And he can definitely take care of that.

Tish Shute: That would be fun. Thank you!

Recently, AR browser applications have really caught the imagination of the web community, eg., Layar and Wikitude?  Where do you think the most important market for AR is at the moment, entertainment,  green tech, business, education?

Bruno Uzzan: I think that all that you mention will be important. The first one that did grab my attention is entertainment particularly dual marketing, because they always searching for new ways to interact with players or the consumers.  But it’s just the tip of the iceberg, you know, I mean medical applications could be huge using augmented reality. Education, and edutainment is definitely using more and more augmented reality components.  And I will just be submitting with big companies – that are considering using augmentation for education. Museums are very important too. Also augmentation as a kind of free sales tool, you know there are so many applications, design, architecture – so many directions that it’s hard to say today which one will take the lead.

But I do believe that on the short term the ones that are really really moving fast are the entertainment business and the digital marketing business.

Tish Shute: What do you think are the biggest shortcomings with current augmented reality and what are the obstacles that no one has solved yet?

Bruno Uzzan: I think the cell phone is not fully ready for augmented reality – a lot of people are working on that but there are still a lot of constraints to get the augmented reality working on a cell phone and I think that from what I heard a lot of manufacturers and a lot of companies are working from direction that are going to help us a lot to develop some great cell phone applications.

And I think that’s one of the biggest part of the game. All the applications that you see on cell phones so far are just gimmicks – the next big key is how to transform some gimmick cell phone application to a real, industrial, robust application that’s going to work on a cell phone. So I think that’s a big challenge for this year.

Most of what we see now is just matching and overlaying some 2d components in a video. This is not what I call AR.  You’re far away – with this kind of application, you are far away from doing the registration that we need to do – you can’t do it. So here’s the challenge: “how can you get a Topps is an application working on cell phone. That’s the big challenge  How we can make that work!” You can’t today get a real AR Topps application working on cell phone because there’s no cell phone  that’s actually ready. But we are working on it and the first one that can make that work, it’s going to be huge.

When you are working with good AR components you need a lot of CPU and GPU programs. So today new cell phone have started to be more and more ready for augmented reality but you need a really good cell phone to make it work. You can’t choose an old cell phone to make it work because you have some recognition, you have some tracking, you have some rendering, so you can’t choose a Nokia cell phone two years old to make that work. For sure the newest iPhone is the one that can make it work, but that’s it for now. There is a lot of research – from large cell phone companies – to get more CPU and GPU into their cell phone.  But so far we are also waiting for these devices to be released to consumers.

Tish Shute: And the current economic climate has put a damper on MIDs hasn’t it. But who can tell? It depends what price points some new MID came out at right?

Bruno Uzzan: Correct.

Tish Shute: Yes,I agree. But basically what’s interesting, the interesting thing is, the iPhone can deliver so much of what is necessary and even if Apple hasn’t given access to the full power of the iphone to AR developers yet, there is really no going back now – the mobile augmented reality cat is out of the bag!

Bruno Uzzan: You’re right, you’re fully right.

Picture on the left Mirrorshades, picture on the right a Metroid Hud.

“Augmented Reality is like a Philip K Dick novel torn off its paperback rack and blasted out of iPhones,” Bruce Sterling in Beyond the Beyond

“a techno visionary dream come true – those are rare, really rare, you have to be patient,  it’s super cyberpunk”… Bruce Sterling, “At the Dawn of the Augmented Reality Industry.”

The Dawn of the Augmented Reality Industry continues to brighten, and now we have two augmented reality companies, Total Immersion and Ogmento, firmly established in Hollywood – the dream mother of so many of our augmented realities.

Ogmento is the most recent of these two pioneering augmented reality companies to set up shop in LA.  Total Immersion’s CEO Bruno Uzzan moved to LA from France two years ago, although he still has a fifty person RandD team in France.   Total Immersion began 10 years ago in the quiet, lonely, hours before the dawn of an AR industry.  But Total Immersion’s AR toys for Mattel, and augmented reality for Topps baseball cards, fired CNet writer Daniel Terdiman up enough to say, “I have seen the future of toys, and it is augmented reality” (see full post here on CNet).

Recently, I talked with  Ori Inbar, one of the founders of Ogmento and  the premier augmented reality blog Games Alfresco about his new venture in Hollywood. Bruce Sterling, @bruces, had some fun with my invention of brand new augmented reality trade jargon here!  Ori pointed out Ogmento brings two important new facets to the rapidly growing augmented reality field: firstly they are bringing leadership from veterans of the entertainment industry into augmented reality development. Brad Foxhoven and Brian Selzer from Blockade have partnered with Ori on Ogmento.  And, in an another important step forward for a young industry, Ogmento announced they will be acting as publishers for a fast growing cohort of augmented reality application developers and helping AR development teams out there bring their concepts to the market.

So I was very happy also to have the opportunity to talk with Brian Selzer.  Bruce Sterling pointed out in his seminal sermon from the flatlands at the Layar Developer Conference, AR is kind of a “Hollywood scene.” We have seen the web early adopter/developer/blogger community  embrace augmented reality browser experiences in recent weeks in an awesome wave of enthusiasm. Are Hollywood creatives equally smitten? For the answers see the full interview with Brian Selzer below.

Brian Selzer (www.brianselzer.com and twitter – brianse7en ) has an extensive involvement with emerging platforms:

“from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC [user generated content] projects (mashade.com, instafilms.com).”

I began my own career in Hollywood doing motion control photography and creating software that bridged the language of robotics and servo motors with the visions of  film directors. Eventually our little company, NPlus1, moved on to 3D vision systems and image recognition stuff.  So yes, I have been really, really patient waiting for this particular techno visionary dream.  And, while I have been waiting for augmented reality to manifest, I have grown to love the internet.  But now, how awesome, It is OMG finally for mobile AR!

Augmented reality is busting out all over – through our laptops, our phones, on the streets, toys, baseball cards, art installations, sticky light calligraphy and more.

Many of my questions to Brian were directed at at how and when we will see augmented realities with near field object recognition, image recognition and tracking and, of course, the illusive eyewear.  As Bruce Sterling points out we are just at the very, very beginning – the dawn of an industry.  I created the photomontage below on the right to compliment Tonchidot’s illustration suggesting the evolutionary inevitability of holding our phones up (below on the left).  The Evolutionary Reality of AR will not end there.  It is just a step into eyewear, hummingbirds or Nano Air Vehicles, and more…….

The Evolutionary Reality of AR

Cartoon on the left by Tonchidot on the right a collage of a stock photo and the Pentagon’s Robot Humming Birds – “Nano Air Vehicles.”

While we finally we have, an affordable mediating device with the horse power, mindshare and business model to bring AR mainstream with the iphone.  The much anticipated Apple 3.1 Beta SDK to be released in September will not, I am sure, open up the Video API at the levels that augmented realities with near field object recognition and tracking require (I would love to be proved wrong though). But the magic wand to deliver even tightly registered AR graphics/media (that require a lot of CPU and GPU) to a wide audience is in our hands, so full access to may not be far off. And others, of course, can/will/might knock the iphone off its current pedestal.  AR made it’s mobile phone debut on the Android after all.

Like everyone else who loves AR, I wish that Apple would open up faster (and I wish Android would manifest on some rocking hardware). But we will see enough of the iphone Video API open for the next generation of mobile augmented reality games and applications to emerge in the coming months.

One of these will be Ogmento’s. Although Ogmento is in stealth mode, they have released a teaser for their first game, “Put A Spell,” developed by ARBalloon – screenshot below.  Ori did reveal to me in this interview that they are doing image recognition and using the Imagination AR engine.

As Brian notes, Hollywood has had the AR bug for a long time. AR has been everywhere in Science Fiction Movies and video games. Nintendo’s SPD3 head Kensuke Tanabe, “effectively the man in charge of overseeing all the Metroid franchise underneath original co-creator Yoshio Sakamoto,” explains the story of Metroid to Brandon Boyer of Offworld here (an image of a Metroid Hud on the right opening this post) :

“the idea of the different visors you use in the Prime games to interact with the world: the scan visor, for instance, set the game apart from other first person shooters in that the player was using it to proactively collect information from the world, rather than having the story come to them passively, in the form of cut-scenes or narration. “Prime could have adventure elements with the introduction of this visor,” says Tanabe, “That’s how we came up with the genre — first person adventure, instead of shooter.”

But as Brian points out:

“the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear).”

Talking with Brian Selzer

Tish Shute: Bruce Sterling’s sermon at the Layar Developer conference, “At the Dawn of the Augmented Reality Industry,” was absolutely awesome. He spread the future feast/orgy of augmented reality before us  – and described many of the dishes we will tasting both delectable and diabolical.  One of the many things he points out is, AR is kind of a “Hollywood scene.” And, as Ogmento is one of only two augmented reality companies in Hollywood, I am interested to hear how it looks from your neck of the woods. We have seen the web early adopter/developer/blogger community  embrace augmented reality browser in recent weeks in an awesome wave of enthusiam – are Hollywood creatives catching the buzz?

Brian Selzer:  It was a thrill to hear Bruce Sterling mention Ogmento. I devoured all of his Cyberpunk books back in the 80′s, along with writers like Gibson, Rucker, Shirley… To me, sci-fi writers are the visionaries who define and influence our technological paths into the future. They make science and tech sexy enough to want to manifest those experiences in the real world. Clearly Bruce sees the AR industry as being sexy. I love that he called it “a techno-visionary dream come true… and super-cyberpunk.”  And yes, kind of a Hollywood scene.

Hollywood creatives caught the AR bug before they knew what AR was.  Look at science fiction movies and video games to see AR everywhere. Terminator, The Matrix, Minority Report, Iron Man.. the list goes on.  Look at any video game with an integrated heads-up display.  It’s clear Hollywood loves AR.  It’s only been in the past few months though that the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear). So yes, the buzz is here and it’s strong.  With that, we all have to be prepared for the good, the bad and the ugly as AR goes mainstream.

It certainly goes to show how young this industry is when Ogmento and Total Immersion are currently the only AR companies based in Los Angeles. It’s very exciting to be the only company right now demonstrating a natural feature tracking (markerless) iPhone experience in Hollywood. We are in talks to bring some very big brand and properties to the mobile AR space. The goal is to deliver experiences that create added engagement and value to the consumer.

Tish Shute: Also in his landmark sermon Bruce Sterling noted that augmented reality has been around for 17 yrs and now at last we are seeing the dawning of  an augmented reality industry. What inspired you to take up the challenge of launching an augmented reality company in Hollywood?  Oh congrats that Bruce Sterling name checked Ogmento in his list of companies that prove that this really is the dawn of an industry!

Brian Selzer: I’ve always been involved in emerging platforms… from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC projects (mashade.com, instafilms.com). Working with all these great CG game assets, I continued to think about what’s next, and that’s when I started to follow AR very closely and started engaging with those who were pioneering in the space.

I remember swapping instant messages with Robert Rice (@robertrice) right after the 2008 Super Bowl.  We were not chatting about the football game, but rather about some of the commercials that aired during the event as a sign that AR was making its way into the mainstream.  A lot of people became aware of AR for the first time when the GE SmartGrid commercial aired.  There were all these YouTube videos popping up of people blowing on holographic wind turbines.

The commercial that really got me excited though was the Coke Avatar commercial.  In that commercial people in the city were sporadically being portrayed as their digital persona’s, avatars, gaming characters, etc..  For me that spot did a great job showing how many of us already have these ‘alter egos” that live in cyberspace, and how the line between these worlds can sometimes be blurred. I remember watching that commercial and thinking that is exactly the type of experience I’d like to create with mobile AR.  I want to overlap the virtual world into our every-day reality. Why cant I bring my World of Warcraft or Second Life persona with me into the real world?

I am big on the notion of “Games and Goals.” I believe that games have the power to motivate people in a very powerful way. By challenging ourselves while playing a game we can climb mountains.  Augmented Reality is the perfect platform to bring gaming into the real world.  By mixing the virtual world with the physical world, this added layer of perception provides a very powerful experience for something like a role-playing game.

One of my earlier social-gaming projects was a website called Superdudes.  This was a “Be Your Own Superhero” concept that celebrated and motivated kids to create superhero avatar/persona’s online, and we gave members all sorts of games, challenges, and rewards, some of which carried into the real world. The site recognized members for teamwork, creativity, volunteer work and things like that. So the Superdudes were often involved in charity events and benefits to help children.  Everybody called each other by their Superhero names, and the line between fantasy and reality were being blurred.  This project really got me thinking about what happens when you take positive role-playing like this and mix it into the real world.  I started to work on a plan for location-based activist missions for points and rewards, but never got to complete that. So I have some unfinished business here.

I think it would be fantastic to be able to show up to some type of fun event with friends, and everybody could see each others alter ego personas standing before them. When you can turn the world into a playground, and use the power of gaming to make a positive impact on the planet… well, I don’t think there is anything better than that.  These are the types of projects that drive me, and I think AR is the best platform to support these types of social gaming experiences.

Tish: Does Ogmento have any RPGs under development?  I noticed in the Google Wave on RPG someone has been working on doing something with the Dungeons&Dragons API.  I am interested in exploring the web of protocols underlying Wave as a transport mechanism for multi-person, mobile, AR experiences (not requiring downloads), on an open global outdoor AR network. If not Wave, what do you see as the potential infrastrucure and protocols we could harness for an open augmented reality network?

Brian:  Ogmento has a deep background in video games and we interact regularly with most of the major game publishers. As a company we are not so much developing our own RPGs right now, but rather exploring what mobile AR extensions make sense for existing brands.  There are many limitations to location-based gaming, but a global AR network is exactly along the lines we are thinking.  Lots of discussions are taking place on protocols, platforms, API’s, and there are numerous ways to approach this.  We need to be able to use what’s available now and continue to refine and customize for AR’s specific needs and issues as we progress.

In general though, Ogmento is focused on what types of experiences can be had today and over the next couple of years. I still think we are several years out from a truly open augmented reality network.  We are certainly looking at launching our own “Ogmented Network” which would support some fun treasure hunt type experiences, or add an entertainment layer on top of traditional outdoor marketing campaigns.

Tish: I don’t know whether you have read Thomas Wrobel’s ideas for an open augmented reality network that I just published here on Ugotrade.  The principals he talks about are very important for augmented reality to become a major part of our lives – .  Considering the difficulty open networks can pose for emerging business models how can we fund the development of an open framework for augmented reality?

“a future AR Network, I mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.
Where people can just point their phone, webcam, or pair of AR glasses anywhere were a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.”

Brian: I think funding for these types of projects will definitely come from Venture Capital groups in the near future.  It’s early in AR, but the VC’s are watching and deciding which horses to bet on.  Until that time, it’s about service work, and developing AR experiences for others with what is possible today. That work will help fund internal development of original AR products, and platform development.

Tish: How did you get started with Ogmento?

Brian: My first conversation with Ori was actually about my interest in Location Based RPG concepts.   We had a long conversation about the possibilities with AR, and it was clear that we shared similar interests, but were coming from different complimentary backgrounds. The idea of collaboration was exciting, so we just kept talking until the timing felt right. Now, with Ogmento we bring a unique blend of AR development experience with a deep backgrounds in AR technology, animation, video games, entertainment, social media, etc.   I think this is a powerful mix that will allow us to do some great things.

It’s still so early, and things are just getting started in AR. There are only so many webcam magic tricks you can enjoy before you are ready for something else.  The location-based apps have the most potential in my opinion, which is why we are really focused on mobile AR.   We have some board-game type projects, which do not instantly scream location-based gaming, but if you look at something like the ARhrrr board game, you can see how much more compelling it can be when the game invites the player to be actively moving around during the experience.

Tish: I am interested in your perspective on how we can create the kind AR experiences that really embody what has always been so exciting about AR – the tight alignment of graphics and media with real world objects and ultimately a rich immersive 3D experience, so I am going to hit you with a bunch of those, “Is this really eyewear or vaporware?” questions.  The real deal eyewear changes everything!

While eyeware is a big challenge technically and aesthetically,  I am pretty sure that there are several outfits out there that can pull off the optics and projection. 
Will the entertainment industry get excited enough to put a major push into delivering the eyewear in short order instead of the 5 to 10 year project that some people still think it is?    The business development challenge is bigger perhaps than the technical obstacles perhaps? What is your view on this?

And, perhaps, the eyewear is a clear example of a need for partnerships. For example, we have seen efforts from companies like Vuzix and Lumus, and recently a Japanese Company, Masunaga.

I have no reports from people who have tried the Maunaga eyewear yet.  But,  limited by small field of view, and tethered, currently eyewear offerings, available at a reasonable price point, are not workable solutions for augmented reality experiences. But the problems are not insurmountable. What will facilitate the real deal?  


It seems that it is critical to start creating hardware relationships now. The industry is costly and slow moving and as Robert Rice put it to me in a recent conversation, “once the software cat is out of the bag, its going to go wild and if the hardware isnt there, its going to stutter.”

As Ori notes some of the hardware companies like Intel and others don’t seem to be paying enough attention to AR.  Ori points out they don’t see the demand yet.  But in order to create an awesome AR experience and demand from a mass audience, don’t we need to work in conjunction with hardware designers?

Brian: It’s fun to think about who will eventually deliver a great hardware solution for AR glasses. It will happen. It would be cool to see somebody like an Oakley or Nike partnered up with a company like Vuzix to deliver something people actually might wear in public.  Perhaps a hardware manufacturer like Apple or Nokia will bring us something like the iSight or the NGaze down the line.  I’d love to see a set of glasses designed by Ideo.  Microsoft or Sony are already playing with technologies like Project Natale and the EyeToy, so I think its only a matter of time before they deliver an eyewear solution. I would even look to the toy companies to eventually make an investment here.

Gamers will be the early adopters, and in a few years we may start to see people running around in the park wearing glasses with headsets, but it will be acceptable because it’s clear they are using them for a game.  It’s going to take a very sexy and stylish piece of hardware for everyday people to be willing to wear AR glasses in public while going about their everyday business.   It’s like the recent cover of Wired magazine where Brad Pitt is wearing a mobile headset in his ear, and the editors point out that even he can’t pull that look off, so why do you think you can.  When AR glasses come in designer frames, and you can’t tell them from non-AR glasses, to me that’s when things get really interesting from a mass-adoption perspective.   Compare how many people were carrying around a mobile phone in the 80s to now.  I think it will be the same thing with glasses.

I was in an AR pitch meeting the other week at a very significant media company, and brought up the point that today’s handheld Smartphones will eventually evolve into tomorrows Smartglasses. My comment was quickly shrugged off as sort of a sci-fi notion that was irrelevant to the business at hand.  Probably true, but I think it is important to understand where digital media and entertainment is going, so you can adapt quickly, and evolve into those spaces more naturally.  The more we see people walking around with their Smartphones in front of their face (like a camera), the sooner it will be that we make the jump to eyeglasses as a key hardware device for AR experiences.

At Ogmento, we definitely are working on AR experiences with the hardware and software available today.  We will get some product out this year, and 2010 will be a banner year for markerless mobile AR in general.  I think the entire AR community is looking forward to bringing this technology to the mainstream in the form of games, marketing campaigns, virtual docent apps, and much more.  It might not be the full experience we are all dreaming about for some time, but we can see the path and the true potential, and it’s pretty spectacular.

You mention the tight alignment of graphics and media with real world objects.  That is really our focus. A lot of well-deserved attention is going to the browser overlay “post-it” approach right now, which uses compass and GPS.  We are focused on markerless natural feature tracking, so once you identify something that is AR enhanced in your environment, you can interact with that integrated experience.  On an iPhone that can be as simple as using your touch screen to interact.  When you are wearing glasses, it becomes more about visual tracking. There are lots of smart people thinking through these issues. Many of which you have interviewed. It is my hope that there are exciting collaborative efforts to be had in the coming months to get us all there together and faster.

Tish: Bruce touched on some of the hard problems that have to be solved for augmented reality – and he noted for instance security needs to be tackled in the early stages. Robert made a nice list, “privacy, media persistence, spam, creating UI conventions, security, tagging and annotation standards, contextual search, intelligent agents, seamless integration and access of external sensors or data sources, telecom fragmentation, privilege and trust systems, and a variety of others.” Will Ogmento be leading the way in solving some of these hard problems?

And, won’t trying to solve these hard problems for networked AR in walled garden scenarios one company at a time lead to a lot of reinventing the wheel wasted energy?

Brian: These are all important issues, and again there are a lot of smart people thinking about solutions to these problems on a daily basis.  Ogmento is interested in partnering with developers and supporting their efforts as a publisher of mobile AR experiences.  While we intend to roll up our sleeves in these areas, we are currently more focused on taking AR mainstream with the hardware and software available today.  As the industry evolves, so will Ogmento. As the opportunities evolve, our ability to make a greater impact tackling these issues will be realized.

Tish: Another area of development that could really kick AR into high gear might be creating augmented reality hotspots  where we use can deliver the kind of location accuracy/instrumentation necessary to create interesting AR experiences (partnership with Starbucks, perhaps ?!).  Augmented reality hots spots, could deliver the kind of high quality AR experience that isn’t possible ubiquitously at the moment, and may be a real way to get people really exploring the potential of AR now, rather than later?

Brian:  Agreed. I see a great opportunity here with this approach.

Tish: Although there are many obstacles to Green AR – the energy hogging servers at the backend for starters! Last week I had a conversation with Gavin Starks, AMEE, and Robert Rice and Jim Purbrick about how to work with AMEE and the technology available and encourage Green Tech AR development (see an early exploration of green tech AR from Pachube here).

We came up with the idea of holding a competition perhaps centered around a targeted instrumented space. But I would really love to hear your thoughts on the topic of Green Tech AR (the energy hogging servers at the back end being the first cloud on the horizon!.)  Cool GreenTech AR imaginings, social gaming ideas, RPGs, not even necessarily even tied to the immediately practical, would be like rain in a drought!

Brian:  I go back to “Games and Goals”… If you make environmental and other activist efforts fun and rewarding, more are likely to be motivated and participate.  Can you imagine having a personal “carbon footprint stat” floating over your self at all times? Or over your home or factory?  How would that change your behavior?  We all love stats. Look at how the Nike+ campaign has used technology and gaming to motivate people to run.  I think there is a lot that can be done to make being green fun. It starts with the individual, and spreads from there.  Keep me posted on that one!

Tish: I would also like to explore further the RRW suggestion that ambient intelligence is both the Holy Grail of AR and possibly snake oil:

“The holy grail of the mobile AR industry is to find a way to deliver the right information to a user before the user needs it, and without the user having to search for it. This holy grail is likely in a ditch somewhere beside a well-traveled road in the district of the semantic Web, ambient intelligence and the Internet of things. Be wary of any hyped-up invitation to invest in a company that claims to have gotten the opportunity right. What we’ve seen in the commercial industry to date is a rather complex version of a keyboard, mouse, and monitor.”

So Holy Grail, Snake Oil, or a ditch somewhere….?

Brian:  I instantly think of Minority Report, where Tom Cruise’s character is being bombarded with holographic ads personalized with his name and to his current situation.  In the future, Spam is a nightmare, especially when it knows who you are.  I think the key thing here is delivering “the right information”, and we still dont have that down. I do see a day where we can truly customize what comes to us, how we want it, when we want it.  My future vision of ambient intelligence is the ability to “turn everything off” if I want to… block out the stimuli and replace it with images of nature, or natural surroundings, etc.  Where I live in Los Angeles, we have those digital billboards everywhere, so it’s like advertising overload wherever you look (hints of Blade Runner).  I personally don’t mind them, but I know there is great debate on there being simply too many billboards everywhere. So AR would only add to the noise of life by adding yet another digital overlay of information, right?

Perhaps the holy grail is to use technology to filter things out. AR might become a solution to leading a simpler life, or a perfectly customized life if you want that. Ultimately the control needs to be with the individual.  I guess I am talking about something like TiVo taken to the extreme.

Tish: And then that other biggy – augmented reality search! I am asking this next question of  Wikitude and Sekai camera too and now I must also ask Acrossair and several others I guess! Obviously a huge area of opportunity in this broader landscape that uses location-awareness, barcode scanners, image recognition and augmented reality is to harness the collective intelligence – a whole new field of search. There is the beginning of a discussion on this in the comments here.

What will it take, in your view, to become a leader in augmented reality search?

Brian:  I’m more of a content guy, so I tend to focus on things like UI, quality of creative, etc..  From that perspective, I am looking forward to evolving beyond the “post-it” text overlay user-experience we see now in AR search. I was impressed with the TAT Augmented ID concept and hope we start seeing more smart design solutions like that emerging in the space.  There are some great new design approaches coming out of the location-aware space that should be applied to AR search. I’ve been studying the heads-up display designs being used in video games, and re-watching movies like Iron Man for ideas. This is another example where Hollywood has painted a polished picture of what AR can and should look like, and the masses have already accepted these design approaches.  So from that perspective, from my view the leaders in search will be delivering sexy, smart and simple solutions. It’s all about the S’s.

Today, I was very excited when Thomas Wrobel sent me a draft of, “Everything Everywhere: A proposal for an Augmented Reality Network system based on existing protocols and infrastructure.”

Thomas has kindly agreed to let me publish his draft, to open a discussion on this topic. The diagram opening this post (click image to enlarge) shows, “An example of how collaborative 3D-spaces could be shared over existing IRC networks.” It is from Thomas’ proposal. The full text of his paper is included later in this post.

“Can we try to avoid a browser war this time?”

Thomas notes in the closing remark to his paper:

“I am absolutely confident in my belief AR will become at least as important as the web has, and probably a lot more so. It will also face much the same hurdles and challenges getting established as that medium did. But, speaking as a web-developer, can we try to avoid a browser war this time?”

Thomas Wrobel has consistently posted insightful comments on how existing standards could be used for creating open augmented reality networks. But he expressed concern to me that his work and this paper not be overplayed:

“I’m hardly a leader, I’m just an amateur with a load of ideas on AR-related topics, some which might be useful, others might become unworkable. I don’t want anyone to get the impression this is how I think it has to, or should be done.”

I have brought/am bringing up this topic of using existing standards and infrastructure where possible for open augmented reality networks in all my interviews with members of the AR Consortium.

And I am finding agreement on a point that Robert Rice makes, “there is no perfect, ultimate solution *now*, but we have to do *something* to work from and refine/evolve.”

Thomas Wrobel makes what I consider some crucial opening suggestions. I take my hat off to him for thinking about this early, coming up with some clear, elegant, and practical ideas, and doing the work to articulate these ideas so others can participate in evolving them.  Massive props for that, many times over.

Good ideas on standards at an early stage of  a developing industry like augmented reality are like spring sunshine and April showers for new crops. No one knows what storms and pests the growing season will bring – but water and sunshine (open standards) are always a good start. And, personally, I can’t wait to see how this new industry unfolds (see Bruce Sterling’s Layar Conference awesome keynote : “At the Dawn of the Augmented Reality Industry.”)

Thomas Wrobel is:

“a web developer working for a small, brand-new company called Lost Again, which mostly works on ARGs (That is, the alternate reality games, not the augmented reality games, although there’s probably going to be big overlap there in the future). We developed two educational ARG games for the Netherlands with a company called res-nova.”

I have been following Alternate Reality Games  through the amazing work of Elan Lee and Fourth Wall Studios. Like Thomas, I think the intersection of ARGs and augmented realities is going to be very interesting. Thomas wanted me to point out that the website for his company with Bertine van Hövell, http://www.lostagain.nl/, is just a placeholder for now.

“Probably be up fully within a week or two. And, “despite the logo, we aren’t an AR company [yet], or a travel firm. The logos supposed to represent being lost in our minds.”

Thomas has been thinking about the topic of an open augmented reality network for a while now.  He is an artist also known as DarkFlame and his ARN network is included in this augmented reality concept for 2086 he did in 2006 (click on image below to enlarge).

Beyond IRC

Both Thomas and Mez Breeze made extensive and insightful comments on my last post, “Augmented Reality – Bigger Than the Web: Second Interview with Robert Rice.” And in particular they both picked up on something I am very interested in – the potential use of the Google Wave Web of protocols in creating open augmented reality networks.

Mez in her brilliant brainsplosion on social tesseracting takes on the very definition of information:

“Tish, when you ask Robert “…what is your approach to delivering a massively shared real time [augmented reality] experience that is like Wave not confined to a walled garden?” that’s an extremely relevant question + one that needs to be addressed while considering the entirety of the Reality-Virtual Continuum. I’ve recently finished a series of articles addressing this: the framework I’ve developed is termed “Social Tesseracting.”

I have recently begun exploring the Google Wave Web of Protocols which are nicely outlined in this post by J Aaron Farr which includes the very interesting diagram below (so more on Google Wave in another post).

But, as Thomas notes, while he demonstrates his ideas using IRC (Internet Relay Chat) they reach Beyond IRC:

“As mentioned before IRC has some drawbacks, which are due to its age or method of working. As such, future systems might yet prove better alternatives for a open AR network. One example of such a system is Google Wave. It shares many of the advantages of IRC (open, anyone can create a channel of data, different permission levels can be set and its free), while avoiding some critical restrictions. (The data can be persistent). I believe some of the ideas I’ve mentioned, and possibly even the proposed protocol string could be adapted for Google Wave or other future systems. I believe overall the principles are more important then any specific implementation to get to them”

Also Thomas pointed that while he uses markers to illustrate some of his examples, they are just a method for tracking.  What he is presenting is going to be transparent to the methodology of registration/tracking.

Tish Shute: You mostly use marker based examples but there is no reason why the principles you are suggesting will not be just as relevant as we move more into using more sophisticated image recognition tools is there?

Thomas Wrobel: No reason whatsoever. I mostly choose familiar markers as something that could be used now, with a lot of coding library’s already established for them. I think for most future AR use, markers will go completely…especially outside. Either things will be done purely by gps, object recognition, or the (in the case of advertising) markers will look like normal posters.

However, I do think traditional markers might “cling on” as being used for non geographical specific stuff at home. After all, if you need some reference points for moving mesh’s about in real time…(say, when playing a board game with a friend on the other side of the world)….then there’s probably nothing that’s going to be more practical then some simple bits of paper or card.

Everything Everywhere

-  A proposal for an Augmented Reality Network system based on existing protocols and infrastructure.

by Thomas Wrobel

The following paper is my vision of a open AR Network and potential methods to implement it with existing technologies. Specifically I’ll be focusing on a potential for a global outdoor AR network, although the ideas aren’t limited to that.

Of course I call it “my” vision, but I’m obviously not the first to have many of these ideas. I have been influenced and inspired by many things…

[Some of Thomas Wrobel’s influences – watched and played. Images  from Mitsuo Iso’s Denno Coil (Click to enlarge) top,  below from the game “Metroid Prime,” and Terminator, and the last from Buffy the Vampire Slayer!]

The AR Network.

When I speak of a future AR Network, I mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.
Where people can just point their phone, webcam, or pair of AR glasses anywhere were a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.

From this point forward, I will refer to this future AR Network simple as the “Arn”.

The Arn should be an inclusive, and open platform where any number of devices can connect to, and anyone can make and host their own location-specific models or data.
It should allow people to communicate both publicly and privately, and not have their vision constantly cluttered with things they don’t want to see.

There’s two old, existing paradigms that I think can help reach this goal when they are combined.

The Internet Relay Photoshop.

IRC, or Internet Relay Chat was a chat system designed by Jarkko Oikarinen in the late 80′s.

Its a system where people meet on “channels”, they can talk in groups, or privately. Channels can be read-only, or open to all to contribute to. There is no restriction to the number of people that can participate in a given discussion, or the number of channels that can be formed. All servers are interconnected and pass messages from user to user over the network.

To me, this relatively old internet technology is a great template, or even foundation, for how the Arn could operate. Rather then text being exchanged, it would be mesh data (or links to mesh data), but other then that much of the same principles could apply.

People could join channels of information to view or contribute. Families could leave messages to each other scribbled in mid-air on private channels. Strangers can watch AR games being played between people in parks. People going into a restaurant could see the comments from recent guests hovering by the menu items.
None of this would have to be called up specially, if they are on the right channel when it was broadcast, they will see it.

The IRC paradigm becomes particularly powerful when combined with another one common to many computer users; that of a “Layer” in an art program, such as Photoshop or Paint Shop Pro.
As most of us know, layers allow us to separate out different components of a piece of art while editing, either to focus our attention on one piece, or to make future editing easier.

Now what if we simply have each “channel” of information represented as a layer?

Click to enlarge image above.

Having channels corresponding to layers is an easy and intuitive way for the Arn to operate. The user can login and contribute data to any channel, like IRC as well as adjusting the desired opacity and visual range of each layer, like they would a layer in Photoshop.

In this way they can get a custom view of the world, both with shared and personal AR elements visible at the same time.
They would not have to switch between various overlays to their world view, as they could see many at the same time.

Persistence of Data

With IRC or IRC-like system to communicate the data sent is mostly temporary data…broadcast on the fly from user to user and device to device. Retained in the users local logs, but not “hosted” anywhere.

I think for the majority of day to day purpose’s this is not so much a drawback, but actually desired for AR. Most casual communication doesn’t need to be recorded permanently in 3D space and, indeed, if it was, the cost of running such a service would increase exponentially with users and with time. Not to mention, our visual view of the world would get very cluttered very quickly. Imagine what your monitor would be like if it kept a history of every window you have ever opened and their positions!

So for most cases AR space should be treated like a 3D monitor letting us display many pieces of data from remote and local sources, and even to share them with others, but not being, by default, a permanent record for it all.

Most data will be analogous to pixels on a display, and if kept in records its only on the clients devices, not on the network itself.

However, occasionally we do want 3d data analogous to a web-page, such as (in the example above), the map layer. Data here should be persistent and visible to all that have that layered turned on.  I see no reason why hosting this data needs to use anything else but standard web-hosting with the (read only) #channel on the Arn merely providing a route to the data.

As the user logs onto the channel, the server, using a chat-bot, can send them a list of meshes with location data attached, and the Arn browser can simply pick the data to display that’s local to them. (Note 1: By doing it this way around, it allows some degree of anonymity to be possible, rather then the server knowing exactly where you are and feeding the specific correct string to you.)

We simply need to establish standards so this data can be pulled up and interpreted.

For instance, this standard could be as simple as a XML string pointing to a KML file on a server. This could then be then displayed in the users field of view at the co-ordinates specified.

In this way permanent data tied to locations, such as historical overlays or maps, could co-exist on the same protocol as temporary data such as mid-air chat’s or gaming related meshes.

There is also no reason why this shared-space/personal spaces based on channels of data has to be restricted to things given absolute co-ordinates.

(Different ways to access the same mesh)

It could work just as well with Markers and thus relative co-ordinates.

This would be mostly useful for indoor use, letting people logged onto a channel see the same meshes as everyone else on the markers. Thus allowing multi-player AR games, or AR games with observers very easily.

For example; games like Chess could be played between people with no additional code needed; You simply have a set of markers for only your own pieces, and as you move them the channel updates with the new positions, which are displayed in place in your opponents field of view.

This sort of game comes “free” with just having a generic system of shared space supporting markers.

It would also allow AR adverts down the street or in magazines to be viewed by simply logging onto the right AR channel

If markers are designed with URL data in them, this could even be a prompted or automatic process.
“There is visual data in this area on the following channel; #ABCD would you like to view this channel?”

Pros and Cons of using IRC or IRC-like systems

Pros;

• Anyone can write a IRC interface software.
• Anyone can create new IRC channels without cost
• Channels can have read and write permissions set.
• Users can easily have multiple channels open at once.
• Already established with thousands of severs worldwide.

Cons;

• 500-or-so character limit. 3D data must be linked too, not sent.
• Slow update rate. Lines of data can take a whole second or more to send.
• Non-persistent. Good for a 3d-view, not good for storage.

An example of how collaborative 3D-spaces could be shared over existing IRC networks;

Click on the image to enlarge

While in the long run I would hope for a dedicated AR network to be developed, with greater flexibility with persistence of data, there is a lot that can be done with the existing IRC system to implement the ideas mentioned above.

Below I will show an example of simple, crude, pseudo-protocol that could be fairly easily implemented to create shared AR spaces broadcast across IRC channels.

Its important to note, the goal here isn’t to exchange the mesh data itself on IRC, its to exchange links to the data.

Exchanging the mesh data directly within the 500 character IRC limit would be very hard, and liable to errors.

It’s also a waste of network bandwidth, as many people logged onto the channel might not have that object in their field of view, so their clients should not bother downloading it. (it should be up to the client browsers when to anticipate and cache mesh data).

Proposed Basic XML link exchange for AR;

Principle;
As user creates or changes an object, the clients software  posts a simple xml formatted string to
the IRC channel.
Anyone logged into that channel then sees that mesh displayed in the specified location.

This string could be formatted as follows;

<Mesh
ID=”DARKFLAME:1”
Obj=”http://www.darkflame.co.uk/mesh/church/chuch.kml”
Loc=”(49.5000123,-123.5000123)”
Permissions=”None”
LastUpdate=”12/12/0000,2012:12”
/>

This string allows other users client logged into the channel to automatically load the object from the URL and display it at the correct position in their field of view.
If the permissions are set to allow it, they could then move the object themselves, with the update being feeding back seamlessly to other users on the channel.

The objects posted are given an ID, which can be just the posters name, followed by a unique object number for that name. These unique ID’s would allow clients to track different instances of the same mesh, as well as making it easy to implement permissions. (if only the poster should be allowed to move this object, then the clients simply check if ID matches the user name posting the update. If its not, they can ignore it).

Next the objects need to be linked to a mesh.

The location of the objects mesh doesn’t have to be a fixed remotely-hosted url, it could be an IP address and port number of the user posting the mesh,hosted by the application posting the link to the channel.

Obj=”www.darkflame.co.uk/mesh/church/chuch.kml”
Obj=”123,223,14,23::3030”

The objects co-ordinates, likewise, need not be specified as absolute gps co-ordinates, but instead could refer to generic Marker.

Loc=”(49.5000123,-123.5000123)”
Loc=”Marker1”
Or relative to a marker;
Loc=”Marker4 (+0.0023,-0.0023)”
Or relative to a default plane;
Loc=”Default(+0.213,-0.123)”

The AR Browsers could then handle the association between the Markers pattern and its Name.
This way the markers are reusable, they do need unique markers to be printed for every new bit of AR they want to look at.
Users could just keep a set of generic markers handy, which they could simply assign to be Marker1,Marker2 etc for any AR use. (Note 2: As mentioned above specific makers could also contain a default ID name and channel built into their data, letting the Arn browser simply prompt the user if they want to see the model even if they aren’t in the right channel. This set up would be most useful for paper and even billboard advertising.)

The Default location could be a settable region, or marker, on the clients browser that defines a playable/user-able area in the field of view. Mostly useful for home use, this could typical be a square region on a users desk.

So, in the chess-game example, the client of the person making the moves simply updates the position relative to the Default every time they move their marker (which is tied to a chess piece mesh).
Then the (non-owners) clients software could automatically display it relative to their Default plane. This would make games like Chess, Checkers, Go or any other game involving merely moving objects about automatically very intuitive and easy to set up.

So by having meshes settable to absolute gps, marker-relative, or default-relative locations, reduces the bother necessary to experience AR content quite considerably, and makes “non-geo-specific” AR applications and games trivial to implement.

Next is permissions.

Mesh-permissions would be a simple string saying who else can update the data, if anyone.

eg;
Permissions=”None”
Permissions=”RandomPerson1, RandomPerson2”
Permissions=”All”

By default you could only update or move your own meshes. (identified by the ID of first posting). If you attempt to update anyone else’s,  their clients would just ignore it.

Thus in a game of chess, you can only move your own pieces. If you attempted to move your opponents (by reassigning  your own marker to their pieces Ids), the clients would just ignore that assignment. You’d only be fooling your own system.
Likewise, when pinning a message in mid-air for your friends to read, no one else can change that message without your permission, although copying it would be easy.  (Note 3: It’s important to note this sort of object-specific permission system is in addition to the global-permissions, or “user-modes” it’s possible to set for the IRC channels and users as a whole.)

Finally, as object data could change within all sorts of time-scales, the easiest way to keep everyone logged in up to date is to just have a time-stamp of when each model was last updated.

LastUpdate=”12/12/0000,2012:12”

This would not necessarily be the same as the XML string post date, because the models mesh might not be updated, but merely moved, and in such case the Arn browser shouldn’t redownload the mesh.

This sort of arrangement could be used as a standard today, and users wouldn’t have to constantly download special AR programs to view a single AR mesh.

In the long-term I would hope for more advanced methods to manipulate Arn-content online, analogous to Dom manipulation in web-pages. But for now, we should at least establish standard methods for devices to pull up meshes and overlay them in the correct position.

So, having a layered system could give the user a seamless blend of dynamic and static data with which to paint their world with.
I believe this is all relatively easy to achieve using modifications of existing web technology, combined with some basic graphics systems.

Local Data:

However, so far I have only talked about remote data.
What of programs originating on the device itself? This is, after all, how most AR software we have at the moment works.

I think, that just like the remote channels, local software should also be blended into the same list of layers.  People shouldn’t have to “Alt+Tab” out of one view of the world, to see another.
They should be able to see both at once, if they wish.

For instance, if your playing a AR game, why shouldn’t your chat window be viewable at the same time?

If you have skinned your environment with a custom view of the world, why shouldn’t you also see mapping or restaurant recommendations?

So local data and remote data should be blended in the same view.
How can AR software – of which I hope, there will be  thousands – seamlessly be expected to layer their graphics, not only with the real world, but with each other, and with online data too? Will games and software makers need to co-operate to allow their graphics to be integrated together with correct occlusion taken into account? A tall order, no?

I must confess though, my technology knowledge fails me here.

I can only guess special graphics drivers, or 3D APIs,  will have to be developed to let programs share their 3D world with that of a Arn browser.
Maybe programmes should simply treat themselves as a local-sever which the browser can connect too, and let the Arn handle all the rendering itself (although I imagine many games designers would find this quite limiting).
So I leave it as an exercise to the readers to discuss and propose the best methods by which this vision of a layered world could be realised..

Beyond IRC:

As mentioned before IRC has some drawbacks, which are due to its age or method of working.
As such, future systems might yet prove better alternatives for a open AR network.
One example of such a system is Google Wave.
It shares many of the advantages of IRC (open, anyone can create a channel of data, different permission levels can be set and its free), while avoiding some critical restrictions. (The data can be persistent).
I believe some of the ideas I’ve mentioned, and possibly even the proposed protocol string could be adapted for Google Wave or other future systems.
I believe overall the principles are more important then any specific implementation to get to them.

Summary;

⁃    In order for AR to flourish the user shouldn’t need to download a separate application for each mesh they want to see.
⁃     Having url’s embedded into QRCoded markers which point to standard mesh files like dxf or kml would be a way to do this right now.  The QR code would only have to be seen precisely  in shot once, then its borders could be used like a standard marker.

⁃    An augmented view of the world needs to support visual multitasking, and having  layers of information is the best way to do that.
⁃
⁃    Methods need to be devised to allow drastically different software to contribute to these layers, without restricting either the software’s rendering ability’s, or the users ability to pick and choose what layers of information he wants to see.

Last point;

I am absolutely confident in my belief AR will become at least as important as the web has, and probably a lot more so. It will also face much the same hurdles and challenges getting established as that medium did.
But, speaking as a web-developer, can we try to avoid a browser war this time?

Everything Everywhere , draft.
by Thomas Wrobel
Darkflame a t gmail