Blair MacIntyre is one of the original pioneers of  augmented reality and an extraordinary amount of creative work is coming out of his Augmented Environments Laboratory at Georgia Tech – see YouTube videos here.  The screenshot below is from, ARhrrrr, a very impressive augmented reality shooter game created at Georgia Tech Augmented Environments Lab and Savannah College of Art and Design, (SCAD- Atlanta), and produced on the NVidia Tegra devkits – watch the demo here.

Blair has spent much of his career working on immersive augmented reality and more recently the integration of augmented reality with mirror worlds. Blair explains:

“I am interested in the intersection of mobile devices – whether they are head mounts or handhelds – and parallel mirror worlds…I think that parallel mirror worlds are a direct manifestation of the intersection of the virtual world we now live in (the web) and geotagging.  As more and more information is tied to place, and as more of our searching become place-based, we will want to do those searches about places we are not at.  A 3D mirror world may provide one interface to that data.  Want to plan your trip to London;  go their virtually and look around, see what is there (both physically and virtually), teleport between areas you want to learn about, and so on.  More interestingly, talk to people who are there now, and retrieve your location-based notes when you are on your trip.”

But, at a time when many augmented reality developers are focusing on AR apps for smart phones, including Blair (the picture on left opening this post is Blair’s augmented reality iphone app ARf), I was interested in finding out from Blair what the state of play was for the real deal Rainbow’s End style AR, as well as the potential he sees in smart phones to mediate meaningful AR experiences.

There is enormous amount of  innovation in mapping our world, see my post, “Location Becomes Oxygen at Where 2.0 and WhereCamp,” and  Ori Inbar’s  Where 2.0. conference roundup. But as Ori notes, to move augmented reality forward:

My point is not a shocker: all we need is to tap into this information and bring it, in context, into people’s field of view.

And this is what Blair MacIntyre’s work is all about.

Talking With Blair MacIntyre

Tish Shute: There do seem to be broader implications to augmented reality today than when this term was first coined. I am interested to have your perspective on how augmented reality may go beyond some of our early definitions?

Blair MacIntyre: I still think the original definition of the term is useful:  media (typically graphics) tightly registered (aligned) with the physical world, in real time.  Many people talk about many things that relate virtual worlds to places, spaces, objects and people.  There is room for many of them, and they don’t all have to “be” augmented reality.  I like using Milgram’s definition of Mixed Reality as everything from the physical world (at one end) to the virtual world at the other;  it’s a spectrum, and augmented reality just sits at one point.

The reason I like the old definition is I believe there is something special about graphics that are tightly, rigidly aligned with the physical world.  When things appear to stick to the world, and an obviously identifiable location, people can start leveraging their natural perceptual, physical and social abilities and interact with the mixed world as they do the physical world.  We’ve found this with the two studies we’ve done of tabletop AR games (Art of Defense and Bragfish);  one key to those games is that the graphics were tightly aligned with identifiable landmarks in the physical world (gameboard).

Art of Defense (pic on left) Bragfish (pic on right)

Tish: I know that you are involved with ISMAR 2009 which is the key US augmented reality conference.  What do you think will be the hot themes, applications, innovations at this year’s conference? Do you think this will be the year that AR really breaks out of eye candy into truly useful and sustained experiences?

Blair: Unfortunately, I won’t be involved this year.  I was supposed to be helping run the technical program, as well as the art/media program, but sickness in my family prevented me from having the time, so I am not helping this year.

First, I would not agree with the implication of the last question — I don’t think AR has just been eye candy up to now.  I do agree that the “high profile” uses of it have largely been that, which is mostly because of the limits of the technology.  I don’t think we’ll see huge changes in that regard by ISMAR this year.  However, we will hopefully see a mixing of communities that hasn’t happened at ISMAR before, and I do believe that this year (independent of ISMAR) we will see more and more AR apps.  Whether they go beyond eye candy is still a question.  I’m hoping that some folks (including myself and other ISMAR folks!) will help push AR in new directions.  But I also expect many folks new to ISMAR and AR to play a big role, because it is this new blood, especially those folks with real problems to solve, new art and game ideas, and a fresh perspective, that will open new doors.

Tish: You have been working on integrating augmented reality with virtual worlds. You mentioned that the way you use Sun’s Wonderland is really about pulling the virtual world into the real world, i.e., Wonderland, “is just a place to put data.”  How is your use of the persistent virtual space different from what we have become accustomed to call virtual worlds?

Blair: The approach we are taking in our project at Georgia Tech is to use the virtual world as the central hub of the information space, and allow the virtual world to be the element that enables distributed workers to collaborate more smoothly.  This is work we are doing with Sun and Steelcase (and the NSF), and is an outgrowth of a project (the InSpace project) that’s been going on for a few years.

What we are trying to do is use mixed reality and ubicomp techniques to pull as much of the physical activity into the virtual world, and then reflect that activity back out to the different participants as best suits their situation.  So, folks in highly instrumented team rooms will collaborate in one way, and their activity will be reflected in the virtual world;  remote participants (e.g., those at home, or in a cafe or hotel) may control their virtual presence in different ways, but the presence of all participants will be reflected back out to the other sides in analogous ways.  We may see ghosts of participants at the interactive displays, or hear their voices in 3D space around us;  everyone will hopefully be able to manipulate content on all displays and tell who is making those changes.

A secondary benefit, I hope, is that by putting the data in the virtual world and making that the place that gives you more powerful and flexible access to the data (e.g., by leveraging space and giving access to history), distributed teams will begin to have the virtual space become a place they go to work, bump into each other and have those casual contacts co-located workers take for granted.

Creating the Information Landscape of the Future

Tish: At the end of my interview with Ori Inbar he said, in order to have a ubiquitous experience “you’ll need to 3d map the world. Google earth like apps are going to help but it is not going to be sufficient. So let’s leverage people. Google became successful in part by making people work with them.  Each time you create a link from your blog to my blog their search engines learn from it.  So let’s find ways to make people create information that can be used for AR.” What ways do you think people can create information that can be used for AR?

Blair: I think the big part of that is the creation of models and environments, the necessary “baseline” for specifying experiences.  Google and Microsoft are clearly working toward this;  recent videos from Microsoft show them starting to move the photosynth work toward Virtual Earth.  Similarly, I came across a page where people are finally starting to mine geotagged Flickr [see my post, “Location Becomes Oxygen,” and here for more on the “The Shape of Alpha” project from Flickr]  images to create models.  It’s that kind of thing that will be useful first;  using the data we all create to enable modeling and (eventually) vision-based tracking in the real world.

After that, it’s a matter of time till more of what we “create” (e.g., Tweets and blog posts and so on) are all geo-referenced;  these will become the information landscape of the future, the kinds of things people think about when they read “Rainbow’s End”.   The big problem will be filtering, searching and sorting.  And, of course, safety and security.

Tish: You are working with Unity3D to research the integration of mobile location based AR with persistent mirror world like spaces.  What has attracted you to Unity? What is the difference between this and your Wonderland project? I know you mentioned. you will be using head-mounted displays are part of this Unity project. What are your goals for this project?

Blair: We started to use Unity3D because it gave us what we wanted in a game engine.  Most importantly, it’s very open and let us trivially expose AR technologies into the editor.  Similarly, it can target the iPhone, so we can begin to work with it on that platform, too.  The biggest problem with creating compelling experiences is content;  and a show stopper for creating content is not getting it into your engine.  Unity has a nice content workflow.

Unity3D is a front end engine, for creating the game;  Wonderland is both a front end, and a backend.  We are actually looking into using the Wonderland backend with Unity as well.  Wonderland also has growing support for doing “real work” in a virtual world, which is key to our other projects.

Eventually, we’ll be using HMD’s.  The goal for the Unity3D project, initially, was to explore what you can do with an AR/VR mirror-world; this is a project are working on with Alcatel-Lucent, and demo’d at CTIA this year.  It’s continuing to grow, though, and now includes a number of our projects, including some work on mobile social AR and soon, some performance and experience design projects in the area of AR ARG’s.  It’s really quite interesting to imagine what you can do when you have an “MMO of the real world” (which we now have for part of campus) that supports both VR-style desktop access simultaneously with mobile AR access.

Tish: Have you taken another look at OpenSim as a possible backend for augmented reality?  Recently I talked to David Levine, IBM and he is thinking about some possibilities to optimize OpenSim to dynamically load a large amount of objects at once (i.e how fast OpenSim can bulk load into an existing sim) and make it better suited to augmented reality/mirror world type projects.

Blair: I haven’t looked at OpenSim recently.  We will probably look at it this summer.

Tish: Why did you select Unity as a good client for augmented reality?

Blair: Unity is a 3D game authoring environment so at some level it is no different from using Ogre, if all the associated stuff was just as well done. It has integrated physics, scripting, debugging, etc. – you can write code in javascript or C# or whatever.   It has a good content pipeline, as well, and supports a range of platforms.

It has simple networking built in, so multiple unity engines can talk to each other but it is not a virtual world platform out of the box – there is no back end …

Tish: Someone described Unity to me as a great client waiting for a great backend? So what are you going to use as a back end?

Blair: There is no real processing except in the client right now.  We will eventually have to create a back end.  We are thinking of using Dark Star because someone on the Sun Wonderland community forums has already built a set of scripts connecting Unity to Darkstar.

But for us, we are not proposing right now to build a real product.  This is research to demonstrate what you could do if you actually had the back end.

Tish: What are the most important aspects of the backend from your POV?

Blair: We want to simulate a variety of the interesting aspects of the back end.  So I very much care about notions of privacy and security and how these sorts of AR/VR Mirror Worlds would work in practice.  But I care about how those things as they impact user experience, not really about how we would really implement them.

Tish: So looking at some of the big problems from the perspective of user experience? Are we are going to go through the same growing pains that the web and VWs have seen, for example, will we have to type in passwords to get into everyone’s little worlds….

Blair: Well you know the SciFi background to this, you’ve mentioned it in other posts on your blog.  Because when you look at the Rainbow’s End model where you have security certificates flying around, that is in effect what cookies and so on are now.  You can authenticate yourself once and then have those certificates hang around. So you can easily imagine how it could be done.  But the big question is how does that change user experience.  There are all kinds of things that start coming into play – like what happens if nearby people see different things – it goes on and on!

Tish: Sounds Like this is very valuable research.  It seems to me that there will be a lot of investment soon in putting the pieces together to do location based markerless AR and it would be nice if we knew more about it from the user experience POV.

Isn’t it vital for a productive intersection between mobile AR and persistent mirror world spaces for us to have markerless AR?  Aren’t we right at the beginning of people really saying yeah markerless AR is doable now? But it seems to me not many people researching or working on fully immersive AR and its integration with mirror worlds?

Blair: I think some of the AR community is thinking about this. There’s probably people who are doing stuff in some other non technical communities. It wouldn’t surprise me to find out that people in the digital performance or ARS electronica world who are thinking a little bit about these sorts of things. Although not necessarily at the level of actually trying to build it, because they probably can’t right now.  But experimenting with the precursors.  My colleagues in digital media like to point out that this is often the purpose of digital art, to point out new directions and push the boundaries.

Obviously Science Fiction has explored the possibilities because that is what Rainbow’s End and the Matrix were all about.

Tish: and Denno Coil…

Blair: There has been some research – people like my adviser Steve Feiner up at Columbia, Mark Billinghurst in New Zealand, myself and people at Graz University in Austria .  But partly it has been so hard to do mobile AR up to now – so many people mock head worn displays and can’t get past current technology – you have had  to be willing to ignore the bulky back packs and cables and batteries and so on.  That is changing which is good.

My current response to the anti-head-mounted display people is if 5 years ago you told me you told me that fabulously dressed people who care about their looks and wear stylish clothes would have had big things hanging from their ears that blink bright blue light, so they could talk on the phone, many of us would have said you were crazy, because it would be ugly and so on.  But because there is an intersection of demonstrable need and benefit…Bluetooth headsets are really useful and the sort of early gestalt feeling that grew up around them – that people who use them are so important that they always have to be in touch, they wear these things – so people accept them.

It will likely be a similar thing with head mounted displays. And I don’t know if it will be that people wearing them so that they can read their mail while driving, god forbid. But it will be something.  And when we get the 2nd generation of the wrap glasses that look more like sun glasses and are not bulky and so on, we will have the potential for them catching on because you will look at them and you will think that the person is wearing because they are doing x…

X might be surfing a virtual world or reading their email or keeping in touch, or being aware. It will happen. But they have to get unbulky enough and there has to be more  than one important application, not just watching TV.

Picture above shows  an outside view of the KARMA AR system;  the knowledge based maintenance system Blair built in his first year of grad school (“first AR system Steve Feiner, Doree Seligmann, and I worked on”).  Blair noted, “The Communications of the ACM paper on it (from 1993) is a pretty widely cited AR paper.”

Tish: I think the need for  full on transparent, immersive, wraparound, Gucci stylish eyewear with a decent field of view are the elephant in the room in terms of realizing the full potential of augmented reality.  There are a few new players in the field Digilens,  Vuzix, others?  What is the progress in this area and what do you hope for in terms of near term solutions?

Blair: I agree with that sentiment.  I think that, in the near term, there is a lot we can do with handhelds, as we’ve been doing in the lab.  However, because it’s awkward and tiring to hold up a device, even a small one, for any length of time, handhelds will only be good for “focused” uses of AR.  Such as the table-top games we’ve been doing, or the constellation viewing app that I heard came our recently for the Android G1.  I don’t even see something like Wikitude as that compelling (beyond the “gee wiz” factor) for a handheld form factor.   Many proposed AR apps only really become compelling when users have constant awareness of them, and that requires a see-through head-worn display.

I’ve seen the mockups of the Vuzix ones;  they seem pretty interesting, and are getting to were early adopters could use them (they will be cheap enough, and will hopefully be good enough).  Microvision’s virtual retinal display is also promising;  the contact lens displays will be the most interesting, if anyone can ever make them work.   I don’t know of anything else out there.

“its not really a killer app you care about, it is the killer existence that all of the technology and small applications taken together facilitate”

Tish: While location based services are accepted now and people are understanding that it is something that opens up a new relationship to everything, we still haven’t found the experience that will get everyone holding up their mobile devices?

Blair: Well that is actually the killer problem.  Gregory Abowd is one of my colleagues who does ubiquitous computing research here at Tech.   Way back when we started the Aware Home project (Aware Home Research Institute at Georgia Tech) when I first got here about ten years ago, there was always this question of what is the killer app.  So Gregory comment in a meeting once that its not really a killer app you care about, it is the killer existence that all of the technology and small applications taken together facilitate. It is not that any one of these AR demos we see right, whether it is seeing your photos in the world or whatever, is important. Its that when taken together, there is enough of a benefit that you would use the whole environment.

In the original context we were talking about an instrumented home, but it is the same thing here with AR.

The problem with the mobile phone as a AR device is that problem of awareness. If I have a head mount on and I walk down the street and there is bunch of probably-not-useful-but-potentially-useful information floating by me, that’s a good thing, because I may see something that is useful or makes me think of something else.  But if I have to hold up my phone to see if something might be interesting nearby, I will never hold up my phone because at the time there is a high probability that there won’t be anything particularly important there.  You might imagine you can get around this by using alerts or something like that, but then you overload whatever alert channel you use.  For example, I forward maybe 5 or 6  people’s updates from Facebook to my phone – started with my wife, a few friends, my brother, and the net result of that is I never get SMSs’ anymore because when my phone buzzes, usually I ignore it because it is probably just somebody’s random Facebook update. So if we start overloading channels like that with “oh there might be something useful here in the real world, if you pick up the phone and look through it you will see it … and I will buzz you.” People  just start ignoring the buzzes.

So it is a very hard problem if you think about the kinds of applications that people always imagine with global AR — names over peoples heads and other random information floating in the world — until you have a head mount and all that information is around you all the time. That is when those sort of applications will actually happen.

Tish: Robert Rice notes: “AR is inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.” (see my interview with Robert, “Is it ‘OMG Finally’ for Augmented Reality?). And I think the iphone experience has laid the foundation for the increasing desire to experience the network wherever we are – and not be stuck behind a pc.  We cannot perhaps do all we want to do yet. But even in the range of things we can do know, we are not even sure exactly what it is we want to do where yet is it?

“imagine your iphone Facebook client supports AR and that all data on Facebook might be georeferenced – pictures, status updates etc…….”

Blair: Yes that is a huge problem. I have been lucky to be able to teach two fun classes this year that let the students and I start to explore some of the potential that handheld AR might bring.  Last fall I taught a handheld AR game design class — coordinated with a class at the Savanna College of Art and Design’s Atlanta campus — and we had the students build a sequence of prototype handheld AR games, which was a lot of fun.   This spring I taught a mixed reality/augmented reality design class with Jay Bolter (a professor in the School of Literature, Communication, and Culture here at GT).  Jay and I have been teaching this class off and on for about 9 years; this semester we decided to say to the students “imagine your iphone Facebook client supports AR and that all data on Facebook might be georeferenced – pictures, status updates etc…….” and have them do projects aimed at such an environment.

Tish: Not many of our favorite social media today have much sense of location do they? But Flickr  are  utilizing the geo-referenced pictures to create vernacular maps…..The Shape of Alpha

Blair:Yes that is because lots of cameras put geo location data into the exif data so they can extract it…

Some mobile Twitter clients like the one I use in my iphone will let you add your location.  But in general Facebook and other sites don’t have any notion of location. But if you look at all the things people do in Facebook, such as sending gifts and other games, its easy to imagine what these might look like with geo-reference data.  So, the high level project for the class is the groups have to design experiences people might have using mobile AR Facebook.  We told them to assume Facebook as it stands now, but add geolocation and AR to the client.  The class boiled down to “What would you imagine people doing?” So it has been kind of fun.

And we are using Unity for the class too – the same infrastructure I am working on in my research linking mobile AR to persistent immersive mirror world type spaces – and we having the students mock up what a mobile AR Facebook experience would be like.

Tish: Can you describe some of the ideas you class came up with that you think have potential? I know Ori mentioned that from the games class he liked Candy Wars.

Candy Wars

Blair: In the end, they had a nice range of projects in the Spring class.  One created tag clouds out of status messages over spaces, others looked at analogies to virtual pets and gift giving out in the world, one looked at leveraging geolocation to help with crowd-sourced cultural translation, and three groups did straight-up social games.

[See all of the projects from the handheld AR games class on YouTube here]

iphone, Android, or NVidia Tegra devkits or the Texas Instrument’s OMAP3 devkits?

Tish: Is anyone in the class working on Android?

Blair: Nobody is using Android because no-one in the class has the phones. We have ATT microcell infrastructure on campus.  Some ATT people joke that we are better off than them because we have a head office on campus so we can build in the network applications which people even at ATT research can’t do.  But because  we have this infrastructure on campus, and a great relationship with ATT and the other sponsors, we have the ability to provision our own phones without having to pay for long-term contracts, which is vital for research and teaching.

Tish: So does this lock you into the iphone?

Blair: Well the G1 is of course not AT&T but it is GSM so we could probably buy them unlocked and put them on our AT&T network. But the students I work with are much more interested in the iphone right now.

Tish: Is that because the iphone has the market?

Blair: For me the reason I am not interested in the G1 is because you can’t do AR on it – there is Wikitude and a few other apps, but it is all hideously slow.  Worse, because the Java code isn’t compiled like it would be on the desktop, you can’t do computer vision with it, so you can’t do anything particularly interesting on the current commercial G1s.  We could probably take the NVidia Tegra devkits or the Texas Instrument’s OMAP3 devkits (both are chipsets for next gen phones — high end graphics, fast processing),  and install Android on those and we may actually do that yet.  But, it seems like a lot of work right now, for not much benefit.

Augmented Reality shooter game ARrrrr from Georgia Tech and SCAD Atlanta on the NVidia Tegra devkits – watch the demo on YouTube here.

Tish: Everyone seems very excited about the iphone OS 3.0 and the addition of compass. Compass is pretty essential for AR right?

Blair: It is necessary if you can’t do other forms of outdoor tracking, but the problem is that the compass on the G1 isn’t very good, relatively speaking and the iPhone one probably won’t be much better. It does not have very high accuracy, nor is it very fast (compared to, say, the high end 3D orientation sensors we use, from Intersense and MotionNode). As far as I can tell, it doesn’t even give full 3D orientation. I don’t have a G1 (although I have pre-ordered an iPhone 3Gs), but people have told me it only has absolute 2D orientation, so you can only line things up if you are careful.  Your can’t look around arbitrarily…

Tish: You can’t sweep your phone?

Blair: You can look left and right, but if it doesn’t have full 3D orientation, you can’t go up and down. You can’t tilt it in weird directions. It is not fast in the form that you would want to look around quickly.  So it is nice demo.  And it is good for what the Android people use it for which is to let you do your Google street view by looking around, which is actually really useful.

I think there are lots of really useful things you can do with such a compass.

And, it is clear that compass is a necessary feature if we want to do AR.  It’s just not sufficient.

Outdoor Tracking and Markerless AR

Tish: Isn’t it essential for markerless AR? I guess not I just saw this post about SREngine on Augmented Times!

This wasn’t up when we spoke so perhaps you have some comments about what it brings to the table?

Blair: Maybe. The folks at Nokia are working on outdoor tracking, they demoed some stuff at ISMAR last year on the N95 handsets that is all image based.  We are trying to do some work with them, one of my students is working on it.  And probably Microsoft is going to do more on this as well, they had a video up showing that they are also working on vision based techniques.  If you give the phone the equivalent of those panoramic Google Street View images (assuming they are up-to-date) and you are standing at the right place, you don’t really need a compass, you can figure out which way you are looking by looking at the camera video. Ulrich Neumann (USC) did some work on tracking from panorama’s years ago, I don’t know what ever became of it.

Regarding SREngine, that project appears to be a pretty simple first step, but is probably just a demo at this point, and limitations like “only works on static scenes” and “doesn’t work for simple scenes” means it’s probably extracting some simple features out of the image and then matching those to some database.  The trick would be getting this to work on a large scale, where the world changes a lot.   It’s not obvious how to get there.

Tish: So forget RFID for AR…

Blair: RFID is not really useful.

Tish: not at all?

Blair: RFID is useful for telling you what things are near you.  The problem is it doesn’t give you any directional information – it just tells you you’re in range of the tag. So can use it to tell you when you are near a certain product for example.  So it is useful in terms of telling you what thing you are near, and then you can load up a vision system or something else that will recognize that thing.

In that way, it could be useful as a good starting point.

Similarly for computer vision, the compass and the gps are very useful for giving you an initial guess at what you may be looking at that can then speed up the rest of the process.  But, computer vision by itself will not be a complete solution because if I have my panoramic Google Street view (or whatever image database I use for tracking) and you are standing between me and the building -  I am not going to see what I expect to see, I am going to see you.

So I think it is all going to be part of one big package – you are going to see accelerometers, digital compasses, and gps and then combine that with computer vision and other sensors, and then maybe we are going to start getting the things that we have always dreamed about.  I like to show this video from the U. of Cambridge (work done by Gerhard Reitmayr and Tom Drummond) of an outdoor tracking demo because it gives a sense of what will be possible.  Techniques like this will be an ingredient in the future of things.  It becomes especially interesting when you have these highly detailed mirror worlds.  It is sort of one of those chicken and egg problems where if I have an highly detailed model of the world then techniques like they have can be used to track.  But that mirror world needs to be accurate or you can’t use it for tracking, and why would you create the mirror world if you couldn’t track?

Tish: I noticed in your comment to “my interview with Robert Rice” that you said you thought that is was important not to collapse AR into ubicomp – “forgetting what originally inspired us about AR” is, I think if I remember correctly, the suggestion you made. But aren’t ubiquitous computing and AR basically coextensive?

The vision of ubicomp Mike Kuniavsky describes – “sharing data through open APIs and the promise of embedded information processing and networking distributed through the environment” demonstrates how much can be done with very little processing power.” In its most immersive form augmented reality requires a lot of processing power. I think we have all become very conscious about trying minimize levels of consumption.  Can you explain why you think people shouldn’t see AR as the Hummer (energy squandering indulgence) of Ubiquitous Computing?

Blair:  I think there will be a hierarchy of interfaces. You are going to have the rich Rainbow’s End like experience – you are totally submerged in a mixed environment, if you have a head mount on (its not going to be Rainbow’s End for while) but if you don’t have the headmount on that information might be available to you other ways, whether it is a 3D overlay using your handheld or just a 2D mashup with Google maps.  But there will be some circumstances and people who will want to get the compelling experience you can only get with the headmount.

Tish:  Are you doing any research on how all these hierarchies of experiences will fit together – what aspects of this are you looking at?

Blair: The thing that really needs to happen is you need to have this backend architecture that allows you to collect your data from different sources and aggregate it much like the web. Right now Google Earth and Microsoft’s Virtual Earth are much like the old pre-web hyper-text systems that were all centralized. And what we really need is to have the web equivalent where Georgia tech can publish their building models and I.B.M. can publish their building models and their campus models, and your client can aggregate them, as opposed to Microsoft or I.B.M. puts their building models into Google Earth and then somehow you get them out with Google’s google earth browser. That’s just not going to fly.

Tish: so what does it take then to get us to this backend architecture, because I’m in total agreement?

Blair: The nice thing about augmented reality versus virtual reality is that you don’t need everything modeled. You can do interesting AR apps like Wikitude with absolutely no world model.

Tish: So that means we can start with what we have – utilize cloud services without a full blown backend architecture?

Blair: It may very well be that Google Earth and MS Virtual Earth act as a portal because people go and build models and link them with KML, and they can see them in google earth but they can also download the KML’s through some some other channel. So it may be that those things end up being something that feeds some of this along. Then people start seeing a benefit to having these highly accurate models so then you start integrating the Microsoft photosynth stuff and leveraging photographs to generate models.

It’s just keeping up with it and building it in real time is the challenge. A lot of folks think it will be tourist applications where there’s models of times square and models of central park and models of Notre Dame and the big square around that area in paris and along the river and so on, or the models of Italian and Greek history sites – the virtual Rome. As those things start happening and people start building onto the edges, and when Microsoft Photosynth and similar technologies become more pervasive you can start building the models of the world in a semi-automated way from photographs and more structured, intentional drive-by’s and so on. So I think it’ll just sort of happen. And as long there’s a way to have the equivalent of Mosaic for AR, the original open source web browser, that allows you to aggregate all these things. It’s not going to be a Wikitude. It’s not going to be this thing that lets you get a certain kind of data from a specific source, rather it’s the browser that allows you to link through into these data sources.

So it’s that end that interests me. It’s questions like “what is the user experience”, how do we create an interface that allows us to layer all these different kinds of information together such that I can use it for all my things. I imagine that I open up my future iphone and I look through it. The background of the iphone, my screen, is just the camera and it’s always AR.

I want the camera on my phone to always be on, so it’s not just that when I hold it a certain way it switches to camera mode, but literally it’s always in video mode so whenever there’s an AR thing it’s just there in the background.

When we can do that I can have little alerts so when I have my phone open I can look around and see it independent of the buttons and things that I’m tapping and pushing to use the phone. That’ll be a really a different kind of experience.

Of course it is not known yet if the next gen iphone will have an open video API.  And of course, the current camera is pretty low quality, so why would they give it an open API until they put in a better camera?  I am not expecting anything one way or the other until the 3Gs comes out and people start using it.

But there are many things about the iphone 3.0 OS that are hugely important, like the discovery API that allows people to play games with other people nearby, that don’t have much to do with AR.

Tish: You have an iphone AR virtual pet application ARf.

Macrumors wrote it up and suggested that the neg gen iphone will have compass and open video API.  What are your plans for ARf?

Blair: ARf is just a demo right now.  I know what we’d like to do with it, but it would require tons of work;  imagine what it would take to do a multiplayer, social version of Nintendogs?  It’s not clear what we’d really learn by doing that, but there are lots of other game ideas we have that we want to explore.

Tish: I think it was on Twitter where Tim O’Reilly said, “saying everything must have a RFID tag is like saying we can’t recognize each other unless we wear name tags. Look at what’s happening with speech recognition, image recognition et.al. and tell me you really think we need embedded metadata.” What would you say to that?

Blair: I think that whatever extra data is there will be used. So if we put machine readable labels on some objects then they’ll be used if they make the identification and tracking problem easier. But it’s pretty clear that people are already working on tracking and so on.

A lot of these mobile AR apps are clearly putting ideas in people’s minds things that won’t really be doable in the near future. Like being able to look down the aisle of the store and it recognize all of the products. Given the distances and complexity of the scene, the number of pixels devoted to each of those objects, and so on – you just can’t recognize things in that context. But if I’m standing in front of a small set of objects, or looking at one thing, or I’m standing in front of a building, or if I’m in the store and because of the location API — imagine an enhanced location API that can tell me within a few feet where I am, and then combine that with some use of the discovery API that allows the store to tell your device you’re in the toothpaste section. Now you only have to look for different brands of toothpaste. So now you can recognize the big letters “Crest” or whatever. It’s all about constraining the problem.

That’s why I like that particular piece of Drummond’s work, the tracking web site I mentioned above. The general tracking problem of looking around and recognizing objects and tracking is still impossible. But if I know roughly what direction I’m looking in and I have a good estimate of my position, and I have models of what I should be seeing when I look in that direction, then it becomes a tractable problem. And so it’s not that a compass and a GPS are 100% necessary. But if you have them it certainly makes things possible that you wouldn’t otherwise be able to do.

Imagine for example  if there’s a new version of GPS, I just noticed that some of the new satellites going up have this new L5 channel. There’s the L1 & L2 signals  that the military and civilian ones use and they added this civilian L5 signal, which should make GPS more accurate. I haven’t found anything online that says how much more accurate.

But someday, hopefully, all GPS will get to be the quality of survey-grade GPS. Right now, if you get an RTK GPS from one of these companies that make the survey grade GPS systems, they give you position estimates in the range of two centimeters, and update 10 to 20 times a second. When you have that kind of positional accuracy combined with the kind of orientational accuracy you get from the orientation sensors we use in the lab from Intersense and MotionNode, everything is easier because you’ve pretty much got absolute position. You put that into a phone and now when I look up, it’s still not perfectly aligned because there will still be errors (especially in orientation, since the compasses are affected by metal and other magnetic noise). But it does mean if you and I are standing 5 feet apart from each other and look at each other, I can pretty much put a little smiley face above your head. Whereas now, with GPS, if I look at you and we’re 5 feet apart our GPS’s might think we’re on the opposite side of each other because they’re only accurate to two to five meters.

And that depending on the time of day and weather!

Putting RFID tags everywhere is easy; the problem is the readers – they currently require lots of power and they have a limited range.  Sprinkling RFID tags everywhere is fine. But you have to be able to activate those tags and read back the signal.  In certain contexts it works.

Tish: And one final question!  What do you think can be done re beginning to think about standards for AR.  Is there a meaningful discussion going on yet? Thomas Wrobel left this comment on my blog rcently and I was wondering what your position was on some of the ideas he raises?

Wrobel wrote, “The AR has to come to the users, they cant keep needing to download unique bits of software for every bit of content! We need an AR Browsing standard that lets users log into an out of channels (like IRC) and toggle them as layers on their visual view (like Photoshop).Channels need to be public or private, hosted online (making them shared spaces) or offline (private spaces). They need to be able to be both open (chat channel) or closed (city map channel) as needed. Created by anyone anywhere. Really IRC itself provides a great starting point. Most data doesn’t need to be persistent, after all. I look forward too seeing the world though new eyes.I only hope I will be toggling layers rather then alt+tabbing and only seeing one “reality addition” at a time.”

Blair: I agree with him, in principle.  But, I’m not sure there’s a point yet.  It can’t hurt to try, of course, from a research perspective, and I’m interested in the experience such an infrastructure would enable (as we’ve talked about already).

Recently, I met Kyle Gomboy, a former aerospace test engineer turned entrepreneur (pictured above working in Project Manhattan OpenSim), at a realXtend open house in Second Life. I quickly realized that this little bot (his avatar in Second Life is a cute energy efficient robot called G2 Proto) was one of the power houses behind a big vision.

Kyle, with partner Robin Gomboy (the G of G2), is now working with Microsoft, OpenSim, realXtend and the community of over 800 TechNet and MSDN/.NET that has already come together in Second Life to establish:

a technology grid for companies to come together with community members to form a high tech silicon valley virtually

The Microsoft connection, of course, really intrigued me! IBMers have been a movers and shakers in OpenSim since the early days. But Microsoft had not to my knowledge shown any interest in joining the immersive 3D party.

But within hours of meeting Kyle I had an interview set up with Zain Naboulsi (C# Writer in Second Life), a Microsoft Developer Evangelist who is the engine on the Microsoft side for this “community effort.”  Zain pointed out to me that despite his title, he is known as “the guy for Virtual World Evangelism” at Microsoft. I asked Zain how many virtual world evangelists there are at Microsoft?

Ha good question! There is probably about four or five of us right now.

Zain described his role in Microsoft and how this relates to this community effort in OpenSim?

I am relatively young in Microsoft I just celebrated my first year anniversary so I can’t talk about what happened several years ago, all I know is what happened over the year that I have been there.

My job description when I started this absolutely didn’t include anything about 3d worlds or anything of the kind. I became interested in 3d worlds and I think it speaks volumes for Microsoft that they said alright fine lets make it your job. And, that’s exactly what they did and likewise with other folks it is on their commitment now. Our strategy is straight forward, if you you can prove that your efforts in the community work then you are given carte blanche. If you can’t prove it then why be given the resources to waste.

My friend Ben Lindquist, of Green Phosphor, who is the only developer I know that develops in all the major open source virtual worlds, sent me a link pointing out that Microsoft has been interested in virtual worlds for some time.  But I asked Zain about this notion I have that Microsoft has been hiding its light under a bushel re the free form 3D immersive space. Zain responded:

I think we want to be careful. I mean just because we didn’t show up to the party doesn’t necessarily mean we didn’t like the party. Like most very large companies we wanted to make sure it was a viable type of thing. I think what you are seeing now is the beginning of the emergence of that experimentation and that is validation of the fact that we are definitely interested in this space.

As both Kyle and Zain were very clear that this is an open source community project, with no expectations of paid Microsoft developers being involved, at least in the short term, I was very interested as to how Kyle and Zain saw the big question of how such community driven Open Source development will reward people for their efforts (also see Second Thoughts for an alternate position to my own Open Source advocacy).

The full transcript of my interview with Kyle and Zain follows here. But, I will highlight one of Kyle’s responses. He is an experienced open source developer and entrepreneur and this is the answer he gave to this billion dollar question.

That’s a great question and since I got involved with communities and Open Source it has been a tough one to figure out. But I came to grips with when I found out that you could create this cool java script and that anyone could open new source and take what you did and that annoyed me. And I remember thinking many many years ago, how am I going to get anywhere if everything unique I make people can just take?

But then I began to thinking I can do that to. I can look at their code and then we can move up the whole technology the whole effort together, we’ll move forward faster and that will benefit us all because we will be able to send twice the product to market that we could have if we all stayed in our own little cubicles.

As far as the Manhattan project goes anything we develop there goes back to open source immediately and once it is embedded and tested it goes right back in cos OpenSim is open source and that is the ideology it was started with and we are not going to interrupt that at all.

Of course turning back code to an open source community is not always a straightforward process as a recent debate about the integration of realXtend code in OpenSim sparked off by Justin Clark-Casey and Dusan Writer indicates. But another good friend – an astute virtual world developer/evangelist, Peter Quirk, made an important point re the idea that realXtend “may not be giving back what it takes.” He noted too this must also be viewed from the perspective of “how hard it is to merge source trees that are changing rapidly without any stable releases.”

As I have heard on the grapevine great efforts are being made by both OpenSim and realXtend to work out an approach to integration. I cannot say much specifically at this point. But as Dusan Writer notes, I am an “OS promoter” and as such I think the fact that OpenSim, at such an early stage in its development, is being faced with the challenge of integrating such large contributions as realXtend’s is not just about “cracks in the open.”  This is just as much a question of how to deal with an abundance of riches in an unstructured and rapidly expanding community. The arrival of another important new community of developers from the Microsoft .NET and TECHNET will of course bring more riches and challenges too.

Interview with Zain Naboulsi (MIcrosoft) and Kyle Gomboy (G2)

Tish: Are you going to get involved with the OpenSim community?

Kyle: Yeah, absolutely. I’ve been trying to catch Adam [Frisby]. He’s been chatting with me through Facebook. The first thing we want to do is start working with him and Microsoft’s Zain has offered to get support going for some of the packet handling issues that Adam was talking about that might be limiting how many people could be in OpenSim. So Adam was looking to run some code by Microsoft, and Zain was saying that he definitely had some volunteers to help out. We definitely plan to get involved.

Tish: I was interested to know how deep you were going to get in terms of OpenSim development?

Zain: Kyle you want to start and then I’ll take it?

Kyle: I had thrown out to Tish that I had already started discussion with Adam, and first thing he mentioned was some issues with performance that he was hoping if Microsoft got involved they could contribute. Zain had already asked to start working with codeplex and getting some of the files up on there so we could swarm in the .net community and possibly some Microsoft help as we get them more educated on the OpenSim movement.

Zain: Absolutely. You asked how far we’re going to go with this? We’re going to go as far as it lets us go. We honestly believe we’re forging the future here so it needs to go as far as people want to take it.

Tish: Have you thought about where you want to be interoperable with other Microsoft projects, XBox even eventually, Virtual Earth? Have you gone that far or are you just going a step at a time at the minute?

Zain: That’s a great question. Honestly we haven’t thought beyond seeing if people are really interested in this concept and from what Kyle tells me a lot of uptake. And, I know we’ve started some basic experiments, nothing really major, to see. But certainly it’s not inconceivable. I think what it is though, you mentioned the Windows aspect of it, that’s great. But we’ve already proven, if you just look around here, it doesn’t really matter to us that much what the platform is. I think the main reason we’re going to OpenSim is not so much because it all runs on Microsoft stuff which is just a nice bonus, but because I really feel like (and I just love the Lindens by the way so let me get that out!), but I really feel like we’re kind of restricted here [in Second Life]. I think that we can have a lot more flexibility by going the OpenSim route, and working with folks like Adam who I haven’t met personally but Kyle tells me he’s just phenomenal. And everybody Kyle’s met has been cool and talking about all these great things we can do with OpenSim that I don’t think Second Life’s going to give us the ability to do this anytime soon.

Tish: I know some groups are involved in the OpenSim core, are you aiming to be that involved or are you just not sure yet?

Zain: Honestly, this is the very early stages now. [Project] Manhattan is the first real test to see if first of all people respond, secondly if it’s something we want to stay involved in. All indicators are right now that it’s phenomenal. Third, I think the big thing is will this resonate with the community? The main reason we’re in this game right now is about the community. That’s why we built everything you see around you, and that’s why we want to do what we want to do with OpenSim. Then if it benefits the commmunity and the community responds to it, then we’ll stay with it. If the community doesn’t like it, doesn’t think it’s a good idea, then obviously we’ll get out. But from what Kyle has been telling me the community has just had phenominal response to the Manhattan project.

Kyle; Yes, I am pretty much lost to OpenSim at the moment because of all the options and the flexibility and the code familiarity. And everyone I’ve talked to who has gotten their own sim set up, every single person who is getting their ports configured to hook up to our grid agrees, and it doesn’t seem to be a contest even with the quirks here and there, and the lack of some features. There is so much ability – the scripting the c#sharping is everything that we had been talking about needing in SL and the desk top sharing that realXtend is doing. It is exactly the work we want to get involved in. I don’t see myself developing in Second Life, I see myself attaching the work I do in Second Life to OpenSim and everyone else is joining in so far too.

Zain: Let me clarify though too. While OpenSim will be the R&D and experimentation side of what we do, by no stretch of the imagination are we even considering abandoning our Second Life presence. So this sim [Microsoft sim in second Life] isn’t going anywhere. Community is what drives us so now we will have the community folks come here for some things and go there [OpenSim] for other things.

Tish: Just to be absolutely clear, when you say community, you mean your .net and C# developers community right?

Zain: Yes, and the technet folks, all of them, any of the community that wants to come here. But obviously the Microsoft focus people in general, so yes .Net and TechNet.

Kyle: yes we have almost 800 total in Second Life. And Zain is right noone’s leaving Second Life. But this is a community based thing and the biggest number is our c# developers. And what c# developers want to do is get into something and tinker and the top community leaders dove right into OpenSim and have seen past the quirks. So I think it will turn into a socializing place here on the islands [project Manhattan] but I think in the next year you will see a lot of .net innovation coming out of the group that was just meeting here in Second Life.

Zain: Yes I really agree with that. This is a playground now for them but I think it will evolve into something more than that.

Tish: Your focus is really on your developer community isn’t it?

Zain: Yes and I think that we may be the only project from a large company that is hundred percent community driven. I don’t go to outside folks and ask them to build this stuff. The community does it, the community decides on it, the community drives it. That’s the biggest deal. I am here to help the community in any way I can.

Tish: When you say that, it’s not just about people who are interested in Immersive 3D development, it’s people who are working many Microsoft areas right?

Zain: Absolutely. My job as Evangalist is fairly broad in nature. Today I just got done doing a webcast on new features in AJAX. I cover a very broad spectrum of things, but without a doubt an enormous amount of my time is spent with virtual world evangelism. As you aptly pointed out people are getting more and more interested in this stuff so I find more and more of my time is spent there.

Tish: I suppose future directions depend on what the developers pick up on in OpenSim doesn’t it to some degree? Do you have people who have started doing any projects in OpenSim yet?

Zain: I don’t focus internally, yes, Microsoft folks are doing stuff in OpenSim a variety of projects. But my main goal is to get out and encourage the community to get involved in OpenSim. If we can’t get the community engaged and involved in this then ultimately we’re doomed to fail because the community has to want this. The community has to pick this up as early adopters or we’ll never get to the point of mainstream acceptance.

Tish: I know Kyle is interested in something I’m really interested in which is seeing OpenSim as something much more than a 3d virtual world. Some people see OpenSim as something potentially more like a next generation operating system with really far reaching potential for dynamic, interactive data visualization and virtual operations centers and applications like that. I know Kyle you’re interested so perhaps I can ask you about that?

Kyle: I’m one hundred per cent interested in that. You might be able to send an engineer a drawing and then pop in a little active-X control and they can read about the drawing and then just click and have a view of OpenSim and move the camera around and inspect it in 3d. I see more as another output device like an excel graph or something like that.

Zain: I’ve got to agree with Kyle. I think in the short term the only way you’re going to initially get adoption is, to use a metaphor, if we think of the client as the next generation of the web browser. I think by going that path you’re going to get a lot more folks to want to try it. But ultimately certainly it’s entirely possible it becomes the shell or something like it becomes a shell. It might become, not the entire desktop which is probably unlikely, but a major component of the desktop integrated with it just like you get pieces like IM and things like that today. I don’t know if you’ve seen TouchWall yet. One of the technologies we’ve come out with. It’s like surface except it’s a big wall you can use with your interface. And, I can see TouchWall combined with the client we have today, and doing some pretty cool things especially if we can do what Kyle’s talking about – bring in some Exel spreadsheets – basically bringing in the ability to do office type activities into these clients.

Kyle: I don’t mean to say that I rule that out as a possibility, I do think that’s a possibility. I just look at what can I do in the next one or two years with this because whatever I start now that’s about as far as I can really look because the whole game will be changed in three years.

Zain: I totally agree with Kyle on that.

Tish: As far as I can see Intel might have a key part to play now, because I think that the graphics cards are a limiting factor here, particularly for the applications you are talking about Kyle?

Kyle: Yes. The video requirement is pretty steep and not only that I need probably 4 times the resolution that OpenSim has now maybe a thousand by a thousand grid before I can do any real high resolution data visualization. There needs to definitely be some expansion but if you pick the right project and the right application, it’s perfect for a lot of things right now. The other thing that Adam [Frisby] mentioned was some of the threading issues. That would help with concurrency where we can have more than 30 people or so. Even though I’ve read some hidden blogs here and there that someone’s working on a mod that lets you do over a hundred people in a sim. That would help a lot. But you’re right Intel getting involved could mean some help with some of the threading issues.

Tish: I’m really interested when you even start to mention taking this technology into other user interfaces. There’s just something wrong with being stuck in a little 2d window when you have all this 3d power. Anything that gets you away from pushing around a little mouse in a tiny window, that’s where it begins to rock.

Zain: I was in this game place where they had video arcade games and there was this one where they had these motion detectors and you play this karate style game. and you kick and the avatar does what you do. And in the end that’s where all this technology’s headed. You won’t have bulky crap attached to your hands and body like we see now, but you’ll have motion detectors that are finely tuned so that you can move and manipulate things in the virtual world.

I think Phase 1, if we’re really going to get wide adoption of OpenSim, is to treat OpenSim like the conference call center of the future. For the big events that’s where you start getting the budget and interest from internal Microsoft. I think that naturally leads to whole lot of Microsoft folks becoming much more heavily involved.

Tish: I’m assuming at the moment, your OpenSim community just has separate islands. You haven’t gridded them, they’re not related in any way right?

Kyle: That’s right. I’ve only really had the sim up a little over a week now. The initial Manhattan sim. I’m just setting up some different group meeting places within that sim. For example, we’re working on Live-ID integration so you can login with your LiveID and a few other live services integration projects. So I have a designated area, and at that area we have a colored ball that’s a certain color that you know is a wiki for that area or API for that area. You click it and open up the services for that API. Also the inworld scripting area has a link to the approved LSL commands for OpenSim. So the first sim is resources for anything and everything to do with developing in Second Life and OpenSim because we will really be developing for both. I’ve got five or six sims from community members up and running and they are just waiting on me to fire up my grid server. We’ll even try and organize our grid according to different disciplines. There’ll be a .NET developers area with a bunch of islands having to do with .NET work. And then you’ll be able to fly to the TechNet area where there’ll be a windows server and SQL server. So there’s going to be a community grid but it’s going to be organized around the developer community.

Tish: Are you hosting sims for the community for free?

Kyle: Yeah, I’m going to have a few sims up for community experiments and things like that. In addition, since most of the community has at least a broadband connection at home, and a spare computer. Everybody’s been reformatting that old 3 gigahertz single processor computer in the closet and opening up ports, so I’m going to have a grid half made up of hightech companies that want to do their own experiments and the other section of the grid will be community members that just pop in with their own simulator.

Tish: Very nice. So you’re basically showing people how to do a very quick community grid aren’t you?

Kyle: Absolutely. And then any company that has some sort of experiment going on can connect to the grid too. I’m hoping that if Intel develops a grid that they’ll at least teleport link to us or something like that so that we can start out as a community and expand out to Silicon Valley or something like that where we have gateways to all kinds of different experiments. So I’ll let Second Life be the social networking thing and then I’ll have all the Star Trek type of projects going on in OpenSim.

Tish: I love your approach – building a grid based on developer communities – people do need to be very geeky to handle OpenSim development at this level at the minute.

Kyle: Exactly. And you happen to be talking to two developers. Our job is to try and match the technology with the problem. Right now the people most likely to embrace virtual worlds and use it is the technical crowd. That’s just the way the internet came to be and all that. So put it in their hands first, get all the really hyper geeky stuff going, and let it evolve from there down to the regular person. One of the things that is in Manhattan sim now is a meeting place for a standards committee. I want to start right away setting up common standards for how navigation and walkways are done, even making accessibility possible for blind people one day where we use sound to guide you along. And we make sure like on web sites we have the proper alt. text in so audio readers can translate what’s going on. The first step is testing it, putting some standards in so anybody can go into any type of sim and can know how to get around. The we’ll really be able to bring this to the common person who can just pop in there like they can a web page now.

Tish: A question for Zain. I have this notion that Microsoft has a reputation for not liking the free form 3D immersive model?

Zain: I think we want to be careful. I mean just because we didn’t show up to the party doesn’t necessarily mean we didn’t like the party. Like most very large companies we wanted to make sure it was a viable type of thing. I think what you are seeing now is the beginning of the emergence of that experimentation and that is validation of the fact that we are definitely interested in this space.

Tish: A big question for many people looking at OpenSource development is how will people be rewarded for their efforts. You see this as a community development project – how will the members of the community be rewarded. This is a question for Kyle I think!

Kyle: That’s a great question and since I got involved with communities and Open Source it has been a tough one to figure out. But I came to grips with when I found out that you could create this cool java script and that anyone could open new source and take what you did and that annoyed me. And I remember thinking many many years ago, how am I going to get anywhere if everything unique I make people can just take? But then I began to thinking I can do that to. I can look at their code and then we can move up the whole technology the whole effort together, we’ll move forward faster and that will benefit us all because we will be able to send twice the product to market that we could have if we all stayed in our own little cubicles. As far as the Manhattan project goes anything we develop there goes back to open source immediately and once it is embedded and tested it goes right back in cos OpenSim is open source and that is the ideology it was started with and we are not going to interrupt that at all.

Tish: Are you going to become involved with the interoperability efforts like the one launched by Linden Lab – The Architecture Working Group that is making efforts re: the interoperability of OpenSim and Second Life, in particular?

Kyle: It is projects like the LIveID integration that we are working on and many people are interested in OpenID. There may be a lot of third party type of log ins that handle authentication. So you may see that even Linden Lab uses multiple methods for people to authenticate into their grid. And I think that this new group that they have started [AWG] is just proof that they know that they must open up and embrace these other grids and get ahead of the game so that they can work with these other grids right away.

Zain: And as far as standards go i think we have a ways to go before we start getting down to the hard core standards path. First we have to establish that this medium is going to be viable to the market in general. And then start bringing it together much like we did with the web. The web took off and there was a governing body that evolved into the world wide web consortium and I think that eventually you will see that with this who knows maybe even W3C will take over aspects of standards for this medium.

Tish: A question for Zain – how do you describe the community development in OpenSim and how does this differ or follow on from other Microsoft involvement in OpenSource communities? Oh and how do you describe your roles what you do for Microsoft and how this relates to this community effort in OpenSim?

Zain: From my personal experience, I am relatively young in Microsoft I just celebrated my first year anniversary so I can’t talk about what happened several years ago, all I know is what happened over the year that I have been there. But without a doubt I think our approach is simple straight forward and elegant. And as a developer evangelist I am hired to interact with the community. My job description when i started this absolutely didn’t include anything about 3d worlds or anything of the kind. I became interested in 3d worlds and I think it speaks volumes for Microsoft that they said alright fine lets make it your job. And, that’s exactly what did they did and likewise with other folks it is on their commitment now. Our strategy is straight forward if you you can prove that your efforts in the community work then you are given carte blanche. If you can’t prove it then why be given the resources to waste.

Tish: So you actually have the title Metaverse Evangelist?

Zain: No I am still a Developer Evangelist but I am the guy for Virtual World Evangelism which is what we call it internally at Microsoft and I guess externally as well. It is known as Virtual World Evangelism.

Tish: So how many Virtual World Evangelists are there at Microsoft?

Zain: Ha good question! There is probably about four or five of us right now.

Tish: So have I missed asking you any important questions!

Zain: No I think you have hit the big points! If nothing else the one key point I would walk away with from this is the emphasis in my virtual worlds evangelism on the community. That’s really what we are about and that is why we are embracing folks like Adam [Frisby] and the realXtend guys because we want to come in and get more community involvement and get people excited about this. We are not interested in anything but really fostering a good environment for making this stuff happen.

Tribal Net, which goes into public Beta at the end of this week, not only brings us the long awaited OpenSim on your PC but by creating new back end protocols for OpenSim the Tribal Media team has introduced a key innovation to OpenSim – the decentralized grid. I interviewed the two founders of Tribal Media last week, Darren Guard who is also the founder of OpenSim and Stefan Andersson who was the first to join Darren on the OpenSim project. Stefan also led the development of “Playahead Island” in Second Life â„¢ (A registered Trademark of Linden Lab). Stefan has deep roots in web development and Playahead is one of Sweden’s largest web communities. Stefan explained the heart of the Tribal Net concept:

Everybody brings their own computing power [to Tribal Net] and we’ve packaged it for end users. I mean that basically joe schmoe can install it, set it up, and run it. Because OpenSim’s been very tech heavy, our goal with Tribal Net is to make Opensim more accessible for the wider layers so to speak. Also we’ve done some work on the map so that now when people go online their regions show up on the map. When they go off line the region disappears [it can also be persistent]. That is also a radically different approach from Second Life .

Yes! It is very easy to install and you will find your region immediately embedded in a network of other regions running on other PCs.

After I had set up my Ugotrade region, I clicked on map and it was a real thrill to see the Ugotrade come up in a neighborhood of other regions (even though I am apparently one of the first five people to try it outside of Tribal!) and immediately begin my first adventure in sim hopping – NOT across a grid run on a bunch of servers NOT on some huge server farm somewhere, but simply by teleporting to other peoples regions run on their own PC’s located across the globe. W00t!

Ugotrade Jr. got into it immediately and his terraforming and building skills are blossoming! Picture below shows his mountain retreat 2.0.

Tribal is still using the Second Life client but Darren said they hope to support the realXtend client at some point. Check out the awesome new avatar technology from realXtend in a new video out this week that shows off character morph controls, inverse kinematics, and clothing physics.

Below is a picture of Stefan And Darren enjoying a game of chess in one of the first regions I visited – a Tribal Net region running on Stefan’s laptop. Stefan explained:

The chess game application installed on the desktop is developed in c# by a third party – and anybody can create their own set of pieces and share it as an xml file

Tribal Media’s Vision

Tribal Net is the first public application from Tribal Media but it is only the beginning of their venture. Stefan explained their vision:

We see a virtual web world much more like the Web works today. Instead of a closed bubble, we should have an open, networked model. One size never fits all, so people should be able to make content on their own computers and share it with others, professionals should be able to make their own applications and run on their own servers, or to have it hosted on reliable hardware. Content should also be much more moveable, people should be able to transfer objects not only between worlds, but also via the web, blogs or e-mails. Tribal Net lets them do that. Our goal is to supply the tools to make this vision come true.

And as Darren noted: “At the moment our back end is basically customized for this one application. Each new application is going to need slightly different customization. Stefan also talked about a Facebook application they have been working on. There is not a general solution at the moment. Darren explained some of what they are hoping to achieve with Tribal Net.

From a engineering side, I think the main point is at the moment we are working on making it easier for people to start up and host their own region. TribalNet is our first demo of that process. Then we hope to make it easier for people to host their own small grids with this easy hosting of regions, so say any school or college could have their own small grid without the admin level that is needed currently for opensim. A important part of our concept is having a GUI for the regions, so that we can later provide add on modules for these GUI’s so for instant maybe we would provide a game construction toolset add on, or a presentation addon , that made it easier to host and control presentations. Some of these then at a later time could move into the viewer. Then at a higher level we have our set of extension api’s which I think its a bit too early to go into detail of.

What follows is the full transcript of my interview with Darren and Stefan. The first bit is about the history of the OpenSim project and then we discuss a number of topics including interoperability with Second Life, the Tribal teams’ view on virtual economies, asset development on Open Source grids, and what application they are most important in Tribal’s vision.

Interview with Darren Guard and Stefan Andersson.

Tish: How did OpenSim begin?

Darren Guard: For a number of years, I had thought about starting or getting involved with a open source virtual world project. But none of them seemed to be going anywhere. One of the problems was trying to create a client and server at the same time. Then in January 2007, I was looking around again at the various options, for use by my ex-employer. At the same time Linden Labs released the code for their client. Which I think from a legal point of view made creating a server that it could connect to a much more easy “sell”. So I started work on writing a very quick prototype server, to see if there was any problems with getting the SL viewer to connect to it and be able to move around. A number of people had the same idea of writing a server which was compatible with the SL viewer, it was even listed in the roadmap on libsecondlife’s website.

So, as much as we all moan about the SL client and wished there was something at least more generic, we do owe opensim’s existence to them releasing the source code. As I just wouldn’t have started Opensim if the
client was a closed proprietary one.

My initial thoughts on what I wanted to use the platform for, were very much on what would be useful in my old job. Simulation (Robotics) and 3d visualisation of data. This is why one of the stated goals of opensim is
it being a 3d application platform and not a SL clone. I see the social grids as just one small subset of the applications that a 3d virtual environment platform could be used for.

This is one of the biggest points of confusion when it comes to opensim. Most people think its goal is just to clone SL and behave exactly how that does, and support all the features out of the box. And we have to repeatedly tell them that its not the goal and that opensim will most likely never have all the features of SL as part of the main project. Other people will have to create the modules to add those features.

The goal of Opensim and Tribalmedia is to produce a common server that can be used as the base of a lot of different applications. I don’t think we can even really start guessing at what the most sucessful applications
will be in the future.

Tish: How did you meet Darren, Stefan?

Stefan Andersson: I’ve been working quite a lot with web, I mean I was quite early in the web development and I’ve been doing that for quite a lot of years. And, I was very interested in web and 3D integration. That’s actually how I stumbled upon Darren, when he first came on-line in lib second life chat room, and said hey I’ve got a prototypic open source second life server here.

Tish: Was the big moment in February 2007, Darren?

Darren: It was January.

Stefan: And I jumped on it. The first thing I did was to tweak his code, and give him some patches so I could connect his Second Life server to my then employers user database. The Employer at that time was Playahead. It’s one of Sweden’s largest web communities. So basically what I did was within an hour of getting my hands on the zip file, Darren’s first unpublished zip file, was to make sure that I could log into the SL viewer with my web community name and my web community password. When I came into a personalized world where all my friends and my friend lists were avatars, and when I chatted with them and I got guestbook messages into their guestbooks, that’s when I sat there and said WOW!

Tish: Not everyone recognised the potential then?

Stefan: I think they still don’t. You ask what’s important to us. We are very application oriented. We are very integration oriented. At the moment OpenSim is not very accessible and understandable for a large audience. We’re trying to show the world that there can be commercial application on this not just social networking applications, but actual 3D application. It’s a bit like the web, everybody thought that the web was about static HTML pages, and then now today we do much of our daily work on HTML. That’s kinda like the grand big hand motions. What we did when we started OpenSim, we had a very clear agenda, that we wanted this to be a shared experience 3D application platform.

Tish: When you say application, are you thinking more vertical applications or are you thinking vertical or horizontal on these distributed grids?

Darren: We don’t see one big monolithic grid, we sort of see, like Stefan said, like a web where you can link from one web site to another. You wouldn’t really say that every web site is part of one grid. When we talk about applications, its custom, i.e., you might go into one application to do one function then hop to do something else.

Stefan: So for example, now we have actually done Tribal Net as a showcase for our product Tribal server. And we did another showcase that we haven’t publicized yet, but basically that was web community integration.

Tish: What do you mean by web community integration?

Darren: That was a Facebook integration.

Tish: There are 2 basic models it seems being tried, a) embedding Opensims within the web or b) embedding the web in Opensims. Which model do you lean towars, or do you see a sort of heterogeneous mix of embedding in web pages and people who grid OpenSim out into larger communities and embed web pages in them?

Stefan: Actually that will probably be very much a per application decision. Some application are very suitable for 2 dimensional presentation and some application are very well suited for 3 dimensional. and we’ve done some prototypes with integrating web and 3D and obviously there’s going to be a lot of that coming.

Tish: How do you differentiate yourself from say an initiative like RealXtend?

Stefan: RealXtend? They are trying to do quite a lot of things. We are not competing with RealXtend. We would probably use Real Xtend.

Darren: Yes. From what I understand of Real Xtend, they’re focusing on the client. They have a central avatar system. I don’t really have that much knowledge of that. The main focus that we spoke about is on the client side. So we’re not really competition for them, we hope to support their client sometime.

Tish: I have heard on the grapevine you are using dynamic sims. What is a dynamic sim?

Stefan: Dynamic Sim! That can mean anything! We have quite a lot of concepts.

Darren: I think what they’re talking about is we’ve got this concept where we can bring a region up quickly when it’s needed so when you login we can bring your region up so that you wouldn’t even know it wasn’t up all the time. If there’s nobody there we can take it down.

Tish:
How have you done that?

Stefan: If you think about how Second Life works right now, how everybody seems to think about that, It’s kind of like a static model, the thing is there, whether anybody wants it or not. So you have like thousands of regions producing air. It’s a terrible waste of CPU. We wanted to do something like a web page, a dynamic web page. It’s constructed when you need it.

Tish: Are other people using this concept?

Darren: I think Adam has done something like it I’m not sure what exactly. But from what I hear Adam has similar ideas.

Stefan: That’s the thing about dynamic regions, It’s a feature of our tribal worlds platform. We used that as a proof of concept in that Facebook application where anybody could add a Facebook application and then they just went into their own private region, which was constructed on the fly for them. And when the last person leaves the room, he just turns the lights off.

Tish: And what kind of concurrencies can you get in these dynamic regions though?

Darren: I don’t think we’ve actually pushed it, but we can certainly get more because it’s basically where Second Life is based on total regions and they have to have a server up for each region, we only have to have a server or an instance of a region up for every person who wants to let a region have somebody in it. So if there’s nobody in a region we just don’t have it up.

Tish: I know everyone’s talking about putting OpenSim in clouds. Is this going to be workable with that idea too? Just in terms of being more efficient about the server side of this. Is that something that works with this or are you another direction than that?

Darren: I think it can work yes. It’s complimentary to our project.

Stefan: What we did with Tribal Net – It’s kind of like OS grid. Everybody brings their own computing power and what we’ve done is that we’ve packaged it for end users. I mean so that basically joe schmoe can install it, set it up, and run it. Because OpenSim’s been very tech heavy, our goal with Tribalnet is to make OpenSim more accessible for the wider layers so to speak. Also we’ve done some work on the map so that now when people go online their regions show up on the map. When they go off line the region disappears. That is also a radically different approach from Second Life.

Darren: To clarify that a little bit. What he means is that instead of a region having a set position on the map, we have a center of the map and the regions brought online are thrust around the center and if a region goes off it’ll be replaced by a new region that comes up.

Tish: What it’s sounding like when you describe this is you’re trying to use some of the ideas of P2P in a distributed grid.

Stefan: Yes definitely. I used to say Hey look Microsoft messenger, it should be like that. I know Darren isn’t that fond of that. But I think of it like that.

Tish: The development curve seems to have been rather slow thus far in P2P virtual worlds like Croquet and Solipsis. How do you see a lot of rich and interesting assets being built up on this kind of distributed grid

Darren: I think it’s too early for even us to say. Yes we’re more sort of a model of decentralized rather than a big monolithic grid like Second Life. One problem with Second Life is all the assets are centralized. That makes you responsible for making sure any users are updated with current textures etc. And you’ve got the problem of trying to police that to make sure there’s no textures that you don’t want there.

Stefan: We’re trying to move to P2P because that’s the only viable solution if we’re going to see web scalability. It just is. And we can’t really have stuff like central avatar repositories and things like that. We have to have a base case, which is the single server, and the single client. And then just have to grow from there. But what I wanted to say is that Darren made a brilliant choice way back when he was pondering what he would do. That was to take something that had proven itself on the market and to the user base. That is the Second Life client. What Darren did was that he combined that with another immensly popular and available technology. And that is .NET (dotnet). Basing this off .NET made it reasonably available to a community of programmers.

Tish: Second Life is the treasure chest of assets at the moment and also great content developers do not want to have to develop different content with different tools for tons of different VWs. What’s your stance on encouraging good content developers in Tribal Net? Are you aiming to be potentially interoperable with Second Life? Are you part of the Linden Lab Architectural Working Group initiative? Or are you going to try and go it alone?

Darren: First of all I need you to clarify the question. Do you mean like some people want to have OpenSim regions that are part of the Second Life grid like IBM, is that what you mean?

Tish: Oh I’m almost assuming from what you have said so far that you’re not going to do that. But are you going to aim for some level of interoperability?

Darren: I’m part of a working group that’s trying to get a common client protocol. We’ve got a sister project called Open Viewer. It’s started a few weeks ago but we’re attempting to incorporate elements of many clients like Croquet in an effort to achieve some degree of universality. Peter Fin from I.B.M. do you know him? He arranged it. We have a common protocol that you might not be able to do every feature in well. But you can at least connect, see the world and move around. We are also part of the Architectural Working Group but the focus is on Second Life there.

Stefan: If you look at what the architecture group has come up with so far, it’s Linden Lab’s protocol version 2 or not even that, it’s Second Life 1.5 – a big wish list basically. What we have, I guess, you could say that we have the small company rogue – a bit decentralized slash anarchistic – approach. We have a saying in OpenSim, I don’t know if you have seen it. I think actually I coined it but it’s “let a thousand worlds bloom.”

What we mean is that it’s too early to start drafting universal protocols. So what we’re doing with OpenSim is, and we’ve been very clear about this often repeating it over and over and over again, not to try to build a free and open source Second Life. We’re trying to build a platform, (we are not trying to make THE protocol) so that people who want to make protocols, should be able to do that with less effort. So when we’ve had our taste of applications, social applications, business applications, marketing applications, everything, then somewhere in there we can see probably something like http.

In Tribal net we have our own backend protocols. We have changed large chunks of the communication stack at the backend, the regions you install on your pc – some parts we have changed because this application needs other data and other processes. So I think we have implemented four different stacks. So we talk from experience.

Tish: Will you be publishing all your protocols?

Stefan: It is way too early to go into that. Right now we do not know what will bring everything forward. This is just one application now.

Darren: At the moment our backend is basically customized for this one application. Each new application is going to need slightly different customization. There is not a general solution at the moment.

Tish: So if it is all in the application, what are the killer apps?!

Stefan: Well we have done integration with web communities and 3D integration, that in combination with marketing applications, for example, being able to go on to say a Toyota site and click on a link and be on a Toyota showroom, not necessarily in the browser like everybody is visualizing because we have seen problems with that. We need something a bit more intelligent. The interplay between 2D and 3D is very intricate.

Tish: Everyone asks me about when they will be able to use OpenSim to create content they can upload into Second Life?

Darren: The big problem with that is SL terms and conditions, you aren’t actually allowed to export your creations out of SL. Now while it would be possible to import creations from OpenSim into SL, really until Linden Labs allows creations to be exported, there isn’t really the reason for people to work too much on adding those features to opensim. As only Linden labs really gains, as its one way traffic

Tish: Is there anything else notable about Tribal Net that hasn’t come up yet?

Darren: From a engineering side, I think the main point is at the moment we are working on making it easier for people to start up and host their own region. TribalNet is our first demo of that process. Then we hope to make it easier for people to host their own small grids with this easy hosting of regions, so say any school or college could have their own small grid without the admin lvel that is needed currently for opensim. A important part of our concept is having a GUI for the regions, so that we can later provide addon modules for these GUI’s so for instant maybe we would provide a game construction toolset addon, or a presentation addon , that made it easier to host and control presentations. Some of these then at a later time could move into the viewer. Then at a higher level we have our set of extenstion api’s which I think its a bit too early to go into detail of.

Tish: My big question is still without a virtual economy what do you see driving rich content production?

Stefan Andersson: Yeah; we’re all about content, actually; Tribal Net is about creating a content producer platform. A complete ‘ladder’ from consumer, over enthusiast, semi-pro and pro. Yeah; basically, we offer empowerment. It’s like ‘your world’ but for real.

I guess you know that content production and systems integration in a third-party hosted environment is a drag.

How many complex games and functions have you seen in SL? Stuff that would be a small thing to code if you had proper tools, becomes a nightmare.

But to me also it seems your definition of ‘content’ is close to the Linden notion of ‘content.’ The Lindens created an economy based on artificial scarcity you pay to stop somebody from sharing something but ‘content’ also often play a role in the execution of a service.

Just an example : you have a medieval battle sim with castles with all kind of nifty storytelling bound to them the busines model is subscription, perhaps; combined with added value like buying weapons well, the actual assets are for free – the service owner couldn’t care less if you walk away with the sword into another world because it only functions in a context, the context of the battle system.

So, yeah, you could have a static snapshot of a weapon or you could have the customizable weapon that actually functions in a system now, the former will be very hard to charge enough for to make up for the production cost but the latter you definitively could. charge enough for to make up for the production cost.

Tish: You have a very innovative idea for a distributed grid. Will people have problems with their firewalls though?

Stefan: That’s the big hurdle we’ve done everything we can to make everything else easy but the server still needs to be accessible from the net.

Now, next week we’re launching the next version, which has a built-in sandbox too so you can terraform and build without being public

I don’t know if you saw that in the gui, but we’re adding ‘load/save’ to it so that you can easily export terrain and objects.

Well the private mode is sharing content db with the public mode. so, basically, you just go public or, you save it to your hard disk and upload it onto a hosted region.

At the moment, it’s just “save all object definitions” and “load all object definitions” but even that’s enough to export objects, tweak them, re-import them that’s how the superprim was done and the chessboard pieces. The chessboard coder has access to objects and object definitions on a whole other level that the SL coder, for example notepad, basically. It’s like, text editing html pages all over again.

Tish: hmmm a very interesting concept…..

Stefan: well, it becomes even more interesting when you have a program creating those definitions.

Tish: Are you planning on Linux support?

Stefan: We’re planning on linux support. It’s just that these versions, that are very end-user oriented, windows is the main demography. Our commercial Tribal Server runs linux just fine. And, as I said, we’re planning for even the end-user versions to run on linux.

At this point in the interview, Stefan encouraged me to try Tribal Net myself! I was at first resistant. After all I already have an OpenSim up. But it was truly a revelation to so quickly get set up and find myself with a region (a full on free form 3D programmable space on a PC!!!) – a virtual world of my own that let me interact with my neighbors and yes, for me, Tribal Net was “an easy-to-install, easy-to-configure express version of the full Tribal Server” as the Tribal Net website proclaims!

Beta Phase

Currently, Tribal Net is in ‘Beta’ phase. They write:

This means we have made the software available to the public in order to get feedback and hunt down bugs. If you want to try the Beta out, or simply want to be notified when the 1.0 version is released, you can ‘register’ here.

See you soon in Tribal Net!!