Blair MacIntyre is one of the original pioneers of  augmented reality and an extraordinary amount of creative work is coming out of his Augmented Environments Laboratory at Georgia Tech – see YouTube videos here.  The screenshot below is from, ARhrrrr, a very impressive augmented reality shooter game created at Georgia Tech Augmented Environments Lab and Savannah College of Art and Design, (SCAD- Atlanta), and produced on the NVidia Tegra devkits – watch the demo here.

Blair has spent much of his career working on immersive augmented reality and more recently the integration of augmented reality with mirror worlds. Blair explains:

“I am interested in the intersection of mobile devices – whether they are head mounts or handhelds – and parallel mirror worlds…I think that parallel mirror worlds are a direct manifestation of the intersection of the virtual world we now live in (the web) and geotagging.  As more and more information is tied to place, and as more of our searching become place-based, we will want to do those searches about places we are not at.  A 3D mirror world may provide one interface to that data.  Want to plan your trip to London;  go their virtually and look around, see what is there (both physically and virtually), teleport between areas you want to learn about, and so on.  More interestingly, talk to people who are there now, and retrieve your location-based notes when you are on your trip.”

But, at a time when many augmented reality developers are focusing on AR apps for smart phones, including Blair (the picture on left opening this post is Blair’s augmented reality iphone app ARf), I was interested in finding out from Blair what the state of play was for the real deal Rainbow’s End style AR, as well as the potential he sees in smart phones to mediate meaningful AR experiences.

There is enormous amount of  innovation in mapping our world, see my post, “Location Becomes Oxygen at Where 2.0 and WhereCamp,” and  Ori Inbar’s  Where 2.0. conference roundup. But as Ori notes, to move augmented reality forward:

My point is not a shocker: all we need is to tap into this information and bring it, in context, into people’s field of view.

And this is what Blair MacIntyre’s work is all about.

Talking With Blair MacIntyre

Tish Shute: There do seem to be broader implications to augmented reality today than when this term was first coined. I am interested to have your perspective on how augmented reality may go beyond some of our early definitions?

Blair MacIntyre: I still think the original definition of the term is useful:  media (typically graphics) tightly registered (aligned) with the physical world, in real time.  Many people talk about many things that relate virtual worlds to places, spaces, objects and people.  There is room for many of them, and they don’t all have to “be” augmented reality.  I like using Milgram’s definition of Mixed Reality as everything from the physical world (at one end) to the virtual world at the other;  it’s a spectrum, and augmented reality just sits at one point.

The reason I like the old definition is I believe there is something special about graphics that are tightly, rigidly aligned with the physical world.  When things appear to stick to the world, and an obviously identifiable location, people can start leveraging their natural perceptual, physical and social abilities and interact with the mixed world as they do the physical world.  We’ve found this with the two studies we’ve done of tabletop AR games (Art of Defense and Bragfish);  one key to those games is that the graphics were tightly aligned with identifiable landmarks in the physical world (gameboard).

Art of Defense (pic on left) Bragfish (pic on right)

Tish: I know that you are involved with ISMAR 2009 which is the key US augmented reality conference.  What do you think will be the hot themes, applications, innovations at this year’s conference? Do you think this will be the year that AR really breaks out of eye candy into truly useful and sustained experiences?

Blair: Unfortunately, I won’t be involved this year.  I was supposed to be helping run the technical program, as well as the art/media program, but sickness in my family prevented me from having the time, so I am not helping this year.

First, I would not agree with the implication of the last question — I don’t think AR has just been eye candy up to now.  I do agree that the “high profile” uses of it have largely been that, which is mostly because of the limits of the technology.  I don’t think we’ll see huge changes in that regard by ISMAR this year.  However, we will hopefully see a mixing of communities that hasn’t happened at ISMAR before, and I do believe that this year (independent of ISMAR) we will see more and more AR apps.  Whether they go beyond eye candy is still a question.  I’m hoping that some folks (including myself and other ISMAR folks!) will help push AR in new directions.  But I also expect many folks new to ISMAR and AR to play a big role, because it is this new blood, especially those folks with real problems to solve, new art and game ideas, and a fresh perspective, that will open new doors.

Tish: You have been working on integrating augmented reality with virtual worlds. You mentioned that the way you use Sun’s Wonderland is really about pulling the virtual world into the real world, i.e., Wonderland, “is just a place to put data.”  How is your use of the persistent virtual space different from what we have become accustomed to call virtual worlds?

Blair: The approach we are taking in our project at Georgia Tech is to use the virtual world as the central hub of the information space, and allow the virtual world to be the element that enables distributed workers to collaborate more smoothly.  This is work we are doing with Sun and Steelcase (and the NSF), and is an outgrowth of a project (the InSpace project) that’s been going on for a few years.

What we are trying to do is use mixed reality and ubicomp techniques to pull as much of the physical activity into the virtual world, and then reflect that activity back out to the different participants as best suits their situation.  So, folks in highly instrumented team rooms will collaborate in one way, and their activity will be reflected in the virtual world;  remote participants (e.g., those at home, or in a cafe or hotel) may control their virtual presence in different ways, but the presence of all participants will be reflected back out to the other sides in analogous ways.  We may see ghosts of participants at the interactive displays, or hear their voices in 3D space around us;  everyone will hopefully be able to manipulate content on all displays and tell who is making those changes.

A secondary benefit, I hope, is that by putting the data in the virtual world and making that the place that gives you more powerful and flexible access to the data (e.g., by leveraging space and giving access to history), distributed teams will begin to have the virtual space become a place they go to work, bump into each other and have those casual contacts co-located workers take for granted.

Creating the Information Landscape of the Future

Tish: At the end of my interview with Ori Inbar he said, in order to have a ubiquitous experience “you’ll need to 3d map the world. Google earth like apps are going to help but it is not going to be sufficient. So let’s leverage people. Google became successful in part by making people work with them.  Each time you create a link from your blog to my blog their search engines learn from it.  So let’s find ways to make people create information that can be used for AR.” What ways do you think people can create information that can be used for AR?

Blair: I think the big part of that is the creation of models and environments, the necessary “baseline” for specifying experiences.  Google and Microsoft are clearly working toward this;  recent videos from Microsoft show them starting to move the photosynth work toward Virtual Earth.  Similarly, I came across a page where people are finally starting to mine geotagged Flickr [see my post, “Location Becomes Oxygen,” and here for more on the “The Shape of Alpha” project from Flickr]  images to create models.  It’s that kind of thing that will be useful first;  using the data we all create to enable modeling and (eventually) vision-based tracking in the real world.

After that, it’s a matter of time till more of what we “create” (e.g., Tweets and blog posts and so on) are all geo-referenced;  these will become the information landscape of the future, the kinds of things people think about when they read “Rainbow’s End”.   The big problem will be filtering, searching and sorting.  And, of course, safety and security.

Tish: You are working with Unity3D to research the integration of mobile location based AR with persistent mirror world like spaces.  What has attracted you to Unity? What is the difference between this and your Wonderland project? I know you mentioned. you will be using head-mounted displays are part of this Unity project. What are your goals for this project?

Blair: We started to use Unity3D because it gave us what we wanted in a game engine.  Most importantly, it’s very open and let us trivially expose AR technologies into the editor.  Similarly, it can target the iPhone, so we can begin to work with it on that platform, too.  The biggest problem with creating compelling experiences is content;  and a show stopper for creating content is not getting it into your engine.  Unity has a nice content workflow.

Unity3D is a front end engine, for creating the game;  Wonderland is both a front end, and a backend.  We are actually looking into using the Wonderland backend with Unity as well.  Wonderland also has growing support for doing “real work” in a virtual world, which is key to our other projects.

Eventually, we’ll be using HMD’s.  The goal for the Unity3D project, initially, was to explore what you can do with an AR/VR mirror-world; this is a project are working on with Alcatel-Lucent, and demo’d at CTIA this year.  It’s continuing to grow, though, and now includes a number of our projects, including some work on mobile social AR and soon, some performance and experience design projects in the area of AR ARG’s.  It’s really quite interesting to imagine what you can do when you have an “MMO of the real world” (which we now have for part of campus) that supports both VR-style desktop access simultaneously with mobile AR access.

Tish: Have you taken another look at OpenSim as a possible backend for augmented reality?  Recently I talked to David Levine, IBM and he is thinking about some possibilities to optimize OpenSim to dynamically load a large amount of objects at once (i.e how fast OpenSim can bulk load into an existing sim) and make it better suited to augmented reality/mirror world type projects.

Blair: I haven’t looked at OpenSim recently.  We will probably look at it this summer.

Tish: Why did you select Unity as a good client for augmented reality?

Blair: Unity is a 3D game authoring environment so at some level it is no different from using Ogre, if all the associated stuff was just as well done. It has integrated physics, scripting, debugging, etc. – you can write code in javascript or C# or whatever.   It has a good content pipeline, as well, and supports a range of platforms.

It has simple networking built in, so multiple unity engines can talk to each other but it is not a virtual world platform out of the box – there is no back end …

Tish: Someone described Unity to me as a great client waiting for a great backend? So what are you going to use as a back end?

Blair: There is no real processing except in the client right now.  We will eventually have to create a back end.  We are thinking of using Dark Star because someone on the Sun Wonderland community forums has already built a set of scripts connecting Unity to Darkstar.

But for us, we are not proposing right now to build a real product.  This is research to demonstrate what you could do if you actually had the back end.

Tish: What are the most important aspects of the backend from your POV?

Blair: We want to simulate a variety of the interesting aspects of the back end.  So I very much care about notions of privacy and security and how these sorts of AR/VR Mirror Worlds would work in practice.  But I care about how those things as they impact user experience, not really about how we would really implement them.

Tish: So looking at some of the big problems from the perspective of user experience? Are we are going to go through the same growing pains that the web and VWs have seen, for example, will we have to type in passwords to get into everyone’s little worlds….

Blair: Well you know the SciFi background to this, you’ve mentioned it in other posts on your blog.  Because when you look at the Rainbow’s End model where you have security certificates flying around, that is in effect what cookies and so on are now.  You can authenticate yourself once and then have those certificates hang around. So you can easily imagine how it could be done.  But the big question is how does that change user experience.  There are all kinds of things that start coming into play – like what happens if nearby people see different things – it goes on and on!

Tish: Sounds Like this is very valuable research.  It seems to me that there will be a lot of investment soon in putting the pieces together to do location based markerless AR and it would be nice if we knew more about it from the user experience POV.

Isn’t it vital for a productive intersection between mobile AR and persistent mirror world spaces for us to have markerless AR?  Aren’t we right at the beginning of people really saying yeah markerless AR is doable now? But it seems to me not many people researching or working on fully immersive AR and its integration with mirror worlds?

Blair: I think some of the AR community is thinking about this. There’s probably people who are doing stuff in some other non technical communities. It wouldn’t surprise me to find out that people in the digital performance or ARS electronica world who are thinking a little bit about these sorts of things. Although not necessarily at the level of actually trying to build it, because they probably can’t right now.  But experimenting with the precursors.  My colleagues in digital media like to point out that this is often the purpose of digital art, to point out new directions and push the boundaries.

Obviously Science Fiction has explored the possibilities because that is what Rainbow’s End and the Matrix were all about.

Tish: and Denno Coil…

Blair: There has been some research – people like my adviser Steve Feiner up at Columbia, Mark Billinghurst in New Zealand, myself and people at Graz University in Austria .  But partly it has been so hard to do mobile AR up to now – so many people mock head worn displays and can’t get past current technology – you have had  to be willing to ignore the bulky back packs and cables and batteries and so on.  That is changing which is good.

My current response to the anti-head-mounted display people is if 5 years ago you told me you told me that fabulously dressed people who care about their looks and wear stylish clothes would have had big things hanging from their ears that blink bright blue light, so they could talk on the phone, many of us would have said you were crazy, because it would be ugly and so on.  But because there is an intersection of demonstrable need and benefit…Bluetooth headsets are really useful and the sort of early gestalt feeling that grew up around them – that people who use them are so important that they always have to be in touch, they wear these things – so people accept them.

It will likely be a similar thing with head mounted displays. And I don’t know if it will be that people wearing them so that they can read their mail while driving, god forbid. But it will be something.  And when we get the 2nd generation of the wrap glasses that look more like sun glasses and are not bulky and so on, we will have the potential for them catching on because you will look at them and you will think that the person is wearing because they are doing x…

X might be surfing a virtual world or reading their email or keeping in touch, or being aware. It will happen. But they have to get unbulky enough and there has to be more  than one important application, not just watching TV.

Picture above shows  an outside view of the KARMA AR system;  the knowledge based maintenance system Blair built in his first year of grad school (“first AR system Steve Feiner, Doree Seligmann, and I worked on”).  Blair noted, “The Communications of the ACM paper on it (from 1993) is a pretty widely cited AR paper.”

Tish: I think the need for  full on transparent, immersive, wraparound, Gucci stylish eyewear with a decent field of view are the elephant in the room in terms of realizing the full potential of augmented reality.  There are a few new players in the field Digilens,  Vuzix, others?  What is the progress in this area and what do you hope for in terms of near term solutions?

Blair: I agree with that sentiment.  I think that, in the near term, there is a lot we can do with handhelds, as we’ve been doing in the lab.  However, because it’s awkward and tiring to hold up a device, even a small one, for any length of time, handhelds will only be good for “focused” uses of AR.  Such as the table-top games we’ve been doing, or the constellation viewing app that I heard came our recently for the Android G1.  I don’t even see something like Wikitude as that compelling (beyond the “gee wiz” factor) for a handheld form factor.   Many proposed AR apps only really become compelling when users have constant awareness of them, and that requires a see-through head-worn display.

I’ve seen the mockups of the Vuzix ones;  they seem pretty interesting, and are getting to were early adopters could use them (they will be cheap enough, and will hopefully be good enough).  Microvision’s virtual retinal display is also promising;  the contact lens displays will be the most interesting, if anyone can ever make them work.   I don’t know of anything else out there.

“its not really a killer app you care about, it is the killer existence that all of the technology and small applications taken together facilitate”

Tish: While location based services are accepted now and people are understanding that it is something that opens up a new relationship to everything, we still haven’t found the experience that will get everyone holding up their mobile devices?

Blair: Well that is actually the killer problem.  Gregory Abowd is one of my colleagues who does ubiquitous computing research here at Tech.   Way back when we started the Aware Home project (Aware Home Research Institute at Georgia Tech) when I first got here about ten years ago, there was always this question of what is the killer app.  So Gregory comment in a meeting once that its not really a killer app you care about, it is the killer existence that all of the technology and small applications taken together facilitate. It is not that any one of these AR demos we see right, whether it is seeing your photos in the world or whatever, is important. Its that when taken together, there is enough of a benefit that you would use the whole environment.

In the original context we were talking about an instrumented home, but it is the same thing here with AR.

The problem with the mobile phone as a AR device is that problem of awareness. If I have a head mount on and I walk down the street and there is bunch of probably-not-useful-but-potentially-useful information floating by me, that’s a good thing, because I may see something that is useful or makes me think of something else.  But if I have to hold up my phone to see if something might be interesting nearby, I will never hold up my phone because at the time there is a high probability that there won’t be anything particularly important there.  You might imagine you can get around this by using alerts or something like that, but then you overload whatever alert channel you use.  For example, I forward maybe 5 or 6  people’s updates from Facebook to my phone – started with my wife, a few friends, my brother, and the net result of that is I never get SMSs’ anymore because when my phone buzzes, usually I ignore it because it is probably just somebody’s random Facebook update. So if we start overloading channels like that with “oh there might be something useful here in the real world, if you pick up the phone and look through it you will see it … and I will buzz you.” People  just start ignoring the buzzes.

So it is a very hard problem if you think about the kinds of applications that people always imagine with global AR — names over peoples heads and other random information floating in the world — until you have a head mount and all that information is around you all the time. That is when those sort of applications will actually happen.

Tish: Robert Rice notes: “AR is inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.” (see my interview with Robert, “Is it ‘OMG Finally’ for Augmented Reality?). And I think the iphone experience has laid the foundation for the increasing desire to experience the network wherever we are – and not be stuck behind a pc.  We cannot perhaps do all we want to do yet. But even in the range of things we can do know, we are not even sure exactly what it is we want to do where yet is it?

“imagine your iphone Facebook client supports AR and that all data on Facebook might be georeferenced – pictures, status updates etc…….”

Blair: Yes that is a huge problem. I have been lucky to be able to teach two fun classes this year that let the students and I start to explore some of the potential that handheld AR might bring.  Last fall I taught a handheld AR game design class — coordinated with a class at the Savanna College of Art and Design’s Atlanta campus — and we had the students build a sequence of prototype handheld AR games, which was a lot of fun.   This spring I taught a mixed reality/augmented reality design class with Jay Bolter (a professor in the School of Literature, Communication, and Culture here at GT).  Jay and I have been teaching this class off and on for about 9 years; this semester we decided to say to the students “imagine your iphone Facebook client supports AR and that all data on Facebook might be georeferenced – pictures, status updates etc…….” and have them do projects aimed at such an environment.

Tish: Not many of our favorite social media today have much sense of location do they? But Flickr  are  utilizing the geo-referenced pictures to create vernacular maps…..The Shape of Alpha

Blair:Yes that is because lots of cameras put geo location data into the exif data so they can extract it…

Some mobile Twitter clients like the one I use in my iphone will let you add your location.  But in general Facebook and other sites don’t have any notion of location. But if you look at all the things people do in Facebook, such as sending gifts and other games, its easy to imagine what these might look like with geo-reference data.  So, the high level project for the class is the groups have to design experiences people might have using mobile AR Facebook.  We told them to assume Facebook as it stands now, but add geolocation and AR to the client.  The class boiled down to “What would you imagine people doing?” So it has been kind of fun.

And we are using Unity for the class too – the same infrastructure I am working on in my research linking mobile AR to persistent immersive mirror world type spaces – and we having the students mock up what a mobile AR Facebook experience would be like.

Tish: Can you describe some of the ideas you class came up with that you think have potential? I know Ori mentioned that from the games class he liked Candy Wars.

Candy Wars

Blair: In the end, they had a nice range of projects in the Spring class.  One created tag clouds out of status messages over spaces, others looked at analogies to virtual pets and gift giving out in the world, one looked at leveraging geolocation to help with crowd-sourced cultural translation, and three groups did straight-up social games.

[See all of the projects from the handheld AR games class on YouTube here]

iphone, Android, or NVidia Tegra devkits or the Texas Instrument’s OMAP3 devkits?

Tish: Is anyone in the class working on Android?

Blair: Nobody is using Android because no-one in the class has the phones. We have ATT microcell infrastructure on campus.  Some ATT people joke that we are better off than them because we have a head office on campus so we can build in the network applications which people even at ATT research can’t do.  But because  we have this infrastructure on campus, and a great relationship with ATT and the other sponsors, we have the ability to provision our own phones without having to pay for long-term contracts, which is vital for research and teaching.

Tish: So does this lock you into the iphone?

Blair: Well the G1 is of course not AT&T but it is GSM so we could probably buy them unlocked and put them on our AT&T network. But the students I work with are much more interested in the iphone right now.

Tish: Is that because the iphone has the market?

Blair: For me the reason I am not interested in the G1 is because you can’t do AR on it – there is Wikitude and a few other apps, but it is all hideously slow.  Worse, because the Java code isn’t compiled like it would be on the desktop, you can’t do computer vision with it, so you can’t do anything particularly interesting on the current commercial G1s.  We could probably take the NVidia Tegra devkits or the Texas Instrument’s OMAP3 devkits (both are chipsets for next gen phones — high end graphics, fast processing),  and install Android on those and we may actually do that yet.  But, it seems like a lot of work right now, for not much benefit.

Augmented Reality shooter game ARrrrr from Georgia Tech and SCAD Atlanta on the NVidia Tegra devkits – watch the demo on YouTube here.

Tish: Everyone seems very excited about the iphone OS 3.0 and the addition of compass. Compass is pretty essential for AR right?

Blair: It is necessary if you can’t do other forms of outdoor tracking, but the problem is that the compass on the G1 isn’t very good, relatively speaking and the iPhone one probably won’t be much better. It does not have very high accuracy, nor is it very fast (compared to, say, the high end 3D orientation sensors we use, from Intersense and MotionNode). As far as I can tell, it doesn’t even give full 3D orientation. I don’t have a G1 (although I have pre-ordered an iPhone 3Gs), but people have told me it only has absolute 2D orientation, so you can only line things up if you are careful.  Your can’t look around arbitrarily…

Tish: You can’t sweep your phone?

Blair: You can look left and right, but if it doesn’t have full 3D orientation, you can’t go up and down. You can’t tilt it in weird directions. It is not fast in the form that you would want to look around quickly.  So it is nice demo.  And it is good for what the Android people use it for which is to let you do your Google street view by looking around, which is actually really useful.

I think there are lots of really useful things you can do with such a compass.

And, it is clear that compass is a necessary feature if we want to do AR.  It’s just not sufficient.

Outdoor Tracking and Markerless AR

Tish: Isn’t it essential for markerless AR? I guess not I just saw this post about SREngine on Augmented Times!

This wasn’t up when we spoke so perhaps you have some comments about what it brings to the table?

Blair: Maybe. The folks at Nokia are working on outdoor tracking, they demoed some stuff at ISMAR last year on the N95 handsets that is all image based.  We are trying to do some work with them, one of my students is working on it.  And probably Microsoft is going to do more on this as well, they had a video up showing that they are also working on vision based techniques.  If you give the phone the equivalent of those panoramic Google Street View images (assuming they are up-to-date) and you are standing at the right place, you don’t really need a compass, you can figure out which way you are looking by looking at the camera video. Ulrich Neumann (USC) did some work on tracking from panorama’s years ago, I don’t know what ever became of it.

Regarding SREngine, that project appears to be a pretty simple first step, but is probably just a demo at this point, and limitations like “only works on static scenes” and “doesn’t work for simple scenes” means it’s probably extracting some simple features out of the image and then matching those to some database.  The trick would be getting this to work on a large scale, where the world changes a lot.   It’s not obvious how to get there.

Tish: So forget RFID for AR…

Blair: RFID is not really useful.

Tish: not at all?

Blair: RFID is useful for telling you what things are near you.  The problem is it doesn’t give you any directional information – it just tells you you’re in range of the tag. So can use it to tell you when you are near a certain product for example.  So it is useful in terms of telling you what thing you are near, and then you can load up a vision system or something else that will recognize that thing.

In that way, it could be useful as a good starting point.

Similarly for computer vision, the compass and the gps are very useful for giving you an initial guess at what you may be looking at that can then speed up the rest of the process.  But, computer vision by itself will not be a complete solution because if I have my panoramic Google Street view (or whatever image database I use for tracking) and you are standing between me and the building -  I am not going to see what I expect to see, I am going to see you.

So I think it is all going to be part of one big package – you are going to see accelerometers, digital compasses, and gps and then combine that with computer vision and other sensors, and then maybe we are going to start getting the things that we have always dreamed about.  I like to show this video from the U. of Cambridge (work done by Gerhard Reitmayr and Tom Drummond) of an outdoor tracking demo because it gives a sense of what will be possible.  Techniques like this will be an ingredient in the future of things.  It becomes especially interesting when you have these highly detailed mirror worlds.  It is sort of one of those chicken and egg problems where if I have an highly detailed model of the world then techniques like they have can be used to track.  But that mirror world needs to be accurate or you can’t use it for tracking, and why would you create the mirror world if you couldn’t track?

Tish: I noticed in your comment to “my interview with Robert Rice” that you said you thought that is was important not to collapse AR into ubicomp – “forgetting what originally inspired us about AR” is, I think if I remember correctly, the suggestion you made. But aren’t ubiquitous computing and AR basically coextensive?

The vision of ubicomp Mike Kuniavsky describes – “sharing data through open APIs and the promise of embedded information processing and networking distributed through the environment” demonstrates how much can be done with very little processing power.” In its most immersive form augmented reality requires a lot of processing power. I think we have all become very conscious about trying minimize levels of consumption.  Can you explain why you think people shouldn’t see AR as the Hummer (energy squandering indulgence) of Ubiquitous Computing?

Blair:  I think there will be a hierarchy of interfaces. You are going to have the rich Rainbow’s End like experience – you are totally submerged in a mixed environment, if you have a head mount on (its not going to be Rainbow’s End for while) but if you don’t have the headmount on that information might be available to you other ways, whether it is a 3D overlay using your handheld or just a 2D mashup with Google maps.  But there will be some circumstances and people who will want to get the compelling experience you can only get with the headmount.

Tish:  Are you doing any research on how all these hierarchies of experiences will fit together – what aspects of this are you looking at?

Blair: The thing that really needs to happen is you need to have this backend architecture that allows you to collect your data from different sources and aggregate it much like the web. Right now Google Earth and Microsoft’s Virtual Earth are much like the old pre-web hyper-text systems that were all centralized. And what we really need is to have the web equivalent where Georgia tech can publish their building models and I.B.M. can publish their building models and their campus models, and your client can aggregate them, as opposed to Microsoft or I.B.M. puts their building models into Google Earth and then somehow you get them out with Google’s google earth browser. That’s just not going to fly.

Tish: so what does it take then to get us to this backend architecture, because I’m in total agreement?

Blair: The nice thing about augmented reality versus virtual reality is that you don’t need everything modeled. You can do interesting AR apps like Wikitude with absolutely no world model.

Tish: So that means we can start with what we have – utilize cloud services without a full blown backend architecture?

Blair: It may very well be that Google Earth and MS Virtual Earth act as a portal because people go and build models and link them with KML, and they can see them in google earth but they can also download the KML’s through some some other channel. So it may be that those things end up being something that feeds some of this along. Then people start seeing a benefit to having these highly accurate models so then you start integrating the Microsoft photosynth stuff and leveraging photographs to generate models.

It’s just keeping up with it and building it in real time is the challenge. A lot of folks think it will be tourist applications where there’s models of times square and models of central park and models of Notre Dame and the big square around that area in paris and along the river and so on, or the models of Italian and Greek history sites – the virtual Rome. As those things start happening and people start building onto the edges, and when Microsoft Photosynth and similar technologies become more pervasive you can start building the models of the world in a semi-automated way from photographs and more structured, intentional drive-by’s and so on. So I think it’ll just sort of happen. And as long there’s a way to have the equivalent of Mosaic for AR, the original open source web browser, that allows you to aggregate all these things. It’s not going to be a Wikitude. It’s not going to be this thing that lets you get a certain kind of data from a specific source, rather it’s the browser that allows you to link through into these data sources.

So it’s that end that interests me. It’s questions like “what is the user experience”, how do we create an interface that allows us to layer all these different kinds of information together such that I can use it for all my things. I imagine that I open up my future iphone and I look through it. The background of the iphone, my screen, is just the camera and it’s always AR.

I want the camera on my phone to always be on, so it’s not just that when I hold it a certain way it switches to camera mode, but literally it’s always in video mode so whenever there’s an AR thing it’s just there in the background.

When we can do that I can have little alerts so when I have my phone open I can look around and see it independent of the buttons and things that I’m tapping and pushing to use the phone. That’ll be a really a different kind of experience.

Of course it is not known yet if the next gen iphone will have an open video API.  And of course, the current camera is pretty low quality, so why would they give it an open API until they put in a better camera?  I am not expecting anything one way or the other until the 3Gs comes out and people start using it.

But there are many things about the iphone 3.0 OS that are hugely important, like the discovery API that allows people to play games with other people nearby, that don’t have much to do with AR.

Tish: You have an iphone AR virtual pet application ARf.

Macrumors wrote it up and suggested that the neg gen iphone will have compass and open video API.  What are your plans for ARf?

Blair: ARf is just a demo right now.  I know what we’d like to do with it, but it would require tons of work;  imagine what it would take to do a multiplayer, social version of Nintendogs?  It’s not clear what we’d really learn by doing that, but there are lots of other game ideas we have that we want to explore.

Tish: I think it was on Twitter where Tim O’Reilly said, “saying everything must have a RFID tag is like saying we can’t recognize each other unless we wear name tags. Look at what’s happening with speech recognition, image recognition et.al. and tell me you really think we need embedded metadata.” What would you say to that?

Blair: I think that whatever extra data is there will be used. So if we put machine readable labels on some objects then they’ll be used if they make the identification and tracking problem easier. But it’s pretty clear that people are already working on tracking and so on.

A lot of these mobile AR apps are clearly putting ideas in people’s minds things that won’t really be doable in the near future. Like being able to look down the aisle of the store and it recognize all of the products. Given the distances and complexity of the scene, the number of pixels devoted to each of those objects, and so on – you just can’t recognize things in that context. But if I’m standing in front of a small set of objects, or looking at one thing, or I’m standing in front of a building, or if I’m in the store and because of the location API — imagine an enhanced location API that can tell me within a few feet where I am, and then combine that with some use of the discovery API that allows the store to tell your device you’re in the toothpaste section. Now you only have to look for different brands of toothpaste. So now you can recognize the big letters “Crest” or whatever. It’s all about constraining the problem.

That’s why I like that particular piece of Drummond’s work, the tracking web site I mentioned above. The general tracking problem of looking around and recognizing objects and tracking is still impossible. But if I know roughly what direction I’m looking in and I have a good estimate of my position, and I have models of what I should be seeing when I look in that direction, then it becomes a tractable problem. And so it’s not that a compass and a GPS are 100% necessary. But if you have them it certainly makes things possible that you wouldn’t otherwise be able to do.

Imagine for example  if there’s a new version of GPS, I just noticed that some of the new satellites going up have this new L5 channel. There’s the L1 & L2 signals  that the military and civilian ones use and they added this civilian L5 signal, which should make GPS more accurate. I haven’t found anything online that says how much more accurate.

But someday, hopefully, all GPS will get to be the quality of survey-grade GPS. Right now, if you get an RTK GPS from one of these companies that make the survey grade GPS systems, they give you position estimates in the range of two centimeters, and update 10 to 20 times a second. When you have that kind of positional accuracy combined with the kind of orientational accuracy you get from the orientation sensors we use in the lab from Intersense and MotionNode, everything is easier because you’ve pretty much got absolute position. You put that into a phone and now when I look up, it’s still not perfectly aligned because there will still be errors (especially in orientation, since the compasses are affected by metal and other magnetic noise). But it does mean if you and I are standing 5 feet apart from each other and look at each other, I can pretty much put a little smiley face above your head. Whereas now, with GPS, if I look at you and we’re 5 feet apart our GPS’s might think we’re on the opposite side of each other because they’re only accurate to two to five meters.

And that depending on the time of day and weather!

Putting RFID tags everywhere is easy; the problem is the readers – they currently require lots of power and they have a limited range.  Sprinkling RFID tags everywhere is fine. But you have to be able to activate those tags and read back the signal.  In certain contexts it works.

Tish: And one final question!  What do you think can be done re beginning to think about standards for AR.  Is there a meaningful discussion going on yet? Thomas Wrobel left this comment on my blog rcently and I was wondering what your position was on some of the ideas he raises?

Wrobel wrote, “The AR has to come to the users, they cant keep needing to download unique bits of software for every bit of content! We need an AR Browsing standard that lets users log into an out of channels (like IRC) and toggle them as layers on their visual view (like Photoshop).Channels need to be public or private, hosted online (making them shared spaces) or offline (private spaces). They need to be able to be both open (chat channel) or closed (city map channel) as needed. Created by anyone anywhere. Really IRC itself provides a great starting point. Most data doesn’t need to be persistent, after all. I look forward too seeing the world though new eyes.I only hope I will be toggling layers rather then alt+tabbing and only seeing one “reality addition” at a time.”

Blair: I agree with him, in principle.  But, I’m not sure there’s a point yet.  It can’t hurt to try, of course, from a research perspective, and I’m interested in the experience such an infrastructure would enable (as we’ve talked about already).

The biggest news at Where 2.0, 2009 came from the Yahoo! Geo Technologies Group. Tyler Bell, announced Yahoo! Placemaker and the opening up of the GeoPlanet data set, “all of the WOEIDs [Where On Earth (WOE) IDs] available as a free download under Creative Commons in June” (see Brady Forrest’s post for more details).

WhereCamp 2009 was held immediately after Where 2.0 and was a great place to chew on the events and ideas of Where 2.0.  In the picture above Anselm Hook addresses the WhereCamp morning circle in the courtyard outside the Social Text offices in Palo Alto. Anselm pointed out to me:

“there are interesting implications of placemaker in combination with other yahoo assets – in particular YQL – placemaker by itself is neat – but placemaker combined with everything else is a natural missing piece that is a big enabler.  Yahoo has been impressive.”

With all the Geo platform power available to us now, also (also see New Geo for Devs from Google I/O), there isn’t a shadow of a doubt in my mind Brady is right when he said, just before the Where 2009 conference: “Location is no longer a differentiator it’s going to become oxygen” (quote from WebMonkey).

The Yahoo! GeoPlanet team at WhereCamp – Tyler Bell, (talking to Brady Forrest in picture on the left) is sporting his spatial junkies T-Shirt. Photo on right, Aaron Cope, Tyler Bell, Martin Barnes, Gary Gale.

WhereCamp was alive with key figures from the social geography movement who knew the power of these new tools (see some of my photos of WhereCamp on Flickr here).

The importance of the Yahoo! announcement really became clear to me at WhereCamp where I attended sessions all day Saturday including the Curating Big Data Session led by Tom Carden, Stamen Design and Aaron Straup Cope, Flickr, (see Aaron’s slides from his Where 2.0 presentation on “The Shape of Alpha” here and video here).

Anselm Hook, a prime mover for WhereCamp, is a leading philosopher of place making and veteran software developer who led Platial engineering and is now at web consultancy http://makerlab.com. If you missed Anselm at WhereCamp he will be presenting on, Ubiquitous Angels at The OpenSource Bridge, Portland, Oregon, June 17th -19th, 2009.

Anselm describes where he thinks the challenges are:

“We should be mapping information that in some ways has been historically unmappable because it is 1) not valued or is 2) actively seen as threatening or is 3) simply too hard to map using traditional tools.”

The WhereCamp Schedule

The Shape of Alpha

Screen capture from Aaron’s Where 2.0 presentation on “The Shape of Alpha. Original photo from Flickr user “NickIsConfused”.

Aaron Straup Copes’s work on “The Shape of Alpha” puts key questions about curating big data center stage.

Firstly, the exploration of what it means to curate/collaborate over meaning from “the abundance of data produced in the precise but distant language of machines” (also see The Interpretation of Bias (and the bias of interpretation). The Shape of Alpha uses a process of reverse-geocoding to translate machine-generated geographic data into place names that people can understand and relate to.

The shapefiles are built with nothing but geotagged photos and some code called clustr (written by the brilliant  Schuyler Erie – co-author of Mapping Hacks). Anyone can make these shapefiles. You can get the shapefiles out of the  Flickr API. Aaron has been keying off WOEIDs (Where On Earth (WOE) IDs) but as Aaron noted you can key off anything you like – tags are an obvious choice.

Wow! You can reinvent mapping with this stuff.

Very importantly, “The Shape of Alpha,” tells us something about how we relate to place versus location. The emotions, disputes and behavior related to place also emerge through crowd sourced corrections.  For more see this very evocative post by Aaron about corrections and treating airports as cities.  There is a glorious thread/riff and ode to the genius of  J. G. Ballard pursued by Aaron and Dan Catt in their posts (also see Dan Catt’s, J.G. Ballard, Flickr, naked singularities and 3-letter airport codes, and Aaron pointed me to this brilliant “geo-detective work” on Concrete Island, by Mike Bonsall .

Dan Catt created geobloggers and “seeded the geotagging community around the Web.” I met Reverend Dan Catt (Twitter @revdancatt ) at Where 2.0 when he was kind enough to share part of his seat so I could join a very interesting discussion with Aaron on The Shape of Alpha.

As Aaron points out they decided to treat “the airport itself as the town…”  not (only) because they admired the work of J.G. Ballard,  “but because it is the right thing to do.”

Dan Catt has excellent blog posts “describing the nuts and bolts of how ‘corrections’ works.”  Aaron points out,  “in the nerdier of the two Dan sums it up nicely by saying”:

“On a slightly more philosophical level, it’s a never ending process. We’ll never reach a point where we can say “Right that’s in, all borders between places have been decided.” But what we should end up with are boundaries as defined by Flickr users.

…

For us, it’s a first small step into an experiment, and actually a pretty big experiment as we’re potentially accepting “corrections” from our millions and millions of users. We’re not quite sure how it’ll all turn out, but we’re armed with Maths, Algorithms and kitten photos.”

Psychosynthography – “Wearing Geography as a Perfume”

Psychosynthography screen capture from Aaron Straup Cope’s Where 2.0 presentation . Original photo from Flickr user,  “NiteLynx.”

As I mentioned before, many of the ideas raised at Where 2.0 were unpacked and worked through at WhereCamp. For example, Aaron introduced a word psychosynthography in the last 24 seconds of his talk at Where 2.0.

So I spent as much time as I could listening to Aaron at WhereCamp, and asking him about psychosynthography and more (post of this interview upcoming).

Aaron urged the Where 2.0 audience to pay attention to the Psychogeography movement seeded by Guy Debord, and “to wear geography like a perfume.”

Joseph Hart writes in a “New Way of Walking” psychogeography is:

“a whole toy box full of playful, inventive strategies for exploring cities…just about anything that takes pedestrians off their predictable paths and jolts them into a new awareness of the urban landscape.”

Curating Big Data

Tom Carden, Stamen, (picture above) paired with Aaron for the Curating Big Data session. Tom noted:

“The Curating Big Data session for me was an attempt to learn from other attendees (as opposed to teach/lead, as with the Stamen session, “Real Time Web-Based Visualization and Mapping”).  Also, it was an excuse to get Aaron to recap parts of the Flickr Shapefile story for WhereCamp folks, and to get *input* on how to do more things like it. I was a bit disappointed that nobody had really good examples for us, but I was happy with Brad Stenger’s suggestion to look into the upcoming census data as a relevant area.”

Aaron’s work on the The Shape of Alpha and The Corrections project shows, as Tom noted:

“what you can do once you have 150 million geotagged photos, and millions of users who are willing to say I took this thing here and my name for that place is …..”

And part of the significance of opening up the GeoPlanet data set is that now:

“we can try and start talking about the same places, as far as, [for example], these shape files go. So if you are interested in what comes out of the Flickr shape files project and but you also have your own opinion about what shape those places are so the IDs have be open you have to be sure that you are talking about the same thing in the first place.”

And, as Tom pointed out, collaborating over geo data informs us about curating any big dataset:

“it should lead to an overarching discussion about any kind of dataset geo or otherwise and ways in which we can talk about it, and think about patterns for improving that data, for collaborating, even on things like cleanup.”

Warp speed geo-genius Andrew Turner, Fortius One, took these excellent notes for the “Real Time Web-Based Visualization and Mapping” (on left) and “Curating Big Data” (on the right).

On my way to Where 2.0 I took the train from SFO to San Jose which was a delight but a little slower than I imagined. So, unfortunately, I arrived on Tuesday just after Michal Migurski (Stamen Design), Shawn Allen (Stamen Design) presented  Maps from Scratch: Online Maps from the Ground Up. This was on my MUST attend list and it was a wonderful opportunity to get into, “Real Time Web-Based Visualization and Mapping.”   I did get a chance to talk to Michal and Shawn a bit later in the conference but I will try to catch up with them soon for an in depth story.  Below is  Shawn Allen’s map of overlapping data sets from, “Trees, cabs and crime in San Francisco:”

Another follow up I am really looking forward to making is with Liz Barry and her work on S+EM, “an environmental mapping and social networking design project that links New York City trees with the people who care for them” (also see, Creating a Greener San Francisco Tree by Tree).  Also I got a chance to talk to another fellow New Yorker (we have to travel to the West Coast to find time to chat!), John Geraci of DIY City who presented DIY City:  An Operating System for Cities.

Machine Intelligence and Human Intelligence

Aaron Cope, Flickr, on the left is talking to Andrew Turner on the right the CTO of FortiusOne (see Andrew’s presentation at Where 2.0, “Your Own Private Geo Cloud”)

Many of the most interesting conversations happened in between sessions at WhereCamp and Where 2.0.

I caught this one in which Aaron Cope and Andrew Turner where discussing some of ideas Aaron raised in his presentation, “Capacity planning for meaning in the age of personal informatics” (see Aaron’s blog post, Tree planting and tree hugging in the age of personal informatics). The core question they were discussing was what happens when you wire the world at the scale people are talking about and it breaks… Aaron argues that you already have a whole class of people in systems operations that can tell us a lot about how to answer this question.

Ryan and Anselm shared the pulpit for the morning circle pulpit with Ross Mayfield of Social Text who was the generous host to WhereCamp.

Social Reality Mining

“As it stands today, we have no idea what terms and limits of a cloud based citizenship of the Google Caliphate will entail and curtail. Some amalgam of post-secular cosmopolitanism, agonistic radical democracy, and post-rational actor microecomics, largely driven by intersecting petabyte at-hand datasets and mutant strains of Abrahamaic monotheism. But specifically, what is governance (let alone government) within this?” from Benjamin Bratton’s talk at ETech 2009 (picture above), Undesigning the Emergency: Against Prophylactic Urban Membranes.

The other big take away from WhereWeek – Where 2.0 and WhereCamp, was not so much news, but a confirmation of something that has been pretty clear for a while now. (Check out Brady’s posts on reality mining at Where 2.0 last year).

We are moving headlong into the era of reality mining with all its myriad possibilities from: “hedonistic optimization” (this term came from Stig Hackvan when I asked him about some of the ideas central to the HeadMap Manifesto -more about HeadMap later in this post); to new forms of marketing (social reality mining the inside to predict if someone is going to trade business cards in the next 120 seconds – Alex “Sandy” Pentland, MIT, Where 2.0);  to stuff that matters to save us from mass extinction like distributed sustainability – greening production and consumption and our cities; to open government;  empowering indigenous communities (also see Rebecca Moore’s Indigenous Mapping: Emerging Cultures on the Geoweb Presentation); and not to be forgotten, the troubling possibility of new forms of social control.

Smart phones are powerful networked sensor devices in the palm of our hand

As Sandy Pentland MIT pointed out in his Where 2.0 keynote, “Reality Mining for Companies, or, How Social Networks Network Best,” mobile phones have created an ubiquitous instrumented reality that goes way deeper than location awareness. Smart phones are powerful networked sensor devices in the palm of our hand that know a lot more about us than location. With proximity, motion, (accelerometers), voice, images, call logs, email – what is enabled is not just knowing where people are but knowing more about them.

Many of the issues raised by Adam Greenfield in Everyware and in my interview with Adam were on my mind during WhereWeek, also questions that were distilled and explored in this presentation by Matt Jones last year, Polite, Pertinent, and… Pretty: Designing for the New-wave of Personal Informatics and Timo Arnall’s presentation, The Web in the World.

Google Wave, Pachube  Feeds, Sensor Networks and Microsyntax!

Visualizing 24 hours of Pachube logs, feeds all around the world -  built with Processing.

I found myself really wishing Pachube founder Usman Haque had been able to come to Where 2.0 this year – Usman was originally on the Where 2.0 schedule but had to drop out. My small contribution to WhereCamp was to discuss Pachube, Natural Fuse and OpenShaspa in the, Urban Eco-Managment session (see my interview with Pachube Founder, Usman Haque here).

Pachube announced – mapping mobile feeds in realtime, with 3d datastream value time & location based graphing just before Where 2.0.

And, as I was writing up this post, I was delighted to see this post by Bruce Sterling on Pachube Feeds and his challenge, offering:

“(((Extra credit for eager ubicomp hackers: combine this [pachube feeds] with Googlewave, then describe it in microsyntax. Hello, 2015!)))”

Also Anselm Hook, who has an extensive background in video game development, made an interesting point about Google Wave to me:

“btw – there is a preexisting metaphor for the wave – the wave is notable in that it is making the web like a videogame – its bringing real time many participant shared interaction to the web”

“Jens, Lars, and team re-imagined email and instant-messaging in a connected world, a world in which messages no longer need to be sent from one place to another, but could become a conversation in the cloud. Effectively, a message (a wave) is a shared communications space with elements drawn from email, instant messaging, social networking, and even wikis.”

For more on microsyntax see microsyntax.org

Aaron pointed out to me re microsyntax:

“This is ultimately the “magic word” problem, which is essentially the semweb vs. google-is-smarter-than-you problem.”

I will have some more questions for Aaron on the the “magic word” problem in my upcoming interview post.  At the moment I am busy studying some of the thoughts in these links.

http://delicious.com/straup/magicwords

http://www.slideshare.net/straup/the-papernet/22

http://www.xml.com/pub/a/2005/02/16/edfg.html

http://xtech06.usefulinc.com/schedule/paper/135

Privacy: Towards a Win Win and Community Sensing

While a key element of  Yahoo! Geo Technologies portfolio of platforms, FireEagle, not only gives an important set of tools to allow people to “share their location with sites and services through the Web or a mobile device” but also offers up some vital privacy tools, the community sensing work of Eric Horvitz takes privacy and data sharing into new terrain.

Eric didn’t have time to discuss his privacy work in his Where 2.0 presentation, Where, When, Why, and How: Directions in Machine Learning and Reasoning about Location, – it came up in his very last slide. But I ran up after his talk with my trusty old ipod recorder in hand, and got the part we missed! Fascinating stuff that will be the subject of an upcoming interview post. Here’s a little taste of what is to come. Eric describes one of the directions his team will be exploring.

“One thing I want to do, on our research team, I’d like to develop something very simple for people to use. A challenging problem with privacy is usability and controls. Aunt Polly and Uncle Herbie just don’t get all these authentication controls and sliders, nor do they want to invest in figuring them out. They also don’t get why they’re being asked with pop up windows to yes or no to various questions and so on. One Idea is having a useable privacy lens, that you can hold up anywhere and it tells you what you’re showing anybody or any organization, what does the world know about you. And you would like to have buttons to turn sharing off for some items. You’d also like to have a way to go back in time and view prior sharing and logging over periods of time, and to have buttons to push to say erase that segment of your logs.”

Understanding the social implications of what it means to live in an instrumented world is a topic that we cannot afford not think about. But luckily there are lot of people who have been thinking pretty deeply about this for a while now.

And I did my best at both Where 2.0 and WhereCamp to seek out as many of geothinkers as I could, and do interviews wherever possible (I have not had time to mention everyone I talked to in this post but hopefully all the interviews will get on Ugotrade soon!)

HeadMap Manifesto

In the bar of The Fairmont on the last night of Where 2.0, I heard some of the history of Where 2.0, GeoWanking, and The HeadMap Manifesto from Sophia Parafina, Director of Operations for OpenGeo and Rich Gibson, programmer, GeoWanker,  Gigapanner and co-author of Mapping Hacks with Schuyler Erie and Jo Walsh (Jo did a lot of key early work on bottom up urban informatics but unfortunately couldn’t make it to WhereWeek this year).

Check Gigapan.org out! “The GigaPanSM process allows users to upload, share, and explore brilliant gigapixel+ panoramas from around the globe.”

Also I interviewed Paul Ramsey, Senior Consultant, OpenGeo, so more on OpenGeo is upcoming (see Paul’s Where ReCap). Justin Deoliveira (OpenGeo) and  Sophia Parafina did a session, GeoServer, GeoWebCache + OpenLayers: The OpenGeo Stack, which unfortunately I missed as it was before I arrived Tuesday.

I met Rich Gibson at Etech 2009 playing Werewolf and Rich introduced me to his co-author on Mapping Hacks and alpha geek supreme, Schuyler Erie, who also wrote the clustr code that The Shape of Alpha uses.

Joshua Schachter founder of Delicious and the GeoWanking mailing list, [and GEOURL - and MemePool!] now at Google came to WhereCamp and was mobbed by a small crowd eager to get their hands on one of the developer G Phones he was handing out from a large box.

GeoWanking, which is now run by O’Reilly Media, has been the incubator for all things location aware and “neogeography” discussions since 2003 – check out ‘sproke for a Paleogeography vs Neogeography (which, as Sophia notes, was a common topic of discussion at Where 2.0) smack down in which geowanking rules in the form of a list traffic comparison.

Sophia and Rich shared some of their perspective on the early days of GeoWanking and the creation of the HeadMap Manifesto with me and pointed me to many other people to talk to. The prime mover of the Headmap manifesto, Ben Russell, has retired from the scene – perhaps bored by seeing a radical vision gone thoroughly mainstream, or exhausted by the rigors of carrying an idea through the early blue sky years, or just s simply doing something else? I don’t know.

The HeadMap Manifesto is still vibrant today even as much of what it envisaged has already been realized. HeadMap assembled the future in a poetry of fragments:

“you can search for sadness in new york people within a mile of each other who have never met stop what they are doing and organize spontaneously to help with some task or other.”

Anselm explained to me what powered all this social cartography revolution, from his POV, was actually IRC.

“We had a channel on IRC called “#geo”.  And many of us met there.  I met Ben Russell at MathEngine in the UK. Ben and I were fascinated by the future of maps.  Ben, Jo and I met Schuyler, Dav, Dan Brickley (who worked for Tim Berner’s Lee who invented the Web), Rich Gibson, Joshua Schachter (who was just a geek at Morgan Stanley at the time ) … and the snowball took off….  many others.

We stormed ETECH ( Schuyler met Jo there).  We got invited to FooCamp. Schuyler was married to Jo by Marc Powell (Food Genome) and lived at his house.  We pushed so hard on the social cartography revolution.

I did a spinny globe for geourl – a project by some hacker named Joshua Schachter…  we were all friends for years and we had never even met.”

“Can AR researchers harness these new approaches to index reality?”

Radiohead’s laser (as opposed to video) clip made using Lidar

If you have read my interview with Ori Inbar, you will know how excited I was to attend The Mobile Reality panel.  The video is up and it is really awesome to hear Raven Zachary (on twitter @ravenme) get into the fray with augmented reality.

The main take away for me from the Mobile Reality panel was that we shouldn’t get too hung up on the difficulties of achieving fully immersive visual augmented reality and twiddle our thumbs waiting for the long anticipated sexy lightweight eyeware – which is still in a coming soon phase (for more on immersive augmented reality see my upcoming interview with Blair MacIntyre). Because, in the meantime, there are plenty of delightful and useful ways to augment our experience of the world – and not all of these augmented realities rely soley on smart phones as John S. Zeleck showed in his presentation on “Wearable Sensory Substitution Device for Navigation.” Also I had an interesting discussion at lunch with Ori Inbar about the use of audio for augmented reality projects.

Where 2.0 clearly demonstrated that we have an unprecedented amount of information from mapping our world, Ori Inbar noted in his conference roundup. Ori writes:

“My point is not a shocker: all we need is to tap into this information and bring it, in context, into people’s field of view.”

As Ori noted Earthmine and Velodyne’s Lidar showed off two new approaches to mapping the world that have potential to create new opportunities for augmented reality:

“Earthmine uses its own camera-based device to index reality, at the street level, one pixel at a time. They have just announced Wild Style City an application that allows anyone to create virtual graffitis on top of designated public spaces. However, at this point, you can only experience it on a pc!”

Lidar, Ori notes, has also embarked on a mission to map the outdoors. But, the question Ori highlights is:

“Can AR researchers harness these new approaches to index reality?”

Brady Forrest inspects John S. Zelek’s “Wearable Sensory Substitution Device for Navigation” at Where Fair before putting it on and being guided by sensory nudges at the cardinal points in the belt.

Coolest Mobile Locative Media App. at Where Fair

Atsushi Shionozaki of Place Engine – “a core technology that enables a device equipped with Wi-Fi such as a laptop PC or smart phone to determine its current location,” demoed the coolest location aware mobile app in Where Fair – Oedo Yokai. Working with ethnologist, Dr. Hiro Kubota and artist Atsushi Morioka, “Oedo Yokai” is Koozyt’s “first attempt to cross IT (Location Information) and Folkloristics.”

“The Japanese “Yokai” are known to dwell and appear at specific locations. They can frequently be seen within the grounds of shrines and temples, believed to be the border between this world and the afterlife, or in more common places like on a hill or at a crossroads. If the “Yokai” symbolize the mystery, legend, and lore associated with places, as our interests fade from actual locations, the rol, es they play in modern day society will diminish, and the “Yokai” might then cease to appear at all.”

I love this idea of bringing the ancient spirits of place back into our lives with our new tools of location awareness.

Odeo Yokai also reminds me of Aaron Straup Cope’s work on “the idea of every spot being a “history box” which he explained is “one of the threads behind the ‘nearby’ project at Flickr.”

The Food Genome

I cannot end this roundup of WhereWeek without a mention of The Food Genome.

“Food Genome is a big hungry brain that scours the internet, trying to learn everything there is to know about food.”

Watch out for the upcoming launch of this project, it stole the show with an exciting presentation at WhereCamp. You can follow @foodgenome on Twitter now.

To get one of the gorgeous Food Genome brochures you had to ask Mark Powell a good question. Notice an eager hand reaching out in the picture below. I asked, “how would the basic building blocks of the food genome be licensed?” I got my brochure and a rain check on an answer to my question.

The Ubiquitous Media Studio

Another highlight of WhereCamp was hearing from Gene Becker about his new project, Ubiquitous Media Studio which will be located in Palo Alto. The project is still in the early stages of devlopment but it sounds really exciting. I am looking forward to being involved from East Coast.  If you’re curious where this is going, follow @ubistudio on Twitter to stay updated.

Nathan Freitas holding a Peek with Oliver+Coady partner David Oliver talking to fans at New York Tech Meetup – Mobile Meets Social

Volunteerism and participation in public life seem to come naturally to Nathan Freitas. Nathan is one of the leading innovators/developers in NYC in mobile strategy/design (for more on his Android development read on). And he is much in demand as speaker who shows others how to realize their mobile experience and design dreams (for upcoming speaking engagements see Nathan’s blog). But also Nathan has spent much of the last ten years working on new ways for causes and non profits to benefit from technology.

Most recently Nathan has started working part time for the NY Senate under, “Albany’s King Geek,” the new CIO Andrew Hoppin:

“The CIO team is organizing training sessions for senators and their staff on social networking platforms and how to pay attention to online feedback. Last week, they hired mobile specialist Nathan Freitas to create new phone applications that will allow citizens to get government news on the go.”

Also, Nathan is currently supporting engineer on, The Extraordinaries, a smart phone application that explores territory “beyond the flattening tendency of online relationships” (see this list from Andy Oram of the Questions on Government participation).  The Extraordinaries is Ben Rigby and Jacob Colker’s prize winning project  – “a smartphone application that delivers volunteer opportunities on-demand.”

Ben’s post, Information Age Volunteerism – Open Sourced! Crowdsourced! and the extensive comments give a detailed analysis and critique of this brilliant and creative new approach to volunteersim in the information age.

Nathan, in my view, is a great example of how to “do well by doing good.” And, I am particularly excited by the work Nathan and his partner in Oliver+Coady, David Oliver, are doing on Android, e.g., Nathan’s new gReporter – opensource, geotagging, media capture report client (you can download the source here).

I first met Nathan when I interviewed him about Cruxy in 2007 (see my post, The Mixed Reality Metarati and “Destroy TV:”  Merging Art, Technology, Politics and Play).  Nathan recently announced that “the fat lady has just uploaded her last song.” Cruxy was an innovative distributed music venture Nathan started with Jon Oakes.  Although, as Nathan explains, Cruxy “never really broke through in the way we hoped.” Nevertheless Cruxy seems to have been a fertile garden for ideas that are coming of age in Oliver-Coady’s current mobile experience endeavors.  As Nathan explains, “the world, including Apple and iTunes, has shifted to embrace some of the ideals we have always had – open formats, more ways to distribute and promote online, more avenues for niche content to be discovered and heard.” Cruxy’s technology platform, built by the incomparable Will Meyer:

“was a great success in my mind, being one of the first to fully embrace Amazon’s cloud and provide a widget-based commerce system that actually worked!”

Nathan has a new company, Oliver+Coady. But Nathan told me that he feels he is over his “start up phase.”

Nathan Freitas: I am just tired of the term “startup.” I’m more interested in being defined as person than a member of a corporation. Also I am more interested in the ideas of cooperatives, and have been working on this idea (see here for more on the New York Creative Cooperative ).

Tish Shute: You do a high percentage of non profit work. Are you still managing to keep the home fires burning in the economic downturn?

Nathan Freitas: There is definitely profit to be made in non-profits because even if you only get paid half of what you get for corporate work, it is worth it in terms of fulfillment, ego, respect, and general contribution back to the planet. However, I’ve also been investing time & energy w/o pay into thinking about how causes can benefit from technology for over ten years. So its not just something you decide to do one day, and suddenly are successful.

Tish Shute: What are some of the highlights of your non profit work recently?

Nathan: Well, The Extraordinaries project is definitely a highlight. It is focused on a whole new approach to volunteering and winning the first prize at the WeMedia Conference for the non-profit tech category was a great validation of the work. I am just a supporting engineer on the effort, which was founded by my good friend Ben Rigby (a longtime non-profit tech guy as well) and Jacob Colker.

Ben wrote this excellent book on mobile tech and organizing, Mobilizing Generation 2.0 He’s done a ton of mobile work with youth voters via his non-profit, Mobile Voter.

The Extraordinaries is really taking all of our joint experience and putting it into a whole new system that is meant to go beyond generic email blasts that just ask you to “send a fax” or “send a link”. it gives people specific tasks they can accomplish on their phone or in their local area using their phone.

Tish: Did you do Twitter Vote Report with Ben too?

Nathan: Oh, no, Twitter Vote Report was with a different group of folks…mostly east coast-based, organized by the TechPresident.com blog. But Ben and I worked on SMS efforts for the 2004 election. We sent 40,000 messages out to SEIU labor members and MoveOn members… really the first time SMS was used in a wide-scale manner to help get out the vote on election day.

Tish: Do you have a new mobilization project planned?

Nathan: Its all about The Extraordinaries right now. We’ve got a big launch coming in June, and are working actively to add more causes that can benefit from volunteers and organizations that have volunteers but don’t know what to do with them.

Tish: I was just looking at your post on Peek too.

Nathan: Yeah… fortunately that is a completely  “for profit” gig.  But I like the company a lot, and think their spirit of providing access to email at a very low cost plays well with the non-profit world.

Tish: So it isn’t just iphone apps that are paying the bills?

Nathan: Nope. iPhone is just an aspect. Everyone is so obsessed with it and how to strike it rich quick, but in the greater scheme of things, there is a huge ecosystem of mobility out there for you to find a niche in, if you are looking.

Tish: Are you able to monetize your work on Android yet?

Nathan: here and there… releasing some for pay apps soon, also including “free” Android ports in some high-profile iPhone apps we hope to have out soon. Some successful iPhone app developers are looking for people to port their apps to Android, as well.

gReporter – opensource, geotagging, media capture report client

Tish: So what are your hopes for Android development in general and your gReporter app in particular?

Nathan: I think Android represents right now what Linux on desktops did in 99 or 00  Though as we all know, cycles of technology seem to speed up. There is huge interest in it at the academic level and there is also a genuine interest in its use by non-profit/development agencies working around the globe.

You have to jump through hoops to get an unlocked, open iPhone w/o contract. Android provides an alternative solution to this, that acts more like a true platform, and not just a consumer product.

Tish: At the moment the Android market place is only for free apps right?

Nathan: No, it now supports paid apps. I just bought one today for $2.99

Tish: What did you buy?

Nathan: An app that allows me to turn my G1 phone into a WiFi hotspot sharing my 3G connection to anyone who connects.

Tish: So what are the most important aspects of Android in your view?

Nathan: There are  two sites to help demonstrate what is really going on with Android that makes it significant

1) Open Intents – this is the ecosystem of developers, all creating services and apps that interoperate, share data, and generally build a very rich Microsoft style platform:
except all these are open-source and built by lots of small developers and not one big corporation.

2) Android on HTC – this is the home for all the efforts to port Android to pre-existing HTC/XDA mobile phone hardware. You can see the status of ports here: http://wiki.xda-developers.com/index.php?pagename=Android_devices  Imagine… taking an old Windows Mobile HTC phone, and then popping in an SD card that reformats it over to Android brand new phone!  For much of Asia, India and Africa, there is huge interest in this.

Tish: Nice! You mentioned earlier that you are thinking of doing SDK for the android sensor API’s?

Nathan: That would be part of the geo report app… expanding it to capture all sensing data and report that when you submit your text, photo or audio report.  Right now it just detects your lat and lon, but no reason it couldn’t also check your compass, altitude and whatever other data the device might offer.

Tish: So what will your geo report do now?

Nathan: It allows you to submit a text, photo or audio report, tagged with geo coordinates, timestamp, and basic user info (name, email, home location, etc) to whatever server it is configured to us. it is the latest release of code used for the TwitterVoteReport and InaugurationReport efforts.

There is also just a lot to learn or use from the code itself, which is available at: http://github.com/natdefreitas/georeport-android

Lots of little lessons learned packaged up into a functioning application

Tish: How many sensor APIs does android have?

Nathan: http://developer.android.com/reference/android/hardware/SensorManager.html

int SENSOR_ACCELEROMETER A constant describing an accelerometer.
int SENSOR_ALL A constant that includes all sensors
int SENSOR_DELAY_FASTEST get sensor data as fast as possible
int SENSOR_DELAY_GAME rate suitable for games
int SENSOR_DELAY_NORMAL rate (default) suitable for screen orientation changes
int SENSOR_DELAY_UI rate suitable for the user interface
int SENSOR_LIGHT A constant describing an ambient light sensor Only the first value is defined for this sensor and it contains the ambient light measure in lux.
int SENSOR_MAGNETIC_FIELD A constant describing a magnetic sensor See SensorListener for more details.
int SENSOR_MAX Largest sensor ID
int SENSOR_MIN Smallest sensor ID
int SENSOR_ORIENTATION A constant describing an orientation sensor.
int SENSOR_ORIENTATION_RAW A constant describing an orientation sensor.
int SENSOR_PROXIMITY A constant describing a proximity sensor Only the first value is defined for this sensor and it contains the distance between the sensor and the object in meters (m)
int SENSOR_STATUS_ACCURACY_HIGH This sensor is reporting data with maximum accuracy
int SENSOR_STATUS_ACCURACY_LOW This sensor is reporting data with low accuracy, calibration with the environment is needed
int SENSOR_STATUS_ACCURACY_MEDIUM This sensor is reporting data with an average level of accuracy, calibration with the environment may improve the readings
int SENSOR_STATUS_UNRELIABLE The values returned by this sensor cannot be trusted, calibration is needed or the environment doesn’t allow readings
int SENSOR_TEMPERATURE A constant describing a temperature sensor Only the first value is defined for this sensor and it contains the ambient temperature in degree centigrade.
int SENSOR_TRICORDER A constant describing a Tricorder When this sensor is available and enabled, the device can be used as a fully functional Tricorder.
float STANDARD_GRAVITY
with a few easter eggs as well
GRAVITY_DEATH_STAR_I
SENSOR_TRICORDER

Nathan: They are all in the API however, there isn’t hardware to support all of them yet… for instance TEMPERATURE is not yet supported
nor is LIGHT.

Tish: and errr what is gravity_deathstar

Nathan: It is a value representing the fictional gravity on the Death Star from Star Wars – geek humour

Tish: That makes me think of this great essay by Julian Bleeker, Design Fiction: A Short Essay on Design Science, Fact and Fiction:

“When you trace the knots that link science, fact and fiction you see the fascinating crosstalk between and amongst ideas and their materialization. In the tracing you see the simultaneous knowledge-making activities, speculating and pondering and realizing that things are made only by force of the imagination. In the midst of the tangle, one begins to see that fact and fiction are productively indistinguishable.“

Picture below is Nathan playing his dream ukulele – designed using the free, open-source Inkscape vector drawing tool (see his open-source Ukulele plans here)
See Nathan’s blog for the whole story of how the Flying V Rockin’ Ukulele Design he posted to Thingiverse a few weeks ago, after being inspired by Bre Pettis’ talk at ROFLThang materialized at the  NYC Resistor “amazing workshop laboratory in Brooklyn where they let anyone come over and hang out at, to learn how to make, build and fabricate pretty much anything. They also have a laser (aka “LAAAZZZOOOR”) which you can think of as an automagic thing cutter-outer!”

so this….

became this …

Nathan and David presented Coovents at NYTM – Mobile Meets Social. They had a large group of questioners surrounding them (see picture below).   I talked to David after the presentation.

David Oliver was a software architect, user experience designer and product manager in the areas of mobile/wireless and electronic payment at IBM for over a decade.  Most recently, he lead the effort to productize a mobile client for IBM’s Lotus Connections enterprise social networking suite.  As a software architect, David was often technical lead for IBM’s business partner relationships with mobile device manufacturers.  Prior to IBM, David was co-founder of the Internet’s first “micropayments” company, Clickshare.

Talking with David Oliver

Tish Shute: How are smart phones are causing us to rethink what networked online relationships are all about.

David Oliver: You know these [mobile] devices are .. there’s a long time we tried to pitch that we’re going to treat them like they’re PC’s, or they’re just like anything else. But they’re really not. It may be the same coding style but the way you think about using them is entirely different. And the way you think about your program. so if you use html, java and that kind of stuff, yes it’s same code type but the way you think about it is entirely different. And to me these little devices make what you said [relationships inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.] a lot more possible than a PC. because in a PC you almost have to sit in front of it and like it controls you. But the device is so little and there’s almost no user interface by comparison. You got to be very smart how you build something so that it’s almost invisible. And of course that’s the beauty of the iphone, Apple will tell you. The idea of ubiquitous computing. Ubiquitous what? Am I really computing? I don’t feel like I’m computing. I feel like I’m interacting or something.

I think twitter is very cool. The real way it’s cool is that there’s no required client. You can access Twitter any way you want.  You can imagine other ways to use it. Tweet Deck happens to be a nice for now. What I like about Twitter is, if you give it a tiny bit of thought, the Twitter network’s complete white noise, just like the internet itself. If you put a probe on the internet it’s all white noise, it’s all unordered packets. It makes no sense. So it’s cool that Twitter is at the level of little bitty conversations, but collectively all white noise. Totally meaningless white noise.  There’s some neat things going on, but I think we haven’t seen barely the first of what you can do with Twitter.

The way I see it is it’s like instant messaging where you don’t instant message to someone you instant message to the network and there are listeners. So normally in the old world of IM like AOL IM I would say Tish let’s talk and I kind of like grab you. Then it’s a narrow pipe you to me. You can add a few people in and make a little group, and that makes a bit of a closed network. But with twitter you just like talk into the air as if I were standing over there and you had a twitter client here, we could have the same interview. Because I would be watching you OH I see Tish’s question. I’d be over there talking  and you’d be picking me up over here. I’ts like you’re talking into white noise, like at this bar. You choose to hear me, this guy is not choosing to hear me right now.

Tish Shute: So what does Android bring to the party?

David Oliver: They have the notion that you have a telephone platform that’s open, and that everybody can use. And it’s got a variety of sensor data – not just location but also accelerometer and compass and more.  So in theory you can almost broadcast that data. It’s connected to a network. It’s easy, open API’s to get at that data. But the question is who are you going to broadcast it to or who are you sending it to. What are they going to do with it? How are you going to control it, and make sure people don’t misuse it? As you heard with the services tonight, there’s a central kind of service necessary to filter and rebroadcast that stuff back out to places that need it, or can use it, or you want to have use it. I think the mobile device is only one piece of this. Nat and I always talk about well we do mobile applications but a portion of it is on the server. And coordinating with the people or the group or the central resource that brings all this data together.

Tish Shute: There seems to be a lot of new location based services – platforms to aggregate location based data being developed (e.g. xtify and viaplace). What do you think about the direction this development is going in?

David Oliver: It’s not conventional wisdom but it’s one of these things where when a crowd of people does something, and that means people themselves are the service providers,  when they all get together the net effect is greater that the individual effect would be. Pooling together makes more sense than doing it individually. Its a little bit like an advanced version of you have to have a password for every single site and you manage your passwords. Location is the same way. If you had to give every single website that you enjoyed your location data or tell them how to get it, what a huge pain. So they’re offering a way to do that in a more general sense. There are humongous privacy issues though. Just like passwords. Would you really trust a place that held all your passwords centrally?

Even with the most basic level of calling. Now that you can call from anywhere. Largely people are getting into a mode where their mobile phone is them. It’s always with them. That’s how you reach me. Forget the home phone, the work phone it’s just a mobile phone. You have an address attached to you, an address I can reach you at that’s location independent. So there some beauty in that and it’s very freeing. It makes your location unimportant, you can call me anywhere. You can text me anywhere, message me anywhere. You can be anonymous. My son told me something recently. “I love going to New York City because I can just walk around and nobody knows me. I’m completely anonymous. That’s the coolest thing”, he says. At one level that is a good thing and a lot of good things can happen that way. But this new thing is sort of the flip side where everybody knows your location. And we haven’t figured out if that’s a good thing yet. But we’re in the throes of that whole changeover happening. And we’ll see. There’ll be some misuse. I’m not an advertising guy, so the fact that everything’s got to be ad supported makes it potentially very creepy and very dangerous. So we’ll see how that evolves.

Is there any model where you can go “Oh this is just like ‘S’”? I don’t see where that’s possible. It’s a new world. Where you’re exposed all the time, potentially. And how do you figure out either as an individual or a larger group, society or whatever, when that works and when that doesn’t. And you know there’s going to be some mis-steps probably. But the tangibility creates some of these interesting opportunities, there are just some amazing things that could happen, really, really good things. But we’re not going to get there in one step.

One of the things that was really a killer for privacy and a killer for in some ways the internet, was during the dot com bust. Prior to the bust, there were web sites that you’d given your name and email, and they said “we promise to preserve this privacy.” But as soon as those companies went bankrupt, their email list was gold. It was value. And a bankruptcy judge, in a court in Delaware, created a legal basis to sell that data. Those things that were formerly private were no longer private – “no no no that’s got value. I’m going to sell it so the shareholders get their money.” So all these web sites who had lists of user names that they promised were private, became public information. That was one of the biggest blows to privacy in the history of the internet. That’s going to happen again and again. Like if MeetMoi goes out of business the likelihood is all your shit’s going to get sold. I’m sorry it’s all going to be sold. It’s all a big joke. And that’s why central services are horrid, and I don’t like anything about a central service.

There are some pragmatic things about the way routing on networks actually works and the fact that the internet has gotten very centralized itself. The core ideas of the early internet which were essentially a survivable telecommunications network, remember it was the defense department that did the original internet? So the original idea of the original internet was survivability. The Russians could bomb the daylights out of the United States, territorial U.S. and we would still have a survivable network. That was the idea. And therefore all the nodes were dispersed and did not count on each other, and could reroute. Well now one company UUNET or whatever they are they own the whole thing. And you can look up all their locations on some internet database. 18 well placed bombs and the whole internet goes down. That’s what happens over time.

Well the whole cloud thing is also kind of a myth. It’s a very neat sounding term, and some aspects of it are different and new. Nate and I do a lot of cloud computing, it’s all on Amazon.

But we’ve always had that. That’s called time sharing. Strictly speaking it’s a thin contractual accompanied by a much much much easier application programming interface. That’s what cloud computing is. It’s a very skinny contract. Timeshare was a  huge contract. Literally it’s legal and a little bit of API ease. It’s just timesharing. But at Amazon and the other ones too, you’re not responsible for your node going down. If it goes down, they push it somewhere else automatically. Your disk goes down. You’re not responsible for backing up your disk, it’s already on 14 copies on 8 continents. They do that. So it’s a higher level of service. Nate and I have this thing called slice host. And we’ll probably build some services on it, and if they get popular, it’s like a vending machine. You just drop in a dime, they give you another slice. No contract at all. It is growth and learning about old ideas. Like this whole idea of software as a service. The company called ADP Automatic Data Processing, who basically in short do payroll for everybody. It’s software as a service. It’s been going on since 1952 or something. It’s more like a reconception using modern tools. It’s like virtual worlds are a different thing. That’s a whole different beast.