Augmented World Expo 2013 was really an amazing experience. I’m co-founder and co-organizer of the conference, along with Ori Inbar, so it has meant a lot to me to see our event grow over the last four years, and thrilling to make such a big splash this year.  There were 1,163 attendees, and the expo show cased an ecosystem of emerging technologies – augmented reality, gesture interaction, eyewear, wearables, and connected hardware of  many stripes, that mark the beginning of natural computing entering the mainstream. It was a unique opportunity to get up close and personal with what it feels like to be an augmented human in an augmented world!

Videos of AWE 2013′s 35 hours of educational sessions and inspirational keynotes are now available on our YouTube channel. I am sharing my own talk (my slides are also up on slideshare here), and a few of my favorites in this post, but there are far to many to post here, so please browse further on the Augmented World Expo youtube channel.

One notable high point of AWE2013, for me, was the showcase sponsored by Meta, a startup developing the first device allowing visualization and interaction with 3D virtual objects in the real world using your hands. It was made possible by the generous contribution from the private collections of Paul Travers, Dan Cui, Steven Feiner, Steve Mann, and Chris Grayson, and passionate volunteers who are helping advance the industry. Sean Hollister of The Verge did this excellent report on the eyewear showcase 35 years of wearable computing history at Augmented World Expo 2013
Also for more on Meta see this article by Dan Farber.

My colleagues at Syntertainment, Will Wright, Avi Bar-Zeev, Jason Shankel, and LaurenElliott all gave great talks. Ironically, we’re not building augmented reality apps or H/W. We all just happen to continue to be very interested in the field.  

Thank you to everyone for supporting the event!

The press coverage was truly extensive:

In the shadow of Google Glass, an augmented reality industry revs its engines
The Verge, Sean Hollister, June 9, 2013, 271 Tweets

The next big thing in tech: Augmented reality
CNET, Dan Farber, June 7, 2013
Pick up on Current News Daily
350 Tweets

AWE 2013 Conference Report: Augmented Reality and Marketing
The Persuaders Marketing Podcast on Dublin City FM, June 23, 2013

AR Dirt Podcast – Ori Inbar AWE2013 Extravaganza Recap
AR Dirt by Joseph Rampolla, June 18, 2013

35 years of wearable computing history at Augmented World Expo 2013
The Verge, Sean Hollister, June 9, 2013
7 Tweets

Augmented Reality: Bruce Sterling, keynote at Augmented World Expo 2013
Wired, Bruce Sterling, June 9, 2013
9 Tweets

On the road for VR: Augmented World Expo 2013
Doc-Ok, Staff, June 7, 2013
3 Tweets

My Interview from Augmented World Expo 2013 [VIDEO] Wassom.com, Brian Wassom, June 7, 2013

Augmented World Expo
ZenFri, Staff, June 7, 2013

AWE2013: Hardware for an augmented world
FBNSantos.com, Felipe Neves Dos Santos, June 6, 2013

Augmented Reality Will Be the New Reality
InvestorPlace, Brad Moon, June 6, 2013

Wearable computing pioneer Steve Mann: Who watches the watchmen?
TechHive, Armando Rodriguez, June 6, 2013

Expo puts augmented reality in the limelight
ABC 7 News, Jonathan Bloom, June 5, 2013

These OLED microdisplays are the future of augmented reality
DVICE, Evan Ackerman, June 5, 2013

Visualized: a history of augmented and virtual reality eyewear
Engadget, Michael Gorman, June 5, 2013

Wikitude announces Wikitude Studio and in-house developed IR & Tracking engine
PapiTV, KC Leung, June 5, 2013

Augmented reality expo aims for sci-fi future today
USA Today, Marco della Cava, June 5, 2013

Augmented Reality: High Dynamic Range (HDR) Video Image Processing For Digital Glass
Wired, Bruce Sterling, June 5, 2013

Will Wright at Augmented Reality Conference: Don’t Augment Reality, Decimate It
AllThingsD, Eric Johnson, June 4, 2013

Philip Rosedale’s Second Life with High Fidelity
CNET, Dan Farber, June 4, 2013

Google Glass competitors vie for attention as industry grows
PC World, Zack Miners for IDG News Service, June 4, 2013

4D Augmented Reality Leader Daqri Announces $15 Million Financing
Press Release, June 4, 2013

CrowdOptic Powers Lancome Virtual Gallery App, Crowd-powered Heat Map
TechZone 360, Peter Bernstein, June 3, 2013

Augmented humans, enhanced happiness?
Crave Culture, Angelica Weihs, June 2, 2013

Metaio & Vuzix to Showcase AR-Ready Smart Glasses at the 2013 Augmented World Expo
Press Release, May 30, 2013

Four ways augmented reality will invade your life in 2013
Quartz, Rachel Feltman, May 30, 2013

Augmented Reality: Augmented World Expoâ„¢ is next week
Wired, Bruce Sterling, May 28, 2013

Strike it Rich with Cachetown and AWE 2013 Playing the Gold Rush 49’er Challenge In Augmented Reality
Press Release, May 24, 2013

Local Community College Student Headed to Silicon Valley to Learn More about Augmented Reality
St. Louis Post-Dispatch, Staff, May 24, 2013

Explore an intricate labyrinth with smartphone AR
CNET Australia, Michelle Starr, May 21, 2013

Dartmouth firm lands super app
Herald Business, Remo Zaccagna, May 21, 2013

Augmented World Expo 2013–The Future of Augmented Reality
Silicon Angle, Saroj Kar, May 17, 2013

I’m really excited that we opened a call for proposals today for Augmented World Expo (registration opens February!).  Our edgy conference on augmented reality has morphed into the world’s first Expo about the augmented world.  If you loved ARE you are going to find Augmented World Expo the most important event of 2013, and if you never got a chance to attend before register early to reserve your spot!

“The way we experience the world will never be the same. We no longer interact with computers. We interact with the world. A set of emerging technologies such as augmented reality, gesture interaction, eyewear, wearables, smart things, cloud computing, and ambient computing are completely changing the way we interact with people, places and things. These technologies create a digital layer that empowers humans to experience the world in a more advanced, engaging, and productive way.

Augmented World Expo will bring together the best in augmented experiences from all aspects of life: health, education, emergency response, art, media and entertainment, retail, manufacturing, brand engagement, travel, automotive, and urban design. It will be the largest ever exposition demonstrating how these technologies come together to change our lives and change the world.

Registration will open in February.”

Augmented Twitter

Presenting Augmented Twitter (see video and slides above) at Jeff Pulver’s 140 Characters Conference (#140conf ) was super fun, and great video makes this a conference that you can enjoy catching up on after the fact.  Jeff Pulver does an excellent job of keeping people to a challengingly short format.  Even I managed to bring my talk in under 5 mins!

#140conf is a real time mobile social crowd, and pretty attuned to Augmented Reality.  Everyone had heard of Augmented Reality in the audience, and while most had never tried an AR app, nearly everyone used a mobile social app like, Four Square, Gowalla, or Latitude.   As Dan Harple (@dharple) – Executive Chairman, GyPSii, said in his  interesting presentation, Evolution of Location and Places,  “everyone get’s connection, and that connection in real time is the thing if we can get it, and that real time connection is innately mobile.”

ARwave aims to push mobile, social, real time connection even further with augmented reality.  As Anselm Hook puts it so brilliantly in his presentation at EComm, “AR is about publishing “verbs” – interactive, actionable, digital agents not publishing 3D models.”  I have some mega posts brewing on this topic.  Augmented Reality will need to support publishing game like behavior, and digital agents that can embody a set of actions and reactions.

This need for augmented reality to publish behavior, and to share and integrate, in one view, multiple real time data streams are just some of the reasons AR Wave uses an open federated protocol.  Federation is also particularly important for augmented reality because, as Anselm pointed out at WhereCamp,  AR will certainly demand very efficient distribution of state change at the systems level –  to move the computation to its lowest latency.

The only other cloud over our Augmented Reality party at #140conf  was that #ashtag kept our co-panelist and panel chair from joining us.   Rita J King, @ritajking, who is Innovator-in-Residence at IBM’s Analytics Virtual Center, the “General of the Imagination Age,” and Dancing Ink Productions, and Joshua Fouts, @josholalia, “Cultural Attaché,” and Chief Global Strategist of Dancing Ink, were on a 5 day trek out of #ashcloud, and, sadly, not there for our panel.

Bu Twitter, once again, was a life line in a time of crisis connecting them to TEDxVolcano, an impromptu unconference with must see presentations from Rita and others, see Rita’s blog for more.

So the two of us carried the flag for  Augmented Twitter.  Myself and Jerry Paffenfdorf, futurist, artist, entrepreneur and swell guy – the co-inventor of the most famous real time social web system you have never heard of (actually I tried and loved it in alpha testing, before it was quote “shut down by blood thirsty investors”).

Now Jerry lives in Detroit Michigan where he works on the Loveland Micro-real estate project which is the simplest, cheapest, funnest way to become a land owner.  At a dollar a square inch it mixes video games and real estate, like Farmville for urban development.

Joshua and Rita, our very virtual panel mates, are the first and largest inchvestors, and creating their own micro city within the project.  Jerry is one of the most creative and original thinkers on the planet, so treat yourself to glimpse of what is on his mind in the video above – Loveland, 3D mailbox, canned augmented reality, and the relationship of virtual worlds to the real time social web.

Jerry also hat tipped one of the most captivating projects and presentations of the conference, Alon Nir’s, The Story Behind @TheKotel, “Tweet Yr Prayers!”  What a great story about the power of Twitter to reach out into the world, and beyond!  I got a chance to chat with Alon at #140conf, and I found out he is brother of augmented reality guru, Rouli Nir, @augmented.  Rouli is known for his sharp and comprehensive AR commentary on Augmented Times and Games Alfresco.  Cool family!

Before I close this post, I want to mention @AndyDixn’s talk on the prison sysetm, A conversation with Andy Dixon: What the prison yard & twitter have in common.  This conversation, I think, is a great example about what makes #140conf special.  As @nwjerseyliz pointed out, we, “hear few voices from those who’ve experienced that side of the issue.”

Thank you @jeffpulver for creating such a cool staging for so many diverse voices.

And before I close here is what the only slide I didn’t have time to show said!

If you liked “Augmented Twitter”
Don’t miss Augmented Reality Event!

2 days, 3 tracks, 40 AR companies, 76 SpeakersArt! Magic! Competitions! Awards!Bruce (the Prophet) Sterling, Will (The Sims) Wright, Jesse (Gamepocalypse) Schell, Blaise Aguera y Arcas (Microsoft Bing) and You! The sneak preview of the schedule is here.

Register today at Augmented Reality Event.com

Discount code for @140 attendees, (and readers of this post!) TISH245 activates $245 price for full conference.

See you there!

Picture on the left Mirrorshades, picture on the right a Metroid Hud.

“Augmented Reality is like a Philip K Dick novel torn off its paperback rack and blasted out of iPhones,” Bruce Sterling in Beyond the Beyond

“a techno visionary dream come true – those are rare, really rare, you have to be patient,  it’s super cyberpunk”… Bruce Sterling, “At the Dawn of the Augmented Reality Industry.”

The Dawn of the Augmented Reality Industry continues to brighten, and now we have two augmented reality companies, Total Immersion and Ogmento, firmly established in Hollywood – the dream mother of so many of our augmented realities.

Ogmento is the most recent of these two pioneering augmented reality companies to set up shop in LA.  Total Immersion’s CEO Bruno Uzzan moved to LA from France two years ago, although he still has a fifty person RandD team in France.   Total Immersion began 10 years ago in the quiet, lonely, hours before the dawn of an AR industry.  But Total Immersion’s AR toys for Mattel, and augmented reality for Topps baseball cards, fired CNet writer Daniel Terdiman up enough to say, “I have seen the future of toys, and it is augmented reality” (see full post here on CNet).

Recently, I talked with  Ori Inbar, one of the founders of Ogmento and  the premier augmented reality blog Games Alfresco about his new venture in Hollywood. Bruce Sterling, @bruces, had some fun with my invention of brand new augmented reality trade jargon here!  Ori pointed out Ogmento brings two important new facets to the rapidly growing augmented reality field: firstly they are bringing leadership from veterans of the entertainment industry into augmented reality development. Brad Foxhoven and Brian Selzer from Blockade have partnered with Ori on Ogmento.  And, in an another important step forward for a young industry, Ogmento announced they will be acting as publishers for a fast growing cohort of augmented reality application developers and helping AR development teams out there bring their concepts to the market.

So I was very happy also to have the opportunity to talk with Brian Selzer.  Bruce Sterling pointed out in his seminal sermon from the flatlands at the Layar Developer Conference, AR is kind of a “Hollywood scene.” We have seen the web early adopter/developer/blogger community  embrace augmented reality browser experiences in recent weeks in an awesome wave of enthusiasm. Are Hollywood creatives equally smitten? For the answers see the full interview with Brian Selzer below.

Brian Selzer (www.brianselzer.com and twitter – brianse7en ) has an extensive involvement with emerging platforms:

“from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC [user generated content] projects (mashade.com, instafilms.com).”

I began my own career in Hollywood doing motion control photography and creating software that bridged the language of robotics and servo motors with the visions of  film directors. Eventually our little company, NPlus1, moved on to 3D vision systems and image recognition stuff.  So yes, I have been really, really patient waiting for this particular techno visionary dream.  And, while I have been waiting for augmented reality to manifest, I have grown to love the internet.  But now, how awesome, It is OMG finally for mobile AR!

Augmented reality is busting out all over – through our laptops, our phones, on the streets, toys, baseball cards, art installations, sticky light calligraphy and more.

Many of my questions to Brian were directed at at how and when we will see augmented realities with near field object recognition, image recognition and tracking and, of course, the illusive eyewear.  As Bruce Sterling points out we are just at the very, very beginning – the dawn of an industry.  I created the photomontage below on the right to compliment Tonchidot’s illustration suggesting the evolutionary inevitability of holding our phones up (below on the left).  The Evolutionary Reality of AR will not end there.  It is just a step into eyewear, hummingbirds or Nano Air Vehicles, and more…….

The Evolutionary Reality of AR

Cartoon on the left by Tonchidot on the right a collage of a stock photo and the Pentagon’s Robot Humming Birds – “Nano Air Vehicles.”

While we finally we have, an affordable mediating device with the horse power, mindshare and business model to bring AR mainstream with the iphone.  The much anticipated Apple 3.1 Beta SDK to be released in September will not, I am sure, open up the Video API at the levels that augmented realities with near field object recognition and tracking require (I would love to be proved wrong though). But the magic wand to deliver even tightly registered AR graphics/media (that require a lot of CPU and GPU) to a wide audience is in our hands, so full access to may not be far off. And others, of course, can/will/might knock the iphone off its current pedestal.  AR made it’s mobile phone debut on the Android after all.

Like everyone else who loves AR, I wish that Apple would open up faster (and I wish Android would manifest on some rocking hardware). But we will see enough of the iphone Video API open for the next generation of mobile augmented reality games and applications to emerge in the coming months.

One of these will be Ogmento’s. Although Ogmento is in stealth mode, they have released a teaser for their first game, “Put A Spell,” developed by ARBalloon – screenshot below.  Ori did reveal to me in this interview that they are doing image recognition and using the Imagination AR engine.

As Brian notes, Hollywood has had the AR bug for a long time. AR has been everywhere in Science Fiction Movies and video games. Nintendo’s SPD3 head Kensuke Tanabe, “effectively the man in charge of overseeing all the Metroid franchise underneath original co-creator Yoshio Sakamoto,” explains the story of Metroid to Brandon Boyer of Offworld here (an image of a Metroid Hud on the right opening this post) :

“the idea of the different visors you use in the Prime games to interact with the world: the scan visor, for instance, set the game apart from other first person shooters in that the player was using it to proactively collect information from the world, rather than having the story come to them passively, in the form of cut-scenes or narration. “Prime could have adventure elements with the introduction of this visor,” says Tanabe, “That’s how we came up with the genre — first person adventure, instead of shooter.”

But as Brian points out:

“the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear).”

Talking with Brian Selzer

Tish Shute: Bruce Sterling’s sermon at the Layar Developer conference, “At the Dawn of the Augmented Reality Industry,” was absolutely awesome. He spread the future feast/orgy of augmented reality before us  – and described many of the dishes we will tasting both delectable and diabolical.  One of the many things he points out is, AR is kind of a “Hollywood scene.” And, as Ogmento is one of only two augmented reality companies in Hollywood, I am interested to hear how it looks from your neck of the woods. We have seen the web early adopter/developer/blogger community  embrace augmented reality browser in recent weeks in an awesome wave of enthusiam – are Hollywood creatives catching the buzz?

Brian Selzer:  It was a thrill to hear Bruce Sterling mention Ogmento. I devoured all of his Cyberpunk books back in the 80′s, along with writers like Gibson, Rucker, Shirley… To me, sci-fi writers are the visionaries who define and influence our technological paths into the future. They make science and tech sexy enough to want to manifest those experiences in the real world. Clearly Bruce sees the AR industry as being sexy. I love that he called it “a techno-visionary dream come true… and super-cyberpunk.”  And yes, kind of a Hollywood scene.

Hollywood creatives caught the AR bug before they knew what AR was.  Look at science fiction movies and video games to see AR everywhere. Terminator, The Matrix, Minority Report, Iron Man.. the list goes on.  Look at any video game with an integrated heads-up display.  It’s clear Hollywood loves AR.  It’s only been in the past few months though that the light bulb has been lit and Hollywood is seeing that the software and hardware are here today to deliver these types of AR experiences in real life (to a lesser extent of course, but the path is getting clear). So yes, the buzz is here and it’s strong.  With that, we all have to be prepared for the good, the bad and the ugly as AR goes mainstream.

It certainly goes to show how young this industry is when Ogmento and Total Immersion are currently the only AR companies based in Los Angeles. It’s very exciting to be the only company right now demonstrating a natural feature tracking (markerless) iPhone experience in Hollywood. We are in talks to bring some very big brand and properties to the mobile AR space. The goal is to deliver experiences that create added engagement and value to the consumer.

Tish Shute: Also in his landmark sermon Bruce Sterling noted that augmented reality has been around for 17 yrs and now at last we are seeing the dawning of  an augmented reality industry. What inspired you to take up the challenge of launching an augmented reality company in Hollywood?  Oh congrats that Bruce Sterling name checked Ogmento in his list of companies that prove that this really is the dawn of an industry!

Brian Selzer: I’ve always been involved in emerging platforms… from launching dot com entertainment sites in the late 90′s to creating early versions of social gaming platforms, or bringing big brands like Spider-Man and X-Men into the mobile space for the first time.  Last year I was focused on bringing video game characters and worlds into the online space as UGC projects (mashade.com, instafilms.com). Working with all these great CG game assets, I continued to think about what’s next, and that’s when I started to follow AR very closely and started engaging with those who were pioneering in the space.

I remember swapping instant messages with Robert Rice (@robertrice) right after the 2008 Super Bowl.  We were not chatting about the football game, but rather about some of the commercials that aired during the event as a sign that AR was making its way into the mainstream.  A lot of people became aware of AR for the first time when the GE SmartGrid commercial aired.  There were all these YouTube videos popping up of people blowing on holographic wind turbines.

The commercial that really got me excited though was the Coke Avatar commercial.  In that commercial people in the city were sporadically being portrayed as their digital persona’s, avatars, gaming characters, etc..  For me that spot did a great job showing how many of us already have these ‘alter egos” that live in cyberspace, and how the line between these worlds can sometimes be blurred. I remember watching that commercial and thinking that is exactly the type of experience I’d like to create with mobile AR.  I want to overlap the virtual world into our every-day reality. Why cant I bring my World of Warcraft or Second Life persona with me into the real world?

I am big on the notion of “Games and Goals.” I believe that games have the power to motivate people in a very powerful way. By challenging ourselves while playing a game we can climb mountains.  Augmented Reality is the perfect platform to bring gaming into the real world.  By mixing the virtual world with the physical world, this added layer of perception provides a very powerful experience for something like a role-playing game.

One of my earlier social-gaming projects was a website called Superdudes.  This was a “Be Your Own Superhero” concept that celebrated and motivated kids to create superhero avatar/persona’s online, and we gave members all sorts of games, challenges, and rewards, some of which carried into the real world. The site recognized members for teamwork, creativity, volunteer work and things like that. So the Superdudes were often involved in charity events and benefits to help children.  Everybody called each other by their Superhero names, and the line between fantasy and reality were being blurred.  This project really got me thinking about what happens when you take positive role-playing like this and mix it into the real world.  I started to work on a plan for location-based activist missions for points and rewards, but never got to complete that. So I have some unfinished business here.

I think it would be fantastic to be able to show up to some type of fun event with friends, and everybody could see each others alter ego personas standing before them. When you can turn the world into a playground, and use the power of gaming to make a positive impact on the planet… well, I don’t think there is anything better than that.  These are the types of projects that drive me, and I think AR is the best platform to support these types of social gaming experiences.

Tish: Does Ogmento have any RPGs under development?  I noticed in the Google Wave on RPG someone has been working on doing something with the Dungeons&Dragons API.  I am interested in exploring the web of protocols underlying Wave as a transport mechanism for multi-person, mobile, AR experiences (not requiring downloads), on an open global outdoor AR network. If not Wave, what do you see as the potential infrastrucure and protocols we could harness for an open augmented reality network?

Brian:  Ogmento has a deep background in video games and we interact regularly with most of the major game publishers. As a company we are not so much developing our own RPGs right now, but rather exploring what mobile AR extensions make sense for existing brands.  There are many limitations to location-based gaming, but a global AR network is exactly along the lines we are thinking.  Lots of discussions are taking place on protocols, platforms, API’s, and there are numerous ways to approach this.  We need to be able to use what’s available now and continue to refine and customize for AR’s specific needs and issues as we progress.

In general though, Ogmento is focused on what types of experiences can be had today and over the next couple of years. I still think we are several years out from a truly open augmented reality network.  We are certainly looking at launching our own “Ogmented Network” which would support some fun treasure hunt type experiences, or add an entertainment layer on top of traditional outdoor marketing campaigns.

Tish: I don’t know whether you have read Thomas Wrobel’s ideas for an open augmented reality network that I just published here on Ugotrade.  The principals he talks about are very important for augmented reality to become a major part of our lives – .  Considering the difficulty open networks can pose for emerging business models how can we fund the development of an open framework for augmented reality?

“a future AR Network, I mean one as universal and as standard as the internet. One where people can connect from any number of devices, and without additional downloads, experience the majority of the content.
Where people can just point their phone, webcam, or pair of AR glasses anywhere were a virtual object should be, and they will see it. The user experience is seamless, AR comes to them without them needing to “prepare” their device for it.”

Brian: I think funding for these types of projects will definitely come from Venture Capital groups in the near future.  It’s early in AR, but the VC’s are watching and deciding which horses to bet on.  Until that time, it’s about service work, and developing AR experiences for others with what is possible today. That work will help fund internal development of original AR products, and platform development.

Tish: How did you get started with Ogmento?

Brian: My first conversation with Ori was actually about my interest in Location Based RPG concepts.   We had a long conversation about the possibilities with AR, and it was clear that we shared similar interests, but were coming from different complimentary backgrounds. The idea of collaboration was exciting, so we just kept talking until the timing felt right. Now, with Ogmento we bring a unique blend of AR development experience with a deep backgrounds in AR technology, animation, video games, entertainment, social media, etc.   I think this is a powerful mix that will allow us to do some great things.

It’s still so early, and things are just getting started in AR. There are only so many webcam magic tricks you can enjoy before you are ready for something else.  The location-based apps have the most potential in my opinion, which is why we are really focused on mobile AR.   We have some board-game type projects, which do not instantly scream location-based gaming, but if you look at something like the ARhrrr board game, you can see how much more compelling it can be when the game invites the player to be actively moving around during the experience.

Tish: I am interested in your perspective on how we can create the kind AR experiences that really embody what has always been so exciting about AR – the tight alignment of graphics and media with real world objects and ultimately a rich immersive 3D experience, so I am going to hit you with a bunch of those, “Is this really eyewear or vaporware?” questions.  The real deal eyewear changes everything!

While eyeware is a big challenge technically and aesthetically,  I am pretty sure that there are several outfits out there that can pull off the optics and projection. 
Will the entertainment industry get excited enough to put a major push into delivering the eyewear in short order instead of the 5 to 10 year project that some people still think it is?    The business development challenge is bigger perhaps than the technical obstacles perhaps? What is your view on this?

And, perhaps, the eyewear is a clear example of a need for partnerships. For example, we have seen efforts from companies like Vuzix and Lumus, and recently a Japanese Company, Masunaga.

I have no reports from people who have tried the Maunaga eyewear yet.  But,  limited by small field of view, and tethered, currently eyewear offerings, available at a reasonable price point, are not workable solutions for augmented reality experiences. But the problems are not insurmountable. What will facilitate the real deal?  


It seems that it is critical to start creating hardware relationships now. The industry is costly and slow moving and as Robert Rice put it to me in a recent conversation, “once the software cat is out of the bag, its going to go wild and if the hardware isnt there, its going to stutter.”

As Ori notes some of the hardware companies like Intel and others don’t seem to be paying enough attention to AR.  Ori points out they don’t see the demand yet.  But in order to create an awesome AR experience and demand from a mass audience, don’t we need to work in conjunction with hardware designers?

Brian: It’s fun to think about who will eventually deliver a great hardware solution for AR glasses. It will happen. It would be cool to see somebody like an Oakley or Nike partnered up with a company like Vuzix to deliver something people actually might wear in public.  Perhaps a hardware manufacturer like Apple or Nokia will bring us something like the iSight or the NGaze down the line.  I’d love to see a set of glasses designed by Ideo.  Microsoft or Sony are already playing with technologies like Project Natale and the EyeToy, so I think its only a matter of time before they deliver an eyewear solution. I would even look to the toy companies to eventually make an investment here.

Gamers will be the early adopters, and in a few years we may start to see people running around in the park wearing glasses with headsets, but it will be acceptable because it’s clear they are using them for a game.  It’s going to take a very sexy and stylish piece of hardware for everyday people to be willing to wear AR glasses in public while going about their everyday business.   It’s like the recent cover of Wired magazine where Brad Pitt is wearing a mobile headset in his ear, and the editors point out that even he can’t pull that look off, so why do you think you can.  When AR glasses come in designer frames, and you can’t tell them from non-AR glasses, to me that’s when things get really interesting from a mass-adoption perspective.   Compare how many people were carrying around a mobile phone in the 80s to now.  I think it will be the same thing with glasses.

I was in an AR pitch meeting the other week at a very significant media company, and brought up the point that today’s handheld Smartphones will eventually evolve into tomorrows Smartglasses. My comment was quickly shrugged off as sort of a sci-fi notion that was irrelevant to the business at hand.  Probably true, but I think it is important to understand where digital media and entertainment is going, so you can adapt quickly, and evolve into those spaces more naturally.  The more we see people walking around with their Smartphones in front of their face (like a camera), the sooner it will be that we make the jump to eyeglasses as a key hardware device for AR experiences.

At Ogmento, we definitely are working on AR experiences with the hardware and software available today.  We will get some product out this year, and 2010 will be a banner year for markerless mobile AR in general.  I think the entire AR community is looking forward to bringing this technology to the mainstream in the form of games, marketing campaigns, virtual docent apps, and much more.  It might not be the full experience we are all dreaming about for some time, but we can see the path and the true potential, and it’s pretty spectacular.

You mention the tight alignment of graphics and media with real world objects.  That is really our focus. A lot of well-deserved attention is going to the browser overlay “post-it” approach right now, which uses compass and GPS.  We are focused on markerless natural feature tracking, so once you identify something that is AR enhanced in your environment, you can interact with that integrated experience.  On an iPhone that can be as simple as using your touch screen to interact.  When you are wearing glasses, it becomes more about visual tracking. There are lots of smart people thinking through these issues. Many of which you have interviewed. It is my hope that there are exciting collaborative efforts to be had in the coming months to get us all there together and faster.

Tish: Bruce touched on some of the hard problems that have to be solved for augmented reality – and he noted for instance security needs to be tackled in the early stages. Robert made a nice list, “privacy, media persistence, spam, creating UI conventions, security, tagging and annotation standards, contextual search, intelligent agents, seamless integration and access of external sensors or data sources, telecom fragmentation, privilege and trust systems, and a variety of others.” Will Ogmento be leading the way in solving some of these hard problems?

And, won’t trying to solve these hard problems for networked AR in walled garden scenarios one company at a time lead to a lot of reinventing the wheel wasted energy?

Brian: These are all important issues, and again there are a lot of smart people thinking about solutions to these problems on a daily basis.  Ogmento is interested in partnering with developers and supporting their efforts as a publisher of mobile AR experiences.  While we intend to roll up our sleeves in these areas, we are currently more focused on taking AR mainstream with the hardware and software available today.  As the industry evolves, so will Ogmento. As the opportunities evolve, our ability to make a greater impact tackling these issues will be realized.

Tish: Another area of development that could really kick AR into high gear might be creating augmented reality hotspots  where we use can deliver the kind of location accuracy/instrumentation necessary to create interesting AR experiences (partnership with Starbucks, perhaps ?!).  Augmented reality hots spots, could deliver the kind of high quality AR experience that isn’t possible ubiquitously at the moment, and may be a real way to get people really exploring the potential of AR now, rather than later?

Brian:  Agreed. I see a great opportunity here with this approach.

Tish: Although there are many obstacles to Green AR – the energy hogging servers at the backend for starters! Last week I had a conversation with Gavin Starks, AMEE, and Robert Rice and Jim Purbrick about how to work with AMEE and the technology available and encourage Green Tech AR development (see an early exploration of green tech AR from Pachube here).

We came up with the idea of holding a competition perhaps centered around a targeted instrumented space. But I would really love to hear your thoughts on the topic of Green Tech AR (the energy hogging servers at the back end being the first cloud on the horizon!.)  Cool GreenTech AR imaginings, social gaming ideas, RPGs, not even necessarily even tied to the immediately practical, would be like rain in a drought!

Brian:  I go back to “Games and Goals”… If you make environmental and other activist efforts fun and rewarding, more are likely to be motivated and participate.  Can you imagine having a personal “carbon footprint stat” floating over your self at all times? Or over your home or factory?  How would that change your behavior?  We all love stats. Look at how the Nike+ campaign has used technology and gaming to motivate people to run.  I think there is a lot that can be done to make being green fun. It starts with the individual, and spreads from there.  Keep me posted on that one!

Tish: I would also like to explore further the RRW suggestion that ambient intelligence is both the Holy Grail of AR and possibly snake oil:

“The holy grail of the mobile AR industry is to find a way to deliver the right information to a user before the user needs it, and without the user having to search for it. This holy grail is likely in a ditch somewhere beside a well-traveled road in the district of the semantic Web, ambient intelligence and the Internet of things. Be wary of any hyped-up invitation to invest in a company that claims to have gotten the opportunity right. What we’ve seen in the commercial industry to date is a rather complex version of a keyboard, mouse, and monitor.”

So Holy Grail, Snake Oil, or a ditch somewhere….?

Brian:  I instantly think of Minority Report, where Tom Cruise’s character is being bombarded with holographic ads personalized with his name and to his current situation.  In the future, Spam is a nightmare, especially when it knows who you are.  I think the key thing here is delivering “the right information”, and we still dont have that down. I do see a day where we can truly customize what comes to us, how we want it, when we want it.  My future vision of ambient intelligence is the ability to “turn everything off” if I want to… block out the stimuli and replace it with images of nature, or natural surroundings, etc.  Where I live in Los Angeles, we have those digital billboards everywhere, so it’s like advertising overload wherever you look (hints of Blade Runner).  I personally don’t mind them, but I know there is great debate on there being simply too many billboards everywhere. So AR would only add to the noise of life by adding yet another digital overlay of information, right?

Perhaps the holy grail is to use technology to filter things out. AR might become a solution to leading a simpler life, or a perfectly customized life if you want that. Ultimately the control needs to be with the individual.  I guess I am talking about something like TiVo taken to the extreme.

Tish: And then that other biggy – augmented reality search! I am asking this next question of  Wikitude and Sekai camera too and now I must also ask Acrossair and several others I guess! Obviously a huge area of opportunity in this broader landscape that uses location-awareness, barcode scanners, image recognition and augmented reality is to harness the collective intelligence – a whole new field of search. There is the beginning of a discussion on this in the comments here.

What will it take, in your view, to become a leader in augmented reality search?

Brian:  I’m more of a content guy, so I tend to focus on things like UI, quality of creative, etc..  From that perspective, I am looking forward to evolving beyond the “post-it” text overlay user-experience we see now in AR search. I was impressed with the TAT Augmented ID concept and hope we start seeing more smart design solutions like that emerging in the space.  There are some great new design approaches coming out of the location-aware space that should be applied to AR search. I’ve been studying the heads-up display designs being used in video games, and re-watching movies like Iron Man for ideas. This is another example where Hollywood has painted a polished picture of what AR can and should look like, and the masses have already accepted these design approaches.  So from that perspective, from my view the leaders in search will be delivering sexy, smart and simple solutions. It’s all about the S’s.

Wilfred Pinfold, Director, Extreme Scale Programs for Intel, and the Supercomputing Conference general chair, is working with some Intel colleagues to make a project called ScienceSim the centerpiece of a special workshop event at the SC09 conference (see Supercomputing Conference, an ACM and IEEE Computer society sponsored event).

Recently, I interviewed Wilf Pinfold (see interview below), Mic Bowman (also see my previous interview here), and John A. Hengeveld (see interview below). I wanted to find out what are the underlying goals of this SC conference program?  Why are members of the SC community being encouraged to participate with the ScienceSim environment? What projects are beginning to emerge? And, what are Intel’s goals in giving infrastructure support to further the conversation between high performance computing and collaborative virtual worlds?

The vision of creating new ways to collaborate and interact with big data does seem to be one of the more significant steps we can take at a time when we find many of our most complex systems roiling and threatening total collapse. As Tim O’Reilly has pointed out – from financial markets to the climate, the complex systems we depend on for our survival seem to be reaching their limits.

But,  how can we get from the place we are now – see this example of an n-body simulation in OpenSim, to the point where we can collaboratively steer from our visualizations big data simulations of climate change, financial markets, or the depths of the universe.  The picture opening this post is a:

Frame from a 3D simulation of gravitational waves produced by merging black holes, representing the largest astrophysical calculation ever performed on a NASA supercomputer. The honeycomb structures are the contours of the strong gravitational field near the black holes. Credit: C. Henze, NASA

Wilf Pinfold explained to me part of the reason to begin a dialogue on collaborative visualization at SC ’09 is that super computing communities (that tend to be highly skilled and visionary) have played key roles in internet development in the past. Wilf pointed out,  key browser technology  developed out of these communities in the early days of the internet – see this wikipedia entry that gives  a background on the role of NCSA (National Center for Supercomputer Applications).

The hope is, while there are many obstacles to overcome, the super computing community has both the skills and motivation to find solutions to creating collaborative environments capable of the kind of rapid data movement that scientific/big data visualization needs. Solving the problems of realtime collaborative interaction with big data will  have many ramifications for the way we understand virtual reality, the metaverse, virtual worlds (all these terms are becoming increasingly inadequate for cyberspace in the age of ubiquitous computing, an argument I will make in another post!).

There have already been a number of blogs on ScienceSim (see Virtual World News, New World Notes, Vint Falken, and Daneel Ariantho). There have also been Intel blogs – see this post by John A. Hengeveld (a senior business strategist working with Intel planners and researchers to accelerate the adoption of Immersive Connected Experiences). And Intel CTO Justin Rattner’s post announcing the project this November.

But to blow my own horn a little, I think i was the first to blog the encounter between OpenSim and Supercomputing (an encounter I to some degree provoked by making the introductions) see this post.  So I have been following the ScienceSim initiative with great interest.

Very shortly after N-Body astrophysicicsts Piet Hut and Jun Makino, creators of  – GRAPE (an acronym for “gravity pipeline” and an intended pun on the Apple line of computers) – a super computer that will become one of the fastest super computers in the world (again), met Genkii – a Tokyo based strategic company working with OpenSim, the first N-body simulation appeared in OpenSim.  And in a matter of weeks  this video went up on YouTube – the result of a collaboration between MICA and Genkii.  But the nirvana of being able to create visualizations using real time data from super computers that can be steered from a collaborative environment is still a ways off.

Super computing communities tend to be geographically very dispersed and researchers often find themselves far from simulation facilities so there is both the motivation and skills to pioneer new tools for collaborative visualization. I know that astrophysicists certainly see their value (Piet Hut has some profound ideas on this). Astrophysicist Piet Hut and others  (see here for more) have been pioneering the use of VWs for collaboration.  There are two Virtual World organizations, both founded by Piet Hut and collaborators, that are currently exploring the use of OpenSim for scientific visualizations.  One is specifically aimed at astrophysics, MICA, the Meta Institute for Computational Astrophysics, and the other is aimed broadly at interdisciplinary collaborations in and beyond science, Kira, a 12-year old organization focused on `science in context’.  As of last week, there are two weekly workshops sponsored jointly by Kira and MICA that explore the use of OpenSim, ScienceSim, and other virtual worlds.  One of them is “Stellar Dynamics in a Virtual Universe Workshop” and the other is “ReLaM: Relocatable Laboratories in the Metaverse.”

MICA was founded two years ago by Piet Hut within the virtual world of Qwaq Forums (see the paper “Virtual Laboratories and Virtual Worlds”). The Kira Institute is much older: it was founded in 1997.  Later this month, on February 24, Kira will celebrate its 12th anniversary with a presentation of talks, a panel discussion, and a series of workshops.  See the Kira Calendar for the main event, and the Kira Japan branch for a special mixed RL/SL event in Tokyo.  During both events, Junichi Ushiba will give a talk about his research in which he let paralyzed patients steer avatars using only brain waves.

Other early adopters of ScienceSim include Tom Murphy, who teaches computer science at a Contra Costa College. Prior to teaching, Tom spent 35+ years working for supercomputer manufacturers. Tom said:

it is very natural for me to find significantly new ways to visualize and interact with scientific mathematical models via ScienceSim and the OpenSim software behind it. ScienceSim also allows us to interact with each other and teach students in new ways.

Also Charlie Peck, chair of the SC09 Education Program, (his day job is teaching computer science at Earlham College in Richmond, IN), is working with Wilf Pinfold, Tom Murphy and others “to explore how 3D Internet/metaverse technology can be used to support science education and outreach.”

Cristina Videira Lopes, University of Irvine, is doing very interesting work  on road and pedestrian traffic simulations. Crista is also the creator of hypergrid in OpenSim,

People Meet People Meet Data: A Conversation With Mic Bowman

Screenshot of ScienceSim from Daneel Ariantho

Tish: How does this work on ScienceSim  fit into a wider dialogue on linked data? Where people meet people meet data, and where data meets data?

Mic: Yeah… that’s hard by the way.  Open integration of data (and more interestingly the functions on data) is very hard if it comes from multiple, independent sources.

That’s the people part. For example, if Crista can build a model of the UCI campus somebody else builds an accurate model of several cars and another expert provides the simulation that computes the pollution generated by those cars in that environment…its bringing people together to solve real problems, no matter how far apart physically.

Tish: You mention three different simulations here. Could you explain why it is difficult to integrate data from multiple sources?

Mic: integrating data from multiple sources has always been one of understanding & interpreting both the syntax & semantics of the data. Even relatively simple things like multiple date formats require explicit translation. More complex formats, like the many formats data is represented for urban planning, are barely computable independently let alone in conjunction with data from other sources (each with its own representation for data). Its often the expertise & the collaboration of bringing people (and their bag of tools) together that solves these problems.

Tish: and in this case the bag of tools is high performance modeling..?

Mic: high performance modeling, rich visualizations and data. Its the three that matter… data, function, and interface.

Tish: Some people have a very hard time wrapping their head aropund the fact that anything that seems related to Second Life can do this.  Can you explain more about the difference between SL and OpenSim?

Mic: OpenSim potentially improves data & function because it can be extended through region modules. Region modules hook directly into the simulator to provide additional functionality. For example, a region module could be implemented to drive the behavior of objects in a virtual world according based on a protein folding model.

We need to work on additional viewer capabilities to address the user interface limitations.

Tish: Yes Rob Smart’s (IBM) recent data integrations with OpenSim (see here) are impressive. Re viewers one of the biggest objections to virtual worlds is the mouse pushing and pc tied interface.

Mic: There are great opportunities for improving the interface

Tish: Yes I really like where the Andy Piper’s experiments with Haptic Interfaces for OpenSim lead, see Haptic Fantastic! And I think that we will have cyberspace ubiquitous in our environment, not just stuck on a pc screen, sooner than we think.

Mic: Mic’s opinion (not Intel): until we get souped up sunglasses with HD screens embedded (or writing directly into the eye) there will always be a role for the PC/Console/TV).  But, it isn’t about the device… its about the services projected through the device… sometimes you’ll want a very rich experience… sometimes you’ll want an experience NOW wherever you are.

Tish: I think people are only just realizing that VWs will be a now and wherever you are experience very soon.

Mic: That’s the critical observation the virtual world is not an application you run… its a “place”… and you interact with it where you are or maybe interact through it. Speaking for Intel… it is the spectrum of experiences that are critical to support.

Interview with Wilfred Pinfold

Picture from National Science Foundation – “Climate Computer Modeling Heats Up.”

Tish Shute: I know your day job for Intel is in High Performance computing. Could you explain to me a little bit more about what you are working on in this regard – a mini state of play for high performance computing from your perspective?

Wilfred Pinfold: My title is Director, Extreme Scale Programs. This program drives a research agenda that will put in place the technologies required to make an Exa (10^18) scale systems by 2015. The current generation of high performance computers are Peta (10^15) scale so this is a 1000x increase in performance and this increase will require significant improvements in power efficiency, reliability, scalability and new techniques for dealing with locality and parallelism.

Tish: The nirvana in terms of linking supercomputers to the collaborative spaces of immersive virtual worlds is to be able to create visualizations using real time data from super computers in collaborative VW environments, and ultimately for researchers to be able to collaborate and steer their simulations from their visualizations.  Where are we at now in terms of scientific data visualization in VWs? And what are the current obstacles to using realtime data from super computers?

Wilf: Being able to steer a simulation from a visualization requires both a visualization interface that allows interaction and a simulation that operates at a speed that is responsive in interactive timeframes. For example a weather model that predicts the path of a hurricane would need to operate at something close to 1000x real time. This would run through a day in ~1.5 minutes allowing an operator to run the simulation over several days multiple times with different parameters in a single sitting to understand the likelyhood of certain outcomes?

Tish: Do you see a networked online collaborative virtual world being capable of being a visualization interface that allows meaningful interaction with the hurricane scenario you describe in the near future (next 6 to 18 months)?

Wilf: I was using the hurricane example to explain the usage model not an imminent capability. Hurricane Simulation: Accurate hurricane simulations require multiscale models able to resolve the global forces working on the storm as well as the microforces that define precipitation. We can build useful weather models today that run faster than real time (anything slower is not useful for prediction) but we are a long way from the ideal.
Visualization: There are excellent visualizations of weather systems but I have not yet seen a virtual world that can track a simulation and allow the scientist or team of scientists to see what is going on at both the macro scale and zoom in to see precipitation conditions. Today’s supercomputers are much better at this than they were a few years ago but they are a long way from ideal.

Tish: Open Source Virtual World technologies are pretty diverse in their approaches, Croquet, Sun’s Wonderland and OpenSim are quite different and have different strengths and weaknesses. As you have become more familiar with OpenSim, what have you found about the technology that particularly lends itself to this project – ScienceSim (Mic mentioned Crista’s hypergrid code for example, modularity is another feature often cited).

Wilf: We have found OpenSim’s client server model is well suited to the visualization model and the ability to put the server next to the supercomputer producing the visualization data is critical. We are however very interested in other environments and encourage papers, demonstrations and research on any of these platforms at the conference.

Interview with John A. Hengeveld

Tish Shute: OpenSim’s dependence on Second Life based viewers is sometimes cited as a limitation, and sometimes as a strength. What are your views on this?  What would a strong open viewer project directed at science applications bring to the picture?

John Hengeveld: There may be more than one strong open viewer project required for opensim compatible experiences.  The strength of the Hippo viewer, for example, is availability and its weakness is the size of the client.  We would love a ubiquitous, client.. that runs on all platforms, but each hardware platform brings tradeoff and restrictions of its own.  Today, probably all of the folks innovating in the space can deal with the size of a very fat rich client ap.. they have big computers anyway.  But as we get into more 3D entertainment and augmented reality applications.. virtual mall, collaboration apps.. etc… there is a great deal of room to optimize for the specific experience.  Balancing visual experience with bandwidth and compute performance available .. tying into standard browsers, etc… people have done some of this work.. and I think all of it adds to the usefulness of these worlds.

Tish: Integrating highend game engines and OpenSim opens up new possibilities. But licensing issues have been an obstacle. Could a project like ScienceSim get a non-commercial license on a high end game engine?  What would that bring to the picture?

John: Anything is possible. Game engines can give a great deal of design power for high value experiences, but the programming of these experiences must be simplified.  Mainstream adoption in enterprise can’t be premised on the programming model of studio games… that’s a big step to get over I think.  There are very interesting possibilities when we take that step tho.  Simulation, training, agents of various types (I just finished watching “The Matrix” for like the billionth time… I think agents are cool…)

Tish: Where does Larabee fit into the picture of ScienceSim and next generation virtual worlds?

John: We are all very excited about the Larrabee architecture and its application to work loads like next generation virtual worlds, both in the client.. delivering immersive reality.. and someday potentially in a distributed architecture simulating and producing these worlds.  For Intel CVC is an all play.  Atom will be used in strong mobile clients.  Core will be used in Enterprise PCs, Laptops and Desktops   Xeon will be simulating these environments and handling the data communication, and Whatever we brand Larrabee… will be enabling compelling visual experiences. Oh.. and our software products (Havoc, tools and others) will be building blocks in knitting all this together.  Larrabee is a part, but there are a lot of other pieces in our vision…

Tish: If the kind of rapid data movement that scientific visualization needs is achieved in virtual worlds, this will be quite a game changer for business applications of VWs too. Also it will blurr the boundaries between what we call virtual worlds and mirror worlds. It seems to me this kind of rapid data movement is a vital step towards what Mic described to me as Intel’s vision of CVC: “Connected Visual Computing is the union of three application domains: mmog, metaverse, and paraverse (or augmented reality).” It almost seems to me that if you achieve your goals for ScienceSim you will change how we think about virtual worlds in general? What do you think?

John: I certainly hope so..  Part of our goal is to stimulate innovation in the technology and usage models that will enable broad mainstream adoption of CVC based applications (what we categorize as immersive connected experiences).   By tackling the scientific visualization problem, we hope to find the key technology barriers and encourage the ecosystem to solve them.

Tish: To me virtual worlds and augmented reality should be complimentary and connected experiences. How do you see this connection evolving?

John: We certainly see them as related.  In the long term, there are many common building blocks.. but they aren’t united per se.  Its about the user experience, and in some usages these two are almost identical…  in some.. they don’t look or feel at all alike… the viewer is distinct by a lot.  Our approach is to enable building blocks that people can quickly build out usages that are robust.

Tish: What is Intel’s vision for ubiquitous mobile computing and an internet of objects?  How can high performance computing be an enabler for this vision?

John: Mobile computing is a central part of our life, culture and community in economically enabled economies.  It feeds the data of our decisions, it connects us to entertainment, it is the access point to our soapboxes, pulpits, economy and families.  This creates a massive increase in data, a massive increase in interactions, transactions and visualizations.  While many HPC applications will be behind the scenes (finance, health, energy, visual analytics and others), HPC will emerge as a part of a scale solution to serving some of this increase… particularly that part where interactions and visualizations are complex or compelling.. or where scale enables the usage per se .. I talked about my love of agents earlier.. and some of that comes in here.  Compute working behind the scenes to help managed the data complexity, manage some of the base interactions between ourselves and technology.  The other thing we talk internally about the “Hannah Montana usage” where millions of people use their mobile devices to access and participate (using the sensors in the device) with an interactive live concert.  When Mylie hears the applause of a virtual interactive audience… and can scream back at them.. we’re there.  Access to ubiquitous compute will be mobile, and interactive experiences will be complex.. and HPC can help make that real.  Watch out for the mental trap that HPC is always high end super compute clusters tho… the “mainstream HPC”.. smaller clusters… high threads, etc… will play a key part in all of this as well.

Interesting that John ended on this point as this just came in from Wired.

Neogence is on stealth mode with an immersive mobile augmented reality platform – “tools, sdk, and infrastructure plus some applications.” They are probably six months away from YouTubing anything according to CEO, Robert Rice.  But Robert rustled up this pic for me – a Google street view of Neogence R&D labs: “the patio on the lower left is where I do a lot of pacing and smoking my pipe and the porch and office upstairs is where  a lot of  meetings have been held.”

Robert Rice (@RobertRice ), CEO of Neogence, recently tweeted:

I’m changing my name to Robert Mobile Ubiquitous Geospatial Augmented Rice. I’m betting on radical changes in next 18 months.

Although Robert’s new AR platform is still under wraps, I think you will get a good idea of what direction he is going in from this interview (full text at end of  this post). Robert is the author of “MMO Evolution” and is a key developer and thought leader in persistent immersive environments, simulations, virtual worlds and massively multiplayer games as well as large scale communities and social networking.

It is OMG finally, at least, for minimally immersive but truly useful AR.

Since the launch of Android a new generation of useful augmented reality applications like Wikitude are emerging.

After the last NYC Tech Meetup, my  friend Nathan Freitas,  (@NatDefreitas),  or rather Nathan Mobile Meets Social Freitas, demoed for me a cool graffiti app  he has developed on Android.   You leave a marker for your graffiti so other people can find view/add their own – a nice primal experience like pissing on the lamp post to let your pack know where you’ve been.  Also the graffiti app taps into a long history of  NYC street culture around tagging and graffiti art.  For more cool mobile projects Nathan is working on – Vote Report and data collection for mass events, a guide to pubs and nightlife in New York City, and more, see his blog, “Nathan’s OpenIdeals. With Camera, GPS, compass, and accelerometer, and APIs on Android for temperature, light meters, (no hardware yet), Nathan says Android:

“seems to be the platform most likely to socialize the idea that sensor data could be a piece of every application.”

As Nathan is fond of saying:

The compass is a killer app enabler!

Also see OpenIntents for some interesting Android Sensor projects.

Wikitude was one of Thomas Wrobel’s two top AR milestones for 2008 (see Gamesalfreso):

Wikitude I think. Seems the first released, useful, AR software.

AR Geisha doll is also a remarkable breakout for AR – but useful, nah.

I asked Robert if he also thought Wikitude and AR Geisha doll as  significant breakthroughs:

Yes,  these are among the first attempts to get away from the novelty of simply rendering a 3D object based on a marker and making it interesting.

Remember, one of the biggest risks that AR has, is being branded as “novelty”, which means “cool for five minutes but ultimately a waste of time.” I think we have a ways to go before something is truly useful, but as 2009 progresses we should start seeing some effort here. I’d guess 2010 before something really useful comes out…at least something practical.

Now, having said that, I should say that I expect entertainment and games to take the lead (as usual), although there are a few companies really trying to leverage AR and video/graphics compositing for marketing (brochures) and location based methods (kiosks, large screen projections, etc.)

So when is it “OMG finally!” for massively multiuser augmented reality?

The picture above is from benjapo’s portfolio on istockphoto – also see the istock video here.

Alex Soojung-Kim Pang (who weighed in recently on the twitter-baby debates – see my KickBee Posterous blog) challenges design assumptions for augmented reality that take as a given the user’s desire for numerous private enhancements to their reality.

Alex points out less will probably be more so that enhancements do not impinge on shared experience.  See his write up of a talk he gave at the Royal College of Art, “and the end of my own private Shibuya.” Photo below by Stéfan, “Karaoke in Shibuya“

Part of the pleasure of these streetscapes is precisely that they’re collectively experienced, rather than individual visions: for even a brief period, we share with other postmodern, globe-hopping flaneurs and expatriates and temporary natives the light of the ABC-Mart sign and storefront.

It is collective experience of enhanced, augmented, virtual or real experiences that interests me too. This is one of the reasons I find Pachube and the EEML project of Haque Design and Research so interesting.

Extended Environments Markup Language (EEML), a protocol for sharing sensor data between remote responsive environments, both physical and virtual. It can be used to facilitate direct connections between any two environments; it can also be used to facilitate many-to-many connections as implemented by the web service Pachube, which enables people to tag and share real time sensor data from objects, devices and spaces around the world.

“Distinctions between virtual and real are as quaint and outmoded as distinctions between mind and body” (Usman Haque)

Usman Haque (founder of Pachube and Haque Design and Research) points out this is an underlying premise of his work – and augmented reality (full interview coming up soon!).

The pictures above show the Haque Design project, Remote:

‘Remote’ connects together two spaces, one in Boston the other in Second Life, and treats them as a single contiguous environment, bound together by the internet so that things that occur in one space affect things that happen in the other and vice versa – remotely controlling each other.

There was a discussion in twitter recently about how the terms like Second Life, Exit Reality, Virtual Worlds are misleading and outmoded. As Robert pointed out we need:

one word please…that sums up virtual and/or augmented reality, interactive, immersive, virtual worlds, mmorpgs, simulations, etc… also, I really don’t like the term “augmented reality” or “mixed reality”. Neither is all that great. And NO “matrix” or “metaverse” please

Robert argues strongly that there is a stultification both in virtual world technology – much of what we call virtual world technology was already, basically, where it is now in the mid 90’s. And MMOGs have devolved into gameplay design “that emphasizes the single player experience and does nothing to take advantage of the potential of the massively connected internet.”

Robert suggested I take a cruise through a new Virtual Space -  CoolIris to find some good pictures for this post (note the partnership between CoolIris and Seesmic to Streamline Video Blogging. I added the Cooliris Plugin to Firefox and typed Augmented Reality into search and soon I was cruising a highway of images and links. The Road Map image grabbed my attention (see below). It shows the continua that the Metaverse RoadMap authors thought are likely to influence the ways in which the Metaverse unfolds. It is “a map of the spectrum of technologies and applications ranging from augmentation to simulation; and the spectrum ranging from intimate (identity-focused)external (world-focused)”

Quite to my surprise, when I clicked out of CoolIris to the source for the image, I found it had been drawn from a post I wrote in May 2007, Hybridized Digital/Physical Worlds: Where Pop and Corporate Cultures Mingle. My post talks about a number of hybridization experiments that were bringing together lifelogging, sensors everywhere, simulation, virtual worlds, and augmentation.

The striking difference from 2007 to now is that we have definitely moved on from mere experimentation. And the poles of the continua intimate/extimate, augmentation/simulation as expressed in the Metaverse Roadmap are now becoming entwined (note the picture above seems to be slightly different to the one used in the road map as published here – perhaps I had an early version?)

“Augmented Reality is not just about overlaying data…” (Robert Rice)

The screenshot above is from TotalImmersions video demoing Augmented Reality with 3D Cell Phones. Also see video of their immersive games, and FutureScope kiosks here and here.

Gamesalfreso noted that Will Wright, delivered the best augmented reality quote of the year. When describing AR as the way of the future for games, Will Wright said:

“Games could increase our awareness of our immediate environment, rather than distract us from it”.

Robert points out in this interview the term Augmented Reality itself has become associated with a very limited understanding of what “enhancing your specific reality,” is really about. Robert notes:

it is inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.

When I talk about AR, I try to expand the definition a little bit. Usually, when you talk to someone about augmented reality, the first thing that comes to mind is overlaying 3D graphics on a video stream. I think though, that it should more properly be any media that is specific to your location and the context of what you are doing (or want to do)…augmenting or enhancing your specific reality.

In this sense, anything that at least knows who you are (your ID, mobile phone #, etc.), where you are (GPS coord or a specific place like a cafe), and gives you relevant data, information, or media = augmented reality. Sure, you can make things more interactive or immersive, but that is the minimum.

So, in this case, yes, I think there will be networked applications in the next 18 months…mostly things that are enhanced by friends lists (you are here, your friend is over there). These will be *application specific*. My team at Neogence is already going beyond this, building a platform and infrastructure for other applications to be developed on…all networked through the same backbone. Now, in this context (the science fiction AR that we all dream about), no I do not see anyone else trying to leap a generation or two ahead of the industry to build a massively multiuser shared AR space. Expect to see things like multi-user AR games, virtual pets, kiosk marketing, magic book, “gee whiz” presentations (tradeshow booths, entertainment parks, etc.), and so forth.

Goggle’s Are Not The Secret Sauce…

AR Cat left and Robert Rice right

What has come to be associated with the term Augmented Reality, in the popular imagination – an idea of 3D graphics projected over markers that has been forever waiting for the advent of “wicked next generation transparent wearable displays” – nirvana for augmented reality. While such displays may be nirvana for AR (and they could be with us in less than twenty four months), Goggles are not the “secret sauce” of AR as Robert points out.

All the glasses are, is another display device. At the end of the day, it doesn’t matter if you are looking at an LCD monitor, an IPhone, a head mounted display, or a pair of wicked next generation transparent wearable displays that magically draw directly on your retinas.

The real tricky stuff is what happens on the backend…making it all persistent, massively multiuser, intelligent, interoperable, realistic, etc. etc.

There has been quite a buzz going around about the new Vuzix Eyewear, and recently Robert talked with Vuzix and checked The Wrap 920AV eyewear out:

Vuzix is not alone in pursuing the ultimate in hardware, at least as far as wearable displays. However, I think they are much farther than the rest of the pack in vision, roadmap, and execution. They have put together a team that has a sense of urgency and ambition that will blow the industry away. After talking to them, I got the feeling that they really know what they are doing and there is a lot of mind blowing stuff in their pipeline. I’m sure they are one of the few companies that really gets it and has a clear vision of the future. Definitely my first choice to work with.

Hybrid Augmented/Virtual Reality

Cory Ondrejka posted this picture of the anybots telepresence robot and “congrats to Trevor Blackwell and the rest of the Anybots team on the launch of QA at CES.”  Cory (one of the founders and former CTO of Second Life) also made some predictions for Virtual Worlds, some optimistic and some less so, including “the increasing need to be able to diversify the Second Life product offering to begin truly rebuilding the code base.”

Robert is unabashedly irritated with the state of play in Virtual Worlds and MMOGS:

Unless both industries (Virtual Worlds and MMOGs) have some serious upheaval or radical new approaches, they will quickly be eclipsed by AR, which will eventually evolve into something hybrid..AR/VR depending on your level of access and hardware.

I’d like to see someone grab an engine like Offset, Crytek, HERO, or Unreal 3, and smack on a fat MMO server infrastructure (Eve or Bigworld)…toss in the right tools, and you would see a revolution and renaissance occur at the same time in the virtual world space. All the puzzle pieces are there, just no one is putting them together the right way.

I did just find out that Nortel’s WebAlive is powered by the Unreal 3 engine. You can try WebAlive out here.

Robert points out how rare it has become to see people really push virtual worlds technology and MMOGs into entirely new directions.  Although, of course, there are exceptions.  I managed to engage some interest from Robert in the possibilities the opensource modular architecture of OpenSim opens up, and the augmented reality experiments from Georgia Tech with Second Life (screenshot below) got praise from Robert for trying to do something new. (Georgia tech have also put out a virtual pet app for the iphone ).

But while Robert clearly has zero patience for virtual world technology which he sees stuck in the mid nineties, he notes:

the innovative and wonderful stuff about SL isn’t SL, it is what people are doing and creating on their own with terrible tools *IN* SL [Second Life].

The immersive mobile augmented reality platform Robert is building, he hopes, will generate this kind of user creativity but with 21st century tools.

So is it “OMG” finally for the Augmented Reality we have dreamed about?

According to Robert:

It really boils down to a markerless solution and a good application.

In the interview below we cover a number of topics including business models for Augmented Reality, e.g., how business models based on micro-transactions and virtual goods will translate to Augmented Reality.

Many of the challenges to becoming mainstream faced by virtual worlds  are similar to the challenges AR must overcome. Robert discusses these including the interface/gui that is a critical element for AR, solving the riddle of one world or many, patent wars in Virtual Worlds and Augmented Reality, the role of Augmented Reality in the future of sustainable computing, and what interoperability is about.

The Back Story for AR/VR…

In case you want to get up to speed on the required background reading for  Augmented Reality. This is Robert’s required reading list and Denno Coil is an absolute must see (feel free to add to this list in the comments, please).

“If you want to see the things that have inspired our vision of what we want to build, check out:

* Dream Park by Larry Niven and Steven Barnes
* Rainbows End by Vernor Vinge
* Spook Country by William Gibson
* Halting State by Charles Stross
* The Diamond Age by Neal Stephenson
* Donnerjack by Roger Zelazny and Jane Lindskold
* Otherland by Tad Williams
* Neuromancer by William Gibson
* Idoru by Wiliam Gibson
* Cryptonomicon by Neal Stephenson

and watch the whole anime of Denno Coil (subbed NOT dubbed!)”

Screenshot from Denno Coil from Concrete Badger.

Interview With Robert Rice

Tish Shute: I am glad to hear that you are working on this [an immersive mobile augmented reality platform]!

Robert Rice: We switched gears from MMO stuff about a year ago and we are finally getting some traction. It is very hard doing anything in this economy right now, but we found an opportunity to take AR to a new level beyond what you see on youtube. AR is still too “cute” and novelty. We don’t want to play around.

Tish Shute: I like Wikitude ‘cos it even manages to do something useful!

Robert Rice: Yeah, useful = traction. Now that we are getting near a prototype we are starting to get a lot of interest even though we are still technically way under the radar.

Tish Shute: r u funded?

Robert Rice: privately funded, some revenues from an early license, and ongoing discussions with several institutional investors. So, we have some funding, but nothing spectacular just yet.

Tish Shute: are you just developing an AR platform?

Robert Rice: hrm, sort of, but not just that. By platform I mean tools, sdk, and infrastructure plus some applications. The idea is to build something that facilitates everyone else making cool things and useful applications for different industries/sectors

Tish Shute: Yes that is the cool thing to do but isn’t that hard to fund!

(Robert grins) Well, that depends on the business model. We’ve got that figured out. I’d be absolutely happy if everyone and their brother were making applications on our stuff that gives us an edge on market penetration/saturation. There are plenty examples that prove the model. If you give people free and easy to use tools, they will run with it. ARtoolkit for example, has tons of people making nifty things and posting videos on youtube that has pushed them to the forefront as THE AR middleware to use right now, or heck, look at youtube free service, and they dominate video sharing.  Sure there will be a lot of “noise”, but there will also be a lot of “signal” that will rise to the top, facilitating and enabling is creating value in its own right.

Tish Shute: But how do you expect to monetize?

Robert Rice: There are a good half a dozen ways to monetize AR or an AR platform.

Tish Shute: What are your top 3?

Robert Rice: hrm, microtransactions, localized mobile advertising, and enterprise solutions (visualization)

Tish Shute: Do you think the consumer market will give the lead?

Robert Rice: I’m not sure. We are getting people from academia, intelligence, defense, border security, and some corporate types knocking on our door already, and pretty aggressively. It may be that those sectors push AR before consumer entertainment really kicks off.

But going back to a discussion we had earlier – yes working with “no markers” is a big deal.

Tish Shute: Can you talk about what you are doing there or is it still under wraps?

Robert Rice: I can say that between some university tech transfer and some of our own proprietary stuff, we are using some fairly common visual tracking technology. if you are really plugged into the AR scene, you will know there are probably half a dozen visual tracking methods out there. We just looked for the best one, licensed it for commercial use, and then started working our magic. This is a very small piece of the overall effort, but worth noting.

The downside with working with university tech is that it is usually based on research, incomplete, and not wrapped up in a nice commercial package on the upside, it can be a good start to build on.

Tish Shute: As you know I am very interested in “technology that matters” in particular tech that can help us accomplish the urgent goal of sustainable living.

Robert Rice:: oh, I’m pretty keen on sustainable living as well…after I sell off a few companies and have money of my own, I’m going to get into arcologies
❦
Robert grins

The interesting thing with the visual stuff combined with our other tech, is that we can make things multiuser, persistent, dynamic, and mobile.
The markers (fiducials) are really really limiting outside of basic applications. You can’t really plaster everyone and everything with a marker.  And they are, by nature, static (even if they are animated or whatever).

Also… our stuff works indoors and outdoors even without a GPS connection.
❦
Robert grins

Tish Shute: Now that does sound interesting!

Robert Rice: Yeah, with visual, you don’t need a compass or accelerometers either. Less hardware : )

You start with wifi triangulation or gps coord to get a “brute” location, and then you use the visual stuff for down to the meter accuracy and that by nature, gives you your orientation and positioning.

Tish Shute: Wow this is beginning to sound very interesting!

Robert Rice: Once you have that, it doesn’t matter where you go, it continues to track and continually refines areas you have been before. We’ve spent the last year figuring all this out. There are so many problems and obstacles that are going to be developing in the future for anyone trying to do what we are, but we have already discovered solutions.

oh, visual tracking = gesture based interfaces too that’s going to take some work, but its doable.  The real pain in the ass there isn’t the actual tracking, it is in the interface design.

That’s something that almost every AR company, venture, and research program is missing out on entirely. They are so focused on making cute things with markers.  They are missing the larger problems of AR Spam, interface, iconography, GUI, metaphor, interoperability, privacy, identity.

Tish Shute: So how are you dealing with all that!!

Robert Rice: We took the backwards approach of trying to think where we want things to be in ten years (and we read all the cool books…Vinge, Stephenson, Gibson, etc.) and then we spent time trying to think of what the potential problems are….like AR spam. Its bad enough when a giant penis flies by in second life, we don’t want that to happen in a global wireless AR platform.

Tish Shute: Do you have a prototype yet?

Robert Rice: hrm, 6 months away from youtubing something. Problem has been slow funding, which equals slow development. We also don’t want to show our cards too soon…too many potential competitors out there.

❦
Robert grins

Tish Shute: when you say microtransactions what is the business potential there?

Robert Rice: hrm last year I think, $1.5B was spent on virtual items. That’s games and virtual worlds. That should hit $5B in a couple of years. That’s basically people buying and selling things like WoW gold or items in SL or whatever. microtransactions, is basically the same thing, but in AR space.

Why couldn’t a 3D artist make a wicked animated 3D dragon, and then sell it to someone else? With AR, you could sit it on your shoulder. With a good scripting engine, you could train it to do stuff. Thats what I want to enable.

tools + sdk + platform = enabling people to make and create. Add in a commerce level (microtransactions) and wala.

Tish Shute: At the moment all of these virtual goods are very platform specific, is that a problem for you?

Robert Rice: Not at all. This is at a higher level. You have to switch mental models when you talk about what AR could or should be. For example, lets contrast the web and virtual worlds. For every virtual world you go to, you have to download a whole new client. Imagine if that model was applied to the web… you would need a brand new browser for every website you went to. That is just so…wrong.

It’s the same thing for AR…people are thinking about it with the same mental and business models and development philosophies as virtual worlds or web.  There are some things and aspects that work fine, but not everything.

Virtual worlds, are, by nature, necessarily different and walled gardens. The idea of 100% open and interoperable virtual worlds is a red herring…it sounds good but in practice it is a really dumb idea.

Tish Shute: I was wondering if you had a way to leverage all the 3D content already created ‘cos that would jump start things in AR wouldn’t it?

Robert Rice: Oh yeah, that’s easy. They all use the same polygons. Any virtual item in any game or virtual world is likely created with 3D studio or maya or something similar would be easy to convert and use.

Tish Shute: So people could bring their WoW weapons into your system?

Robert Rice: Not legally, but sure. Its just a 3D model with a texture.  It doesn’t matter if you use corel draw or photoshop or paintshop pro….or one screwdriver or another. Part of my team’s advantage, is that we are all experienced in MMORPG and virtual world design and development. We know the tools, the tech, and what works and what doesn’t.

Tish Shute: But some of the 3D content created in the social worlds is what has most value to people.

Robert Rice: Right, and that can be exported out easily.

Tish Shute: But back to “real” life applications. Is you platform really markerless?

Robert Rice: Yes.  marker = printed icon or glyph, also known as a fiducial

Tish Shute: But u must have some marker?

Robert Rice: hrm, more accurately, you need a point of reference.

Visual tracking has been around for more than a decade.  Lots of work for robots and other sectors.

Robert Rice: But isn’t the specificity of reference n terms of RL applications a vital key, for example, for a database of things?

Robert grin That is a different problem…tracking, registration, mapping, positioning, etc. That question has to do with mapping which is related to visual tracking, but not the same thing. We have a rather unique approach to some of this that I can’t discuss (patent pending).

Tish Shute: But for example, to create an augmented natural history of food – say I want to point at the slab of meat on my plate and know where that cow came from, what feed lot how it was treated etc

Robert Rice: That is not possible without ubiquitous nanotechnology. Shall I explain?

Tish Shute: Yes please!

Robert Rice: Ok, lets step back a minute and turn that burger back into a cow… the first problem (of this particular situation) is differentiating from one cow to another since most cows look alike, you can either attempt to discriminate visually (cow patterns) or use a much simpler option, like giving each cow a rfid chip in their bell, or hoof

Now, most people would try to figure out how to jam all sorts of info in the rfid chip, which sounds like a good idea, but isn’t, the trick would be to simply use the rfid to store a unique identifier with is then linked to a database elsewhere, or hoof.

That database should continually be updated with whatever relevant information you need so as you get close with your AR laptop, wearable displays, or embedded brain chip, you get the identifier broadcast, then you get the info downloaded to you, and it “sticks” to the cow with the generic visual tracking (object following, even simple bounding box is sufficient for a slow moving cow)

So, up to that point, you can get tons of information about that specific cow, that cow population (remember, AR is not just about overlaying data…it is inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc.) Tie in data visualisation and some farmer tools and all sorts of other things happen. Now, lets move the timeline ahead a bit.

The butcher gets the cow and does his handiwork…because we know all the info about the cow, all of the meat can be properly labeled and marked. Ideally, with a UPC code or a unique glyph (somewhat problematic depending on how many unique glyphs you can create) so, while you are in the grocery store, you can access the relevant shopping data…age of cow, state of origin, type of feed, how many spots, how much body fat, which butcher, whatever not because of what is inside the package, but the package itself.

Getting back to your hamburger, the problem is that it is a burger…there is nothing to distinguish that burger from another one at the table…unless you stuck a rfid chip in it or splattered it with ink and a unique glyph, or maybe a special one of a kind plate.

However, a properly designed AR system could say “hey! that/s a hamburger! and I know I am at Fat Daddy’s Burger Joint in Raleigh North Carolina on Glenwood Avenue, and I know that they cook their burgers this particular way, and their meat supplier is those guys over there, and they usually get their cow meat from a farm out in Utah”

With ubiquitous nanomites or whatever, then its not that far out to consider edible nanos that are in the meat and that broad cast info so a slab of meat can tell you about itself and broadcast that to the general public.

Tish Shute: What useful scenarios can we create without the nanomites?

Robert Rice: If it wasn’t a burger or a consumable organic, the scenario changes.

Tish Shute: What is the time scale on nanomites?

Robert Rice: ehhhhhhh 20 years minimum if we are lucky. They sound good on paper, but there is a whole book worth of problems and why they are so far off…as consumer grade, all over the place, type of stuff.

Tish Shute: Did you see the Nokia Home Control center?

Robert Rice: Yes, I saw the Nokia stuff.

AR for sensors, like security systems, temperature control, etc. all become “sources of data” that a AR system can visualize. So yes, thats easily doable. You could do that in a short period of time with some half decent engineers.

The trick of what Nokia is doing is aggregating sensor data from a building/home/facility, mashing it together, and sending the mobile device alerts and data visualization conceptually rather simple, but no one has done it right or well yet.

It wouldn’t surprise me if Nokia pulled it off.

Tish Shute: yes and if they do and someone does an AR interface to it that would be an inflection point for AR?

Robert Rice: In a roundabout way, yes. You could get data directly from your house, or get it through your mobile device and in either case, use the AR for visualization and control.

The interface/gui is a critical element for AR. That is one of the areas where it, as an industry, risks doing a bad job and turning into just a fad or another novelty like VR.  Virtual worlds have been struggling with that for a while, but MMORPGs have had the effect of extending their life cycle

Tish Shute: Yes VWs have not solved the interface problem.

Robert Rice: The interface is one of their problems yes. Most virtual worlds are stuck in 1996/98

Tish Shute: If AR  is inherently about who YOU are, WHERE you are, WHAT you are doing, WHAT is around you, etc. seems that it is the ideal interface for home control?

Robert Rice: Well for home control, you must know:

1) Who am I? Am I authorized to know this information? Am I a guest?

2) Where am I? Is this my house? or someone elses?

3) What am I doing? Do I want to make all the doors lock? Turn on or off lights? Open the garage door? Trigger the security alarm?

So the same questions apply

I’d say that all virtual worlds are stuck in the mid 90s. They are at least a decade behind the game worlds…in technology, design, implementation, architecture, etc. etc. In my opinion, things like Second Life are shameful in how they are presented as state of the art, innovative, ground breaking, new, wonderful, and world changing.

But thats another topic of conversation : )

Tish Shute: Well for me the contribution of VWs is the presence enabled real time interaction with application (as 3D info machine) and context with other people.

Robert Rice: Oh,there is no doubt that they are greatly useful and have a phenomenal amount of potential.

They *could* be all those things I just said that SL isn’t…the problem is that they are either just existing, or they are meandering around without any real focus or direction. They aren’t evolving.

Even MMORPGs are losing their way and beginning to stagnate terribly

Tish Shute: yes I agree

Robert Rice: But, AR has the potential to change a lot of things.

Im sure you have seen the yellowbook commercials? The technologies you are seeing here are doable in hrm, a year or less maybe. The tricky part is the interactivity and AI…that is, the content. Everything else isn’t a problem. The avatar there could be photorealistic or stylized like a WoW character.

You could do that to some degree with markers for registration but dynamically changing the content linked to those markers is a little weird

(by the way, for the record, I like markers just fine, I just don’t think they are useful for real-world mobile applications)

I also think that the guys that want to dust the planet with miniature rfid chips are on crack and are going about it the wrong way

Tish Shute: A high level of interactivity is hard though. Isn’t it? Even in VWs it is very limited.

Robert Rice: it depends if you can track what the user is doing, and interpret that properly. Interactive is also a very lose term.

Clicking a button and making a light blink could be considered interactive.

Tish Shute: In VWs a high level of interactivity would  be to wield a virtual hammer and have a real nail go in! is physics part of the problem?

Robert Rice: physics aren’t difficult, plenty of middleware out there for it. The problem with that isnt so much the physics as much as it is the scale and purpose

Tish Shute: well for robotics?

Robert Rice: that gets into a conversation about meshes, textures, and volumetric collision detection and stuff

Tish Shute: virtual robotics?

Robert Rice: You mean teleremote/telepresence of real robots?

Tish Shute: yes!

Robert Rice: ah, for that, you need some tactile feedback and some other stuff – doable, but insanely difficult. That’s why you don’t see a whole lot of remote controlled surgery robots all over the place.

They do exist…

Tish Shute: Will AR contribute to sustainable living by freeing us from some of our energy hogging devices?

Robert Rice: AR will ultimately encourage energy saving and recycling. where did I leave a light on at? where is the nearest trash can? what is the UV index outside today?

Yes, computers are energy hogs, but as we start seeing larger SSD drives, more efficient CPUs (even if the number of cores increases in multiples), and so on, the power will go down.

Also, think about this…wearable displays potentially use less energy than LCD monitors on your desk.

Tish Shute: Yes I should pick the brains of my intel chums on energy saving!

Robert Rice: Getting rid of the monitor and switching to solid state drives will save an assload of power. Yes, I said assload.

Tell your intel chums to quit screwing around with single core mobile CPUs. We need multiple cores, that are smaller, faster, and use less power.

Tish Shute: Is AR is the sustainable future of VW and MMOGs?

Robert Rice: The fun stuff will happen when they are both integrated in some fashion.

Tish Shute: So perhaps this is why the Georgia guys are thinking in trying to combine AR and SL (see video here).

Robert Rice: That first video was pretty damn cool. It just pains me that they are using SL for it. And omg, all those markers on the table.

Although, I could care less about seeing my SL avatar on my coffee table. I would rather see an avatar representing ME in the real world, moving around in a virtual world that is a “to scale” replica of the real world. That is MUCH more interesting and innovative.

But even if I don’t like where they are going, or that they are using SL, the important thing is that they are doing something and forging ahead. I have a massive amount of respect for anyone, private, government, or academic, that is doing that.

And yes, the door (or window, or looking glass) has to work both ways for maximum potential, at least, that’s what Id like to see. They don’t *have* to, but it would be rather cool.

And going back to sustainability, AR has the potential to make monitors generally obsolete, laptops too. That’s a lot of power hungry devices with all sorts of metals and batteries inside.

But, even if the tech was absolutely crazy awesome right this minute, it would take a little while for consumer adoption.

Tish Shute: But AR unleashes the mobile device?

Robert Rice: Yes, AR is going to be built on powerful mobile devices for the near future, eventually embedded comps in clothing and whatnot. But that is a ways off

Entertainment is going to be the first huge driver.

Tish Shute: So people will get used to having a pet virtual dragon on their shoulder first?

Robert Rice: Yes, virtual dragon is way cool, easy tech for games, and can eventually be leveraged into a smart agent which becomes a practical application…agent based contextual search, etc. Yes, entertainment will also drive people to get used to the tech

Tish Shute: Oh thanks for turning me on to gamesalfresco!

Robert Rice: Ive noticed that the good stuff usually gets linked to there. They don’t list my blog, but that’s what I get for staying under the radar and not posting often. But anyway, gamesalfresco is the first place I send people that need a crash course in AR. Great site, great owner.

Tish Shute: So are you in agreement with Thomas Wrobel’s positioning of  Wikitude and AR Geisha doll as being significant milestones for AR?

Robert Rice: Yes,  these are among the first attempts to get away from the novelty of simply rendering a 3D object based on a marker and making it interesting.

Remember, one of the biggest risks that AR has, is being branded as “novelty”, which means “cool for five minutes but ultimately a waste of time.” I think we have a ways to go before something is truly useful, but as 2009 progresses we should start seeing some effort here. I’d guess 2010 before something really useful comes out…at least something practical.

Now, having said that, I should say that I expect entertainment and games to take the lead (as usual), although there are a few companies really trying to leverage AR and video/graphics compositing for marketing (brochures) and location based methods (kiosks, large screen projections, etc.)

Tish Shute: Many people would say SnowCrash (metaverse) is now and Halting State (AR) is ten years from now. But you are seeing a development timeline for some popular AR apps in the next 18 months?

Robert Rice: Anyone that says SnowCrash is -now- is living in a box. Virtual Worlds, Virtual Reality, and immersive tech in general stopped innovating in the mid 90s. I’m continually flabbergasted at the number of people that think that things like Second Life are state-of-the-art or innovative. You might as well try to market a walkman as cutting edge, even though we have IPods out there.

Id like to see someone grab an engine like offset, crytek, hero, or unreal 3, and smack on a fat mmo server infrastructure (eve or big world)…toss in the right tools, and you would see a revolution and renaissance occur at the same time in the virtual world space. All the puzzle pieces are there, just no one is putting them together the right way.

Tish Shute: Why doesn’t anyone do that?

Robert Rice: Its not cheap, people will only fund a copy of something that exists already, people fear change and innovation, etc, The list goes on. The right money goes to the wrong people all the time.

Alternatively stated, there is a lot of “right idea, wrong implementation”

MMORPGs carried the torch and have made huge strides on the technology front, but have devolved in design. More often than not the gameplay emphasizes the single player experience and does nothing to take advantage of the potential of the massively connected internet.

Unless both industries have some serious upheaval or radical new approaches, they will quickly be eclipsed by AR, which will eventually evolve into something hybrid..AR/VR depending on your level of access and hardware.

But yes, I’d say that the next 18 months are going to be very interesting with a lot of money being thrown around, new ventures, and plenty of content/applications. I expect most of this will be centered on single user AR experienced through a mobile device with a screen (iphone, android, etc.). I expect that there will be a significant boost after Vuzix releases some of their wearable *transparent* displays, putting Microvision back into the “has potential but is too quiet” position.

Tish Shute: AR conjurs an image in many people’s minds of dreadful head gear!

Robert Rice: Yes, it is either transparent wearable displays (in eyeglass formfactor) or nothing. HMDs with miniature LCD or OLED displays are good for streaming video, but for the mobile ubiquitous AR we all dream about, it has to be something that looks and feels like a pair of Oakleys.

I should also mention that several different types and modes of AR are going to find themselves being defined and refined over the next two years as we continue to blaze new trails, establish a lexicon (we keep borrowing terms from games, VR, virtual worlds, mmorpgs), and really work out the how as well as the why.

Even though the idea of AR has been around for a long time, the technology is just beginning to emerge, and very few people are even looking far enough ahead to figure out the problems and solutions that the tech creates. Really, who is thinking about how to deal with AR spam right now?

Tish Shute: Do you see any successful networked AR applications emerging in the next 18 months?

Robert Rice: Yes and no.

When I talk about AR, I try to expand the definition a little bit. Usually, when you talk to someone about augmented reality, the first thing that comes to mind is overlaying 3D graphics on a video stream. I think though, that it should more properly be any media that is specific to your location and the context of what you are doing (or want to do)…augmenting or enhancing your specific reality.

In this sense, anything that at least knows who you are (your ID, mobile phone #, etc.), where you are (GPS coord or a specific place like a cafe), and gives you relevant data, information, or media = augmented reality. Sure, you can make things more interactive or immersive, but that is the minimum.

So, in this case, yes, I think there will be networked applications in the next 18 months…mostly things that are enhanced by friends lists (you are here, your friend is over there). These will be *application specific*. My team at Neogence is already going beyond this, building a platform and infrastructure for other applications to be developed on…all networked through the same backbone. Now, in this context (the science fiction AR that we all dream about), no I do not see anyone else trying to leap a generation or two ahead of the industry to build a massively multiuser shared AR space. Expect to see things like multi-user AR games, virtual pets, kiosk marketing, magic book, “gee whiz” presentations (tradeshow booths, entertainment parks, etc.), and so forth.

The big thing I’m worried about is AR becoming the next silicon valley trend…once they realize the potential, an enormous amount of capital will flow to a bunch of startups with half baked ideas, weak business models, ten year old tech, and a lot of overhyped marketing. That is the very thing that will kill this technology as something that has true power and potential to literally change the way we interact with each other, our surroundings, information, and media.

Tish Shute: Do you think AR has value for a project like Pachube that helps us connect dtat from lots of different environments and sensor actuator data?

Robert Rice: I think that AR has value as an interface to this data (essentially data visualization based on information streaming from a sensor or source that is interpreted in some dynamic graphical manner that has meaning). This is one of the “big areas” where ubiquitous augmented reality and wearable computing will really shine. I’ll definitely be keeping an eye on Pachube .

Tish Shute: I can’t help it! I am really interested to hear more about the Vuzix glasses?

Robert Rice: Yeah, everyone is getting hung up on the glasses as the end-all be all and having markers everywhere too.

All the glasses are, is another display device. At the end of the day, it doesnt matter if you are looking at a lcd monitor, a iphone, a head mounted display, or a pair of wicked next generation transparent wearable displays that magically draw directly on your retinas.

The real tricky stuff is what happens on the backend…making it all persistent, massively multiuser, intelligent, interoperable, realistic, etc. etc.

I think that we are within 24 months of the magic wearables (these new ones by vuzix are probably the real first generation attempt at doing it right). They wont be perfect, but I expect they will be functional…and once we have functional, we can start doing the good stuff.

Tish Shute: You mentioned you disappointement with VWs and MMORPGs earlier.  Could you tell me more about that?

Robert Rice: Yeah, there was an evolutionary divergence between virtual worlds and mmorpgs a while back. One stagnated almost completely, and the other leapt ahead in one sense and devolved horribly in the other sense. Neither is where the state of the art should be.  That is a whole other conversation, and probably a second book.

Tish Shute: So making AR persistent, massively multiuser, intelligent, interoperable, realistic, etc. etc. that is where your efforts are going?

Robert Rice: Yes. I fully expect that the hardware is almost ready for it. You can cobble together some amazing things in the lab right now, and I think commercial viability is imminent. The real value (as far as I’m concerned) is in making it mobile, wireless, persistent, and massively multiuser. You could argue that augmented reality will take over where virtual reality failed and become internet 3, internet one being the internet, internet two being the web…

mmorpgs are nothing more than single player games in a multiuser environment these days. I’m more than a bit bitter about it. All the right money went to the wrong people, and the best games we have are barely shadows of what we could have had by now.

Tish Shute: Are there any open source AR platform dev projects?

Robert Rice: open source? hrm, Im sure there are multiple ones out there

if not entirely open source, there are plenty of things to experiment with that are generally free if you aren’t trying to sell something, DART and ARTOOLKIT come to mind as very accessible applications.

Marker based AR is very important right now…it is easy, low tech, understandable, highly customizable, and most importantly, accessible to the average joe. Ultimately though, we need a method of pure tracking…no markers glued to everything on the planet, no “billions of RFIDs” embedded in every square inch of every object on the planet, etc.

Tish Shute: What do you mean by interoperability in AR? And what do you think about the development of standards?

Robert Rice: Ooh, good question.

Ok, so the internet is basically computers communicating with computers, and the web is mostly pages linking to other pages (I’m greatly oversimplifying here). Hold this thought for a minute.

Switch over to MMORPGs. If you want to play in one (or a virtual world), you need to download a client that is specific to that world. One client does not work with another world. There are plenty of efforts to change this, but they are all barking up the wrong tree. The specific uniqueness of each world defeats the need and purpose of true interoperability, unless you completely reinvent the whole thing with a common backbone, features, functionality, etc. The very nature of virtual worlds and mmorpgs rebels against this.You absolutely do not want an avatar from second life running around in world of warcraft (for reasons that should be obvious).

On the other hand, with the web, you can use just about any client (browser) to access nearly any website (some requiring plugins or whatever).

The thing with augmented reality, is how do we go about making this? I’ve seen a few people thinking about this from the wrong perspective. There was a question at the last techcrunch to the Sekai Camera guys (a conceptual AR application for the iphone) where someone on the panel wanted to know how website owners would convert their content for augmented reality. BZZZZZT! That is a fundamental misunderstanding of what AR is, or could be, and it falls into the same trap I see a lot of people doing…and that is looking at AR through the web 2.0 lens or the virtual world lens. It is absolutely fundamentally different at the core…sure there are similarities: it has social networking/media applications and properties, and it has 3D graphics, but it stops there.

Ubiquitous augmented reality will be dramatically different depending on which standards, approaches, and philosophies get the most traction first. Will you walk down the street with your AR glasses and have a pop up every 30 feet asking you if you want to access the AR content on another server? Will you then have to register, subscribe, or whatever?

Or will all AR content be mediated by one sole master control server deep in the bowels of google? What about some other option? Will you need different sets of glasses to access different features and content from multiple sources?

At the end of the day, it should not matter what brand of glasses you are wearing, you should never have to deal with AR server popups to join/subscribe, and so forth.

Interoperability, in the context of what I was saying earlier, is the sense of how to build the infrastructure so all of this is seamless to the end user, but still maintaining the features/functionality necessary for all of what augmented reality promises us…I dont want to see everything in AR space, I want to be able to tune in or filter out some things, and I want to customize the snot out of what I see (perhaps changing metaphors or “holoscapes”), and so on. It all has to work together and simplify the end-user experience or it won’t get anywhere

Tish Shute: So what caused the stagnation of new development and devolution of MMOGs in you opinion?

Robert Rice: yes, look at all the hope and hype for the mmorpgs released in the last 12 months really, what is different or better? Now, what is worse?

I bet any decent mmorpg gamer could give you a list of 2 or 3 things for the first question and 20-30 things for the second.

And, VWs seem to be stuck in a feedback loop

Tish Shute: feedback loop?

Robert Rice: Imagine nailing one of your feet to the ground and then trying to run ’round and ’round and ’round.

Tish Shute: Why do you think this happened to VWs?

Robert Rice: Men in suits and flashy watches.

actually, hang on…..

I saw a video clip the other day from a conference about using various virtual and game technologies for simulations and other real world applications several people were talking about “avatar technology” and how theirs was better than their competitions and what not.

Now, can you tell me what “avatar technology” is? Avatar technology is a red herring. Avatar technology is the same thing as calling a toaster a new “fire technology.”

Robert Rice: The problem is that a lot of people that don’t have a clue about what they are doing are selling the tech to other people that have no clue what they are buying, but they feel like the should for some unknown reason.

That is happening all over the government, academic, and industrial sectors now with a few companies selling virtual worlds (again, mid 90-s tech) as the ultimate solution to all problems.

Anyway, getting back to your question

Once virtual reality started getting some buzz, some people got greedy and jumped into the avatar/virtual world thing and tried making it commercial too soon half of the 3D chat worlds were being jammed into platforms for virtual shopping malls.

Most of the money funding tech R&D started funneling towards VRML, and doing 3D in web pages, etc.

Tish Shute: yes horrible idea trying make web pages 3D IMHO

Robert Rice: The money people got involved too soon, and then the greedy people jumped in and tried patenting everything possible. Take a look at the worlds.com patent for 3D worlds.

They filed it back in 2000 or so and it was awarded in 07 (it shouldnt have been in my opinion) now they are suing everyone they can.

Tish Shute: Will there be patent wars in AR?

Robert Rice: Yes, the AR patent wars will be legendary once people start waking up to the real potential here.

The only solution is for everyone to band together and pre-emptively patent or make public domain every possible patentable concept, technology, or implementation for AR otherwise, you haven’t seen anything yet.

Tish Shute: Is the AR community organized enough to do that yet?

Robert Rice: That depends on how my company fares in the next six months.

Tish Shute: Will you patent or make your tech public domain?

Robert Rice: I plan on patenting the snot out of everything we can possibly think of, and then giving away our content creation tools and SDK stuff for free. The whole goal of what we are trying to build is to empower the end user and facilitate the creation of a wonderful world of augmented reality.

There are some things we will make public domain for sure, on top of that

Tish Shute: So back to my question on networked real time experience. Will we have networked Real time AR experiences in the next 18 months

Robert Rice: It is possible, yes. Other than what we are doing, I am not aware of anyone else taking the same approach we are, but the potential for an “under the radar venture” (much like my own company) is definitely there.

Tish Shute: Will you use cloud computing?

Robert Rice: I think that’s overrated and probably another attempt at the whole “thin client” model that some companies have been pushing for the last 20 years.

It sounds good on paper, but ultimately takes power and control away from the end user.

Tish Shute: cloud computing?

Robert Rice: Yes. You know, we aren’t playing around, We are totally building “THE AR” that everyone keeps dreaming about. None of this cute stuff you see on youtube. Actually, if you want to see the things that have inspired our vision of what we want to build, check out:

* Dream Park by Larry Niven and Steven Barnes
* Rainbows End by Vernor Vinge
* Spook Country by William Gibson
* Halting State by Charles Stross
* The Diamond Age by Neal Stephenson
* Donnerjack by Roger Zelazny and Jane Lindskold
* Otherland by Tad Williams
* Neuromancer by William Gibson
* Idoru by Wiliam Gibson
* Cryptonomicon by Neal Stephenson

and watch the whole anime of Denno Coil (subbed NOT dubbed!).

Tish Shute: So scaling the real time experience won’t be a problem in your project hehe

Cos no sharding allowed in AR right

And if you have lots of API calls?

Robert Rice:: haha, sharding is one of the dumbest things to happen to the VW/MMO industry

It is a solution to a technical problem that was relevant 15 years ago.

Tish Shute: so why did it stick (i know men in suits)

Robert Rice: it stuck because “thats what the other guys did” and the mmo designers are too lazy to reconcile gameplay for PvP and RP gamers

However, there is a curious problem between dealing with “one world” and “anyone can start their own custom AR server”

Tish Shute: Now that is a very interesting problem the one world and own AR server

Robert Rice: It took me a few weeks of not sleeping to figure that one out. It gets back to the interoperability issue

Tish Shute: What did you come up with?

Robert Rice: a solution. Thats all I can say for now on that.

Tish Shute: eeextra seeekrit!

Well I will definitely have to bug you on that.

The problem has produced some creativity in OpenSim with people coming up with hybrids of p2p and oneworld

Robert Rice: As far as virtual worlds are concerned, they need to look at the problem from a different perspective. They are trying to make all virtual worlds interoperable intead of creating a new model for interoperable worlds that new ones will be created to adhere to.

Tish Shute: well some people are. I would say most OpenSim developers see their modular approach doing this.  And you choose to interoperate based on what modules you have activated and then social agreements…

Robert Rice: hrm, thats a start, but that only works on a functional and social level – doesn’t account for content (story, mythos, game rules), unique data (my +3 sword), or the concepts of commerce, inherent value, and intellectual property

Enabling my WoW avatar to run around in SL and vice versa creates more problems than it solves.

Its like two alien races working hard to make sure that their two spaceships can dock but no one is paying any attention to the fact that race A breathes nitrogen and race B breathes sulpher.

Its technically possible, but they are missing the boat on the content side of the problem.

Tish Shute: Yes but don’t you think when a modular open source tech for virtual worlds  becomes pervasive, what will happen is that those interested in a similar genre will increasingly use the module in ways that allows their content to interoperate if they want it too

Robert Rice: everyone has to use the same backend tech, and the front end clients need to adhere to the same standards. Bu I have to admit, I haven’t been paying much attention to the vw space in the last 9 months or so.

Oh I have to run now.  But download and install cooliris. I promise you will be blown away and will start using it to search for images and videos

Its frigging awesome.

Tish Shute: Will do!  Thanks so much great talking to you. I can’t wait for your launch.

Junichiro Makino, University of Tokyo, leads the way into the National Astronomical Observatory of Japan (NAOJ) in Tokyo. Piet Hut, Institute of Advanced Studies, Princeton, is right behind with the Genkii team. Jun Makino has offered the use of a server at the observatory to set up an OpenSim environment. History is about to be made.

Thanks Genkii for the photos! Genkii is a Tokyo-based strategic consultancy focusing on social media and virtual worlds (see here for my interview with CEO of Genkii, Ken Brady and COO, Adam Johnson).

Next day Piet Hut announced to the Meta Institute for Computational Astrophysics MICA:

Adam Johnson and Jeff Ames, developers of OpenSim, and members of the Tokyo Genkii team, have succeeded today in tweaking their physics engine in OpenSim to let stars dance according to Newtonian Gravity.

On a Mac laptop, they let hundreds of stars move in real time, as a gravitational N-body problem (yes, a few hundred!). This is a historic watershed.

A video was soon up on YouTube here. But as Adam Johnson noted:

I don’t think the video can capture the pure fun of this thing, it makes computational astrophysics approachable to a 3 year old… like legos for astronomy. And it really puts OpenSim in a new light! This same method can be applied to other areas too.. think protein folding visualizations.

Computational Astrophysics is a field that has long been associated with some serious number crunching. Jun Makino is holding up a piece of the new GRAPE (an acronym for “gravity pipeline” and an intended pun on the Apple line of computers) – a super computer that will become one of the fastest super computers in the world (again).

Later in this post there is an in depth interview I did in Second Life with Jun. His Second Life avatar is Makino Magic. Jun discusses the future of computational astrophysics, and how this may be tied in with virtual worlds

The photo above was taken by Adam Johnson of Genkii. Adam noted:

We want to use super computers like this (the Cray and GRAPE) to offset the simulations and push the results to OpenSim. That way it can simulate thousands, or millions of stars/planets

The picture below shows the Cray at the National Observatory of japan on the right, and on the left Jeff Ames is shown working on the N-Body simulation in OpenSim. I heard Jeff implemented most of the code for Newtonian Gravity in OpenSim on the train ride to the observatory!

Genkii, in addition to their work on OpenSim are developing, with 3Di and other open source virtual world developers, an OpenViewer a “from scratch” that should allow people like scientific researchers, game developers, educators, etc to make fully customized viewers more easily than they could before. Adam explained:

Open Viewer is (BSD Licensed) using, at the moment, OgreDotNet for rendering and LibSL for the protocol.

But you can use any protocol you want actually and any rendering engine – ideally we want to allow it to support numerous virtual worlds with one viewer. We have been talking to HiPiHi guys about getting their protocol working with it as well, and soon will talk to some IBM guys in China to see if they want to take part in that action.

MICA – Pioneering Astrophysics in Virtual Worlds

Piet Hut has been evangelizing the potential of virtual worlds for astrophysics, and bringing astrophysicists into virtual worlds through the Meta Institute for Computational Astrophysics (MICA) for quite a while now (see my previous post). And, Piet and Eiko Ikegami have just published a fascinating paper about Japanese History and Second Life.

MICA, begun in 2007, is a virtual astronomy institute featuring many MODEST-related activities. Aimed at harnessing the capabilities of virtual worlds and 3D collaborative environments (such as Second Life, Qwaq, Sun Wonderland), it fosters interaction among astrophysicists with interest in large-scale simulations, including dense stellar systems. Outreach and educational activities are also major MICA goals. MICA weekly events include popular talks, computational astrophysics lectures and Journal Club discussions of recent astro-ph papers. The MICA wiki, containing more information, schedules of events, and links to related pages, can be found here.

Also, there is an excellent video of Piet’s talk on the “Scientists and Science Outreach in Second Life,” that was part of the SL5BD events posted on YouTube here.

In the picture above, MICA members Prospero Frobozz (Prospero Linden a.k.a. Robert Knop in RL), Tara5 Oh (me, Tish Shute in RL), Pema Pera (Piet Hut in RL), Peter28 shostakovich (Peter Teuben in RL), Paradox Olbers (Spike MacPhee in RL) see also Paradox’s Spindrift in Scilands blog, Pan Numanox (Alfred Whitehead in RL), Jazz90meteotl Loon, Eamu Godenot (Will Farr in RL), and Lagrange Euler (Steven McMilland in RL). Thank you MICA member and SL photographer Kirk Smythe (Tom Deluca III in “real” life) for this picture, and for the portrait of Piet’s avatar Pema Pera below.

I have been attending MICA meetings as an observer since spring when their activities were mostly focused in Qwaq. Now the focus of MICA is Second Life and OpenSim. There is a regular schedule of events. In fact, I was invited to speak about Open Source virtual worlds at one of the daily “coffee” meetings. I later heard that my talk was in fact the trigger for the collaboration between Piet and the Genkii team!

I invited Adam Johnson to come to the discussion after my talk when the conversation focused on OpenSim. After I introduced Adam to Piet, they realized they were both in Tokyo in RL, they met for lunch and the rest is history.

I have blogged and attended a number of MICA’s events including Dr. Rob Knop’s (a.k.a Prospero Linden a.k.a Prospero Frobozz) talk, “The Power of the Dark Side: How Dark Matter and Dark Energy dominate our Universe.” Dr. Knop was on the team that discovered the accelerating expansion of the universe.

Why does MICA want to do simulations within Virtual Worlds?

I asked Piet (Pema Pera in Second Life, picture below) why MICA wants to do simulations in virtual worlds and, why N-body simulation is so important to astrophysics?

Pema Pera: For two reasons:

1) Traditionally simulations were where scientists spent years and visualization was an afterthought, a few pictures in a journal, never the right tools to really mine the data, not enough time, money, tools. So we can use VWs to start with visualization and then have the simulations follow. Another example of inverting a traditional priority, like going to what-you-see-is-what-you-get. Most breakthroughs in computer use are like that. So starting with the enormous investment already in the visualization aspects of Virtual Worlds, you can then run your simulations within them, or so it seems. Whether they actually run in there, or in a hidden way elsewhere on a Cray or GRAPE is of no real concern for the user.

2) Collaborative code writing and debugging. Traditionally, observers collaborate — can’t build a telescope and observatory with one person. Theoretical folks now have to learn to collaborate. You can’t write a whole simulation package in one PhD three-year period.

Most breakthroughs are inverting something, first direct addressing to make the computer fast, but a pain for humans, then you switch to indrect addressing, makes more sense for the human, slower at first for the computer, but the advantages in software productivity are overwhelming. Same with going from assembly code to compilers, same from complied languages to scripting languages

Theorists are learning to collaborate and they have to write computer code together! What better way than to do pair programming in a virtual world, so pair coding and pair debugging becomes possible within a VW, especially when you can see the results in your simulation.

Tara5 Oh: oooh that is interesting, so being able to “play” with a physics engine could be very key!

Pema Pera: YES! You simulate at large scales VERY directly into the fabric of the simulated world iteself. It is the difference between having a stage on which you play and having the stage itself play for you!!!!

Tara5 Oh: Why are N-body simulations so key to astrophysics?

Pema Pera: Ah, great question! Gravity dominates. On short scales we have electricity and magnetism and friction and much more but on large enough scales only gravity is felt. Electricity cancels + and – charges but gravity is only attractive, hense a gravitational N-body problem. You cut up the world into N parts as large an N as you can handle and then you simulate the UNIVERSE.

Will Farr (MIT) implements the “Hermite Algorithm” in OpenSim

The Thumbnails of screenshots above (in order) are from a simple N-Body simulation of 32 bodies in OpenSim. Basically, you can see the bodies collapse, eject some of their members, and form a cluster of the remaining bodies in the sequence of photos

It wasn’t long before MICA members were deep into OpenSim. Notably, Will Farr, Eamu Godenot in Second Life is a MICA member and a 5th-year graduate student at MIT working on numerical relativity and N-body simulation algorithms, “lots of theory and numerics, very little observing, unfortunately,” he told me. Will was one of the first MICA members to pick up on the N-body work in OpenSim.

Tara5 Oh: Could you explain what you have been doing with n-body simulation in OpenSim?

Eamu Godenot: When Adam and Jeff wrote a Newtonian physics engine for OpenSim, this was a couple of weeks ago. they showed that it was possible to run gravitational simulations using a custom physics engine, but they didn’t know a lot about n-body simulations. I don’t know a ton, but I know more, and I was coming to Japan to visit Piet/Pema and Jun/Makino. So, Piet suggested that we try to improve the algorithms that the physics engine used. Maybe enough to do some physically relevant simulations

Tara5 Oh: You all met in Tokyo?

Eamu Godenot: Yes, it was exciting—I met Adam and Jeff two days ago, and spent some time learning about OpenSim and the code from them. Plus pestering them with lots of questions about genkii, programming, and OpenSim in general. Then, on Wednesday and some of Thursday, I started improving the n-body part of the physics engine. They had implemented a nice algorithm for evolving the bodies, but the standard ones used in the field are more efficient and accurate. In particular, Jun Makino invented a technique called the Hermite algorithm that is, more-or-less, the “industry standard” So, I implemented that.

Interview With Jun Makino (avatar Makino Magic)

Tara5 Oh: MICA has been doing some work on N-Body simulation in OpenSim in which you have access to the physics engine. How doe this open up new possibilities for astrophysics to use a general simulation platform in conjunction with super computers?

Makino Magic: Well, what Piet and other people have done so far is mainly to implement some simple models directly as simulation modules in Opensim. That itself is nice, but not quite enough to share and work on really high-end simulations. However, once we learn how to show things in SL/OpenSim, it *should be* not too difficult to connect the Opensim program directly to data visualization or analysis program which runs on any other computer, or actually connect it directly to a supercomputer or GRAPE.

Tara5 Oh: Could a general simulation platform like OpenSim ultimately become an operating system/user interface to super computers? And do you think this could bring a new kind of interactivity to number crunching?

Makino Magic: I think it could. That’s rather similar to the web browser becomming an general user interface for whatever programs people develop. Well, I think the important thing here is the possibility of people in different places working on a single dataset or simulation in really real-time.

Tara5 Oh: Yes it is really my big dream for virtual worlds, or rather for the free form, programmable, 3D collaborative space exemplified by OpenSim. to make the previously invisible world of software becomes visible in a shared interactive environment!

Makino Magic: Yes, not just dataset and simulation, but the possibility to share the program for simulation itself and to do the development in that way like pair- or “Extreme” programming, so more than two people physically far away that can really change the way we do research and development.

Tara5 Oh: Yes! What is you role in MICA?

Makino Magic: Well, at this point not much. I have been too busy with my real-life duties….

Tara5 Oh: What do you think MICA could achieve re integrating virtual worlds in astrophysics? And what to you think you colleagues who are the first evangelists of the use of virtual worlds in astrophysics may be overlooking?

Makino Magic: Well, at this point I cannot predict. But the simulation stuff implemented in OpenSim looks very promising

Tara5 Oh: Do you have any different ideas from Piet re the the role virtual worlds can play?

Makino Magic: Hmm, I must say I do not know all the details of Piet’s ideas, but for me the first step would be to use the environment to really do some collaborative research and discusssion, and we do not yet have simple tools to do so…

Tara5 Oh: And what are the most urgent tools you need now?

Makino Magic: A very first thing, which might be actually there, is to really work in multiple windows much in the same way as yo do in the pair-programming. You should be able to use one editor window, a graphic window to show result, etc, and should be able to edit one file….

Tara5 Oh: and secondly?

Makino Magic: Well, that is hard to predict… But clearly we need a “virtual” 3D “screen” , in which we visualize 3D results and see, walk into or fly.

Tara5 Oh: And is the N- Body simulation in OpenSim the first step to this last requirement?

Makino Magic: Yes, I think so. And that is useful not only for real research but for educational or public-outreach type stuff. Which is also important to get the attention of wide range of people, many of them are better programmers than astrophysists, like the people from Genkii who implemented the first N-body simulation in OpenSim.

Tara5 Oh: Do you have an OpenSim running in your own lab yet?

Makino Magic: Well, the server is running, but I have no client PC in my office yet (My windows PC in my offiice is toooo old and slow…)

Tara5 Oh: What is the contribution that being able to play with physicis engine so easily in OpenSim makes to N-Body simulation research?

Makino Magic: The most important thing is that you can show the result to people, including yourself, much easier than without OpenSim. well, hopefully…

Tara5 Oh: Piet said something to me that I was interested by: “If we really want to reverse simulating and visualizing, we can IMAGINE doing everything on the visualizer and then using a “simulations accelerator” (the opposite of a “graphics accelerator”) to get speed.” He went on to say, “similar techniques can be used in visualization as in simulations, actually, where also you don’t compute the force on one particle from all the billion other particles in a large simulation, so we have a lot of experience there already both conceptually and practically.”

Makino Magic: Hmm, Well, for me it is not too important where the simlation is done, but it must be reasonablly fast. Currently, really cutting-edge large-scale simulation can take months on the largest Cray or GRAPE.

On the other hand, lots of things can be done real-time, and my guess is Piet is mainly interested in those kind of things. There are many interesting problems for which we need to do simulations, but not the largest ones on supercomputers.

Tara5 Oh: So there are many interesting problems for simulation that do not require a super computer?

Makino Magic: Yes. for example, just by solving many three-body problems we can learn a lot, and can even publish an interesting paper in Nature (we did in 2004).

Tara5 Oh: What will scientific computing look like 25 years from now?

Makino Magic: Well, Piet has been in on that for more than 25 years, and I’m approaching to that many years, but it is hard to predict….

Tara5 Oh: But just some speculation, hehe!

Makino Magic: Well, one thing is that it’s becoming more and more difficult to develop large simulation programs. Much of the code we use now has been maintained for more than 20, or 30 years. In that sense, even though the hardware changes, the software changes very very slowly, and that is the problem.

Tara5 Oh: Do you think the intereactivity of virtual worlds could be useful in solving or preventing the problem (frequent in science) of 25 year old code still being in use?

Makino Magic: Well, interactivity itself might not be the solution, but the way SL/OpenSim works and the way it talks to other simulation programs might be able to change the view. In some (many) cases, the program to be solved is really simple, like one line of and equation of motion, resulting in millions of lines to be able to efficiently solve a partucular kind of initial condition with very limited computing power. I do not say there will be any real solution, but there may be some alternative approach.

Tara5 Oh: Do you think that a virtual astrophysics organization like MICA could apply and recieve funding for this kind of research on their own, or would they neeed a RL university partner?

Makino Magic: In the case of MICA it is not difficult to get RL partner, IAS or Drexel or whatever.

[The MICA steering meeting began to convene at this point]

Tara5 Oh: Have I missed asking you any important questions do you think?

Makino Magic: Hmm, no.

Tara5 Oh: Thanks so much Makino!

Makino Magic: Okay, this was fun! Thanks a lot!

Tara5 Oh: And I look forward to hearing about the continuing story in the future!

Makino Magic: Yes, bye for now.

Imagine an L-System forest, a herbivore simulation and a carnivore simulation all developed separately without each having its own graphical front end. Each object in the separate simulations would communicate locally or via the network using some agreed upon protocol. Next, picture one or more 3D front end “view portals” with all the bells & whistles that visualize what is going on in the engines and traffic, putting any local “area” together into a coherent scene.

If it existed, such an A-life system could be run as a true grid, an “Evolution Grid” or “EvoGrid” if you will, with the computation not limited to one processor or one 3D scenegraph’s rendering step clock. Developers could focus on their areas of strength while the quality of the collective simulation grid would improve much faster than any one individual effort. And perhaps best of all, new developers could connect their engines, protocols or view portals into the grid or take up development of existing engines and protocols so that no projects need stagnate or die. So with this vision in hand, is something like the EvoGrid possible, workable, desirable, and doable? (Bruce Damer, 2008)

When Bruce Damer told me he is working on evolution technologies (ETs) that will come “alive” towards the end of the 21st/beginning of the 22nd Century, I pricked up my ears!

A world renowned guru of our digital past and future (see of Bruce’s projects at his personal page Damer.com and his Digibarn Computer Museum), Bruce is in the advance guard of many emerging fields, including: social visual computing – avatars and virtual worlds (see his book Avatars, 1997 and compendium of Avatars events); NASA research – surface robotics, spacecraft and mission design, agent-based modeling, and real-time physics (see DigitalSpace for space projects from 2000-2008); and, artificial life – cellular automata, complex and emergent systems (see Biota.org and the Biota Podcast with Tom Barbalet).

Bruce Damer’s architectural notes for the EvoGrid router, a finite state machine that will consume XML

EvoGrid – an evolution technology grid

Bruce envisages a grid (EvoGrid.org) in which current work on artificial life research being done with teams at NASA, universities, and in the artificial life developer community, e.g., the work of Ventrella (see below), can interact in a common ecosystem.

Bruce plans to build EvoGrid on an open source framework, communication grid and protocol (Zhengyou & Yichuan 2004) allowing future developers to extend the EvoGrid and add their own objects or virtual creatures.

By running the simulations without visuals, the evolutionary algorithms of EvoGrid will be able to develop huge populations that can interact with other large populations evolving in real time. But, with their emergence into the social visual 3D space of a virtual world, they will hit the wall of physics.

The public 3D immersive portal into the EvoGrid will support the simulated evolution of biologically inspired forms. The portal will be a virtual space where they will interact with human users.

This window into the human world raises many interesting questions. Will the algorithms/artificial life forms themselves decide when to emerge into the public eye? Or, will they be pushed out by other life forms, or summoned forth by human voyeurs/god(s)?

But, regardless of how artificial life algorithms eventually emerge into cyberspace, this will be an important step in exploring the far reaching implications of the possible emergence of artificial life from algorithms into atom space.

From Algorithms to Atom Space

J. Doyne Farmer defined a living thing as a pattern in spacetime, able to reproduce itself using a stored information blueprint, employing an internal metabolism driving interdependent parts to interact with and deal with a chaotic environment. Above all, he put forth that a lineage of living things also possesses the ability to evolve through time (Farmer & Belin, “Artificial Life: The Coming Evolution,” 1991).

Our world and ourselves are products of this collective natural technology (whether one believes it is guided by an unseen God or not). Others have argued further that human culture, whether it is in the form of writings, music, ideas and the arts also employs some of the same underlying methods to spread and evolve (Richard Dawkins, The Blind Watchmaker, 1986). Therefore, beyond the physical laws of nature, the most powerful force shaping the universe is what we might call “evolution technology.” (Damer 2008)

The picture above is from Jeffrey Ventrella’s website.” JJ Ventrella is a programmer-artist doing virtual world design and artificial life research. He was Principle Inventor and second co-founder of There.com, and most recently, Senior Developer at Linden Lab. “Ventrella writes papers and chapters on topics centered around evolutionary computation and creativity.”

Ventrella is also the creator with Brian Dodd of Darwin Pond.

Darwin Pond is an imaginary gene pool, a primordial puddle of genetic surprises. More technically, Darwin Pond is an Artificial Life Simulation: a virtual world exhibiting the emergence of life-like behaviors. But it’s more than just a fun and informative thing to watch, you can participate in this artificial life simulation by building scenarios and setting up experiments.

What are the possibilities for artificial life?

Evolution technology is the use of the principles of evolution as seen in nature to rapidly develop new software, chemicals, genes or materials, devices or full robotic machine systems.

Bruce is developing EvoGrid to ask some big questions about our future: Can simulation be used to deduce how life came about? Can simulated biological environments be used to create powerful and transformative technologies? Can artificial life evolve into semi-living machines that can clean our atmosphere and heal our bodies?

“Evolution is a powerful tool,” Bruce notes. It can be used in constructive or destructive ways. “We should use it to make tools – the mechanisms by which we will survive and thrive.”

When our bodies are married to this kind of technology we may live for hundreds of years.

Bruce sees the next generation of space exploration emerging out of these artificial life forms born in cyberspace.

They should be able to work in actual hardware – intelligent manufacturing done at the lowest molecular level.

If we are truly going to travel and live beyond the earth’s biosphere we have to go beyond the 19th century technology that space exploration has depended on up to now. Our spacecraft would be recognizable to the great steamship engineer Isambard Kingdom Brunel, made of pressure vessels and other hard parts. These craft are fragile and subject to “single points of failure” (i.e, one seal goes and there goes the mission).

In order for longer term survivable spaceflight, especially for human crews, these craft will have to almost be alive, or at least be made up of billions of individual micro or nano-parts that are self monitoring and self healing. In this scenario, human crews are going to be like the brain organs in a larger biologically inspired vessel. I believe we are decades, maybe even centuries, from this kind of technology, but it will come.

In addition, evolved biologically-inspired robotic systems will mine outer space resources and prepare the solar system for Earth-life.

Picture trillions of flakes of solar collecting chemical nano-factories working something like an “ET lichen” coating the surface of a richly endowed asteroid, processing its stores of water ice, organic compounds, or metals. Human crews would stop by such asteroids to allow themselves (and their ships) to “feed” on the ET lichen. Indeed if the ET lichen manage to hollow out the asteroid and generate the correct mix of gases then the human crew could step inside for a break.

Other versions of “ET lichen” would have the potent capability of terraforming our own planet enabling us to cope with climate change and other effects of our civilization. As many science fiction writers and Hollywood directors has shown us, out of control ET lichen may also lead us to total annihilation.

The Artificial Life Programmer, the New Alchemist?

Like the medieval alchemists before them, programmers developing “Artificial Life” software (often shortened to “A-life”) are drawn to the elusive yet seductive proposition that they have the power to animate inanimate matter (Steen Rasumussen), except that in this modern incarnation of alchemy; the inanimate medium is a microscopic substrate of billions of transistors. (from “God, Science, and Intelligent Design,” chapter by Bruce Damer, upcoming in World Scientific, Singapore)

Bruce points out there is frequently confusion between the two fields of Artificial Life and Artificial Intelligence. But this confusion, he notes, is a fertile field of inquiry.

A-life is a “bottom up” approach, wherein developers simulate “a large number of simple interacting components employing relatively simple rules from which complex behaviors of whole systems emerge (Chris Langton et al). AI on the other hand has tackled the ever receding goal of creating a “conscious” entity with which we would one day be able to communicate.”

God in the A-Life Universe

In his article for an upcoming book, “God, Science, and Intelligent Design,” Bruce undertakes a thought experiment in which he draws insights from the field of A-Life into a broader Intelligent Design/Creationism vs Evolution discussion.

The open question, “what is life?” underwrites the field of A-Life much as the question “what is consciousness?” does the field of Artificial Intelligence. And, these questions beg others on the role (or absence of role) of God(s).

Will Wright’s Spore: “God as the Intelligent Designer”

The screenshot above (see CNET for more) is from Spore the much anticipated new game from “Sims” creator Will Wright. Electronic Arts just announced that Spore, released through its Maxis Software brand, will go on sale on the weekend of Sept. 7, 2008. It is billed as “massively single-player” game, that “lets users create a universe, evolving from tiny organisms into civilizations capable of intergalactic travel.” (Canadian Press)

Bruce compares Spore with Karl Sim’s Evolved Virtual Creatures and argues they demonstrate two kinds of God in the A-Life universe:

Karl Sims’ – God the Mechanic setting up the initial conditions and then returning only occasionally to view the current state of the simulation; and the Will Wright, Intelligent Designer God, constantly providing opportunities to use and outside intelligence to steer the direction of the virtual universe.

The properties of A- Life software in its early phases can be represented along a continuum which at one end can be represented by Karl Sim’s evolving virtual creatures (see below) and on the other end Will Wright’s game Spore.

Karl Sim’s “Evolved Virtual Creatures” – “God the Mechanic”

Karl Sim’s creatures start life as a simple pair of hinged blocks in a virtual universe that simulates basic physical properties such as gravity, collisions and surface friction. From that point on the simulation was allowed to continue on its own without human intervention (although random mutations were introduced automatically into the “genome” of creatures between generations).

Bruce Damer’s: “God As the Intelligent Adapter”

Bruce analyzes “the copying rule,” as a fundamental principle of life in his article for the forthcoming book “God, Science, and Intelligent Design:”

A living organism differs from bare rock, gasses or a pool of liquid in ine very specific way: the living organism contains instructions that are copied, for the most part unaltered, from one version to the next.

Bruce argues, quite brilliantly, that the Copying Rule, along with the “Laws of Nature” and the element of uncertainty leads us to a notion of God not as an influencer of the future but as an “adept adapter” . And if you don’t see God in the picture the understanding of evolution by cumulative adaptation is even more remarkable for the fact no hand guides it.

Those who wish to celebrate the presence of a God in their lives and in all nature can believe that, God as the Brilliant Adapter, played a hand in the survival and glorious diversification of life on Earth as well as the blossoming richness of human culture and technology. Those who see no need to place an actor like God in the picture can celebrate and seek to better understand the process of evolution by cumulative adaptation, made even more astonishing by the very fact that no hand guided it.

“God who created all things in the beginning is himself created by all things in the end” Wrote Olaf Stapledon in 1937.

Performing the Future

The questions evolutionary technologies raise for human and planetary future are vast and far reaching. To explore the huge social and philosophical questions (Damer 2008) raised by the outputs of more advanced EvoGrids, as well as an enumeration of how evolution technologies will impact life on Earth and in space in the future, Bruce is planning a performance piece that will tour the world – “After the Evogrid.”

As a kind of cyber-hippie empresario (see Bruce above on right and his friend and Digibarn collaborator Al Lundell on the left) of nine virtual world conferences including one just three weeks ago featuring Virtual Worlds and Space for NASA, Bruce has always been deeply involved in firing the public’s imagination about the future. Now he intends to take his involvement with the public space to new heights with, “After the Evogrid,” – a “multimedia performance piece with a spoken word narrative, sound and music, and animated visuals.”

“After the Evogrid” will present both the promise and the perils of Humanity living in a symbiotic existence with the products of the new field of “evolution technology.”

The performance piece will bring together a unique combination of evolution technologies, personal and societal impacts, full biosphere implications and the expansion of life beyond the Earth. The EvoGrid itself will create the first open extensible grid protocol for evolution simulations.

So stand by for launch into an even weirder future, brought to you by “Doc Damer”?!

Recently I participated indirectly in the The Foresight Insitute’s Vision Weekend. I joined Melanie Swan’s tour of Nanotechnology Island on Second Life which she conducted live as part of her virtual worlds presentation to the unconference.

I was delighted to meet Melanie, the principal of MS Futures Group, for breakfast in New York City a few days later. She is a futurist and hedge fund manager based in Silicon Valley. Melanie has a highly nuanced understanding of the intersection of technology with social, political, legal, regulatory and economic regimes (see her post here). Melanie is also one of the organizers of She’s Geeky. Unfortunately I couldn’t make the lunch last Saturday here in New York City to discuss She’s Geeky EAST this winter.

Melanie and I had a fascinating discussion over breakfast about many aspects of the metaverse including it’s potential use as a platform for Artificial General Intelligence. Ben Goertzel (see my post on Ben Goertzel’s work here) also presented on “AGI meets the Metaverse” at the Vision Weekend and Melanie mentioned the very interesting work of Monica Anderson who presented her Artificial Intuition approach.

Picture is of Melanie Swan in NYC and on Nanotechnology Island in Second Life.

Last Saturday I went to the Book Party (and after hours party) celebrating the launch of, “Second Life Herald, the Virtual Tabloid that Witnessed the Dawn of the Metaverse” by Peter Ludlow (Urizenus Sklar in SL) and Mark Wallace (Walker Spaight in SL).

The book, a history of the Second Life Herald which began in 2004 as The Alphaville Herald in The Sims Online, comes out at a very interesting moment.

The sun is rising higher on the metaverse(s) and there is much speculation about a bright day to come in some quarters (e.g. Ugotrade), more skepticism and a wait and see approach from others (e.g., Gartner), and fears of a “high noon” kind of show down between a “bottom up” user generated creation culture versus “top down” corporate control (e.g. Second Thoughts). For a thoughtful look at “Do virtual worlds liberate us?” see Ren Reynold’s post on Terranova.

Ludlow’s and Wallace’s book not only looks at a crucial time for the metaverse, its birth, it is also a study of some of the most important questions about the metaverse’s expansion. One question that motivates my own writing is quoted by Ludlow and Wallace in their intro. Legal scholar Lawrence Lessig in his 1999 book, Code and Other Laws of Cyberspace:

the very architecture of cyberspace is up for grabs: “Depending on who grabs it, there are several different ways it could turn out.”

Ludlow/Wallace’s approach to this challenge is very different from my own. I focus on the blurring of virtual and real worlds and how this access and control to data and meta data that will certainly empower business and government can also be available to benefit people and the planet. Also I try to keep people who do not yet have access to cyberspace in the conversation where possible. Ludlow and Wallace, on the other hand, focus on stories from some of the first people who began living much of their lives in the metaverse and “the conflicts between the owners of virtual worlds and their users, and between groups of users, and between individuals.”

I found myself aware of The Herald’s “mission” very soon after I began blogging about Second Life. Prokofky Neva, with a typically irreverent Herald turn of phrase, dubbed me “the chirpy whitewasher from Ugotrade” (here ) for taking too lenient an attitude, apparently, to a notorious griefer. And, in the body of the post, I was lumped fairly and squarely in the camp of the chief Herald antagonist, Philip Linden, (aka Philip Rosedale, CEO of Linden Lab) who got a wicked Herald tongue lashing for what was described as his “granola crunching fatty huffing way” of dealing with the same griefer.

For those of you who were in Second Life during the time this book covers hopefully it will bring up narratives and memories that you might have otherwise forgotten. For late sleepers who missed out on the “dawn,” Ludlow and Wallace provide an opportunity to catch up (also see LA Times Review).

For me the book provides an opportunity to look deeper into the question of what the metaverse has to teach the paraverse and visa versa by providing an intimate “Herald style” history of a metaverse, Second Life, that has truly succeeded in creating communities around user generated content.

Zha Ewry, a key metaverse architect and thinker I met in the Architectural Working Group in Second Life, said something that I have really taken to heart recently. Though Zha herself said it with a *chuckle*:

I sometimes, when I am feeling.. difficult.. assert that I don’t really trust the judgment of anyone who has never

1) Lived in Second Life or Everquest Online, or The Sims Online long enough that they can get 20 or 30 residents who count them as someone they know by name and behavior

2) Cracked at least half way up the level structure in World of Warcraft, Everquest, or similar

3) Managed some sort of small social community (wiki, bbs, moderated maling list..etc)

The Ludlow and Wallace team have done all of these three things and done them deep. So when Ludlow and Wallace talk I prick up my ears. On the left Urizenus Sklar and on the right Walker Spaight at his wedding to Destroy TV.

The Metaverse and The Paraverse

Exactly how and when the metaverse(s) and paraverse(s) like Google Earth and NASA’s World Wind actually evolve (and likely merge) to become a phenomena that millions or rather billions of people participate in is unpredictable at this juncture. But the consensus is that is will happen soon.

My own optimism for the future of the metaverse is based on an underlying proposition that the blurring of the lines between “virtual” and “real” worlds can be an exciting and liberating juncture for humanity and the planet (see many previous posts). I asked Peter Ludlow the same question I asked Cory Doctorow in London (see previous post):

1) What happens when Virtual Worlds become flooded with data from “real life” objects, geo- positioning, etc., and extreme life–logging enters virtual worlds? Or as Cory D. rephrased it: “What happens when cyber space everts?”

Peter Ludlow:

well, the blowback of info from RL might be useful for some applications of virtual worlds, but I always found virtual worlds to be fun and interesting precisely because the bandwidth of communication with the real world was *narrow*. I don’t want that crap coming into my virtual space — it kills the atmosphere and sense of presence.

I don’t doubt that massive info blowback will have a role in virtual worlds, but that is the point where they aren’t really virtual worlds anymore but just boring communication devices — information rich telephones.

So if the blurring of the virtual and the real is inevitable (which in my view it is) and I agree with Ludlow that mere blowback of data into virtual worlds is potentially a boring phenomenon: “What can the metaverse teach the paraverse?” And, “How do virtual worlds avoid becoming just another boring communication device?”

“What is most likely to become boring when the lines between virtual and real worlds blur is the physical world.”

As David Orban pointed out as we chatted in skype:

My view actually of the blurring is not that the online worlds will be invaded by the physical worlds’ data but absolutely the other way around. The richness and variety of the online worlds will explode into the physical via interfaces and mashups and we will look back and see the physical world as boring and static.

“Huh, a tree that doesn’t even tell its own species?” without the augmentation, or “How could you meet people who didn’t send ahead their v-agents?”

“If you can drape real information across the physical world there is no reason why you can’t drape imaginary information over the real world.”

Mike Liebhold from the Institute for the Future said this in his presentation at the Stanford University’s Metaverse Meetup (much more on his presentation later in this post).

This room may be a conference room in Wallenberg Hall. But with a click of a mouse or a flip of a switch, I could convert this room into a meeting room on Starship Enterprise. Or right outside the walls in the quad in Stanford you could have a Medieval Tournament going on. You can drape total fantasy, total fiction, total imaginary reality on the physical world.

He showed an example of some dramatic new thinking in the world of video games – a mock up of a game idea from a Nokia sponsored research program in Finland. And Liebhold noted, this is only a hint of the kind of ideas people are working on.

How Not To Be Just Another Boring Communication Device.

And even if cyber space everts it will not become just another boring communication device if the read-write culture that has defined the metaverse (exemplified by Second Life) continues to flourish.

Again I refer to the brilliant Larry Lessig who in his TED talk points out that read only culture was ushered in with the telephone. Lessig demonstrates how the digital age has created new opportunities for read write culture again, even though many of our laws are at odds with this.

In my view, whether virtual worlds remain the heart of a reemergence of read write culture or turn into “boring communication devices” is not so much about “massive info blow back” itself, but more about how the culture that has arisen around social networking and user generated content, again the great exemplar of this is Second Life, is worked out in the confluence of metaverse, paraverse, and meat space.

My interview with Peter Ludlow was conducted by email because the book party was too much fun. I could not ask one of the hosts and a man in demand to retire into a quiet corner. The book party was also a metaverse meetup and packed with Second Life movers and shakers including, Nathan Freitas of Cruxy, Joshua Fouts, Rita J. King of Dancing Ink Productions (Eureka Dejavu in SL – see her blog for more on meetup), Marvel Ousley (see he post on SLNN), Andrea Foster of The Chronicle of Higher Education, Eric Reuters, Jessica Segal (aka Pica Paperdoll, Electric Sheep Company, Andy Fundiger, Marshall Sponder (see his post), Morton Swimmer, California Condor, Donald Schwartz Image Link Productions, Dean Pence, and many many more. I posted some pictures to Flickr.

Notably the party was held in 3rd Ward the artists/entrepreneurs city in a warehouse that is home to WelloHorld – the start up that is the brainchild of co-author Mark Wallace, with Christian Westbrook and Jerry Paffendorf. They are on stealth mode so Mark declined an interview until their launch. But I did snap this chart pinned to their office door that might give some clues to their direction.

The after hours party that Peter hosted back in the Marriot by the Brooklyn bridge was also too entertaining to interrupt. It included both a screening of Peter’s screenshots taken through “dawn of the metaverse” and a very rock ‘n roll drama with the hotel security who were bent on ending the party early. In the picture below Ron Blechner (aka Hiro Pendragon) talks with Peter Ludlow about Peter’s early experiences in Second Life. Mark Wallace is on the left and Boris Kizelshteyn of Combined Story (aka Adonis Bussy in SL) is seated on the couch.

Interview with Peter Ludlow.

I collaborated on questions for Peter with my friends Gwyneth Llewelyn, David Orban and Hiro Pendragon. I will indicate which are their questions.

1) Who/what will be the future competition to the SL – based metaverse?

As you probably know, there are lots of alternatives to second life under development, but I continue to believe that ultimately Trevor Smith of Ogoglio has it right: the metaverse is not going to take off until we have widely available web 3.D development tools in the hands of tens of thousands of website designers. When that happens we will each be building our own little corners of the metaverse and supporting them on our desktop computers. Communications protocols will govern how we move between these worlds and what we can take with us.

2) Your current work is in RL on Philosophy of Language? The new book you are working on – is it a collaboration with the Prof. from NYU I met briefly at the party?

There were a couple of profs from NYU at the party. I’m not collaborating with them, however David Velleman has interesting things to say about narrative and avatars and agency, so I recommend that your readers check out his web site.

Right now I’m working on a book on the philosophy of generative linguistics, which has to do with conceptual puzzles that arise in computational/representational linguistic theories like Chomsky’s. Basically I’m obsessed with that at the moment.

3) What is the future of the SL Herald as an SL institution and what will be your role in it? Will the Herald go the way of SL Insider and start to cover 20 other MMOGs like Massively. That is will it become the Metaverse Herald? (This question was suggested by Gwyn)

In the past the Herald has covered other MMOs, but in the way that a hometown paper covers other towns. Typically we would only cover events in other MMOs if they involved a political protest or some dispute with the game company.

The Herald will stay in Second Life at least until our readers and the people we report on move elsewhere. That is, we aren’t really reporting on Second Life so much as a community that currently resides there. We followed them from TSO, and if they go nomadic on us again we will follow them.

4) What is the relationship between Peter/Uri -similarities/differences? Do you have alts and avatars in other verses? (Hiro’s question)

We all have many different avatars that we use in everyday life. We dress and act differently depending on whether we are conducting business, socializing or whatever. If you think of these ways of acting and dressing as modes of presentation, then you see that it is the same as using an avatar to present yourself or mediate your social interactions with others. Like you I have lots of different alts and avatars even in the real world. You’re talking to one of them now!

5) The digital doesn’t fossilize in one out of a billion specimens, but allows perfect preservation of time-sequences, in the changing metaverse. This means that unavoidably in time it is going to become richer than the physical world itself. How are we going to equip ourselves in coping with this? (Question from David Orban)

it can’t become richer than the actual world because information has to be encoded in physical states of the world. That having been said, I’ve never been impressed by the preservatory aspects of the digital so much as the fleeting and fragile aspects of it. This has been made salient to me by my years in Second Life. I’ve seen so many interesting builds and groups come and vanish. Part of the project of our book was to preserve some of this history. But rereading the book yesterday I was reminded of a lot of events that didn’t make it into the book and may be lost forever.

More generally though, digital media does not give us perfect preservation. You can’t fight the second law of thermodynamics. High entropy will trump low entropy, and there will be lots of bit rot between now and the heat death of the universe.

6) Are we living in a simulation? (David Orban)

There’s a philosopher at Oxford who says that there is something like a 75% chance that we are. But I’m going to go out on a limb and say “I don’t think so.” Actually, I know we aren’t, but if you want that story you have to take my course on skepticism.

7) What is the future of identity and IP in the open metaverse? The call for identity authentication grows louder by the day.

Identity shouldn’t be a problem. It should be possible to establish identity even for avatars using a version of public key encryption for digital signatures. IP is another matter. Technologically, preserving IP is getting close to impossible, but on the other hand if the US Congress keeps passing draconian laws that “give” IP rights for patents like crustless peanut butter sandwiches well then maybe it can be preserved by old fashioned meat space head cracking. Zero tolerance for crustless peanut butter sandwiches. Sell one and you go to prison. The only question is how much of that kind of crap people will put up with. When they finally figure out it is a scam (of if they do) then that will be the end of IP.

8) Is the blurring of the lines “between us and them” – human and machine the “high noon” of the metaverse? Or as Ben Goetzel writes here in his post on Global Brain Memes. ” I think this ties in with Ray Kurzweil’s point that by the time we have human-level AGI, it may not be “us versus them”, it may be a case where it’s impossible to draw the line between us and them…” (Also see David Orban’s Conversation with Ray Kurzweil on YouTube).

I don’t buy this for a nanosecond. First of all, is there even such a thing as “general intelligence”? I’d be surprised. “intelligence” is just a covering term for a basket of cognitive abilities that we prize. If you are impressive at enough of those abilities we say you have intelligence. It’s like athleticism. There is no single property of athleticism, there are rather lots of different physical abilities that we prize. If someone has enough of them we call that athletic. When we say something is “intelligent” we are just saying that we are impressed by it. I remember when playing tic tac toe counted as “intelligent” in AI. It doesn’t anymore because the problem is too easy. We aren’t impressed by it anymore.

Now, on the question of whether we are becoming indistinguishable from machines (and I can’t help but think of Blade Runner here) I am also dubious. First of all, I seriously doubt that we will ever see a machine that can pass the turing test for any significant amount of time and broad range of contexts. But that just goes to the question of whether we could be fooled in conversation. The real question is whether machines are actually like us, and here the real problem is that we have no idea what *we* are like. We have just a glimmer of a picture of the nature of our cognitive architecture and zero idea how that architecture supervenes on our wetware. Well, if we don’t know what we are like, then it is difficult to know how to build something like us. It is not an engineering problem. It is a basic science problem. If we knew *what* to build I don’t doubt we could build it. But what to build?

The State of Play

If you are unfamiliar with the state of play between paraverse(s) and metaverse(s) Susan Kish has an excellent roadmap. My friend VJ also has a nice collection of paraverse links tagged in Delicious. In her report, “Virtual Worlds: Second Life and The Enterprise,” Kish notes, “The combination – whether a Google Life or a Second Earth or another similar entity – could be the ultimate enterprise in Virtual Worlds.”

The question is also: Will this confluence be as important and beneficial to non-profit centered enterprises. For example, the notion of Amazon.org is a social software entity that Bruce Sterling evokes in Shaping Things.

And, of course, very importantly, the question that Peter Ludlow raises – will the confluence NOT be boring.

3D Data For Real Virtual Worlds

I was fortunate to attend the very inspiring presentation of Mike Liebhold from the Institute for the Future titled, “3D data for real world virtual worlds” at the Stanford University’s Metaverse Meetup organized by Henrik Bennetsen. The meetup was streamed into the International Spaceflight Museum in Second Life last week. It was an amazing lens into the state of play in the paraverse. Henrik published the talk abstract before the event:

Abstract 3D data, maps, and software will change the way we compute and interact with spatial services. Moving beyond simple texture mapped terrain and boxes, new 3D mapping frameworks are rapidly evolving into platforms for real world virtual world media, interaction, commerce, and science. In this talk I’ll review work of various groups who are building different components of a 3D Geoweb. I will first describe how their 3D data and software will work as a platform for a 3D real world virtual world, and then, what kinds of new applications and user experiences might be developed on these platforms, and then finish with a brief discussion of prospects and mechanisms for data interoperability allowing users to create, discover, use, and exchange 3D data across platforms.

And Leibhold truly covered everything outlined above! The fascinating talk will hopefully be posted to the web soon here. But there is a very entertaining and thought provoking post up on Wrxli FlimFlam’s Second Life blogSecond Front already. I chatted a little with Wrxli who is a performance artist with Avatar Orchestra Metaverse during the meetup and look forward to more conversations.

Highlights of the talk – Leibhold’s responses to some of the questions.

Once again I asked the question that came from call to action that Cory Doctorow made in my previous post:

1) How can the kinds of data visualization and aggregate statistical information about the world that are frequently only available to big companies and used by them in order to realize profit and control also be put into the hands of individuals?

Leibhold’s response was concrete and detailed but due to the sound cutting out in parts I will have to refer to the recording myself when it is posted here later this week for all the details. But Leibhold mentioned several examples including police crime maps that were increasingly available, and the sensor web project at Microsoft where they have networks of all kinds of environmental sensors out there available freely in GRSS format on Microsoft Virtual Earth. Leibhold pointed out the sensor web architecture at Microsoft is built on common standards will work on a variety of sensors. He continued:

We are also seeing a lot of sensor data collected by life scientists and physical scientists available. A lot of biological information and weather information is going to come on line. There are citizen sensing projects Eric Paulos at Intel Labs, Berkley who has being doing all kinds of things using mobile phones as sensors. There is a group at UCLA called CENS (Center for Embedded Network Sensing) that has a whole project to allow citizen sensing. And Nokia has a project called Sense Web, I think. And they have sponsored research programs at about ten universities world wide to come up with interoperable standards and mechanisms for ordinary people to create and share 3D sensor information and to visualize it as well.

Another very interesting point he made re a larger vision of interoperability was that:

while there was division between the worlds of geospatial standards, the worlds of scene rendering and Hollywood, the video game worlds, CAD, Google will prosper. And Google is creating defacto standards around KML and Collada that we are all going to have to live with.

But when I raised the notion that Second Life’s expansive vision for a new open grid architecture might mean noting that, in my view, “Second Life is also the furthest along re open sourcing of the 3D immersive worlds” (someone from Sun disputed this assertion pointing out Project Wonderland has been open sourced top to bottom since March, and I realized I should have limited my assertion to previously closed immersive virtual worlds). But Leibhold’s response was interesting:

I would dispute the fact that Second Life is furthest along. I think that quite frankly I believe that any day now Google is going to announce avatars and avatar based social networks for Google Earth and the rumors are rampant that they have already tested it. And if that is the case they are farther along. There are structural problems with the computer server architecture in Second Life that restrict the kinds of applications you can run. I think that Second Life is one of the greatest social experiments but technically I think they are going to be eclipsed.

I had IMed Ginsu Linden at the start of the meetup to offer him a TP (teleport) if he wanted to attend. But unfortunately he was busy. But, of course, immediately I shot off an IM to him reporting this prediction of Second Life’s eclipse by Google’s imminent launch of avatar based social networks for Google Earth! Ginsu sent me back this reply:

[8:32] Ginsu Linden: Thanks Tara5. I am actually really looking forward to Google’s entry into the market. Will give people something to chew over.

Yes, it will! And IMed my friend Zha Ewry too reporting this prediction of Google supremacy. In response she pointed out how much depended:

19:25] Zha Ewry: on how Google approaches things,and how much freedom they give their residents, if they are even at all residents, not merely transitory avatars. It will interesting to see how they do at running it.

And of course there is the Linden Lab initiative to restructure the Second Life grid to be watched and participate in through the collaborative effort of The Architectural Working Group.

To return to the theme of this post:

The very architecture of cyber space is up for grabs, and: “Depending on who grabs it, there are several different ways it could turn out” (Lessig).

Don’t Miss the Next Stanford Meetup!

The event is taking place on Thursday, November 29th, 2007 from 6:00pm – 7:30pm SLT/PST and to attend via Second Life you go here:

http://slurl.com/secondlif e/Spaceport%20Bravo/66/74/184/

Physically it is at:

Wallenberg Hall, Stanford University

Jamais Cascio writes about the intersection of emerging technologies and cultural transformation, focusing on the importance of long-term, systemic thinking. His work regularly appears both in print and online, and he has spoken around the world on issues such as the global environment, technological transformation, and political change. In 2003, Cascio co-founded WorldChanging.com, the Utne Independent Press Award-winning website identifying models, tools, and ideas for building a “bright green” future. In March, 2006, he started OpenTheFuture.com as his online home. Cascio presently serves as a research affiliate at the Institute for the Future, as the Director of Impacts Analysis for The Center for Responsible Nanotechnology, and as a founding fellow at the Institute for Ethics and Emerging Technologies.

Talk abstract: The Metaverse — what does it include, where is it going, and how will it change our lives? Based on my work for the Metaverse Roadmap Overview, I’ll look both at the underlying technologies of the Metaverse and at the social, cultural and economic impacts it could have.