This post, and my previous post about integration of OpenSim into Web 2.0, explore how immersive virtual worlds, through a full architectural integration into Web 2.0, will become part of the fabric of everyday computing.

The diagram above shows where OpenSim sits in Web 2.0 (click on the diagram to see a readable enlarged version!). The following interview with OpenSim developer, Teravus Ousley, describes some of the work being done to create documented protocols that will make OpenSim fit seamlessly into Web 2.0 architecture.

OpenSim is in the news a lot these days, explicitly as in the case of the announcement last week by 3Di of their “3Di OpenSim” Standard (for more see here and here), and implicitly with the launch of ChinaQ.  Adam Frisby, OpenSim, pointed out to me if you download the ChinaQ client that it is based on OpenSim, it connects nicely to OSGrid too. There is speculation the client is a rebranded version of the realXtend viewer (which is based on the open source Second Life viewer) as all the version numbers are the same.

So OpenSim is not only attracting the interest of business giants like IBM, Microsoft and Intel, it is becoming the architecture of choice for virtual world initiatives from Chinese and Japanese telecoms (see here and here for more). Also, see the press release about Nokia and the City of Oulu, Finland, joining as supporters of realXtend.

But, as Raph Koster in his post commenting on 3Di’s OpenSim announcement notes, the question how immersive virtual worlds can go from strong niche or enterprise markets to mass adoption in consumer markets must be answered.  As Raph points out, Lively, Whirled, SmallWorlds, Vivaty, and yes, Metaplace have a very different architecture that they hope will attract broad consumer markets.  (I did a long interview with Raph on this at The Virtual Worlds Conference and Expo in LA which I will post as soon as it is transcribed, so more on this soon!).

Architectural integration into the heart of Web 2.0, I would argue, is the key to mass adoption for immersive virtual worlds. While architecture alone will not guarantee the necessary breakthroughs in usability for widespread consumer adoption, it will create the ideal conditions for the innovation through which usability obstacles will be overcome, and the enormous potential for immersive, real time interaction over the internet will be realized.

Interview with Teravus Ousley

Tish: What has been  the most fundamental problem re virtual world architecture that has kept immersive virtual worlds isolated from web 2.0 to date? 

Teravus: a lack of standardization, licensing issues, and the difficulty of entry into the industry.

1) Standardization

Tish: In order of importance what in your view are the priorities for standardization?

Teravus: Probably the same order that OpenSimulator was tackled in, basic connect (current state of OGP – Open Grid Protocol).  Basic Service (interaction standards).  Advanced connect/mashup/aggregate extensions.   Preferably people will have working code in the various spaces there to use freely under various licenses..

Tish: Can you show me where OpenSim will fit in this drawing of Web 2.0 architecture? [Teravus makes some modifications on the drawing I send him from Dion Hinchcliffe’s presentation from his Web 2.0 Expo workshop, see original here]

Teravus: The modified diagram [now opening this post] is a great view of how it will look.

Tish: Why is the TCP stream left out of the original drawing? [For more about Transmission Control Protocol (TCP) is one of the core protocols of the Internet Protocol Suite see here.

Teravus: It is left out because the person who made this diagram had web pages in mind.  Static large files, or small changing files. In the the drawing the fact that TCP streams are smaller then HTTP is on purpose.

Tish: I have heard different opinions on the percentage of the communications for virtual worlds that can be done over HTTP?

Teravus: The fact is that the biggest usage of communications in virtual worlds is transmitting images that’s the number one bandwidth usage. So, if we’re counting by ‘usage’ I say 91%.   If we’re counting by services that use http.   I say probably 75%  I definitely think that http should be evaluated for use on new things ‘first’. But, there are a few places where HTTP doesn’t shine.

I am skeptical about replacing things in the UDP with HTTP  thinking that they’ll ‘perform better. [For more about User Datagram Protocol (UDP) another of the core protocols of the Internet Protocol Suite see here.]

I think there’s been a huge test going on now and for the last 5 or six years with regards to the UDP protocol and it really has performed admirably.   In the last year and a half, I’ve seen attempts to convert several things to HTTP that have failed, and failed somewhat spectacularly sometimes.  In the end the items get reverted back to the UDP protocol. One such item that sticks out in my mind is CAPS(HTTP) based inventory retrieval. The capability to do that in the client has been available since before February. And, it’s been turned on and off on ‘Agni’ at least once in the process. Additionally, we (OpenSimulator) enabled http inventory, and, the  inventory failures rose pretty steeply.

I think some services are really just not ‘right’ for HTTP.. . particularly where a ‘poll’ methodology is used, or, the data is significantly dynamic enough that it makes caching useless.

Anyway, as far as the future is concerned, I do want to see some services over HTTP. Other services, it would be more appropriate to have a TCP stream. Stock market data, for example, uses a TCP stream. The Scalability of the stock market, is just one example of a scalable TCP stream.

Tish: So you see TCP  as the communications protocol that would do the work for the parts of virtual worlds not suitable for HTTP. At least that is how you have shown it in our Web 2.0 architecture drawing. But should there also be a UDP stream?

Teravus: For the virtual world of tomorrow? .. probably not.

Tish: Why not?

Teravus: You have less control over the quality of service when it’s delivered over UDP then TCP.

Tish: What is the exact relation between TCP and UDP.  My understanding is UDP a lower level protocol.

Teravus: TCP offers guaranteed delivery through flow control, while UDP does not.  One of the failures of UDP, is the ‘resend’ technology we’ve put on top of it to try and make it reliable.   TCP does this automatically and better then we could at a lower level but it does also cost up to twice the bandwidth depending on what is being sent. HTTP is a layer on top of TCP.

Tish: So just like the HTTP/TCP discussion there has to be a TCP/UDP boundary discussion …so it is HTTP then TCP then UDP and the boundaries have to be worked on.

Teravus: Those are the orderings in my mind…  probably if UDP uses any..  it should use less then 0.5%.

Tish: And the current Second Life architecture what does it use if it isn’t using HTTP? [to see the work of the Architecture Working Group on the future Second Life architecture here]

Teravus: UDP or HTTP

Tish: and TCP?

Teravus: Well, TCP is a layer under HTTP.  As far as I know, SL doesn’t use TCP streams directly

Teravus: Instead, it uses HTTP polling.  This is one of the places, that I’ve highlighted where it doesn’t shine.

Tish: Polling does sound slow?

Teravus: Polling is essentially..     (connect) Got any data for me? No?(disconnect), (connect) Got any data for me?  No?(disconnect).

Tish: So what is the path to standards for this then?

Teravus: Distilling what we know works and what we actually intend on supporting as far as adoption under these standards.

Tish: Where does MPEG-V fit in?  Have you read their document yet?

Tervavus: MPEG-V is interesting reading…     but is there any working example? I have just the overview. But I’ll read it over to have a better determination of how to ‘keep it in mind’ for the future. It looks like they’ve only really defined the requirements of the MPEG-V spec. The MPEG-V spec looks quite far reaching..  but  the documents so far are requirements and marketing talk aimed toward business people – obviously intended to get more people interested in working on them.

But I have a feeling that any format with MPEG before it will be onerous to support. ..for me it’s too early to tell. It’s quite far reaching…it isn’t anything like ’signal processing’ which the MPEG group is most famous for.

Tish: The whole top down approach of the MPEG-V initiative seems counter to Web 2.0 principles to me.

Teravus: Well, remember..  that even if there’s a virtual world format war (reference to DVD-HD vs BlueRay) we still need to win over the rest of the web.

Tish: Yes and don’t you think the way to win over the web is to use as many existing standards as possible?

Teravus: Well, it’s to use as many existing standards as ‘fit’ though.. KISS, as always (K)eep (I)t (S)imple (S)tupid if we have 30 different internet standards..     people looking at it will @.@

Tish: But it is just lack of documented protocols that has created isolation from Web 2.0?  And really doesn’t it boil down to standardizing that small percentage that is outside HTTP – the TCP and UDP stream that we talked about earlier where the real time stuff that virtual worlds bring to the web happens?

Teravus: no..  actually the HTTP standardization is just as important.

Tish: You mean even though SL used HTTP it isn’t standardized?

Teravus: Not documented specifically.

Tish: And OpenSim is that documented?

Teravus: Not well enough probably to define a standard.

Tish: Is AWG (Architecture Working Group) doing the documentation?

Teravus: working on it..

2) Licensing Issues

Tish: It sounds like some of this work has to go on across client and server.  Are we running into the issue of BSD for OpenSim and GPL for the Second Life viewer?

Tervaus: Well, some of the issue here is license choice.  One of the reasons that libOMV was able to achieve what they did was they did it /before/ the client was open sourced.

Tish: So open sourcing the client actually became an obstacle!!???

Teravus: I don’t think so in a whole.  I think it was great for the community.  I do, however think that C++ UDP stacks will be scrutinized more for GPL license violations because, of course, the client is GPL and C++ .

Tish: It is my understanding that Linden Lab is open to discussions on making the licensing more efficient for the open source community?

Teravus: Well, the client, in a whole, should not be changed as far as the license.   JUST the things that they expect people to adopt should be made more open. If they expect people to adopt PRIMs, then there should be an efficient implementation available for anyone to use..   at the very least, in LGPL format. Otherwise, the die hards are forced to re-implement them from scratch, and most people will just choose something more open.

Tish: Has anyone ever put together a list of the parts that need to be LGPLed?

Teravus: Well, I think it’s there in a few places.  There is at least one jira open on it.

Teravus: A few that come to mind for me..   is the UDP stack and the prim to mesh/UV code.   I think there are some things that can definitely be improved about the UDP Stack.  There are some things, (images come to mind), that would be better over HTTP

Tish: Do you think if the UDP stack were L GPLed that would be a significant help to integrating OpenSim better with the web?

Teravus: Well, it would certainly be adopted by more clients. GPL + (your own code) = GPL Licensed client. LGPL linked library + (your own code) = Your own license.
You still need to mention that you used LL’s UDP stack, and provide the source code for it at request.

The general client itself should remain GPL, it’s better for LL that way.  Just the items that they want people to ’standardize’ on. It would help..   if it was at least LGPL

Tish: And the value to  LL on LGPLing these parts is it helped spread their basic technology while protecting the rest of their viewer?

Teravus: It furthers their goal of standardization on their systems because it allows more people to adopt it for their own uses without worrying about GPL-ing their own client.

Tish: It is hard to standardize without access to the low level parts of the client right?

Teravus: The general population of Developers..     will want a libX that they can plug into their application for communicating.. .  libY to deal with object data..

Tish: Hence your requests for LGPL were  UDP stack and  the prim->mesh/UV

Teravus nods

Tish: and at the moment they only have openmv?

Teravus: That’s the only ‘truly’ open standard right now as far as the LL technology is concerned. OpenSimulator’s use of that data..   could also be seen as a standard..

Teravus: But we have not published anything beyond code..   neither have they  really..  technically..  but their organization of the way things work is very very clear

Tish: What are the most significant limitations of openmv?

Teravus: Probably..  just it not being in c++.   c++ has it’s benefits and it’s pitfalls.  Changes in c++ usually take longer then ones in C#.  But, of course c++ is always faster.  With libOMV It isn’t always clear about what packet is used when.  However, with some experimentation, you can figure it out in 30 minutes or less..

Usability

We didn’t spend much time discussing some of the innovation in usability that this architectural integration into Web 2.0 will enable (more to come on that!). But, Teravus mentioned one interesting use case he is working on.

Teravus: You might also stick a ‘cloud renderer’ into the graphic [Tervaus was looking at the diagram (from Dion Hinchcliffe) that opened my previous post on “Web 2.0 to OpenSim Made easy”  click on the thumbnail below].

Some people have discussed having a ‘video stream’ that’s rendered on the cloud and providing that to flash clients would be the best solution to it for them.

The cloud renderer is for organizations that have large pools of servers with GPUs so would allow for very powerful rendering. The servers can render the scenes and stream them to the low end browsers. It would allow extremely high quality rendering for really low end browsers..  such as ‘cell phones.’

Tish: Is that possible now on OpenSim?

Teravus: Nope.  But it’s something that in the future, I intend on working on. It would essentially be a video [streamed to low end browsers].

Tish: Is that different from what Vollee is doing? The mobile client for SL?

Teravus:  It appears that they are, indeed, pre-rendering the client’s view and streaming it to the mobile device

OpenSim was in the spotlight at Virtual Worlds Conference and Expo, LA 2008. OpenSim won a conference Innovation Award and the OpenSim booth was a hub of activity.

At one time I saw conference attendees from Microsoft, Intel, and IBM all in conversation at the OpenSim table. A video of the OpenSim integration with Lotus SameTime was a centerpiece to Colin Pariss’, IBM,  keynote interview with Erica Driver of Think Balm. And, just to back up what might seem my own seriously partisan opinion, I will note Wagner James Au made OpenSim the first of his top three picks from the conference for GigaOM

And, if you want to know why OpenSim is sizzling hot these days and you can manage some serious gearhead discussion here is an audio recording of our “Open Source, Interoperable Virtual Worlds” panel.

I was facilitator for a stellar group of  virtual world uber geeks/phenom developers from OpenSim: Adam Frisby, OpenSim/SineWave, Mic Bowman, Intel, Justin Clark-Casey, Fashion Research Institute, Mike Mazur, 3Di and David Levine, IBM.  See Dusan Writer’s blog for more on the heights of geekdom that were reached in the discussion, and for some interesting thoughts on the intersection of code, standards and social policy.

I open this post with a picture of Adam Frisby, Joshua Meadows, myself and Tess Linden enjoying some pre-conference Venice Beach time. Thanks Tess for showing us that you can be a phenom developer and get beach attire right!

But, the heart of our panel’s message was straightforward and powerful (yes I am obviously caught up in campaign season and on a message kick) . But in case you missed the LA Virtual Worlds conference here is the essence/message of our panel again, from my perspective, and in my words, and in short (relatively!!).

Open Source and Interoperability are the Basis for Innovation in Virtual Worlds.

This picture is from Adam Frisby’s blog of OpenSim “booth babes”  with the  Innovation Award.  They were dubbed “booth babes” by Dusan Writer in counterpoint to David Levine’s comment on seeing this photo, “What a bunch of nerds!”  David is in the center. And from the left Justin Clark-Casey (justincc), Adam Frisby (afrisby), Charles Krinke (ckrinke), David Levine (zha ewry), Adam Johnson (adjohn), Mike Mazur (mikem), Jeff Ames (chil1ken)

Basic building blocks and interoperability work will make Virtual Worlds part of the fabric of everyday computing

Big players like IBM and Intel are often asked why so much interest in OpenSim, in particular, when their goals obviously reach well beyond one platform. The answer to this, panelists explained, is that OpenSim is providing the best environment currently available to explore what will be the basic building blocks similar to http that will enable virtual worlds to emerge as a mainstream phenomena.

In conjunction with the open source Second Life viewer (despite the need for licensing changes and a new modular architecture which was discussed) OpenSim is the virtual world test kit supreme. The modular open source architecture allows for components to be disaggregated and reaggregated into new formations easily and rapidly. As I have mentioned before,  Stefan Andersson, Tribal Media AB,  has a catchy way of putting this,  “OpenSim is a rapid prototyping testbench for wannabe paradigm shifters.”  Stefan, one of OpenSim’s founders, notes this has been the heart of OpenSim from the start.

Interoperability work is the path to achieving virtual world standards

Achieving standards will take virtual world innovation mainstream.  The innovation emerging from open source to produce basic building blocks, and the standards emerging from interoperability efforts, panelists explained, will be the magic brew that transform virtual worlds into part of the fabric of everyday computing.  As Mic Bowman, Intel, describes it (interview upcoming in Ugotrade). The goal is that virtual worlds will ultimately be taken for granted much as when say “browse the web” – we take the “web” for granted (it is the applications YouTube, Flickr etc that gets our attention).

Our panel argued strongly that achieving virtual world standards will emerge from a combination of “rough consensus and working code” rather than “Vapor standards” – worked out by committees in the abstract. Both David Levine, IBM, and Adam Frisby noted “vapor standards” have been lost causes in the history of the internet.

Interoperability work allows for these basic building blocks of virtual worlds to be expressed across various computational fabrics which creates an environment for what David Levine describes as “parallel innovation.” David explains:

The point about parallel innovation is that there are several ways to evolve things. Open Source, tends to foster the addition of features, in common to the main trunk of development. Its very much about everyone building together. Sometimes, we need to or want to build stuff which works with that main trunk of development, but doesn’t share code, or has to fork. That’s where standards come into play.

While the OpenGrid beta has begun with Linden Lab and OpenSim, OpenGrid is just the beginning of much more extensive efforts to test and create protocols across different computational environments that will include increasingly different models of virtual worlds from OpenSim and LL’s model.

So another key message from our panel is: “Heterogenity fosters the creation of standards and more innovation.”

Metaplace and Second Life TM/OpenSim in Dialogue: Play Metaplace on a Prim!

One of the highlights of Virtual Worlds, 2008, LA, for me, was a forty minute conversation between Raph Koster, Tess Linden and myself about Metaplace (the pictures from the Metaplace blog are screenshots from worlds Metaplace testers have made.

I will have a more detailed post on this interesting dialogue soon.  I think Raph illuminates some of the subtleties of what, Dusan Writer noted, came across as somewhat of a Metaplace v Linden Lab debate on the Future of Virtual Worlds panel. As Raph explained, one of the the things they like to say at Metaplace is that, “virtual places are going to be first class citizens of the web next to tags, audio, video, images….that’s a very different kind of picture of where this kind of stuff is headed”  (see Raph’s comments on the panel here).  But as a little warm up to more on Metaplace soon, here are some of the remarks Raph made about integrating OpenSim with Metaplace hehe!

If somebody took an OpenSim server and made it speak the tag language out.  Then every Metaplace client would be able to talk to the OpenSim one. And visa versa, if OpenSim can open a  bog standard gpt socket, because we don’t even use a binary protocol. We use absolutely bare bones of the web. If you can connect by telnet you can connect to Metaplace. So that means an OpenSim client could become a Metaplace client, parse the tags, and play them on a prim.

RealXtend bring a cool new demo to LA

In the picture above Ryan McDougall now senior software developer with RealXtend, Tomi Kujanpää (LudoCraft/realXtend Art Director & Avatar Specialist), Hannu Hollström (ADMINO technologies) have a drink with Gene Yoon (Ginsu Linden), VP of Business Affairs, Linden Lab at The Metaverse Mixer Meetup at the Hotel Figueroa.  Cheers!

Kudos  to Linden Lab for their big picture thinking demonstrated in the work they did open sourcing their viewer at a time when few other companies would have made such a move.  And, more recently, for their efforts on interoperability with the OpenGrid initiative.

And, congratulations to RealXtend who have pushed the envelope on innovation with the open source Second Life TM viewer and OpenSim server this year. RealXtend showed up in LA with another awesome “undersea” demo that will be public on September 23rd (screenshot below from Ludocraft Ltd.)

Also more exciting news from RealXtend this week. Mac’s Virtual World reports:  “OpenLife will be the first grid based on the OpenSim technology to use RealXTend features as far as I am aware of.”

Augmented Reality will bring Virtual Worlds out to the mainstream.

For an important look at how instead of more people going into virtual worlds, virtual worlds will be coming out to meet the people soon, see the Augmented Reality Panel video above. For more see Roo Reynolds and David Orban’s blogs.

Here is the video of our last Metaverse Meetup: OpenSim & Virtual Worlds Interoperability 7.23.08 (from Vimeo). The video of this landmark event was produced thanks to the awesome Annie Ok, Artist, Creative Director, Curator, Video Director, Metaverse Evangelist/Consultant, Co-Organizer of Metaverse Meetup.

While Annie’s first love is art, she has been involved in an extraordinary number of projects (see her bio here). Notably, Annie Ok, with Jeff Crouse, & Stephanie Rothenberg made the documentary and helped with the amazing Invisible Threads project which shows how excellent Second Life is for such innovative mixed reality installations. The documentary premiered at Synthetic Times. Annie also created the interactive, collaborative Timeline of Virtual Worlds that the whole community can help with.

Photos of the meetup are now posted here on Flickr and some nice portraits here on Facebook.

With the video Annie sent out a great write up about the meetup.

Annie noted:

“Adam Frisby and David Levine gave us incredible insight into OpenSim and shared compelling details that really expanded on what has previously been known about its amazing potential and revolutionary role in the future of the metaverse.”

And she was very kind about my really minor supporting role!

“Tish Shute was great as the guest moderator, asking key questions and adding salient commentary.”

And I really agree with Annie’s synopsis about what is at the heart of our metaverse meetups!

“It was so nice to see all the familiar regulars as well as meet the new ones. In true Metaverse Meetup style, we migrated en mass to a local bar where we continued the conversation about all things metaversal and had fun hanging out with fellow avatars until the late hours.”

Thanks to everyone and especially Global Kids for making the meetup possible.

Please be sure to check out the list of Metaverse Meetup links on the new About page. There are now Metaverse Meetup group on LinkedIn, Flickr and FriendFeed, as well as a list of Metaverse Meetup chapters in other cities for those of you who are not based in NYC.

Looking forward to seeing you at the next meetup!

Eben Moglen – Open Source, IPPI panel in Second Life

Life 2.0 Summit Spring ’08 will kick off with the Open Source, IPPI (IP and Privacy/Identity) in Virtual Worlds On Sunday, March 16, at 1 PM PST, with special guest Eben Moglen (his avatar pictured above).

The event will be held in the CMP Amphitheater at CMP 1, 2, 3, 4 ( SLURL ). To attend the Second Life events or watch video you must register for Life 2.0 here.

Eben Moglen, is Director, Chair and Chief Counsel of the Software Freedom Law Center. Moglen, professor of law and legal history at Columbia, is a pioneer of the opensource movement, former general counsel for the Free Software Foundation, and one of the architects of version 3 of the GNU GPL.

Zero Linden

Eben Moglen’s co-panelists on Sunday will include Zero Linden, a.k.a Mark Lentczner, Linden Lab. Zero is one of main the architects of Second Life’s evolving infrastructure. Zero recently published the first draft of Second Life Grid Open Grid Protocol a.k.a. SLGOGP a important step forward on the path to opening up the Second Life grid (see Tateru Nino’s post on Massively, Tao Takashi’s post at mrtopf.de, and mindblizzard).

The brilliant and very elegant Zha Ewry (a.k.a David Levine, IBM Research) will be joining the panel from JFK airport while he waits for his flight to San Francisco. David Levine and Eben Moglen had an interesting conversation back in December that you can find on Ugotrade here. They explore some of the problems of defining digital public space and issues of privacy on the internet, offering many suggestions on how to implement online privacy enhancing technologies and insights as to how we could design the next generation of these technologies in responsible ways.

Also Zha was a interviewed recently on Metanomics with Beyers Sellers (a.k.a Robert Bloomfield). This interview is highly recommended as some of the key issues facing Second Life’s Architecture Working Group (AWG) and a future Open Grid “that will ultimately allow the cohesive operation of both Linden-operated and non-Linden-operated Second-Life style simulators and grids.” (see Massively for more) are unpacked.

Download the video (Quicktime)
Download the audio (MP3)
Read the transcript
Metaversed video archive at SLCN

Also, OpenSim and Linux guru Sean Dague (IBM) will be a panelist. Sean Dague has been a member of IBM’s Linux Technology Center since it’s inception in 2001. He has worked on numerous Open Source technologies over the years including: Cluster Management (SystemImage and OSCAR projects), Hardware Control (OpenHPI), Virtualization (Xen), and now Virtual Worlds with OpenSim. Sean has been an active member of the OpenSim project since July 2007.

Sean’s avatar (picture below) is Neas Bade.

Tara5 Oh (below – that’s me!) will moderate with CMP’s John Jainschigg (John Zhaoying).

See you there! (SLURL)

Tony Manninen, the CEO of LudoCraft games studio (the client side development division of realXtend) who has being doing all this amazing recent development on OpenSim, has a vision for Avatar 2.0 that he is bringing to OpenSim. The possibilities for the future integration of realXtend features (that include meshes and the ability to import proper 3D models) with Second Life is currently under discussion – more on this soon.

We have tried to keep the rexviewer as compatible as possible. We totally appreciate what Linden Lab has done and we are trying to do our best to co-exist with their beautiful social innovation.

When I interviewed Tony (see interview later in this post), I was very excited not only by the scope of his vision and his devotion to enhancing the user experience and possibilities of virtual worlds, but also by his absolute determination to manifest this vision in code with the the utmost speed

The next release of realXtend server and client will be published on 29th of February, 2008 at 4pm (Helsinki time). The features that will be in this release and a roadmap for the future are here.

Virtual Worlds that make our “real” and “virtual” lives better

What do I mean by an exciting vision for virtual worlds? There are many visions for virtual worlds floating around and some of them are not so appealing to me, like the notion of a virtual world as a walled garden/company town put forward by the CEO of There.com, Michael Wilson.

When Michael Wilson was interviewed in Second Life by Beyers Sellers (Robert Bloomfield, Cornell University) for Metanomics this week, he gave the impression that creating a controlled environment for marketing is the number one priority of There.com.

In my view, which seemed to be shared by many of the Second Life audience participating in the group chat, this notion of designing virtual worlds as “big brother houses” is the epitomy of “the metaverse gone wrong.” You can see the full interview here on SLCN. Also see here for Christian Scholz’s (Tao Takashi in Second Life) excellent summary and analysis of a virtual world according to There.

I hope there are some more positive aspects to There.com than this interview conveyed.

Despite ups and downs and growing pains, Philip Rosedale’s vision for Second Life has always, in my view, had at its heart the motivation to make people’s “real” and “virtual” lives better. See The Making of Second Life by Wagner James Au for a fascinating look at the early days.

While Linden Lab is still working hard to break free of the “walled garden” model, with OpenSim and realXtend, the expansive vision that they have pioneered has been open sourced even before they have finished their own open sourcing project. And now, an open source development community of genius and depth (including Linden Lab) is rising to the challenge of taking this vision of making our “real” and “virtual” lives better to the next step.

There are several Open Source virtual world platforms available now, including Ogoglio, Croquet and Sun’s Wonderland. In the news today is the Open Virtual Worlds Project from The New Media Consortium (NMC) and Sun Microsystems which as VWN points out is similar to efforts announced by the Immersive Education Initiative to create an Education Grid across Second Life, Wonderland, and Croquet.

But OpenSim stands out from the pack because it inherits from Second Life an awesome set of tools to facilitate user generated content. This emphasis on user generated content is, of course, key to how Second Life has become the largest and most highly developed 3D immersive world to date. The fact that OpenSim is a platform built for people to build in was of one of the reasons that realXtend chose to work with OpenSim after they investigated all the available open source options. Also important, Tony noted, was the depth of the development community in OpenSim and Second Life.

Philip Rosedale speaking in Second Life at the MacArthur Foundation event on The Future of Philanthropy in Virtual Worlds

The Future of Ubiquitous Game Design

There are many inspiring alternative visions for virtual worlds emanating from Game Developers Conference 2008 which have been percolating through the blogosphere. Particularly interesting, to me, are the new possibilities for avatar expressiveness and some interesting ideas on the future of ubiquitous game design.

I was thrilled to discover that Tony Manninen has been doing research on avatar expressiveness since 2000 (see his papers here). Rich Interaction in networked virtual environments, avatar expressiveness, and the future of ubiquitous game design are his forte. How very cool!

Tony’s vision is to take social gaming to the next level and to produce games that are so heavily collaborative that they reach deep into our experience of the pleasures doing things together, our enthusiasm for team sports, and childhood memories of the enjoyment of playing together. He has been exploring these ideas of team play and community in earlier LudoCraft games like AirBuccaneers.

Ray Kerzweil’s keynote GDC 2008 laid it on the line: “Games are the harbinger of everything.”

But some of my favorite GDC 2008 quotes are from Jane McGonigal (see her blog post “Reality is Broken” – My GDC Rant) and in Liz Lawley’s copious notes on “Pouring Fuel on the Fire: Game Designers’ Rant.”

Jane McGonigal: I’m not mad at game designers. Compared to the rest of the world, we have it all figured out. Our medium kicks all other media’s ass. We make more people happy than any other platform or content in the world. (If you don’t believe that, you’re not paying attention to what’s happening.) We’ve won. Games have won. As an industry we’ve spent the last 30 years learning how to optimize experience. Brains, bodies (recently), and hearts are all engaged. That’s the good news.

The bad news is we rule the virtual world only. Reality is broken, and we’re not fixing it, we’re offering alternatives to it. We offer better experiences, better socialization, in virtual experiences. That needs to start changing. If reality is broken, why aren’t game designers trying to fix it? It’s our responsibility to design systems that make us happy and successful and powerful in real life? We have the power and the responsibility.

Jane shows an image of her favorite “I’m not good at life.” graffiti (from Liz Lawley’s notes).

Tony Manninen is an innovator committed to bringing the power of games into arenas we are yet to dream of.

Jane McGonigal has some nice examples to stretch your mind in this direction. For example, the Sniflabs collar remembers other dogs you’ve encountered. “We can play games with our dogs. What if I could play an MMO with my dog?”

Why will it make our “real” and “virtual” lives better to take play into every nook and cranny of our experience? Because, as Kurzweil points out, “Play is how we principally learn and principally create.” Jane McGonigal goes further:

I have spent the last year doing research on happiness. Instead of trying to figure out what’s broken, these people are trying to figure out what makes us happy. Every positive psychologist has found the same thing. Happiness is 1) having satisfying work to do, 2) the experience of being good at something, 3) time spent with people we like, and 4) a chance to be part of something bigger than ourselves.

“If you are a game designer you are in the happiness business.”

Jani Pirkola, realXtend project manager, works on innovation in the offices of LudoCraft.

LudoCraft = The Art of Designing Games and Play

Ludo – Theories of game and play

Craft – Art of game design and development

Reinventing the technologies of expression and experience.

At the Metaverse Roadmap meetup, Mitch Kapor pointed out “3D cameras would make virtual worlds easier to use.” And Bob Moore (on left above, me on right) asks in this excellent post, Is “free gesticulation” for avatars here yet?

When I asked Tony this question he replied:

“Our avatar will have enough “bones” for full facial expressions, etc. When the actual base architecture of the avatar is fully functional, there’s a possibility to use webcam, voice, or other input devices to control your avatars facial expressions. It can be true 1:1 mapping, but of course it can be something else as well. You can be yourself, or, you can change your “output” to something else.”

Tony, through realXtend, is reinventing the technologies of expression and experience.

Man Upside Down from The Book Of Urizen, William Blake

Interview with Tony Manninen, LudoCraft

Me: Second Life as a platform has been pretty much ignored for game development up to this point. Do you imagine transforming OpenSim and Second Life into platforms suitable for MMOG?

Tony: I am running the company and also making sure the realXtend development reaches the required quality and performance standards you would expect from MMOGs.

We’d definitely love to make games for Second Life, but at the moment the end-user experience is not exactly what you would expect from a game system. Therefore, we’ll pay heavy attention to things like responsivity, graphics quality, frame rate, etc. If we manage to keep up the momentum of realXtend development, then I’m sure there will be some interesting games spawning up in the near future…

Developing a sophisticated game engine is not an easy – or cheap – task, so there’s a loads of challenges ahead. But I truly believe that is the only way forward. With game-like interfaces and features you’ll be able to get much more heightened experiences.

Me: Have you worked out anything with LL yet re keeping the realXtend viewer compatible with SL?

Tony: We have tried to keep the rexviewer as compatible as possible. We totally appreciate what Linden Lab has done and we are trying to do our best to co-exist with their beautiful social innovation.

Me: But will you be able to work out a licensing deal under the GPL so that they can integrate your code into their browser?

Tony: The whole licensing scheme is still undergoing some serious thinking processes. We will try to find the best possible option in order to satisfy the needs of hungry virtual world adopters. GPL has its challenges. But, on the other hand, everything invented by a man can be re-invented by another. I am sure there will be a fruitful solution to the licensing issue. At the moment we are concentrating on releasing all the code to the general public, so that all the enthusiastic developers out there can join the forces and increase the momentum.

I’m pretty sure the pieces will eventually find their right places. I am really happy about the response we’ve got from Linden Labs – it’s great to think we might be able to give something back for them.

Me: Is it safe to say the licensing issues are on the table and being worked out?

Tony: There’s definitely some serious working-out being done, so I suppose it is safe to say that.

Me: I know Will Wright creator of Sims online and Spore has spoken a lot on the spiraling costs of content production and that diminishing returns for content development at these high costs. He has gone to procedural programming with Spore to take gaming on another path. But, it seems to me that you are taking another approach by trying to bring SL up to gaming standards – is this correct, or are you doing something different?

Tony: I guess that’s quite correct assumption – at least in relation to LudoCraft. This is not necessarily a conscious decision. It’s more like the costs of licensing a decent game engine are generally so high that they more often than not fall out of reach of start-up companies and small developers – let alone universities, communities, etc.

We have tried to find a suitable platform for our collaborative games, but since there were no perfect solutions, we decided to try and make one. Not alone, but by joining forces with our partner company Admino and several other keen developers.

The OpenSim merger will increase the development base even further, so there’s a great chance we’ll actually pull this off.

Like I said, there’s loads of challenges, huge amount of work and some design issues involved. But the Open Source communities have proven themselves earlier, so why not now.

Me: What do you think are the chief design issues to be addressed?

The realXtend project includes LudoCraft and Admino, plus we have several sub-contracting developers doing work with us.

The main issue is the divide between social 3D worlds (like SL for example) and MMOG. The gamers tend to avoid these social virtual worlds for obvious reasons [the quality of the experience from a gaming POV]. However, if we manage to develop a platform that can serve both purposes, then I’m sure things will change.

The main design issues, therefore, are the performance, audiovisual quality, rendering, frame rate stability, responsivity, interaction, etc.

Numerous issues that are not necessarily critical in purely social virtual world, but are absolutely essential in any multiplayer game environment.

Plus, if well taken care of, these issues will boost the whole end user experience in the non-gaming situations as well.

Me: You have gone a long way with the rendering and meshes what will be the next most important features and when do they arrive?

Tony: Inverse Kinematics and procedural animation are essential features if we want to have truly expressive and adaptive animations. We see avatars as main tools for self-expression. We’ve researched the issues since 2000 and we believe we are on the right track. With flexible and powerful expression potential and accurate controllability, the users will be able to communicate with their avatars so much more than is possible nowadays. The concept of Rich Interaction is something we will utilize here and it will be interesting to see the results when the system is actually usable by the general public.

Vehicles and projectiles are really important so they sit heavily on our roadmap.

Me: Are you going to do 3D face mapping etc – I know that this is getting close to doable now?

Tony: Oh, the required list of features is endless. There are several key features in terms of game development, but also some interesting stuff for more serious applications.

Our avatar will have enough “bones” for full facial expressions, etc. When the actual base architecture of the avatar is fully functional, there’s a possibility to use webcam, voice, or other input devices to control your avatars facial expressions. It can be true 1:1 mapping, but of course it can be something else as well. You can be yourself, or, you can change your “output” to something else.

Me: OpenSim is very attractive for a number of vertical applications – building automation is one you mention. How focused are you on getting the OpenSIm platform ready for the huge mirror world, 3d info machine/command center market?

Tony: In terms of the mirror world, we already have some interesting pilot projects that target those features. It may be less juicy than making cool games, but it provides some highly interesting scenarios.

Me: What are your pilot projects?

Tony: I don’t think I should go into details at the moment. We will include a list of initial projects and collaborations for our next announcement, which is due 3rd of March (latest). We want the third parties to have their saying first before making these public.

Me: The realXtend avatar server is a new way of connecting sims – could you tell be more about your vision there?

Tony: The main vision is: there should be only one. Only one avatar, one user account, one login, etc.

It is ridiculous that I need to have an avatar in SL, character in WOW, nickname in IRC, and whatnot. I mean, it’s ok if I want all these different identities, but what if I would be, eh, me in all these worlds, perhaps doing business, shopping, hobbying, and so on. There should be an option for me to jump from world to world without the hassle of logging in and logging out

Me: And how will the work you have done on the avatar server alleviate this problem unless SL, WoW and other cooperate on interoperability?

Tony: Think of it more like the 3d web. realXtend/OpenSim is like the Apache of virtual worlds, rexViewer is the Mozilla or Firefox of whatever. When “surfing” the web, you are not constantly required to prove and change your identity when loading different pages.

Me: But because there is no equivalent of http for virtual worlds yet, how will this work re the other worlds?

Tony: Ah, that’s the critical question: it won’t. There is some interoperability work going on in terms of creating the standards. The problem is, this work tends to be slow. We are not willing to wait that long. We want to see some action. That’s why we thought that we’ll start the wheel rolling and then see what happens.

Me: So at least now OpenSim worlds and hopefully Second Life will be able to connect this way?

Tony: The openess is the key to the lack of unified protocols. When the interfaces and APIs are open and transparent, people are free to develop applications, converters and bridges between different systems.

Right, at least OpenSim is now reachable, plus we want to get the teleport between SL and OpenSim/realXtend up and running, at least in the quick-and-dirty way so that people will have a chance to experience the feeling of world-to-world hopping.

Me: So in terms of this experience of avatars hopping between OpenSim worlds how do you think this will change the need /or not for large grids like SL or any of the new ones like OpenLife that are springing up?

Tony: I think there’s still need for various technical solutions like there is need for lighter and heavier websites. There are different applications, different scenarios and different user requirements. It is still really enjoyable feeling to be able to actually walk – ok, virtually at least, from one server to another. With hopping there may be some loading involved (like when surfing the web). So I guess the big grids won’t go anywhere. We’ll just have more modalities in terms of servers/worlds.

Me:So what are your expectations for concurrencies on OpenSim in the near future?

Tony: Hmmm… I’m not sure yet. OpenSim is an excellent initiative and there’s already a substantial user base. There are a lot of work to be done in terms of integrating realXtend codebase to the OpenSim, but since the visions and missions of the people involved are aligned, the hard work is a minor detail in the process.

The excitement in the open source virtual world developer community has been bubbling over since realXtend brought many OpenSimulator developers over to Finland to reveal the extraordinary amount of work they have done “extending” the function of the OpenSimulator project in just 4 months.

See here for an account of this amazing meeting in Oulu, Finland, by one of the participants from the OpenLife grid. Pictured above is a screenshot of the new mesh avatars being conformed inworld. Adam Frisby explained to me:

Meshes behave just like another prim type – you can drag, scale, and rotate meshes around using the same method that you use on primitives and object groups. You can link meshes with other primitives, save them to inventory and use them in pretty much the same way you use primitives.

Second Life is built with prims, so I asked my friend, Zha Ewry, for a 101 on meshes versus prims. “A prim, is a solid, defined by an equation – a cube, and such with permutations, like twist and taper, and cut. Whereas a mesh, is a series of triangles in 3d which conform to the shape of the object.”

Meshes are groovy because “they can do arbitrary surfaces. You can lay a mesh over any shape, add more triangles, increase resolution. A prim you can’t make any shape, only what the math bends it to, so you blend them into shapes by composition.”

SL originally went with prims and not meshes “because they are more compact, lower cost to render and easier to map to the physics model.” But, “many 3d programs build meshes by default and nothing builds prims,” so the introduction of meshes facilitates one of the key new realXtend features – support for proper 3D models.

I asked Zha if meshes would have a negative effect on concurrency. Zha said the answer to that question is not entirely clear yet, “but its not as bad as it seems because its prim+texture vs mesh=texture.”

Zha explained that a lot of people think of meshes as much more expensive when its not prim vs. mesh but rather, prim+texture vs mesh+texture so the difference is smaller, “and even more to the point, for many things, its Prim+prim+prim+prm+textures vs mesh+texture set. Often, even for small objects, you can do several prims worth of detail in a mesh.”

I have heard from OpenSim developers that the expectation for OpenSim is for concurrencies of 100 at least in the next six months.

Two realXtend companies, Admino & Ludocraft, have been dedicating 20 personnel (programmers, designers and content creators)‘ to extending’ the function of the OpenSimulator Project, both server side and client side respectively. The project is backed by the vision and generosity of Juha Hulkko, who has another fascinating project Radio Arkala (also in Second Life on Arkala Island).

Click on the screenshot above for the realXtend video demo. There is a full list of realXtend features on their site here. RealXtend give a summary of some of the most important functionality they have added.

The most important realXtend-specific development areas were the media tools such as VoIP and document sharing, OGRE renderer support and, of course, the separate realXtend avatar storage / authentication infrastructure that enables seamless transition from one world to another.

Having support for proper 3d models as well as simpler primitive type objects is an asset in many different ways. Most importantly, just about all the professionals and advanced amateurs who create content suitable for virtual worlds like to work with proper 3d modeling software. There’s already a lot of existing material that can now be directly transferred to the virtual reality if for example a movie maker wants to create a virtual world based on an animated film and the same goes for computer game developers. Thanks to proper 3d model support, practically all of their material should be easy to transfer from games to realXtend based virtual worlds and vice versa.

Another central feature is the global avatar architecture. A network of interconnected virtual worlds simply doesn’t exist until users can take their virtual representatives with them wherever they choose to go. Architecturally this means separating the avatar and their asset storage functionality from the virtual world server. Avatar service will provide the users with identity and authentication independent of the world they happen to be in at any given time. In practice the role of the avatar service will be somewhat similar to what email is today. Security is naturally a major issue whenever someone’s identity or asset ownership is concerned and it has been an important consideration in the design of the avatar architecture.

The picture below is of the web page as a texture feature.

Under testing for the next release (29th of February 2008)

• Free-form non-humanoid avatars
• Global avatar mesh, skeleton, textures, attachments and animations
• Single sign-on to multiple worlds for teleporting
• Avatar generator
• Avatar attachment tool to help set 3D meshes to different bones
• Unlimited amount of attachments per bone
• Teleports between realXtend and Secondlife
• Avatar storage to move avatar appearance between realXtend grids/worlds
• Mesh tool to scale and set pivot of 3d models
• Server launcher and configuration application
• Home automation example using X10 technology
• Bot with Python scripting
• Media library for world builders
• Server status window

Pictured below is the the location of this historic meeting which was also attended by Chris Collins of Linden Lab. I did hear some very interesting anecdotes about the visitors introduction to Finnish culture – flaming drinks, saunas, and hot metarati rolling in snow!!! But, the focus of this post is the road map for an open standards virtual world’s future, so I will leave it to others to tell those tales.

What follows is an extract from realXtend press release, and then I move on to an interview I did with Adam Frisby who was in New York last weekend. He shared some of his thoughts on the future of the Open Simulator project with me on another snowy weekend of skiing and snowboarding in upstate New York (lots of snow again but no flaming drinks, just Ribena, Mario Galaxy, and snowball fights with Ugotrade Jr).

Oulu, Finland – even a blizzard couldn’t delay the meeting

Team up to make open source Virtual World standard

Open source virtual world developers OpenSim are building a global standard to power the 3D internet, now joined by Finnish developer’s realXtend, OpenSim gains many new key features and technologies.

OpenSim is a free and open standard virtual world server founded at the beginning of 2007 by a small group of developers. The OpenSim platform can be used for creating and deploying immersive 3D Virtual Interactive Environments. The realXtend project’s mission is to assist in bringing forth the next generation of virtual reality development by focusing on interoperability technologies, usability and real-life application support. realXtend brings to OpenSim professional experience and development in the realm of 3D engine programming, voice over IP and network support, and is contributing many new features such as improved 3D graphics and voice chat.

“I welcome the realXtend project to join us. It is my pleasure to see that the OpenSim gets a huge professional code contribution. The future of the OpenSim project looks bright as the developer base continues to grow with giant leaps like this”, says OpenSim project manager Darren Guard (aka core developer Michael Wright).

realXtend is contributing all the server side code for their developments to OpenSim and continues to do so from now on. This collaboration of two projects with a similar vision enables realXtend and OpenSim to focus on common issues, and solve them more quickly – leveraging experience and knowledge from each party. realXtend project manager Jani Pirkola comments on the joining as “I see this as a great possibility to quickly make OpenSim the global de facto standard and to significantly speed up the global technology development in this area. Our common goal is to create the best open source virtual world server platform, and to continue the rapid evolution of OpenSim”.

Interview with Adam Frisby of OpenSim

Snowball fight in the Catskills while “snowed in with Open Sim”

Me: Could you tell me more about the “avatar server?”

Adam: The “avatar server” improvement allows you to sign in remotely from another region and bring your avatar & attachments across with you. It allows you to sign onto a OpenSim with an address like: adam frisby@mygrid.com — and then bring across your avatar & appearance. Inventory/etc is going to come in with this sign on method as well.

Me: You also mentioned to me how this would really change the concept of griding and introduce a different model to the Second Life grid.

Adam: Right, that’s tied in with that [the avatar server] a little – you don’t need a big central grid in the way they are operating. Their philosophy has been more akin to the world wide web — lots of interdependent sites that can link to each other. The avatar server is the first step towards that, since it allows you to login to multiple independent installations with one account.

Me: Do these changes, potentially, signify more of a fork from Second Life or not?

Adam: Definitely. This is the first time we’ve broken compatibility with SL feature-wise and implemented things that go far beyond SL.

Me: Will the Rex viewer work with SL?

Adam: You can still connect to SL with the Rex viewer, but your missing lots of nifty functionality, like the new rendering engine.

Me: What do you think are the implications of that and do you think the fork will continue to get wider or not?

Adam: Well, the subtle irony of it is that Linden Lab can’t take the changes back because of their client being GPL’d and they still rely on non-GPL’d components, i.e., FMOD and other bits and pieces. They can take BSD [all the realXtend server side work is BSD, but the client source code is GPL’d at the moment.
If we released the patches as BSD licensed, which RealXtend intends to try to do, then yes they could take those ones back.

Me: So why the conflict exactly?

Adam: LL use a set of proprietary components in their official release, for example the Kakadu JPEG2000 library – which means their official release is actually GPL-incompatible. RealXtend have to extract out any bit of Linden IP or derivative work in order to be able to release it under a more permissive license. (The GPL is actually fairly restrictive). In copyright law, you can generally make a open source license more restrictive, but never the other way around – it’s one of the reasons that OpenSim is licensed under one of the least restrictive Open Source licenses. If LL took the GPL’d code RealXtend had done back, they would be under the terms of the GPL via RealXtends IP.

Me: So taking the client code back could cause problems?

Adam: Well, if they took it – yes. But unfortunately they really can’t. They could in their pure-open-source release, but not in their mainline release.

Me: What do you think is the best path forward re interoperability?

Adam: I think the best path forward for a standard is to release everything to either the public domain, or as close as you can get. It allows all parties to use the code for any purpose whatsoever.

Me: So you mean no GPL no BSD?

Adam: BSD is fairly close legally – it’s effectively public domain with a “Don’t sue us” clause. There’s a small group of licenses which are essentially public domain:
- New BSD (what OpenSim/libsl are under)
- MIT
- X11
and a few others.

Me: Can Linden Lab switch from GPL to BSD?

Adam: They can – as far as I know they have never accepted anything into their codebase which would prohibit it. They have been discussing lately switching a few key files to BSD, which would definitely be appreciated – but the whole client would be better. A lot of people do – I’m not going to go into names (you can if you want), but there’s a fairly large group of people who dislike the GPL. At least for this kind of standards setting thing.

Me: So what is the road map for OpenSim now? What do you see as the 3 month goals to be, and then 6 month and then one year?

Adam: 3 month goals – we’re going to try and align the two codebases we have now into one seamless codebase that works well both with the official SL client, and with the improved Realxtend one.

We’ve also got a few short term goals in terms of stability, getting features working properly, and maybe with a little luck, doing a bit of work on our client networking. Of course – anything I say has to be taken with a grain of salt, open source means people can work on what they like effectively.

In six months time, I’d like to have the majority of the features in the official client supported, and to be reasonably stable enough for large scale deployments.

I think that might take longer than 6 months, but you never know.

A years time, I’d like to be working towards next generation features – what sorts of things we think the standard virtual world must support, better support for linking environments has to happen over the next year – and we definitely need to be well on our way towards a really good standard infrastructure that can scale to the size of the web today.

Me: Re the discussion of standard features – what do you think is the best forum for this discussion? And, has realXtend committed to more development?

Adam: They have committed to working on this until their backing runs out, but it sounds like they might have found a way to continue working for a good while yet.

Me: And, the forum for the discussion of standards – what do you think is best for that?

Adam: I think that’s going to just happen on it’s own.

A lot of the standards bodies that have been setup seem to be moving no-where, and while we are all happy to participate in them – I think more is going to be achieved by doing things, testing them, seeing what works.

Me: And sorting out a more permissive licensing situation seems like it would help?

Adam: It would help – because it means we don’t need to spend so much time reinventing the wheel. We have to reinvent everything LL has done to be able to license it permissively.

Adam: The Rex serverside changes – those are all BSD, and we’re incorporating it back. Their viewer changes – those are GPL’d by necessity, and we’re not touching those. The RealXtend company has two different companies working on it to avoid any IP contamination. The serverside stuff was done by Admino, and the clientside by Ludocraft.

We’ve got two pieces of software here:
- OpenSim
- Rexviewer
and the server side changes to list them are the avatar server and….. and all the meshes etc. are on the client side with Ludocraft. The server side changes are things like the Python scripting, backend support for the meshes, avatar server, etc. The client side changes are the new rendering engine, client side mesh support, shadows, improved lighting, etc.

Me: Does Windlight goes out if Rex comes in?

Adam: Yes, Rex are looking at integrating Windlight properly into their viewer at the moment. But they can integrate that in without too much of a problem because that’s strictly a client side thing, and that’s already GPL’d.

Me: My friend Ben Goertzel of Novamente who is developing Artificial General Intelligence projects in VWs wanted to know if the skeletons allowed “the ability to script fine-grained control of skeletons controlling characters of different types.” He wants to control characters via inverse kinematics not just by launching pre-fab animations because without that you can’t have naturalistic or adaptable motor control. Will the mesh avatars allow that sort of thing?

Adam: IK is on RealXtend’s feature list – they haven’t implemented it yet, but they are working towards it. What your friend might appreciate is that you can have completely customised bones under the realxtend avatar.

The default skeleton has 220 bones in it, but you can make your own custom skeletons for things like anthropomorphic avatars.

One of the avatars we got shown was someone wandering around as a giant collection of mushrooms.

Me: Can you control the angles of motions of the joints?

Adam: Yeah, you can.

Me: Cool! Ben is going to be so happy. AI just moved a big step forward!

Adam: Shiny.

Me: Ben also said if I want to stress you are ask if you can implement realistic physics of fluids for him

Adam: We’ve got a physics engine that supports it, but client will take a lot of work!

Click on the screenshot below to download your own OpenSimulator and get started. I have. You can download the realXtend client at realXtend.

Mirror Worlds will transform the meaning of “computer.” Our dominant metaphor since 1950 or thereabouts, “the electronic brain,” will go by the boards. Instead people will talk about crystal balls, telescopes, stained glass windows, wine, poetry, or whatever – things that make you see vividly. (Mirror Worlds, David Gelertner 1992)

As the meaning of “computer” transforms so will software. Gelertner talks about software as an embodied information machine. And, as virtual worlds come of age so will this notion of software as 3d info machines that we can walk around, tinker with, and hang out in with other avatars and agents in real time. But, exactly how the most complex, crucial, and up to now invisible, parts of our society become embodied in all their glory is not clear yet.

The photo above is used with the permission of Dan Slater. It is taken with an experimental camera Dan built, called the Spherecam. It is a one of a kind ultra wide angle camera that records a scene in all possible viewing directions (4pi steradians). The camera uses a pair of hyperhemispherical fisheye lens to record the scene in all directions.

In Gelertner’s vision the transformation of computers into seeing machines will empower people to understand and work with the machinery of their society. Edward Tufte describes, “Beautiful Evidence,” as “how seeing turns into showing.” But for this to happen, new metaphors for representing complex information will have to emerge. We will have to imagine new ways to deal with the multiple views that are already part of complex modern software (e.g., the four plus one mode view pictured below – see Grady Booch, Turing Lecture, 2007).

And, we will have to imagine radically new ways to view software.

Software today offers assistance to the specialist (in everybody) not to the citizen. The mere citizen deals with the increasingly perilous complexity of his government, business, transportation, health, school, university and legal systems unaided. Mirror Worlds represent one attempt to change this state of affairs (Gelertner, Mirror Worlds, 1992).

Design Patterns:The soul of software architecture

I contacted Grady Booch to ask him about the role virtual worlds may have in the next generation of software design. He is one of the giants of software design and methodology (known for developing the Unified Modeling Language with Ivar Jacobson and James Rumbaugh, and the Booch method of software development). But also he is undertaking an archeology of the essential piece of software architecture – software design patterns – in a project called The Handbook of Software Architecture. The website is under construction and “this is a work in progress, and the chapters of the Handbook will be exposed as each archeologic dig for each system is finished and vetted by the original development team.”

But, I was fortunate to get a sneak preview. The Handbook, while illuminating the brilliance and short comings of software’s past is, in my understanding, about software’s future – discovering the raw materials of software’s future through a meticulous excavation of the design patterns of the past.

“Some people collect stamps, Grady collects software architectures”

The picture above, from The Handbook of Software Architecture, is of an early design drawing for Google. Google are notoriously secretive about their architecture. The “family jewels” of page rank are not elucidated! Grady pointed out the importance of the “barrels” which are the individual servers.

Contradictions of Software-Intensive Mechanisms

The cartoon/design pattern above is from xkcd, A Webcomic of Romance, Sarcasm, Math and Language – hat tip to Cory Ondrejka for this link. The next generation of software design is pushing up whole new classes of applications. The semantic web is not the last word. But, we are confronted by a set of complex challenges and contradictions. Grady points out several of these (see them in full in Turing Lecture,, Grady Booch, 2007).

1) The internet has changed the way individuals communicate and collaborate but it also creates new opportunities for griefers (for a thoughtful look at some of the issues see Identity in a new era, and for a critique of the way griefers get glorified in the media see here). Mitch Wagner of Information Week, (Ziggy Figaro in Second Life) said to me recently:

I certainly despise griefers — they seem to like to spoil things for other people through no other motivation than mean- spiritedness. A devout Christian friend says that spammers and phishers are just plain thieves, but griefers prove the existence of Original Sin.

2) The web provides unprecedented mechanisms for social networking but also new opportunities for theft, fraud and the exploitation of the vulnerable, especially children.

3) Software-intensive systems permit real time and distributed access to information but this can erode privacy and other basic human rights.

4) Email and other software-intensive mechanisms increase the velocity of communication but email and the aging of digital archives threatens the preservation of history.

5) Software-intensive systems create new forms of artistic expression but piracy can dilute the intellectual property of artists.

6) Software-intensive systems enable and accelerate scientific research but they are also at the center of a new generation of offensive and defensive weapons.

7) Software is part of the very fabric of civilization, living in its interstitial spaces but its complexity continues to grow impacting the users as well as the stakeholders in its development, operation and deployment.

“What is software?” and “What are its limits?”

Through the dedication of Grady Booch (Alem Theas in Second Life) – his archeology and anthropology of software’s past, the past is beginning to speak. The archive not only reveals a valuable history: “What worked and what didn’t?” “What was brilliant and what was a failure?” But by creating this unprecedented access to history, Grady gives us a unique opportunity to chew on the big questions and discover the cross cutting zones from which the future will emerge.

There is a grand vision in David Gelertner’s book Mirror Worlds. A vision of a software revolution in which the underpinnings of our global society, the invisible machinery of software, “becomes visible and is transformed into a beautiful, poetic experience that empowers people to understand and work with the machinery of their society.”

In Mirror Worlds, Gelertner points out, ordinary people will be able to poke around in the workings of society, business and government. You will meet software agents and other Mirror World visitors and you will be able to enter a Mirror World through any household computer.

But as Gelertner pointed out this is not “hazy science fiction” – “the tools and materials for Mirror World building are in hand, and the job is underway.” And the challenges are more social than technical. They are at their root challenges of human imagination.

Last week Grady made a presentation on Software Architecture from Second Life to a group of Canadian developers. Grady’s Second Life avatar was streamed into an IMax Theater for a live audience who watched both the stream from Second Life and a PowerPoint presentation side by side. Through a fascinating set of slides of different software architectures, I saw the process of how successful systems grow and emerge. How software architectures manage to change. How the ability to change allows them to endure over time.

Grady argued that natural forces lead to optimal developments. And, for a given domain there is a reasonably optimum architecture for that domain. Also, how domains have tendencies to grow in particular directions, e.g., How Amazon’s investment in hardware and software was applicable to some domains, e.g. Cloud computing.

Grady illuminated the characteristics of the design patterns that animate the software architecture that invisibly guides our society, the mobile phone – “wickedly complex” yet very resistant to change because customers are fickle. Mars Pathfinder – a classic example of “subsumption architecture as Pathfinder works in a semi-autonomous way, Google, E-bay, Amazon – the titans of the web centric development and web centric retail, Citibank, Visa, and air traffic control, MMOGs, and many more.

How software architectures learn

One of the insights of Grady’s archeology is that architecture that can change is architecture that endures over time. And key to this ability to be flexible and to scale, is componentization. Grady pointed out how game architecture and virtual worlds are increasingly discovering componentization. Grady links increasing maturation of architecture to increasing componentization. A quick look at these two design drawings of the Second Life architecture – one of the architecture today, and the other the new design Zero Linden (Mark Lentczner in Real Life) presented, in September, 2007, show this trend (see more of the new structural design drawings here).

Zha Ewry (David Levine, IBM research) explained to me some of the challenges of introducing an increased separation of concerns and componentization to Second Life architecture while maintaining the essence of Second Life, collaboration and dynamic content.

To a point yes, we’ll see more and more componentization. But.. there are some deep limits, driven off of the need to do state melding.

State melding is the dynamic state updating – what makes Second Life the amazing creative, social space it is. The old web works by exporting state on a per state basis. But Second Life takes inputs from 10 – 40 AVs, and 40 times a second spits out a new state. I asked Zha what are the key ways the new architecture design is different from the old?

Separating out a bunch of non state melding activity from the state melding to start. That’s the agent domain

Some examples of non melding activity are Inventory, IM , Estate management, Profiles, Search and Map. As Zha pointed out:

Anything that doesn’t involve generating the next frame of the sim’s state, or has stand alone capabilities based utilities, e.g, buying Linden. The agent domain can fetch you a capabilities (a short term secure access to a web service) to the lindex . Right now, they all route via the sim.

Many Second Lifer’s favorite design drawings from the next generation architecture for Second Life displayed on the Architectural Working Group wiki are the ones showing the designs for running your own region from a home computer.

3D Information Machines

The tools of modern software such as Rational and Eclipse with complex UML models are currently deeply lodged in the 2D realm of the user desktop. But, the third dimension is crucial to Gelertner’s vision for the future of software:

You set up a software mirror wherever you like, then allow some complex real-world system to unfold before it. The software faithfully reflects what it going on out front. But this is a three dimensional kind of reflection: The program reaches out and engulfs some chunk of reality. Like a child- sized play village modeled precisely on a real town and tracking reality’s every move, the Mirror World supplies a software object to match and track every real one.

When I wrote to Grady requesting an interview I asked him the following question:

Do you think one day we can dispense with all those 2D docs and replace them with living 3D software that we can collaborate on in real time?

Grady replied:

and why do you suppose we are not already living in such a world but don’t have the ability to see it! (hehe..one of my fav books is “Better Than Life” by Grant Naylor; it’s the sequel to the book “Red Dwarf”)

The topic of the virtualness of what we call “reality” has been coming up a lot in the WoK forums, led by Piet Hut and Steven Tainer, that I have been attending in Qwaq. And, as the worlds we call virtual become increasingly “real” in ways we have not yet imagined, what we call “real” will be experienced as increasingly virtual. So things are definitely getting very interesting.

In a few years there will be enough computation cycles for ray tracing and avatars will be more of a “real” person tied to bodily movements. Already the riddle of multi threading is the theme of much discussion and talk in Second Life (see this series of Second Life presentations by Intel’s multi threading gurus.)

These mirror worlds will become increasingly rich and, in Gelertner’s vision, they will mark a new era in humankind’s relationship to the human made world. And, “They change that relationship, for good.”

While some would argue that 2D repositories can be transformed usefully into 3D architectures there is a disruptive discontinuity in the phase shift from 2D programs to 3D as 2D respositories do not have a 3rd or a fourth dimension.

Another approach is to root the design process in 3D from the start. From this perspective the sim itself acts as the OS or middleware. Mirror Worlds are native to the 3D environment. And, prototypes for Mirror Worlds, small scale examples for the moment, are already appearing in Second Life and OpenSim.

“Beyond wickedly cool” – A VNOC (Virtual Network Operations Center) in SL

“Beyond wickedly cool,” was Grady’s Booch’s assessment of the 3D info machines of Illuminous Beltran (Second Life avatar), a.k.a Michael Osias, IBM. These are not merely visualizations they are assemblies driven by “live” or simulated data. “When real business logic is in these machines they become more than “visualizations” and models. The become 3D information processing machines.” I asked Illuminous to describe his work for me:

Nearly every technical engineer has used a program like RAD and Visio to structure and describe the architectures they are building. These drawings may be static or updated from an underlying metadata repository, and render information at different levels of abstraction based on the phase of the project. These 2D, semi-static artifacts are shared among members of the team working on the project. Via collaboration tools such as email or content management systems, wikis, etc, these models are refined, new revisions created, and the cycle starts again. Ultimately, these artifacts need to end up in a deployed system, which involves yet another set of tools, skills, and people.

There is a thought that, using Virtual World technologies, the collaboration cycle time between design, refinement, development, and deployment can be drastically reduced. The idea is that, 3D elements of what we may consider 2D software, are built in the virtual world. What does software ‘look’ like? With the phrase ‘function drives form’, they look like what they are. Components such as sockets, servers, subsystems, and applications all have well defined structural aspects that can be represented in 3D. That is not to say, every line of java code becomes a 3D object, but rather atomic elements – be they objects or collections of objects, subsystems, or other ‘normalized’ elements that represent structure and function, but also make sense.

You could stop there, and say we have a static ’3D software sculpture’, and gain some value by having multiple avatars collaborate on the structure of the system, making changes in realtime. This provides great deal of value by allowing everyone involved in the collaboration to understand and see in realtime, the structure of the system. Taking it a step further, why not give these ‘sculptures’ behavior that mirrors the ‘real’ components? Such as data flows in and out, animations, color changes, even adding in tiny screens, consoles, buttons, network connections, gauges, meters, and even sounds. These elements can be programmed to respond as the ‘real’ component responds. Therefore turning our 3D sculpture into a ’3D machine’. Now we can determine, collaboratively, how the system evolves over time given various stimulus and behavior. Connecting these components together, like connecting wires to electrical equipment, allows composite systems to be built.

You could also stop there, with significant value in not only understanding the structure, but how the state of all components evolve over time given specific stimulus. Or, you could feed key data elements from the APIs of the ‘real’ systems to create an operational mirror image of the system. Finally, to take it a step further, you could begin to move function into the 3D machine, and begin to turn off the functions of the ‘real’ system…after all data is data, and logic is logic, no matter if the runtime is in java or the Virtual World. The data and algorithms, and the behavior of the machine, come to represent not just the ‘living’ architecture, but it becomes the ‘operational’ architecture as well. Of course not all algorithms and data volumes are appropriate for the Virtual World 3D Machines. In these cases, a hybrid of real processing versus manipulation of high performance systems via APIs is an option. This final phase, is where the design, development, and deployment, could become a single entity.

New Forms of Collaboration

The biggest problem in computational science, which is quickly becoming all of science, is to find ways to let scientists write software together. (Piet Hut)

There is much research going on in many different virtual world platforms on how virtual worlds can best be used for collaborative software design. Piet Hut is working with astrophysicists in Qwaq and Second life in a project called MICA, (Meta-Institute for Computational Astrophysics), to explore the possibilities for scientists.

The IBM Project Bluegrass from Li-Te Cheng, Steven Rohall, and John Patterson, The T.J. Watson, Collaborative User Experience group, is looking at virtual worlds for developer collaboration and to support distributed work. They are focusing on some of the complex aspects of visualization of work flow processes that are part and parcel of modern software design, and the critical social aspects of working across culture and geography.

Collaborative applications such as Rational Jazz for software development and Lotus Notes for business processes, provide team support for “heads down” work. However, as teams become more distributed, it is important to support “heads up” work–the kind of social interaction that is achieved by seeing people in the hallways when they are collocated.

A video demonstration of Project Bluegrass can be downloaded in Second Life at IBM Codestation. Bluegrass is a research project. They are considering the strengths of various platforms. But, their basic approach is the creation of a virtual world client installation, a VW plug in for the 2D collaborative environments of Lotus Notes and Jazz. At the moment, they have a research prototype using Torque which is currently available for use by IBM teams.

New Collaborative Tools Evolving From 3D Environments

Bluegrass uses virtual spaces to find ways to enhance the 2D environments in which developers work today – modern developers are dealing with a level of complexity that has already usurped the real estate of their screens. But there are some interesting experiments on design and collaboration native to the 3D environment emerging in Second Life.

These experiments, currently, may seem mere toys in relation to the needs of enterprise software development. But because they are evolving directly in the 3D environment – these under-featured solutions may contain the seeds of innovation that will kill todays giants. (see Innovators Dilemma).

The following pictures show Vision Raymaker (Marco Vanadia in RL), and JonnyBee Cioc’s, Spatial Mind Map project. Their source code is freely downloadable here and the latest version is here. It’s released under the Creative Commons license. They developed Spatial Map using this LSL editor – an off-world IDE integrated Developer Environment developed by Alphons Jano (SL Avatar), Alphons van der Heijden in RL. Vision told me he considers this the best off-world IDE for LSL. “This editor allows you to edit/compile & debug one or more Linden Scripting Language sources in its simulation environment (when you press a F key you can test almost all functionality as if you were in SL). It can update itself every time you open it via internet and offers help from an extensive help file, or the LSL wiki. “Vision (Marco Vanadia is becoming the first (perhaps!) Italian scripting mentor in Second Life.

Spatial mind map to public brainstorm about SL land joint venture

Talking about Singularity through a mind map

The spatial map mind mapping project was born on Vulcano, “an open community, where experimentation is encouraged to flourish, and the consequences of applying common sense and bottom-up self regulation, enable creativity to mature.” David Orban the founder of Vulcano told me: ”

I am very proud of this concrete example of its fertility. An idea is just a humble starting point, as we know, and before we can measure its validity, in the physical world there are enormous hurdles to clear, many of which don’t relate to the idea itself, but are inherent to the rules we are accustomed to obey. In online worlds on the other hand, many of these barriers do not exist, and ideas can evolve very quickly, as they are more easily tested in their utility. I fully expect Vulcano to give birth to many other excellent ideas as well, and my best wishes to Spatial Map and all the others is to succeed, and progress!”

Malachi Mulligan is working on another mind map project on Vulcano. And, see also the opening of the Pyramid Cafe, another awesome manifestation of Vulcano creativity (read more here in Italian and here in English).

Spatial Map Creators JonnyBee Cioc and Vision Raymaker

Software Architects Meet to Play Go in Second Life

Last Thursday, a group of software architects including Saijanai Kuhn (Lawson English in RL), from the AWG, and Zha Ewry, the IBM representative to AWG gathered on the Second Life Go sim created by Zarf Vantongerloo and BamBam Sachertorte for a great match between Zero Linden (as his alt Zarf) and astrophysicist Piet Hut. It was a brilliant and wonderful match to watch. Zero is a 9K player and Piet is Shodan level. The conversation during the match was fascinating too, and touched on the aesthetics of game patterns and the best way to improve one’s skills by replaying the games of the great masters until the patterns become internalized. Thanks Zha for these great pictures! Click here visit Go in Second Life (and see the newsblog and the website for a schedule of events).

The emergence of many forms of virtual worlds will be a notable trend in 2008. But Second Life as the largest and most highly developed user generated, 3D immersive world will continue to blaze the trail for the most world changing potential for virtual worlds – collapsing geography.

How virtual worlds will change understandings of national power is detailed masterfully by Cory Ondrejka in his article, “Collapsing Geography: Second life, Innovation and the Future of National Power.” Cory writes:

Networked innovation and collaboration means quantity may have a quality of its own. As education systems around the world approach parity, nations will finally be able to maximize the skills and potentials of their populations… No nation state will be able to compete counting only the people within her borders. The most successful 21st century nations will be those that redefine what it means to be a citizen and build the largest network of innovators.

Cory’s new blog is also called Collapsing Geography. But to truly fulfill the dream of collapsing geography the challenges to the scaling and interoperability of virtual worlds must be met.

Scaling the Second Life grid is vital if it is to fulfill the expansive vision of its founders. But scaling must progress along with goals of interoperability. No-one has the hubris to suggest that one homogenous grid should service the globe! As Cory Ondejka has left Linden Lab the scaling of Second Life is no longer his concern. But it is interesting to note that Cory has already indicated that it is likely he will be working on the other critical aspect of Virtual Worlds ability to collapse geography – interoperability.

Interoperability may come up as part of general discussions in APOC [Anneberg Program on Online Communities]. I think it is quite likely that I will be working on projects related to interoperability separate from my time at USC.

Scaling, adding new features, ensuring grid stability and developing interoperability will often seem to be competing values that Linden Lab has to juggle in 2008. Whereas Second Life pundits and residents often demand stability at all costs. It is not going to be that simple. The only truly stable worlds in the fast emerging virtual landscape of 2008 will be a small, closed, dead worlds, or perhaps, 2D worlds.

I came to the conclusion that 2008 really could be the year of interoperability, or at the very least the beginning of interoperability starting with increasing levels of web services for the SL grid, after talking to many of the movers and shakers on Second Life including Zha Ewry, IBM representative to the Architectural Working Group, John Jainschigg Exec. Director of CMP Metaverse, Cube Inada creator of Starbase C3, Aleister Kronos, Illuminous Beltran IBM, Gwyneth Llewelyn and others. I also attended many of the Architectural Working Group meetings.

I spent some time reading some of the excellent predictive posts for 2008 from meta thinkers such as, Cory Ondrejka himself (also see Cory’s review of his 2007 predictions in Terra Nova, Hamlet Au of New World Notes, and the Virtual Worlds Management team that has put together a 36 page Industry Forecast for 2008 (see Virtual World News for more on this and how to order a free copy), also some interesting predictions on Second Thoughts, Virtually Blind, Caleb Booker, Not Possible IRL, and Second Tense (the last is a somewhat tongue in cheek look at what 2008 may have in store for us). But only Hamlet of New World Notes makes a specific prediction re interoperability suggesting that a Second Life port to Sony Xbox 360 will be created.

Some Second Life commentators are making the prediction Linden Lab will not open source the server side architecture of Second Life in 2008. And while they may be right on this one, 2008 maybe more of a preparatory year devoted to cleaning up code and protocols, this probably won’t be an obstacle to achieving at least some of the goals of interoperability.

The fast development of OpenSim makes the open sourcing of Second Life server code something of a moot point. See my interview with Adam Frisby of OpenSim for a look at recent progress and some of the strengths of OpenSim architecture. The picture of OpenSim below is from NixNerd (see OpenSim Wiki). It is taken using Windlight.

Interoperability between OpenSim and Second Life in 2008?

Full interoperability between Second Life and OpenSim in 2008 is unlikely. I would put my money on 2009 for this. But a leading member of the Architecture Working Group suggested the following:

Three Achievable Goals for 2008.

Demos for:

1) Cross login for authentification – meaning that Tara5 Oh and her password would be the same in Second Life and an OpenSim that shared authentication with Linden Labs’s authentication servers.

2) Fetching assets between domains. This would mean being able to fetch assets from one space to another. For example I would be able to take my AV and my clothes from Second Life to OpenSim and back. Right now LL will only do this if they have a legal agreement with you not to steal stuff. The problem with off grid back up is that it is instant copy bot so one of the non-technical challenges is setting up the rules for connecting grids. (This also probably needs extended permissions so people can flag if they wish assets allowed off the SL grid.)

3) A tp (teleport) between an OpenSim and a Second Life sim. 1) and 2) are needed to do 3).

Integrating SL with The Web: SL and RESTful principles

I frequently attend the Architectural Working Group meetings on Second Life where Linden Lab, IBM, OpenSim and others meet to discuss open sourcing Second Life, open standards and interoperability. I am a silent observer as such gearhead matters as RESTful principles are usually what’s on the agenda.

Zero Linden’s office hours and Architecture Working Groupies’ meetings are where the warriors/artists of interoperability are to found. And the Linden Lab protocols are the clay from which standards are being molded. If you are serious about interoperability rolling up your sleeves and working with other teams whose objectives may not be exactly the same as you own is the challenge. As one AWG member put it:

Interoperability will emerge battered byte by battered byte from the hands of grubbie techies each with an agenda. Except on Second Life some of us are blonde, with a pert smile but yeah….

The Architecture Working Group is the first virtual world coalition that I know of to dig in and begin some heavy lifting re interoperability. AWG Meeting 2 is tentatively scheduled for 2008 Jan 31. The first AWG outlined some basic concepts for a new Second Life grid architecture.

One area of interoperability that will probably make big advances in 2008 is the better integration of SL architecture with WWW architecture which explains the talk of RESTful principles at AWG meetings! But as I have learned, the core activity of Second Life state merging delivered by the web is odd in relation to current web concepts. For the most part, the web for the last 30 years has largely been about delivering static content i.e. most of what people see stays the same for minutes, hours even days on end. The essence of what Second Life is about is collaboration around dynamic content. A good chunk of REST is about exporting state on a per state basis some of which you can’t do on SL as it doesn’t fit the model. As an AWG architect put it:

What SL does for a living is take inputs from 10 – 40 AVs and 40 times a second spit out a new state.

So Second Life is a very different animal from WWW. It doesn’t really match the core REST models. You can build on top of REST but the last bits are going to have to be different.

As far as I know no-one has really licked the user generated content plus 3D, plus REST equation yet. REST is.. 90% about saying: “Here is how the web works, and why as a consequence, it scales and models well.”

Mashing the physical, the web and the online virtual

A different aspect of interoperability, mashing the physical, the web and the online virtual, is beginning to produce some interesting directions. And, what serendipity! The first link to my blog in 2008 came from Marta Lyall one of the prime movers behind a way cool project wrlds.com that is printing physical art objects representing big stock market moves, and creating a social network around this. The picture above is from the Wrlds web site.

Within the WRLDs System, participants can generate both virtual 3D and physical objects from their trading data. Translating trading data into these new forms creates a shareable social object from the symbolic language of trading

Artificial Intelligence Applications in 3D Virtual Worlds

While the development of distributed artificial intelligence in virtual worlds is certainly going to spawn a variety of killer apps one day soon, it is still early days for this. At the Virtual Worlds Conference and Expo, 2007 in San Jose, Ben Goertzel’s startup company Novamente LLC announced their collaboration with Electric Sheep Company to bring artificial intelligence agents (virtual pets) to online virtual worlds (see BBC News Coverage). Harnessing the wisdom of crowds in the rapid prototyping environment of a user generated, 3D world like Second Life presents an extraordinary opportunity for the development of Artificial Intelligence applications (see my post here on Artificial General Intelligence in Second Life).

The development of OpenSim and creating interoperability between OpenSim and Second Life in 2008 will create many new opportunities to create artificial intelligence applications in virtual worlds that require a secure and public platform. Already use cases and prototypes for energy management, predictive maintenance, building automation and network operation centers that are being designed to be integrated with AI are being developed on OpenSim (see my post on Eolus One’s work on building automation and Illuminous Beltran’s Virtual Network Operations Center).

In the picture above Illuminous Beltran (Michael Osias, IBM) is discussing with Zarathustrapoalypse (Ben Goertzel of Novamente) the possibility of having a virtual Artificial Intelligence system administration operation center that could diagnose problems and be able to describe them in an abstracted, qualitative format and then having conversational AI avatars to describe the problems to sys-administrators (and indicate them gesturally in the 3D sim world).

As Ben Goertzel of Novamente also has considerable experience applying AI to financial trading and understands both the data and the psychology of traders pretty well I asked what he thought of the Wrlds project, and how it could be developed with AI. Ben came up with a very interesting, off the cuff, AI tie in.

There is an AI tie-in, since AI can be used to analyze and extrapolate data, and their stuff could then be used to visualize the results…

But that would be a different sort of product, I would say… and a great one … imagine a data-analysis toolkit whose interface was part of a virtual world…

You view your data in the virtual world, a la the wrlds.com methods

You choose methods to analyze your data, via a metaphor of choosing physical tools in the virtual world…

You apply the tools to the data and visualize the results

I note the “toolkit” metaphor is constantly used in the data-analysis world…

Finance would indeed be the first market to look at here, since there is a big and mature market existing for financial data analysis tools…

Melanie Swan is also doing some interesting work on the visualization and analysis of financial data in virtual worlds (also see Melanie’s post on virtual world killer apps). She is developing a 3-d dynamic display of stock market data in her virtual office in Second Life. Go here for the on-demand real-time 3d stock charts in Second Life.

Recently I met with Eben Moglen, the founder, Director-Counsel and Chairman of the Software Freedom Law Center, and David W. Levine, a researcher at IBM’s Thomas J. Watson Research Center and IBM representative to the Architectural Working Group, for an informal conversation that looked at many of the fundamental social, technological and legal questions of building 3D immersive online spaces like Second Life.

I live only a couple of blocks from the Software Freedom Law Center. And, as the opening sourcing of the Second Life Architecture is pressing forward, I decided I must at least try to get the thoughts of my neighbor who is the great advocate for the role of free software as a fundamental requirement for a democratic and free society.

I was delighted when Eben Moglen said that I could stop by and ask some questions. But he made clear, from the outset, that he wasn’t the optimistic advocate for immersive virtual worlds that I am. But the stage was set for what I felt could be a very important debate, so at the next Architectural Working Group meeting in Second Life I asked David W. Levine (a.k.a Zha Ewry in Second Life) if he was willing to come along and take part in this discussion. The photo of Zha below is by Noelani Lightfoot the proprietor of Quixotic Photography in Second Life (see her great Flickr stream here).

David is one of the most experienced architects, not from Linden Lab, involved in the Architectural Working Group’s efforts to open source the server side architecture of Second Life and develop open standards for virtual worlds. What followed was a fascinating and wide ranging conversation in which Moglen and Levine discussed how the choices we make about the design of virtual spaces and avatar interfaces in general can affect the whole path of human society.

Moglen and Levine explore, in depth, the problem of defining digital public space and issues of privacy on the internet, offering many suggestions on how to implement online privacy enhancing technologies and insights as to how we could design the next generation of these technologies in responsible ways.

A Conversation Between Eben Moglen and David W. Levine. (instigated and transcribed by Tish Shute, Ugotrade)

In this interview David Levine is speaking personally, not representing IBM’s official position on any of the issues under discussion. Also, neither Eben Moglen nor I (Tish Shute) represent or have any affiliation with IBM. We are all speaking from our own perspectives. And I apologize in advance for any errors I may have inadvertently introduced through faulty transcription and/or editing to the speech of Eben Moglen and David Levine.

David: This is entirely Tish’s idea, I’m tagging along to ask interesting questions and hear your insights.

Tish: Yes I roped David in because in terms of the Architectural Working Group in Second Life and the open sourcing of the server side architecture as he is one of the most experienced architects not from LL.

David: Yes IBM has an interest in promoting virtual worlds as standards based environments.

Eben: On the theory that avatar based interfaces have some broader purpose?
David: Right. Something akin to Second Life or variations of it in five or ten years is going to be deeply disruptive. Exactly which piece of it, I’m not making predictions actually, or exactly how it becomes disruptive.

I think it turns into the flavor of things like right now you go into Amazon, you go to book reviews. And there are 50 or 60 people who very carefully putting their opinions down and there are maybe ten other people who are curious about the author — who currently are thinking about buying their book. But you can’t see them on the web at all. But they’re there side by side pulling the pages reading the reviews commenting on them. But you have no chance of interacting with them at all. If in fact we can say now that you’re interested in talking about this author and this book, there are ten people happen to be in the internet context, interested in doing that.

Can we bring you together and have a conversation. You can ask a question rather than simply read someone’s written report. Why do you think the character is interesting? Was it because they do interesting things? Was it because their thought processes as described by the author interesting? What makes this a compelling book to you personally? And we can have a dialog rather than a static web.

Things like that, I think, are going to happen. Do they look like giant wolves wandering around freely through the Internet – which is half game space and half social a space, half social interaction spacee. Partially because in fact because Linden Lab didn’t set out to create that thing, people did, so I figure some of that meets human needs.

But the disruptive change is this weird blend between Wiki, comment pages, chat, talk, and 3D helps along with it. I’m deeply opposed to calling it 3D internet. I think you focus on avatars and visualization and you miss the fact that it’s a blend between what My Space does and classic internet chat that’s really exiting -which a lot of social community building.

What followed at this point was a wide- ranging discussion on whether Virtual Worlds may effect fundamental aspects of human neurology -“Is this an interface that grows the mind or shrinks the mind?” – and underlying issues and problems with privacy on the internet. Two key questions Moglen raises are: “Is store it yourself fundamental to freedom in the 20th Century?” and, “Who gets the logs?” I touched on these questions in an earlier post, “If The Metaverse Goes Wrong….” And there is an excellent podcast of Eben Moglen speaking on these issues at the MySQL conference. Also see David F. Flanders blog.

To focus this post on some very practical suggestions re privacy and avatar rights that came up in this particular discussion, I begin the transcription where the discussion began to dig down, both conceptually and technically, into the architecture of digital public space.

Eben: When I teach about this with law students, I say there are three elements that are mixed up in privacy and we tend not to notice which one we are talking about at any given moment.

There is secrecy – that is the data should not be readable by or understandable by anybody except me or people I designate. There is anonymity which is the data can be seen by anybody but about whom it is should be knowable only by me or people that I designate. And there is autonomy which isn’t about either secrecy or anonymity but which is about my right to live under circumstances which reinforce my sense that I am in control of my own fate. And this form of privacy is actually the one we talk about in the constitutional structure when we talk about the right to get an abortion or use birth control.

The Supreme Court says this about the ability to make our own life choices about having control over the serious consequences of circumstances in our lives in which we make moral judgments. But in the original companion to Roe against Wade, Doe versus Bolton, which was the Georgia Law that was being challenged at the same time that the Texas Law was challenged, Mr. Justice Douglas wrote the opinion for the Supreme Court in which he talked about the freedom to walk, loaf, or stroll meaning that it was more than just a question of some particularly important life decisions.

There was a feeling of a constitutional liberty in making ones own decisions about all the matters in ones life, which I think is also the liberty interest that the court protects in a case like [[inaudible]?? The guy simply wanted to walk about LA in the night time and got tired of being asked to show his drivers license at every step because he was a six foot tall black man with dread locks. He sued over the right, which I thought we used to have in the US, to walk around without having to identify ourselves to people all the time which isn’t just an anonymity problem but becomes a problem because it forces certain kinds of choices on us.

I think that the real issue here is about the forcing on choices on us. I see again and again the ways in which people now find themselves unable to make certain life choices easily because there digital self has acquired an inflexibility that constrains their non-digital self.
Lets start with really simple stories of minor behavior.

A law student comes to me and she says: “I had this bad break up with my boyfriend. And I didn’t want to be on My Space all the time looking at all the photos of him with his new girlfriend. So I dropped out of My Space and I went away. But the problem was that I was the photo collector for my little circle of friends which wasn’t so little. And, I had eight thousand tagged photos that was everybody’s life in law school. And, I was the one who was the keeper of everybody’s photo album. And, when I dropped of My Space they couldn’t get to those photos anymore.

So this was a real cost to people and they were really pissed about it. And they needed to decide how to go ahead and persuade me to come back to My Space. But of course they had already graduated and spread out among law firms and cities in N. America. So they had to coordinate the campaign to get me to come back to My Space. So they made a make so and so come back to MY Space page, in My Space. And, they went and hashed out all this over their photos because they weren’t going to leave all their photos from law school in a place which they couldn’t get to them because of my break up with my boyfriend.

Now there is a point that a fundamental decision occurs that she feels pretty seriously about as an individual. But she is being subjected to a campaign of peer pressure to hold her in circumstances that she is not going to like in order to get the photo album back.

Oh we might say oh there are a million other ways to solve that problem you can upload them all to Flickr and get the hell out of there. But what is actually happening she wanted to leave town and she couldn’t.

We have got to understand that when she wanted to leave town and she couldn’t. The digital self was trapped by a fence that the physical self had no problem passing through and moving on from.

We don’t want that to happen to people. We understood when the Soviet Empire decayed that all over it were places where people felt trapped in webs of surveillance and betrayal and interaction that had a kind of sinister feeling even if there is no Gulag and there is no shooting. And many of us feel very uncomfortable with the changes in the society we live in the United States in the past several years where for us there is no Gulag, no shooting, no being swept away with out charges.

We don’t actually think that like some poor lawyer in Oregon that our finger print is going to be misidentified as that of a suicide bomber’s accomplice in Spain. But we are aware that these webs of knowledge about us are beginning to control us because our digital persona is subject to leverage and to being interfered with in ways that matter.

I think that the question is there a power in projecting yourself into new environments in which you can meet people in new terms and in rich ways has no second answer, and there is no downside to it. But we owe it to people to build safe spaces.

When you said its Anaheim and they own it. We are touching on one of the really important problems for me as a lawyer which is these concepts of property.

I have just got done with guys who think protocols are property. Now I am going to have guys who think music is property. And after that I am going to have guys who think carriage is property. And at the other end is this sort of “you are on my digital island and I can look through your clothes if I want to because it is my island and that’s the way I make the rules.”

Now we have got a very strange notion here of property and we have a very unusual structure because most of the property that they could have in Anaheim is subject to governing. And even if it is Celebration Florida they still have only as much Home Rule as the Florida legislature will give them. And if there march against Alabama or even Logan Valley shopping center there’s rules that say what you can do in a company town about controlling people’s speech. I feel absolutely certain that when Google does it there will not be a guaranteed right to carry a sign that can be read by everybody in town explaining why Google sucks. It’s just not going to happen that way.

David: This is the “I write the TOS for people coming onto this public space. And there is no constitutional guarantees in this space.” Argument.

Eben: And I get the benefit of calling it a public space by which I actually mean my private space, real estate I invented.

David: Now in fact if you wanted to do something interesting there. This is to toss something into the tools to play with. And to toss it to you, in particular, because you are good at playing with these kind of tools. Which is, if I was going to attack that, I would go and attack that under common carrier. If you want to be a public space, if you want the benefits in law of being a common carrier then you have to accept these constraints on you behavior.

Eben: Well this is why this network neutrality thing rolled up out of nowhere. This is where this came from. The problem with this it is that it already begins by saying what you really like is innkeepers and stagecoach guys. This is an attempt to throw your selves 800 years in the past in order to come up with an analogy.

I think what we really want to say is something like this. If you are talking about a public space your talking about a thing that has not just a TOS contract but a social contract.

It’s a thing which has to do with what you get and what you give up in order to be there.

There ought to be two rules about. One: Avatars ought to exist independent of any individual social contract put forward by any particular space. And two: Social contracts ought to be available in a machine readable form which allows the avatar projection intelligence to know exactly what the rules are and to allow you set effective guidelines about I don’t go to spaces where people don’t treat me in ways that I consider to be crucial in my treatment.

Its one thing to say that the code is open source – let’s even say free software – it is another thing to say that that code has to behave in certain ways it has to maintain certain rules of social integrity.

It has got to tell you what the rules are of the space where you are it has to give you an opportunity to make an informed consent about what is going to happen given those rules. It has got to give you an opportunity to know those things in an automatic sort of way so I can set up my avatar to say, you know what, I don’t go to places where I am on video camera all the time. Self, if you are about to walk into a room where there are video cameras on all the time just don’t walk through that door. So I don’t have to sign up and click yes on 27 agreements, I have got an avatar that doesn’t go into places that aren’t clean and well lit.

David: So if I am going to walk into a space there is an “Ehum – you about to go into a space you really don’t want to go into.” So I can make an informed decision whether the trade off is worth it or not.

Eben: But that’s fine but you are going to have to go even further by saying here is the reason why you don’t want to go in there. Here is what you said in the past about why you don’t like places like this. You got to have an angel sitting on your shoulder – a code angel.

David: But before we can even do that we have to mark the world in such a way that I can make – that I can detect that I am making that choice.

Eben: That’s right you have to force the existence of social contracts in terms that are explainable. It is like I say to my students: “Should there be informed consent for My Space or Facebook?”

If I were doing this as a University project – if I was setting this up inside Columbia University – and building one of these space, and if we were going to have people there, volunteers, I would have to go to the IRB. I would have to explain why this was permissible human subjects research. And I would have to show the IRB an informed consent disclosure that showed what the risks were and allowed people intelligently to decide whether they wanted to run them.

Now I have got libertarian colleagues around the university, in the Law School in the Philosophy Dept in other universities who think this IRB stuff is creeping totalitarianism because it is inhibiting the rights of researchers to find out about the world.

And I understand their point of view. Just as I also understand why we have decided that there is some unethical human subjects research conducted in the past of which we consider so abhorrent that we won’t allow researcher to use those numbers. We consider that data to be the fruit of the poison tree. Not even to save our own pilots shot down in cold water will we use the studies about what happens when you put people in freezing water because we think that evidence comes from experiments you shouldn’t be allowed to conduct.

Now I look at these immersive experiences for children that the Times was writing about last week where you have four and five yr olds buying virtual gear in immersive spaces. And I think that is unethical activity. I think that the rules about children television are weak and not very important. But they are way stronger than that would let you get away with that kind of stuff on TV.

I think that is really serious screwing around with children’s wiring to explain to them that they have to be consumers of stuff that doesn’t exist where the only reason that they can’t have what they want is because the software is programmed not to give it to them unless they pay.

That’s a fundamental education in control of life by unfree software so elegant that Stallman probably doesn’t even know it is there because it is so horrible. But that is another perfectly imaginable outcome of this stuff. We create these things we create beautiful 3D amusements parks. We sell children rides for actual money extracted from real parents because the ride won’t go unless you put another nickel in the slot. And we going say – well its ….Disney Land.

Tish: Yes the children’s worlds can be very extreme….

Eben: Well Disney Land is a little extreme. When you go to Disney Land they open a file on you and they follow you through life. Disney buys everything that is can buy about all those children who have ever been to Disney land and they have got a reason which is bringing them back.

The average number of visits to Disney Land by people brought to Disney Land as children is four in a lifetime. Disney’s goal over the next 20 years is to make that six. And in order to make that six all they have to do is keep buying everything they can buy about all those people they opened a file on as children looking for opportunities by data mining to find an opportunity, like the oncoming 65th birthday of a parent or an oncoming retirement or a this or a that. They just have to play the siblings off against one another – you’ve never been to Disney Land but Johnny has. … They are going to do what they can do to create feelings around that person that says remember what it felt like being in Disney land when you are child – do that again.

Tish: So what can we do about this – for example now we have an architecture working group – a community group – that is working with Linden Lab on creating an open architecture that aims to get away from the current model – the one that you are talking about right now that is currently the model for LL and everyone ….

David: And Linden Lab are better than most in that they are actually relatively explicit.

So, for instance one of the things that came up in their kick off for doing their next generation of architecture was what are the Avatars Bill of Rights what things do we prevent the technology from letting happen to avatars. They actually had that discussion.

And Lessig who was relatively enlightened about this had some comments about “Well we don’t want to force going to a public space that somebody else hosts other than Linden Labs to be a way of taking away rights from avatars explicitly.” Which is good. But none the less the model for 90% of the service providers is always going to be: “We have got you’re data. We have got your chat stream, we may have your voice stream, and we have your action stream. How much do you realize we data mine?”

Tish: Is the architecture group thinking about these issues?

Eben: One way of imagining that business …is that there is a “screen” and somebody is going to get it. Another way of looking at it is there may be some element in which the avatar projector is itself the guy who has a part of the screen.

If you want to be really aggressive about it lets assume we have infinite processing power and can slow things down to a crawl. We can take that screen and split it up in a shared secrets manner so that you could reconstruct the screen out of the data held by all of the people who were there. You could even reconstruct the screen if you had opt in co-operation from say 4/5ths or 5/6ths of the people who were there but no one person who was there could reconstruct the whole screen.

This is the sort of thing which cryptography was designed to make it possible to do. Its how escrowing systems in the high security world are supposed to work.

Lets just go ahead and finish this up. For every one of these that is built in the non secret world there is going to be one built in the secret world. So in the Government world we are going to have secure structures. One possible way of having secure structures is there is a trusted server and that will work for intranet kinds of use.

But lets just imagine for a moment what I think of for a conceivable reason for having this stuff is for the conduct of diplomacy, actual meetings. There is a whole lot of stuff that diplomats do because which is very expensive to do because it involves sitzplatz.You send guys to Vienna and they meet every week for thirty years for no good reason . You could do a lot of that in other ways – some of it by video conference – but lets us just for a moment assume that there are things you could do by means of certain kinds large meetings that occur in a neutral environment.

You could set it up so that no one party at the negotiating table actually has a record of what happened. Only by cooperation of multiple parties is the record of what happened there reconstructable. Because the way the space works is that it automatically divides all of the stream data up amongst the avatars and gives it to them in a shared secret structure. So without the cooperation of avatars you don’t have a reconstructable event.

At the moment, I am talking about technology that I can spec but it would be a little bit burdensome in performance terms to implement. But that doesn’t mean anything if we just keep doubling the speed of the chips.

David: Presumably if any of us get together and we have a conversation at my client end I see everything that is said, in some form. The question is how you prevent me from keeping that stream.

Eben: yes right we are going to be living inside an encrypted environment ..

David: Yes right and right down to the eyeballs of course because at some point the digital stream is there and I can capture it…

Eben: maybe although but remember the data mining isn’t just have having the chat stream…

David: No I agree, so the question is how much can you reconstruct – you can reconstruct a lot of it. I can pretty well capture the stream of everything my avatar saw. I can’t see what the other people had in the background conversation without their cooperation. But I can do everything that is said in public.

Eben: Yes, of course, and the most important thing is that, that is as much as the space manager can see too. And the space manager can see no more than what happened in public which is usually what we are concerned about.

David: So one set of reasonable desiderata here would be that stuff that we build enable a lot of that – which is to say that assets held by private individuals it should be possible to put them on private servers and pair wise or group wise discussions should be securable in and of themselves.

Eben: The protocols ought to be agnostic) But, it would be enough if the protocols were agnostic with respect to how that collection occurs. If the protocols of operation don’t necessitate that the data of operation wind up in any particular place then you got a protocol consistent with both what is now the familiar model and the less familiar model where more information stays at the end place.

David: Right now every single bit of data that’s in Second Life is owned by Linden Labs because you put it on their server and they fetch it back out.

One thing we are going to do for lots of good reasons is go to a web centric model where lots of the assets get pulled on to web servers traditional old fashioned web servers. At that point there’s absolutely no reason why I can’t say these are my assets I will let people come to me and ask for permission to see them. And many of them I will share freely, because the clothing I am wearing is the clothing I want them to see the clothing I’m wearing. But the contents of my pocket, my wallet, I’m not going to be so eager to give them. And it ought to be possible to have this office in a virtual world in which you say…

Eben: My filing cabinet is locked.

David: Right! The furniture, everything in here is public. But I’m going to pull out this document in which I’m going to show you some really cool ideas about this law suit we’re engaged in, and you can read it here, and you can walk away with what I gave your client to read. But you can’t find out the rest of my file cabinet at all because it’s safe in my machine.

And you may not even be able to take that document off the island as it were, out of the office. You may be able to take the pages you’ve read from it. The other thing that’s equally important is can I do the chat pier to pier rather than through the central server. Is there any reason why Instant Messaging as opposed to public chat gets dragged through public space. It certainly shouldn’t be.

Eben: Right. The protocol design ought to take as a feature that there is no requirement for intermediation of any communication among parties.

David: Right. Other than that which we need to do to actually get the consensual state. What we’re doing is taking my word, your words, Tish’s words, putting them into one spot and then sending them back out.

Eben: There are things we can agree to do together, but once we are together there are things we can agree to do apart.

David: Right! But there is some core tapping. If we were each sitting on a computer 500 miles apart, in order to have this discussion we do have to bring the space together and spread it back out. But we don’t have to let the guys over there see it unless we want to.

Eben: What we have is both point to point and many to many kinds of conversational modalities and that there shouldn’t be any intrinsic reason to block them off. Now there may be situations since that famous parental controls system. I don’t actually want people sideling up to my children and whispering in their ears in public places. I’m content to let my children be publicly addressed in public places because transparency is an opportunity to avoid certain kinds of misbehavior. And there may be locations which are transparent locations. And if you enter a transparent location, that should be something known to the avatar in a technical way as an entry point and it should be possible for avatars to either seek or avoid such spaces.

I think the goal here is to confine the conception ‘property’ to meaning something like traditional right to exclude and not let property in the form of the greater includes the less since it’s my property I may define everything that goes on here, and I may make whatever rules I please here. There is some community of communities which defines ‘states’ which it is OK for places to be in.

There was a story in the New York Times a day or two ago, about a public park in Oregon that had gone bad. It was written in a slightly frivolous but not entirely frivolous way. It’s an ex-urban park, meant originally as a kind of highway rest stop park, far enough from everybody else that bad things began to happen.

People began to have sex in public, there are child molesters who pitched tents there and lived there, and so over time it becomes more a locus of crime than a locus of recreation. And what the parks authority of Oregon does is they shut it down. They close it up. They chase everything away from it. And now they’re in what they think of as a sort of rebooting period for the park. And then it’ll come back and a different community will be attracted to it and it will be a different park.

This is an example of ‘space gone bad’ in meat space places. You can certainly imagine renegade space in a immersive world. Places where operators are not following the rules. Places where the operators are taking advantage of the free software nature of space building to build spaces which are deceptive about how they look.

They seem to be spaces of type X and they’re actually spaces of type Y. That should be at least a regulatory interaction if not a crime. That should constitute socially actionable misbehavior. Making spaces that don’t behave the way they seem to behave, that don’t give you what they seem to give you. And if we come out of all of this saying the FTC will occasionally make an order about this, I’m going to feel very dissatisfied.

David: There are a couple of interesting things lurking there again, one of which is, in the web today we don’t generally think of it as having two parts. There’s the very public part of the web, where you go to a page like CNN and you get the content and it’s assumed to conform to fairly common broad norms. And then there’s a different part of the web where people know that that’s not true. All pornography and large portions of adult community which are not necessarily pornographic but have some content of theirs that’s restricted and they tend to have click throughs often. So you get something that says you need to be informed at some level that something’s happening.

At the moment none of that interestingly enough is in the protocols, there’s nothing in http that says I have to mark my page as mature, or I have to mark my page as you might not want to bring your child here.

Eben: Right. And as you may recall Larry was himself a big believer in picks and the idea of privacy platform standards built in the web pages. In the end the theory was look there’s a big first amendment problem with making people grade their web pages.

Sandra Day O’Connor went for this theory of doors and locks and walls in cyberspace in the first child decency case. Larry realized he had started the United States Supreme Court down a very dangerous crooked highway and to climb back out again, it’s never happened. And Tim Burness Lee and the others weren’t enthusiastic to say the least, so it never occurred.

Now here it’s a little bit different and the reason it’s a bit different is your public accommodation law kind of aspect. You’re making a public space OK? We’re not telling you grade your web pages, we’re telling you if you’re going to have a public space you’ve got to obey the fire code. You have to post maximum occupancy. You’ve got to avoid chaining the fire doors shut.And you can’t put a surveillance camera in the ladies room. You can’t do it! And that’s sense. Not it’s my space I can do whatever and if you don’t want to take a piss in my monitored men’s room, you don’t have to. That’s not going to fly.

David: Right. There’s a series of layers here just like we say look there are certain things that you can never do in your public space or even in your private space. We don’t allow you to say this is my 5000 acre ranch therefore I can shoot people. We say there are limits to your right to use your space as you desire. Even if you have your Celebration – home rule ends with the norms of the state.

Eben: You can even own a city but you can’t own a city which is an island in space all by itself. It’s got to be part of the rule of law it has to acknowledge sovereign power

David: Except for Mogda Dishu perhaps.

Eben: Well — and that’s what we call a failed state. And I guess what I’m worried about is building a…

David: failed digital space.

Eben: That’s right.

David: That I think there is an argument here that is interesting and worth exploring which is what would be good ways of – you know – we don’t have good rules for digital space we have none in fact. They are a Wild West in fact. There isn’t any sense that the constitution will apply in digital spaces.

Eben: This is why the whole political evolution of free software turns out to be interesting. Stallman creates in effect a constitution of the project, that’s GPL. It functions very narrowly but within its very limited range, it’s supreme, as Judge Marshall would have said right.

Debian free software guidelines and the Debian social contract tried to imagine the constitution of a distribution. Tries to imagine a constitution of a state solely about software building. Privacy policy in the United States which his supposed to be market driven and grow out of positive interaction with the FTC gives you the Ebay privacy policy, which is basically an attempt to imagine the constitution of a flea market.

David: And it is a reasonable thing to do if you are running a digital flea market.

Eben: And what you just said is that Second Life didn’t have a constitution because it it overdrove its headlights. And now your questions is so as that explodes as it inpupalizes as it loses its org chart and begins to float free as technology what would it mean to imagine the constitution of digital spaces in a multiplicity of reproducible forms ? That would be when it franchises.

David: What would it mean if in fact we succeed at the desire to say now we have an open software specification that says anybody can host a chunk of content they can create a public space using this technology and they can invite people to come into it and can bring content into it and share it. What would be good social contracts for those space is a very cogent question.

Eben: So one aspect of this is the debate between those who believe in the so called Aferro GPL and those who don’t. Let’s go back to this as a question of Free Software specialism.

I am going to make a pitch that if you don’t copyleft the software you have got another whole problem. Because in effect we all then l go to work for the guy who wins the race to the bottom. If we imagine this a BSP software we all become developers for the guy who has the one that is most abusive and least free.

That is why, in situations like this where we are talking about software with profound but multiple social consequences, I feel strongly about the utility of copyleft. The second thing I would say that if you imagine within the world of copyleft the fundamental difficulty of the, “its my server I made private modification I want to run this code for people who are willing to have this program run for them,” you can sort of see why even if you’re not distributing the software, if you’re offering services with this code, you have got to release the source code so we can see the modifications and learn from them becomes an important property in licensing.

That’s why the young guys like Mako Hill and others around Stallman are partisans of the Aferro GPL, because they see this in the long run as a really important freedom related question.

Google of course has entirely the opposite point of view of resting on rights which they want to exercise. And Google increasingly says, “well we’re rich enough to re-implement anything we want to, If it’s got a license and we don’t like on it, we won’t use the free world version. We’ll just write our own.

And that raises questions about when do you have to regulate regardless. Now in the world of e-bay because the free market takes real money for real goods and is patently engaged in interstate commerce nobody wants to rumble with the federal government and so everybody has a chief privacy officer and everybody acknowledges that there’s a regulatory role to play.

I think you could get to a point where that would be less obvious than true with respect to other kinds of entrants to the field. And I’m not convinced that the way you can get this done is by licensing the code correctly. But I do think that licensing of code is probably a part of the solution. I think the question of who is the steward of the code also matters. That is to say I think it probably does matter who the legal personalities are who maintain the spaces.

That Central Park out there, that’s a very municipal park but it’s got a bunch of people called the Central Park Conservancy, who are rich people who sit on a board of directors, some of whom hold public offices, some of whom don’t. But they put up a lot of money to keep that park and they have some kind of mysterious rights in Central Park that I’m not entirely sure I understand very well.

Just as I’m not very sure I understand this business improvement district for Lincoln Square that has these little rent-a-cops who patrol my neighborhood. As a local small business, I’m supposed to be very happy about those guys. But I’m perfectly aware that that’s a layer of government in which I am complicit but which I don’t control. And if I see one of those rent-a-cops kicking a homeless person it’s not exactly clear to me whose telephone number I’m supposed to dial. Even though in theory I’m the owner and supporter of those guys.

So I think that there is an ethical advisory board kind of question. I think there’s a dialogue. I think it’s a structured dialogue. I think it interacts with architecture at technical levels. And I think it interacts with structures of licensing and IP control. And I think it interacts with the strategists who invest on the basis of expectations about business laws. Because I think among other things it has a public informative role to play.

I’m sure that we’re going to enter here at maturity a dialog which looks a lot like some American style discussions about regulation. The people who think that disclosure is enough, the people who think that rules are enough, the people who think that you need agencies with quasi-prosecutorial powers to unearth wrongdoing, the people who think you should reward whistle blowers and so on.

The part that most interests me is the engineering because I believe in the durability of technology which is path dependent more than I believe in the congressional will to get things done or to stay where you’re put. The most corrupt man in Abraham Lincoln’s cabinet Simon Cameron Said, “the definition of an honest politician is a man who when bought stays bought.” and I’m very dubious about the honesty of our politicians.

David: There are clearly some really nice social activism open source related questions and any of that is…

Eben: And I don’t want to press this decline of reading stuff either because we agreed that is whatever it is so now let’s try and talk about the stuff you came to talk about.

David: By the end of the day I think there are 2 questions here one of which on a personal basis is as technologists one should have a responsibility to build technology that’s socially responsible. If you don’t you should stop doing it frankly. There’s a fairly good question: “Are we in fact as stewards of the technology doing the right thing architecturally?”

Eben: Well your proposition about disruption is the right proposition in response. Something is going to happen.

David: Right. But as technologists we can’t prevent the disruption necessarily. But we can shape it, shape some of the consequences of the disruption and thoughtful voices like yours saying here are things to look for.

Eben: It’s what lawyers usually do. Having thought about this let’s now let me try and tell you about things you’ve written [I submitted some written questions from myself and other Second Lifers prior to meeting Eben in person].

Tish: Well when I mentioned that I was going to talk to you several people had some questions in Second Life. This one came from Gareth Nelson (a.k.a Gareth Ellison in Second Life):

LL get people to sign a contrib agreement allowing them to relicense their work commercially. They also use trademarks to make it very difficult to release a product based on the viewer without their consent in order to release a non-SL product based on the viewer you’d have to remove all mentions of SL in the code or LL have stated their
policy is to enforce the trademark. It would be interesting to get a legal opinion on that – i.e is it reasonable for them to deny forks with such means?

Eben: you can imagine somebody so embedding their trademark material in a GPL program that the difficulty of removing it is so severe that in practice they’ve essentially made it impossible. And, if they then behave in a way which attempted to inhibit distribution of the software through the exploitation of their trademarks they would in fact have in one sense or another made the GPL a nullity.

But here we have a complexity because it’s theirs to start with. They haven’t got any third party code in there. So until they have some third party code in there they could have not had GPL at all and that would have been perfectly acceptable. And they can GPL it but nonetheless make their proprietary commercial license business attractive to people by creating incentives to use the proprietary licensed version instead of the GPL’d version.

Since removing all the trademarks from the GPL’d version is a job that only has to be done once, because after that all you have to do is remove the trademarks from patches, and removing the trademarks from the gifs doesn’t mean going through the whole codebase again. The general likely outcome is I’m going to say no foul has occurred. It’s their code you can scrub it once.

It would be highly desirable if they took say the step that Red-Hat has taken of finding ways to segregate all their trademark graphics and other ancillary elements into one branch of the code tree so that you can come along and lop that branch off and replace it with other graphics or other materials that don’t bear trademarks and be done with it.

And if in the long run it’s the Linden Lab position that they want to live as easily as possible with the community that’s what they’ll do. But I don’t think the short way across is to say they can’t do what they’re doing now. The short way across is to say if they inevitably continue to do what they’re doing now and they mean it about aggressively protecting their trademarks, they will wind up defeating the community they seem to want to have.

Tish: This next is a question about virtual assets causes a lot of controversy when it comes up in discussions on Second Life:

How can the Architecture Working Group (a community alliance working on an open architecture for Second Life) think about assets in a broader world than SL? And how can AWG encourage and make fair content creation? What properties would be desirable, in such a scheme, and what legal issues show up?

David: Let me toss in a little background. This is a question I’m particularly interested in, which is people are busy creating digital assets and they want to share broadly and in particular they want to sell them. And, in some sense, one can argue whether encouraging content makers to do content for money is a good thing or a bad thing. But it is certainly what we do for a world these days. But, How do we deal with music? How do we deal media? How do we deal with digital assets that people want to share?

Eben: So the problem again is that we live in a world in which duplicating bit streams is easy and asset value consists of the artificial hardness that we put into the work of duplicating the bit stream.

We create a scarcity which we protect by technological means. The more that scarcity value increases, the more desire there is to subvert the mechanism that inhibits the doing of the actual trivial act of copying the bit stream. The consequence of which is that we get a whole lot of weight on the paratechnology of protecting bitstreams against doing what networks do to bitstreams which is copy them. That’s one of the things which bears down on this.

You have to have everything on one server, because then it’s the operating system code on the server that protects against the copying of the bit streams. So the Fort Knox of the asset system becomes the server which has little rules about who’s got what.

Unfortunately there is no actual difference in the world between Second Life assets and first life assets because money has also ceased to exist in the twenty-first century. Money in most of the long history of human beings was stuff with scarcity value in lumps which could be weighed and measured. After that money was the fiat currency of governments whose credit was either infinite or knowable which was hard to counterfeit.

In the 21st century money is information flows arbitrarily assigning to me the right to purchase so many goods in the world and to you the right to purchase so many goods in the world, and so on. And the banking system is essentially a system for securely keeping sets of information books which if hacked, or otherwise interfered with, can radically result in the transfer of immense amounts of money in no time because all that’s happened is some bits have changed.

So we’re moving very rapidly into a world in which the Fort Knox’s of first life and the Fort Knox’s of Second Life are the same. They’re numbers in protected files protected by access control levels of one sort or another.

The good news is again through shared secret kinds of technologies or other ones we can break those assets up and distribute them across a million servers so that it doesn’t have to be centralized in order to be as safe as it is when centralized.

I can’t spend the money, you can’t spend the money, and she can’t spend the money because no one person or even two out of the three of us can reassemble the check. Bring us all into one place, make her J.P.Morgan, you Morgan Stanley and me the borrower and suddenly we can put our pieces together and the lost dollar bill is resuscitated.

Again it’s not a perfect solution to the problem. The 21st century economy is the first time there wasn’t anything that prevented me from stealing all the gold in Fort Knox. Sooner or later all the gold in Fort Knox is going to get stolen.

We faced in the first round of crypto wars the presence of the thing called the briefing which is what the incoming president and vice president used to get from the NSA about how we saved the planet through surveillance and if you had unclean encryption there’s going to be nuclear destruction.

I can remember when Mike Nelson of IBM got ‘the briefing’ as an incoming Clinton administration figure and was then sent to Phil Zimmerman and to me, Oh if you only know what I know now you’d give up the fight. And they said well publish it Mike and maybe we will. Instead we constructed a thing called the alternate briefing which was a briefing to policy makers which said because you don’t universalize strong encryption in the world because everybody doesn’t know how it works and everybody doesn’t have access to it, a secret hole built up last night in the world banking system three trillion dollars had disappeared. Nobody exactly knows where it is this morning. Markets open up in three minutes what do you do now Mr. President?

The problem with the Second Life asset is really the problem of the [First Life] asset. It’s not different. And one of the scary things about the 21st century is that it isn’t different. Having said all that, I agree that this question of making money out of nothing, so that assets come into existence suddenly is scarier for many of the people who run the world’s monetary system than the sudden disappearance of money.

It’s really scary that somebody could be making something that would have value in the system that they built because they think they’re the only people with right. In theory it could get to be a really important problem, The good news is that most of the people on earth can’t eat virtual food. In the end the power of the real economy is going to keep this thing restricted to a certain kind of luxury production. And we are probably good at that. That may cushion some of the difficulties that arise in the world where Pay Pal does more business than all the Forex markets everyday.

The stupidest piece of economics I ever read in my life was the paper which when I got to law school was the most admired piece of economic reasoning of the past generation which was Milton Friedman’s classic article about why currency speculators contributed to currency stability Friedman wrote this paper and most of the economists who taught me when I was in college in the seventies thought that that paper which was then 16 years old attacking the Brenton Woods’ system on the ground that currency speculation is the primary means of maintaining currency stability was the most brilliant thing they had ever read.

Bernie Saffron, my teacher at Swarthmore, who had served on Jimmy Carter’s council of economic advisors gave me a B+ in micro economics in 1979 after the A+ I got in macro economics in the fall because I wrote on the exam that Milton Friedman’s paper was the stupidest thing I ever read. Now as George Soros can explain I was correct. He was the one who got rich knowing that Milton Friedman was an idiot.

Currency speculators don’t contribute to stability of the world currency reserves. And the reason that they don’t is because they are not the marginal arbitragers that Milton Friedman thought they were. Milton Friedman’s logic was revealed to be horseshit the minute that the Forex traders started trading more every year than the monetary reserves of the large countries because that made central bank intervention irrelevant. And that’s what shelved the Rubel.

And it is what produced the great problems of currency in the late 1990s. And it is what’s about to produce the enormous difficulty we are going to have with the United States dollar – which is governments don’t control currencies levels anymore. Friedman thought that these arbitragers, these beautiful capitalist business men operating at the edges of a system whose gravity is over here can do nothing but stabilize it. Soros said wait a minute the axis has been flipped governments can’t even stabilize what I can trade.

Now we about to live in a world where that is square. And there is effectively no way to get larger than Pay Pal or the other bit traders eventually get because they have every business from two kids in a bar sharing a bar tab. I watch these guys in my twenty something crowds in LA first then in New York. When they go out for a round of drinks they Pay Pal the money back and forth to themselves from their handhelds at the bar. That is how they square the drink stab at the end of the night.

They have everything from that micro handle to all the business on EBay to the point at which if I want to get money to somebody elsewhere in the world I am way better sending it through Pay Pal than almost any other mechanism. It is almost one thirtieth the cost of sending a wire transfer internationally, and my bank is going to give me all this bullshit, and the Federal Government, and the know your customer and all that stuff.

Now admittedly the United States Government crawled inside the ass of Pay Pal when [inaudible]? were running it in the first place. They had it totally wired. But that’s OK it is less paper work than I have with dealing with J.P. Morgan.

Now we come to the Second Life asset stream look at what happens. We have virtualized the entire world …this stuff is stuff over in the corner but you can’t eat it. In the end you can’t trade it in for the stuff you can eat at anything like the speed that would be necessary in order to build it as a hole in the world financial system. When it is a hole in the world financial system you will suddenly meet some very powerful people making a real fuss.

Is anybody from one of the major investment banking houses sitting in on the Architectural Working Group.

David: No they are not.

Tish: But they are hovering around Second Life.

Eben: Well I will make a prediction. They will show up. And they will show up very heavy, very hard. If you are lucky they will show up through the securities industries financial management association or one of their sort of combined IT shops. If you are not lucky they will show up with the storm troopers because this is their stuff you are fucking with.

David: It is only a million a day turnover today so that is not scary to anyone. But if it goes from 1 million, to 10 million, to 100 million over the next ten years, they need to understand what is going to happen.

Eben: And they can expect to either be looking at an unregulated wild west that they would approach with both vigor and prudence or a highly regulated environment in which they will look for a way to use the house edge. And what they each want will depend on forces I can’t foresee about what Merrill Lynch and Citicorp are each thinking.

David: If I personally had the decision about this I would say we have pretty good micro-payment systems – called Pay Pal – why on earth would you make a new one.

Eben: Well yes… except that that one has its roots dug deep in the actual financial system which means the minute you go there, there is no fire wall. The play money aspect of the thing makes it possible for a whole lot of stuff to happen.

Your guys who are all so interested in their assets values they haven’t met the tax authorities yet.

David: Some have.

Tish: Yes in Europe with the VAT.

Eben: Yes I have heard that.

David: But the vast majority of them haven’t worried about the tip they get for playing live music being taxed the way they should be.

Eben: What will happen if that thing gets its hooks into the financial system, if it did share a payment platform with the real world, the taxing authorities would be there immediately. Look at the United States government’s attitude about gambling.

The asset values built up in an Antiguan casino online are the asset values in second life. They’re paper profits and losses generated in a flow of offshore money which if it has an entree to the American Credit Card and bank account is the story. So you could model this with the online poker explosion.

My CTO Bradley Coon was there every step of the way going up and every step of the way going down. Bradley used to sit at home in the evenings and play 12 tables while his wife was doing the Live Journal. They had a beautiful little digital console. Bradley played blindfold poker 16 tables while working for me. And of course made lots of profits which as the United States government began to close down the online poker shops and arresting British business executives on their way through the United States on their way to Acapulco and stuff like that, slowly closed down. And figuring out how to get your money out of each of the online poker houses as they each surrendered to the government and they effectively quarantined peoples winnings became what all the poker players were doing.

If Second Life actually ran head on into the tax system there would be a sudden recalculation of a whole lot of trading positions. This whole problem is the problem with the internet sales tax moratorium. It’s the problem of the bit tax not just the VAT on the goods but the VAT on the telecommunications services which is the great unregulated question in European tax laws.

David: And Australia for that matter. They have the same regulatory scheme, pay for bandwidth, pay for each bit.

Eben: And that proposition leads to the clear sense that the telecoms provider ought to calculate cost going in and cost going out and pay the VAT on the difference which produces an enormous windfall to the public fisc and which has uncalculable consequences for the nature of the digital economy. So people back off.

In an European law faculty there will be a person with a tenured chair in VAT that’s a separate legal specialty OK? And I’ve heard the European VAT professors when they get together and start fighting about the bit tax. They use services. Sales taxes on services have a very powerful European history. There was an 18th century system for sales tax on services which were brilliantly effective. It was called stamp taxes. It meant putting little pieces of revenue stamp on the pieces of paper on which services were performed. And if you were a service businessman all your documents in and out had revenue stamps on them and stamped paper was what you did business on. British have been paying stamp taxes since the early 18th century in perfect peace.

When you attempt to introduce sales taxes on services in North America you get trouble. It was done by the British Empire in the late 1760s and there was trouble. It was done in Florida in the 1990s and there was trouble. Our state advertising agencies didn’t like being taxed by Florida sales taxes on services and they contributed a great deal of advertising to making it very hard for the government of Florida to continue until it withdrew the sales taxes on services.

We on the other hand paid poll taxes in the United States until the Supreme Court outlaws them with the greatest happiness in life, in part because it helped us to keep black people from voting. Poll taxes were imposed in England twice, once in 1381 resulting in the peasants rising, and once under Margaret Thatcher resulting in the end of Margaret Thatcher’s regime.

Which cultures will pick what taxes is a very hard thing to understand. Nobody knows what virtual communities will bear and how they will pay their taxes. But when their values rise to a certain point they will pay their taxes and the question of that imposition will be disruptive in another way.

So I think that people ought to go cautiously they ought to understand that the big players with the big power, the people who use the money system around the world with a great deal of muscle and the people who use the tax system around the world with a great deal of muscle have not arrived at this party yet.

If they start making their rule under the assumption that those guys are never going to show up, they’re going to be very negatively surprised by the outcomes.

What they really need to be doing is inconspicuously and thoughtfully planning for what their positions are going to be when the big guys show up. And they need to get ready for that more than they need to have pipe dreams of their own about how they’re going to use technical means to preserve their asset values in currencies that will never make it to market, and that will never be subject to income tax and VAT. It’s going to look very different when it reaches equilibrium.