We are in the age of unearthing and uncovering data, and only just at the beginning of the age of processing data and dealing with it (see my interview with Anselm Hook, Part 2 upcoming).  O’Reilly’s Strata Confernence 2011, will explore, “the change brought to technology and business by data science, pervasive computing, and new interfaces.” It is, perhaps, one of the most important events of 2011.

Data is driving a revolution much as coal, oil, and steel powered the industrial revolution.  And the world changing insight from Karl Marx that “the industrial revolution polarized the world into two groups: those who own the means of production and those who work on them,” is taking on on new life, as  Alistair Croll, co-chair of Strata 2011, points out in his post,  “Who Owns Your Data?”

“The important question isn’t who owns the data. Ultimately, we all do. A better question is, who owns the means of analysis? Because that’s how, as Brand suggests, you get the right information in the right place. The digital divide isn’t about who owns data — it’s about who can put that data to work.”

Strata is where a vanguard will be meet, not only to discuss this revolution’s futures, but to define how to create, handle, and build the platforms and experiences that will harness the data. My flight is booked!  (Also check out BigDataCamp which takes place the night before Strata.)

The picture opening this post is from Michael EdgeCumbe’s  Fall 2010: ITP Winter Show Project.  A project exploring ways to intuitively get the feel of what it going on with big data sets using “the gestural manipulation and stereoscopic visualization of complex data to create a meditative state for data analysis.”  Michael project will be part of the Science Fair at Strata.  For more on Michael’s work see Noise Derived. I also have a number of the  interesting new interface sessions at Strata in my schedule.

The daily Strata Gems on O’Reilly Radar are great place to get a gestalt of some of the Strata themes, and this post by Edd Dumbill, program chair for Strata, Three key data trends for 2011, looks at the year ahead.  This week, I got the chance to ask Edd a few of the questions that I will have on mind at Strata – see his responses below.

If you have been reading Ugotrade, you will know I am interested in our mobile social augmented futures and there is no question in my mind that these will be unleashed by our new capacities to work with data (see my post here).

Data is the how.

The pic above is from “Secrets of BackType’s Data Engineers.” This post on ReadWriteHack by Pete Warden, an ex-Apple engineer, and founder of OpenHeatMap, really lives up to its title.  Check it out if you want to know how  “three guys (the BackType team ) with only seed funding process a hundred million messages a day?”

I asked on Quora, “What would be the most important developments for Augmented Reality in 2011,” Michal Avny, Strategist & Real Time search expert, wrote:

“AR strongly relies on localized personalized real time information.

Having a stream of tweets based on keyword search, location or circle of friends doesn’t really make the AR experience; it is the processed real time relevant information that will make AR useful and intensify the experience.”

In 2011 Real Time search and Social Search will drastically change to provide the infrastructure required.”

I followed up on Michal’s Quora answer with some more questions – see below in this post.

Also note the response from David Molnar, here is an excerpt:

“2. A wave of actionable, important data APIs opened up, enabling useful non-gimmicky AR apps for the first time. Think geoloqi.com , or the work Max Ogden has done with Portland civic data. Plus of course face.com , email providers and calendar providers, etc.”

Amber Case, one of the founders of Geoloqi, is on the programming committee of Strata and will be speaking. Be sure to catch her session! Posthumans, Big Data and New Interfaces, and if you haven’t already seen it, Amber’s TED talk is a must see.

Geographic proximity is a powerful filter, as is route, and time. But clearly social proximity, social relevance, and shared tastes are also key dimensions for location based experiences, (see my convo with Schuyler of Simple Geo, upcoming).

While the whole business of location based search and curation of augmented mobile social experiences is still, for the most part, uncharted terrain, the danger of key points of control being only really accessible to elite players looms large. I asked Sophia Parafina, a pioneer in the open geo space for some thoughts on real-time local /geosearch and geomessaging, and the future of openess & big data (see Sophia’s response below).

Is the market ready yet for P2P cloud computing?

This is another question I’m following,  Is the market ready yet for P2P cloud computing? It is one of those questions that we seem to have been asking in various forms for a very long while now, but without a  major shift in sight.  The pic above is from, The Cloud Made Open Source “Invisible” This Year.  But, perhaps, we are at the point when open p2p clouds will find a place in the market because of their potential importance in real time social search and discovery. Borislav Agapiev, Search Entrepreneur and founder of Vast.com, writes on Quora:

“I believe a P2P cloud is ideally suited for social & real-time search and discovery.

Consider MapReduce, a very interesting and popular paradigm for distributed computing. MapReduce is very much about bringing computation to data i.e. doing computation at nodes (map) and then aggregating results through network (reduce).

It is very clear now that user attention data (what they click on) is very valuable for search and discovery, yet a centralized model relies upon uploading all that to a single location and then doing a supposed local MapReduce. Clearly, MapReduce could be done across the network, without any centralized uploads.

In addition to the efficiency argument raised here, it is even more important to consider privacy issues. Uploading massive amounts of user attention data to a centralized location is not something that is going to make users warm and fuzzy as we are increasingly seeing.

In a P2P cloud, there is no big brother watching over anyone, all computation and data storage is done in the cloud, fragmented in many, many small encrypted pieces ala BitTorrent.”

Picture above from Brynn Marie Evans, “It takes two to tango: review of my social search panel“

The Delta of Now – Transforming Search into a Social Democratic Act

Picture of Maneki Neko “beckoning” cats from Journeyetc

New ecologies of human and machine intelligence are beginning to change basic social structures – see the Future of Work (Biewald and Chirayath Janah 2010). And projects like Swift River, using search and machine mining to filter out streams on topics of interest that can then be subsequently curated by human beings. This may be extended to the curation of real-time data streams and employment of machine learning algorithms based upon the explicit relationships.

Augmented mobile social experiences are a new frontier in which ideas and practices from a number of fields collide, including: ambient findability (Morville 2005), urban psychogeography, narrative structures, ambient games and devices, 4d (time-space), explorations of place and memory, enchanted objects and people (Kuniavsky 2010), and designed animism (Laurel 2010), to mention just a few.

Mobile local interaction presents an opportunity to invert the search pyramid and to transform search into a social, democratic act (see my interview with Anselm Hook upcoming).  Up until now search has been predicated around a very narrow revenue model. Google has an implicit model of a B2C – business to consumer brokerage. We are only just beginning to get a glimpse of the disruptive potential of C2C – consumer to consumer brokerages. Mobile local C2C brokerages that allow us to transact in a trustworthy way over our local geography in close to real time (Hook 2010) have the potential to enable new forms of social organization. Bruce Sterling’s short story about a networked gift economy, Maneko Neki, is a brilliant glimpse at the disruptive potential of such re-imaginings.

Augmented experiences that shift or change a person’s situated geolocal experience of social reality, and change our relationship to the people and the place by augmenting engagement in, and reputation through, socially driven consumer tie ins and game dynamics, like Four Square, & Gowalla are beginning to emerge, as Kati London pointed out in her excellent keynote at Web 2.0 Expo. And, while the integration of mobile local interaction and an augmented view that shifts our geolocal experience visually will involve creative solutions to some well churned mobile, tracking, mapping and registration challenges, the exploration and development of new dimensions through which we can filter and create trusted and meaningful augmented mobile social experiences is vital, whether you are considering a mobile screen, map, camera view, or futuristic HUDs and gestural interfaces.

Talking with Edd Dumbill

Picture from O’Reilly Community.

Tish Shute: First congratulations on Strata! On the Strata homepage there is a quote from Jason Hoffman:

“My gut feeling is that we’re going to look back at the upcoming Strata Conference like we do at the Web 2.0 Conference in 2004/2005.”
—Jason Hoffman, CTO/Founder, Joyent, Inc.

Why do you think Jason’s comparison might be prescient?

Edd Dumbill: Web 2.0 is a development that ran through every brand that has a web presence and radically changed the way business is done for many companies and brands.

Strata will have a similar impact: every business has data, every business collects an increasing amount of data. This data is the new oil – a valuable raw material that when refined or combined creates value and opportunity.

Tish Shute: The rise of real time was one of your three key data trends for 2011. Hadoop is bringing the capacity to work with big data to more than just a few elite players. But the challenge is still real time. You mention we will be seeing a hybrid approach to real time and batch MapReduce processing. Will we hear more about these approaches to real time at Strata? And, what do you see as the most important conversations on real time data analytics emerging at Strata?

You point out “open source projects and cloud infrastructure means developers can evaluate and learn to love technologies without requiring support or approval from above.” What are the most exciting developments on the horizon for open source tools?

Edd Dumbill: Here are some projects worth watching, in the key areas of real time, cluster management and Hadoop.

* Cassandra and MongoDB — NoSQL databases that will prove vital for anybody with real time big data needs

* Mesos — a compute cluster management tool, modeled after that which powers Google

* Hadoop ecosystem’s continuing maturation, especially HBase and Hive.

Tish Shute: Do you think the market is ready for p2p cloud computing?

Edd Dumbill: The market is emerging for decentralized and distributed cloud computing, and P2P technologies are one way of achieving that. They key trends will be moving computation nearer the data sets or nearer the point of user consumption of the result.

P2P is a difficult model for anybody wanting to commercialize a service, so I think it will tend to form part of a hybrid solution.

Tish Shute: We have seen enormous strides in our ability to work with giant unstructured databases recently. Do you think, perhaps, that the dream of a web of linked data – “a web of data that can be processed directly and indirectly by machines,” will be attained through brute force – i.e. through our ability to harness the power of massively parallel processing, as much as by Semantic Web approaches focused on machine readable metadata? [Also see my question on Quora, “Is this a good approach (www.dist-systems.bbn.com/people/…) to use Hadoop to build a scalable, distributed triple store?”]

Edd Dumbill: I’ve been an observer of the SW for over a decade and I tend to believe that on the web, data means to you whatever meaning you give it as the consumer. With that model, the links are made by the consumer rather than sitting out there explicitly. Some links become de facto standards, and some very few become web standards.

I think the actuality will be a mix of both explicitly stated metadata and that which is inferred. The Semantic Web is a great framework for certain operations, especially interoperable exchange of metadata. A great many more private meanings, never intended to be shared, will be created by consuming software.

There’s no question that machines will learn how to process most of the Web. Furthermore, machines will learn how to process most of the physical world we’re in. And that by the end of this decade.

Talking with Sophia Parafina

Picture of Sophia at Where 2.0

Tish Shute: Sophia you have worked in the trenches for a long time now to support the growth of open geo data. What do you hope to see emerge in 2011 in the field of geo-data?

Sophia Parafina: Better support for displaying and handling location data across multiple apps. Fred Wilson recently blogged about content-shifting, he talks about overcoming content silos across devices. We’ve worked very hard to reduce data silos via formats, but devices are creating their own silos. I would like to see a standard method for sending geo data and geo information to mobile devices.

Producing content for mobile is different from producing content for a computer browser. Web 2.0 produced a lot of infrastructure for browser based interfaces, but in mobile devices that gap has been filled with apps which is fragmenting how data is handled by various devices. What is even more interesting in the mobile space is that devices can push data back that contains location, user updates, photos and even sensor data.  If mobile data standardizes, it could lead to browser based applications and stem the continued fragmentation of the mobile application market.

Tish Shute: Simple Geo and Factual are startups emerging in the geodata space. What do you see on the horizon in terms of both the growth of business opportunities and an open geo data community?

Sophia Parafina: In the near future think we’ll see startups providing curated data + API and in response we will also see companies that provide a single interface across multiple data providers. We saw this when everyone released a mapping API and companies such as Mapufacture provided a single interface across multiple APIs.

We will see a resurgence in data providers repackaging the the 2010 US Census data in different ways to respond to market segments, some of this will be open data but all of it will be provided through an API instead of file. Additionally, we’ll see more data from outside the US.

Tish Shute: What are the biggest obstacles to having the open geodata sets available that we need to enable mobile local interactions and social augmented experiences?

Sophia Parafina: Licensing for both crowd sourced data and private curated open data will become an issue. We recently seen VLC, the open source video player, pulled from the Apple app store because of licensing issues. Also, licensing of content by geography will be problematic, limiting searches by geographical location. In addition, how will licensing of data that is updated by crowd sourcing work?

Multiple APIs for accessing data sources. The current trend for each provider to create an API for their data sets will result in data silos – there needs to be a single sign-on equivalent for requesting data.

Size of data on the wire, the current models for delivering data is based on broadband connections. However, as mobiles increasingly become the way people use the web, the data needs to be sized accordingly. This also goes for mobile interfaces. Have you tried to shop on a mobile device, or buy a train or plane ticket? It’s frustrating and error prone. There is a large untapped market of people who only use the Internet on mobile devices.

Tish Shute: You pointed me to this link in Strata Gems re “an interesting and pertinent (also a competitor to GeoLoqi),” – the Android Tasker app. What do these emerging services bring to the table in terms of the next generation of location based services?

Sophia Parafina: This app let’s your device interact with the environment. I think that this is a great way of using the sensors on existing platforms to increase interaction and to implement ambient findability. The basic premise of Tasker is that some action happens in response to an event in an application, time, date, location, event, or gesture. Tasker has defined 180 actions that can occur based and number or combination of events. This can provide a basic vocabulary for interaction between the user and the device and more importantly between users. Tasker also can use Android script plugins, which lowers the bar to creating your own ambient application.

Programs such as Tasker can provide a way for people to interact with social networks beyond sending messages. People can use their mobile devices to interact with their surroundings with out having to interact with the device.

Tish Shute: We have had many conversations about emerging ideas of geo-search, geo-messaging and geo-fencing. What are the most interesting developments in these areas and what do you see on the horizon for 2011?

Sophia Parafina: The map will fade into the background and become less important. Display of information will be context aware, that includes location. For example, let’s say I make a grocery list, when I’m at the grocery story, the list will just pop-up without the need for me to find the app that has the list. Or reminders or offers pop-up when you are near a place at a certain time, let’s say you need to buy a present for a birthday party for a child, you could send out a request that you are looking for an item and retailers could offer “on the spot” discounts if you are in the area.

Geo-search, geo-messaging, and geo-fencing are geared to towards mobile devices, so I expect to see them soon as part of apps. Building generic applications that implement geo* will fail because that sort of information is useful only within a context. Geo* apps are solutions looking for an problem. The killer mobile app will use these functions transparently to reduce the cognitive load of the user who is busy moving around in the world.

User data gathered from multiple web applications will become consolidated profiles that will used for context aware applications. For example, there could be a service which matches prices of items that you have shopped for on the web, so for example the service would have access to your cookies, know your favorite retailers, things you have shopped for, your location and activity patters (when you are at home, work, restaurant). When you are in the vicinity of a brick and mortar retailer with the same or similar items, the service can send you alert to match the price of the item you found on line. So your digital life will become more closely linked with your day to day activities.

Talking with Michal Avny

Tish Shute: At Web 2.0 Summit, one of the highlights for me was the, Q&A:The New Search Insurgents lunch where Charlie Cheever of Quora, IMO, stole the show. I tweeted:

“One of my takeaways from #w2s is that #quora points to future of augmented mobile social experiences – a search filter for experience! #AR”

In your view what are the biggest challenges for location Q&A to emerge as a search filter for location based experiences?

Michal Avny: The biggest location Q&A challenges yet to be conquered are immediacy (real time dynamic data), relevancy (strong personalized filters) and user experience (simplified interface).

Location Q&A enables different use cases. The most prominent are Follow (follow places, topics and friends to learn about a location), Interact (meet new people based on common interests), Plan ahead (plan a trip, night out or a shopping day by asking and searching for local information) and On-site (check for recommendations, friends, deals, events and traffic nearby).

Unlike Follow, Interact and Plan ahead that can be added to existing Q&A platforms (such as Quora) by attending location specifics as they share similar characteristics, the on-site mode introduces a completely different experience, first and foremost it requires immediate attention. It is real time based and the nature of the data is dynamic. Traffic updates, current events, nearby friends, all that changes constantly. Posting a location question on-site implies the response should be in real time (e.g. best kid friendly restaurant), the normal Q&A response latency wouldn’t work.

Strong relevancy filters are required to accommodate for the overwhelming flood of information. Moreover, some of the data should be filtered by user behavior and preferences, check in notifications (type of relation), restaurant recommendations (type of food, price level, etc), shopping deals (commercial categories) and more.

Mobile experience requires ease of use and simplicity. A new Q&A interface and query language that allows for posting questions should be defined as well as coherent summarized response interface. User on the go should not have to post lengthy questions, browse through tens of results or search for the right service, but instead use a simple intuitive tool.

Tish Shute: Real- time location based search is in its infancy. Real time questions can be answered using different services such as Yelp, TripAdvisor, Waze, Foursquare, IMDb and more. But what are the challenges to moving forward with aggregating these sources and then into “locals” that are able to process and deal with vast amounts of information?

Michal Avny: Using some of the leading location services to answer question is sufficient to start with.

In order to provide broad coverage (worldwide) and reliable information, aggregation of the different services is required for instance to normalize product and service rank, aggregate classified, and more. This is quite challenging as there is no one standard available.

When location Q&A user base is big enough, I foresee a tendency to rely more on ‘locals’ input as the base of information. As the platform grows, communities will be formed with different cultures, relationships and trust levels, making the information more valuable and customizable. Some of the challenges I already mentioned are implementing filters, query language and interfaces to enable using the vast amounts of real time data in a mobile environment. More of the challenges lying ahead are integrating the ‘locals’ data with location based services as they are integral components of the Q&A ecosystem. Merging trust levels and relationships while adhering to different privacy guidelines is a challenge yet to be explored. (This should be discussed in more detail under the protocols topic).

It is quite evident that Quora is now facing growing pains and is struggling to maintain its character. Same as with Quora, it will also be a challenge to support and maintain the ecosystem while allowing for massive scale-up.

Tish Shute: I have been very interested in exploring protocols that will be enablers to micro local interaction and mobile social interaction for AR – particularly the XMPP extensions and operational transform work of Google Wave (now Apache Wave), and PubSub protocols like PubHubSubbub and Erlang based RabbitMQ. We are beginning to see protocols emerging that could enable new real time local services. What do you think are some of the most valuable use cases for “locals” that this new generation of real time protocols can enable?

Michal Avny: AR is about interacting with digital information; the AR ecosystem is composed of layers and components such as devices, platforms, browsers, applications and content. For the different components to interact new protocols, security guidelines, and privacy policies must be in place. A standard will enable local vendors and service providers to publish specials, deals, updates and events for any application to broadcast, identify people and places by proximity (without having to use the same application or device), local recommendations will be shared by services, devices will be able to interact, location based platforms, such as Q&A, will have access to vast breadth of information, geo aware devices will provide consistent experience globally, and much more.

Tish Shute: What do you think are the biggest challenges to going mainstream for this emerging field of real time social discovery?

Michal Avny: The biggest challenge is building towards real time, geo-aware, localized, personalized ambient data. Discovery is in its infancy, location social based Best, Top, and Trending lists with some basic filtering options are available, and this is great as people are getting accustomed to information surrounding them. To some degree it can intensify the AR experience, for instance suggest the most popular dish in a restaurant, or map the best coffee shops nearby, but it is customized at best by friend recommendations and depends on the coverage and broadness of the specific discovery service.

There is a need for the next generation of discovery, customized geo social aware discovery that filters the vast amount of real time data by learning user preferences and behavior (built on top of the much needed local social real time open protocol)

Tish Shute: Who are your favorite startups/upstarts in the the field of real time search and why?

Micha Avny: My6Sense - My6sense provides a sharper and better way to experience your information from feeds you subscribe to (Social Networks, News, RSS feeds, etc.). It’s personal – Content is ranked based on what’s relevant to you. It learns what’s valuable to you by translating your consumption behavior into a personalized ranking function.
My6Sense – because it is a personalized prediction filter, a critical foundation for AR

Topsy – Topsy is realtime search powered by the social web that finds the most relevant conversations happening online. The site’s underlying technology examines popular links as well as the influence of each person citing a link. Topsy augments traditional search engines by finding information that people are talking about.
Topsy – because its ranking is based on retweets and influencers, a great social experience

Collecta – Collecta is a real-time search engine for the social web. It monitors the update streams of popular realtime blogs and sites like Twitter, WordPress, and Flickr, and shows results as they happen. Results can be filtered by status updates, comments, stories, or photos. The entire engine is built around the XMPP standard, which pushes out data on a continual basis, so that for every search you end up watching a stream that keeps updating itself.
Collecta – because it is built around XMPP, a real time experience

In a conversation with Tim O’Reilly and John Battelle (Federated Media Publishing) at Web 2.0 Summit 2008, Al Gore suggested that only the aggregate bandwidth of the internet could supply us with the kind of emotional intelligence we need to respond with appropriate urgency to the challenges of our times, for example, the CO2 targets necessary to avert catastrophe.

“People hear these things, and there are many other similar signals, and then the next day it’s gone. Now the neuroscientists have explanations for why that is ….. The urgency center of the brain is geared to snakes, spiders and fire and things that evolution posed as tests to our species…

But when we have to use our neo cortex to connect dots in an abstract pattern and then push that down to the urgency and fear center – that’s just a little footpath.

Its  like the internet, mostly, it’s an asynchronous connection.  There is a big connection going from the fear center to the reasoning process but just a very small pathway coming back.

It needs to be stored in the cloud. It is the aggregate bandwidth than counts. We need to have the truth – the inconvenient truth, forgive me, of this challenge stored in the cloud so that people don’t have to rely on that process and so that we can respond to it collectively.”

Tim O’Reilly responded: “Who knew you were the guru of Web 2.0 as well as global warming. You have totally outlined our premise here.”

(Photograph opening this post of the Former Vice President Al Gore on stage with Tim O’Reilly and John Battelle at  Web 2.0 Summit 2008, co-presented by O’Reilly Media and TechWeb. Produced by Good Company Communications. Photograph copyright James Duncan Davidson – see Duncan Davidson’s Flickr stream for a complete photo essay of the event.)

I was trying to find a word to express how powerfully  Al Gore addressed the Summit audience.  And I was discussing this with a legendary serial entrepreneur, Richard Titus, who is also a great admirer of Al Gore, at the closing party. Richard came up with the phrase I was seeking.  “He was totally naked,” Richard said.

Al Gore described himself as a recovering politician.  And yes, he seems totally recovered from the “woodeness” of politics and utterly at home with the “nakedness” of participatory culture.

Al Gore made clear that to change the world we have to change ourselves (he did).

Bertrand Russell is often attributed with the following quote:

The mark of a civilized human being is the ability to read a column of numbers and then weep.

Gore’s exhortation that the internet needs to be a puppy with a purpose resonated with his audience.  From climate change, global issues of health care, to rethinking global economies we desperately need to optimize our collective and individual intelligence.

Instrumenting the World: Life on the Cloud

Kevin Kelly’s  High Order Bit – a brilliant impressionist view of the internet’s next 6537 days describes what “Life on the cloud” will be like.

“If you are producing some information and it is not webized, i.e., it is online and not related and shared to everything else, it doesn’t count.”

This is already the case to some degree. And the challenge of understanding where our networked identities begin and end is with us. But Kevin Kelly points out, “life on the cloud” will heighten our dilemmas.

Nat Torkington’s presentation to the Privacy Forum in Auckland , New Zealand, “Web Meets World: Privacy and the Future of the Cloud” looks at our changing idea of identity through the lens of privacy – both “the nature of privacy” and “how expectations change over time.”  Nat cites William Gibson (interviewed by Rolling Stone on their 40th Anniversary):

“One of the things our grandchildren will find quaintest about us is that we distinguish the digital from the real, the virtual from the real.  In the future that will likely become impossible.”

The critical layer between this database of things and the ultra, mega cloud (see Kevin Kelly’s slide below) is the web of shared intelligence. This is where the transformation will emerge with its dangers and opportunities.

Brian Solis, in his excellent post, “Barack Obama, The Social Web, and the Future of User Generated Government,” proposes Zappos and their “public and transparent customer-focused culture” is a good model for how government can use the internet not only to push out its message but to create a whole new culture of participation.

Far fetched?  Watch Tony Hsieh’s High Order Bit for yourself. The idea that every interaction at Zappos has relevance to the value exchange between consumers and producers is a very interesting idea to apply to the relationship between government and citizens.

“Ecological Intelligence”

Instrumenting the World requires new models of data sharing. Last year, Cory Doctorow described to me an instrumentation model of data.

An Instrumentation model for data differs from a surveillance model of data sharing.  Instrumentation is “when you know a lot about the world,” in contrast to surveillance – “when people in authority know a lot about you”.

(Note: Mashable has an interesting post on the theme of a “instrumentation,” see:  Seventeen Killers Apps for Taking Control of Your Government:“Government is increasingly putting much of its public records online, creating opportunities for developers to build useful applications for citizens.”)

But corporate culture and governments around the world have embraced the surveillance model of data up to now.  I was fortunate to have the opportunity to ask Larry Brilliant, Google.org, a question about how the tables might get turned.  After his conversation with Tim O’Reilly,  I asked:

“What would motivate corporations and governments to participate in the kind of data sharing and transparency that could produce the changes that our world needs, particularly in the area of health and climate change? For example, why would corporations reveal the aspects of products we use and the food we eat that have negative effects on our health and our planet?” (This is more succinctly phrased than my original question!)

Larry Brilliant replied:

“I don’t know how many of you know Dan Goleman? He created emotional intelligence [quotient] – EQ. He is coming out with a book which I have just had the pleasure of reading in draft form which deals specifically with what you are talking about.

How we can have commercial intelligence. How we use the power of corporations and their various different stakeholders, including their customers to drive corporations to do the morally right thing by losing the commercial support of customers who won’t support them unless they are more green, fairer to women, respect gay and lesbian rights, do the things you would like them to do whatever that happens to be, so that you can vote with your dollars.

It is really a fascinating book:  “The Application of Ecological Intelligence to the Commercial World.”  I don’t know what the final title will wind up being but I recommend it to you.

Dan Goleman’s new book: “Ecological Intelligence: How Knowing the Hidden Impacts of What We Buy Can Change Everything,” will come out in April, 2009.

An Extraordinary Gathering At An Historic Time

Web 2.0 Summit was a brilliantly orchestrated gathering of many of the thought/business leaders and entrepreneurs who have shaped the internet as we know it today.

As my friend Jonathan Hochman, Wikipedia, said on Day 1:

“If everyone here [Web 2.0 Summit] shut down their website it would be the end of the internet!.” (See my upcoming interview with Jonathan on Wikipedia and Jon Brouchard on Wikitecture and what these projects can teach us about participatory culture).

But also in this elite crowd of “C” level execs were the next generation of entrepreneurs who are working on a hunch and prayer to create the future Web.

And this year, as the Web 2.0 Summit architects explained in their intro, the decision was made to extend the scope of the Summit even further:

“….our world is fraught with problems that engineers might charitably classify as NP hard—from roiling financial markets to global warming, failing healthcare systems to intractable religious wars. In short, it seems as if many of our most complex systems are reaching their limits.

It strikes us that the Web might teach us new ways to address these limits. From harnessing collective intelligence to a bias toward open systems, the Web’s greatest inventions are, at their core, social movements. To that end, we’re expanding our program this year to include leaders in the fields of healthcare, genetics, finance, global business, and yes, even politics.”

Truly an extraordinary gathering at an historic time – commencing the day after Barak Obama became President Elect, it seemed the causes and conditions for participatory culture and sustainable living were coming together at last!

Virtual Worlds and “The Web Beyond The Web:” Creating “A Supple Approach to Sharing Identity”

Virtual Worlds were not on the schedule.  But this is not surprising as their potential contributions to the very big problems at the heart of the Summit’s theme are only just beginning to emerge.

But new forms of  participatory culture were a recurrent theme of the Summit.  And Virtual Worlds at the high bandwidth tip of the pyramid of global connectedness and SMS at the bottom of the pyramid have a lot to teach us about participatory culture.

Crista Lopes recently co-founded with Christer Lindstrom a company, Encitra, that is focused on improving urban planning processes, starting with transportation, using virtual worlds. Christer Lindstrom has been a key evangelizer of PRT (personal rapid transit – see photo above).

Crista Lopes is Associate Professor at the University of California, Irvine, in the Department of Informatics (full interview coming soon).  Crista is using the dynamic shared viewpoint of virtual world technology to offer a way for the many stakeholders involved in a city scale transportation infrastructure change to participate in the process of planning. Crista is working with OpenSim – see the video of  “Encitra – Creating Immersive Worlds.”

There are a number of use cases for Virtual Worlds in sustainable living being developed. I have written several posts on Oliver Goh’s work,  “The Path to Sustainable Real Estate.” See my earlier posts here, and here, and IBM’s Virtual Network Operation Centers.“

Also see the recent announcement from Intel Research to create ScienceSim using OpenSim (more on this soon). Justin Rattner writes:

“Wilfred Pinfold (an Intel colleague and general chair of Supercomputing 2009) announced to the Supercomputing 2008 conference attendees plans to create a new virtual world called “ScienceSim.” Supported by Intel and the conference committee, this collaboration aims to use these immersive, connected environments to further cutting edge scientific research.”

George Jobi, Intel, writes in his post on ScienceSim: “Intel is one of the founding members of OpenSim and had been building its vision of open standards based 3D web architecture around OpenSim.”

The Achilles Heel of Web 2.0…….?

As Crista pointed out:

“The  Achilles Heel of Web 2.0 is trying to build the concept of person in a platform that doesn’t have people, at the center of the architecture. With Web 2.0 we go through a lot of hoops trying to integrate basics concepts of identity and storage onto a platform that wasn’t designed for it.”

Most of us have bits of our identity scattered all over the web, e.g., partial friends list here, there and everywhere. Some of us have literally hundreds of different log ins and profiles. Our list of applications with pieces of our identity locked up in them might look something like the slide below from the High Order Bit of Beerud Sheth, Webaroo Inc.

In contrast, Crista noted:

“The key component that a Virtual World offers you is that you can take your identity from place to place and the presence of people is at the center of the whole thing.”

Crista has already submitted code that introduces hyperlinks to OpenSim (see here). Crista is computer scientist of many accomplishments including being the co-inventor of Aspect-Oriented Programming.

There is a long conversation in the comments on my interview with Tim O’Reilly about whether the concept of avatar is the Achilles Heel of Virtual Worlds. So I asked Crista:

“Are avatars the Achilles Heel of Virtual Worlds?”

Crista explained why she thinks this is not the case in the modular open source architecture of OpenSim at least.

“The concept of people is not tied to the concept of avatar in OpenSim: One of the important parts of the OpenSim architecture is that the concept of user is very different from the concept of avatar.”

In OpenSim, Crista noted:

User = identity +storage

When I asked David Levine, IBM, what Web 2.0 could learn from virtual worlds re sharing identity, David, who works on interoperability and protocols in the Architecture Working Group, said:

Immersive spaces, are the real time, multi-user online component of Web 2.0, and identity is deeply part of that……..virtual Worlds teach us, as they expose more resources to Web 2.0, that there needs to be increasingly “supple” ways of sharing identity that go beyond simply anchoring it on gmail or openID, or such.

Social media has been one of Web 2.0′s success stories – giving the impression that Web 2.0 has people at the core of its architecture. But, as Crista pointed out, this is not the case.

There is no way in Web 2.0 to do identity at the level of platform, at the moment. As soon as you want to create identity on the Web there is a big mess.”

Participatory Culture at the Bottom of the Pyramid: “The Web Beyond The Web”

The “Web Beyond the Web,” Beerud Sheth, Webaroo Inc quipped, is not his announcement of Web 3.0. Rather, Beerud is describing the parallel innovation at the bottom of the pyramid where lower prices on mobile devices rather than new features drives adoption and voice and SMS (short messaging service) rule.

SMS is the web of the people for most of the world.  The current ratio is 10:1 with 10 people using text messaging to every 1 that has web access and the SMS population is growing at a much higher rate than web users. The  innovation at the top of the pyramid, where a plethora of Web 2.0 apps are built on top of  http, looks like the unreadable slide above with a forest of applications.

In contrast innovation at the bottom of the pyramid, until recently, has been limited to ringtones, wall papers, and voice response mechanisms.  So Beerud introduced a new service GupShup.

Gup Shup = Chit Chat

“Think of GupShup as another cool word from the language that gave you yoga, nirvana and karma sutra,” Beerud said.

GupShup is a “Twitter for India” but on a vastly bigger scale (only 18 months from launch they are up to 12 million users).

But, Beerud points out, don’t just file away GupShup as another twitter clone.  While they have Web and WAP site, they are deeply intergated into SMS as the lowest common denominator. GupShup can be used entirely from mobile which is vital as they have more users already than the total number of web users in India.

This idea of fully integrating into the lowest common denominator medium, SMS, has allowed GupShup to grow extremely rapidly. And, interestingly, when you look at the use cases you see the end users are deploying many of the uses cases that are familiar from the web,

Beerud left the audience with the take away that all the use cases are surprisingly similar to the web as are the ways of monetizing them,  This is creating enormous opportunity for creativity and entrepreneurship in building out this web beyond the web.

He invited those who already know the possibilities of the web to come and join this new adventure.  The enormous scale of the “web beyond the web,” and the fact people are connected almost continuously, creates vast opportunities for participatory culture to expand beyond the small triangle at the top of the pyramid.

On the “web beyond the web” the potential of 160 characters is explored on a scale unimaginable on Web 2.0 where Twitter, for example, is just one app in a vast ocean of other possibilities.

Crossing the Chasm Between The Top and the Bottom of the Pyramid

This total separation between the top and the bottom of the pyramid is, in part at least, constructed through the current web culture of web exclusive subscriptions.

It is perfectly possible to write an app that would accept SMS text and post it on a web page without ever requiring a web visit from the SMS subscriber. The same app could also accept text input from a web page and send it out as SMS to one or many subscribers that have never visited a web page, thus enabling communication across this gap.

Oxygenating the System: Monetizing Doing the Right Thing

The VCs, business leaders and  entrepreneurs at Web 2.0 Summit had their entrepreneurial Spidy Senses (as John Battelle calls them) tuned to the challenges and opportunities of Web Meets World.  Some of the winners of the Web 2.0 Launch Pad Competition explored the premise that doing the right thing can be monetized.

Danny Kennedy’ Sungevity was the overall winner.  Sungevity’s aim is to “scale solar electricity as a solution to climate change.”  Their use of a Virtual Earth feed to streamline the installation of solar panels and ambition to be the SalesForce.com for the solar industry was a very winning combo.

Good Guide, a really excellent service (also available as an iphone app) providing a guide to all products from the perspective of their healthfullness, greeness and other socially valuable criteria clearly scored a 10 on doing the right thing.  But Good Guide’s ability to succeed on the monetizing side of the equation was questioned by one of the VC’s on the Launch Pad panel.

Carbon Networks pitched with the mantra “do the right thing and enhance the balance sheets in the process.” But the difficulty there, it seems to me, is that there are many questions re the benefits, or lack of them, of global carbon trading markets.

Carbon Networks argued that carbon markets, which are already a giant industry, present enormous opportunity for companies to monetize doing the right thing.

I asked Gavin Starks (who I interviewed recently about his venture AMEE – a BIG project to aggregate the world’s energy data) about the problems of carbon markets.

“They have high levels of inappropriate use even for a new market area,” he commented, noting:

“There are some superb projects out there, but it would be fair to say there has been good dose of snake oil in the space – which has certainly not helped to build consumer confidence. However, markets are necessary to engage with the scale of investment that is needed to address the issue – it’s the use of funds that needs more scrutiny and greater transparency needs to be given to the whole process.”

There are projects working with Voluntary Emissions Reduction which aren’t tradable on proper carbon cap-and-trade markets, “though in theory the step up to CERs (certified emissions reductions) isn’t too great a thing,” Gavin noted.

MicroEnergy Credits the  initiative presented on the Track Me panel by April Allderdice, co-founder and CEO, is a good example of this.

Gavin pointed me to CheatNeutral and their YouTube video for a hilarious and razor sharp look at the problems of carbon offsetting. The text below is from the CheatNeutral site.

Gavin also explained a new initiative Sandbag (beta). Sandbag aims to take the permits that allow polluters to pollute out of the system.

“Thanks to policy makers in the UN and Europe levels of pollution are now controlled. Permits must be bought by polluters to let them keep polluting. But there is a finite number of them in circulation and the good news is anyone can buy them. So by taking a permit out of the system we can reduce the amount of pollution taking place and force industry to invest in cleaner technologies. One less permit means one less tonne of greenhouse gas in the atmosphere.”

AMEE is working with  Sandbag

Consuming Less and Redefining Prosperity

This picture is from the Sustainable Mobility Panel at the ASPO-USA Peak Oil Conference.

Perhaps nowhere is it more clear than when we look at the reports that link catastrophic climate change to the assumption of growth that what is really at stake in terms of averting catastrophe is not just retooling our energy infrastructure, but fundamental changes at the level of culture and identity.

Consuming less may be the single biggest thing you can do to save Carbon Emissions, Tim O’Reilly said, in his Tweet on “Why politicians dare not limit economic growth.”

A growing band of experts are looking at figures like these and arguing that personal carbon virtue and collective environmentalism are futile as long as our economic system is built on the assumption of growth. (New Scientist)

But few of us are willing to contemplate what a sustainable economy and averting the catastrophe of climate change require – redefining prosperity and reducing consumption (see Redefining Prosperity).

Web 2.0 Summit took on the challenge of reimagining giant industries like energy, food and transportation and how we might  be able to shift away from a culture of food and energy consumption that is basically killing us and our world (see Michael Pollan’s brilliant High Order Bit on the culture of food in the US).

The Summit gurus urged that taking risks and tackling very big problems has always been what Web 2.0 is about and indeed cultural shifts of the magnitude needed would be hard to imagine without a Web 2.0 perspective

Shai Agassi, Better Place, explained how paradigm shifts require new business models. See Shai’s High Order Bit here on the evolution of “Better Place,” -  by giving away free electric cars he is creating a new business venture that will bring clean cars into the mass market.  New business models not just new technology are required to drive change.

Breaking News From Reuben Steiger’s blog

First Israel.  Then Denmark.  A few weeks ago, Australia.  Today,  Mayor Newsom along with Governor Schwartznegger and the Mayors of San Jose and Oakland, announced that they would be making a major move towards bringing electric vehicles and the Better Place network to the Bay Area.

Please, visit Planet Better Place to sign the petition,  join the movement and bring Better Place to your town or country.

To motivate yourself and others how important it is to change patterns of consumption see Saul Griffith’s High Order Bit here and Project Wattzon

“…..from flying, driving, powering a home, eating, shopping, working and even one’s share of the energy necessary to make our society function. WattzOn helps users understand their personal impact on the environment and how they rate compared to others WattzOn users, as well as global averages.”

“The Secret Sauce”: New Business Models for Web Meets World

I spent some time talking to Don Dodge, Director of Business Development, Microsoft’s Emerging Business Division, about the future of  virtual worlds and what technologies he thought would play an important role in developing the participatory architecture of the web (full interview coming soon!).

“The question is how do you apply these technologies? Where is the best use for them? And this is the hard part.  When you look at social media and social networks and things like Wikipedia, don’t look so much at the technology because that is fairly simple.

Look at the rules of social interaction and how people interact, and how you put protections in there so that people don’t game the system or do bad things.  Look at the processes because that’s really the secret sauce of how it all works.  The technology is simple. It looks easy from a distance, when you start getting into how it really works from a social perspective that’s the secret sauce.”

(screenshot above from Threadless )

Also I caught up with John Battelle, Federated Media Publishing (see his Data Bill of Rights here), and Jennifer Pahlka, TechWeb, at a small press conference. I managed to squeeze in a couple of questions!

Tish Shute: If marketing has been the oxygen of the system up to now, what will oxygenate the system of the new participatory culture of Web meets World”

John Battelle: I don’t think marketing ever stops being one of the most significant pieces of the economy -  because it is, of the whole economy. So what I do think will happen, and this is the company that I run, I do think marketing will shift very dramatically in terms of its approach and how it is a part of the value exchange that occurs around goods.

One of the reasons that I had Tony Tsieh from Zappos was to show that.   Tony shows how every single human being in his organization is a marketer and sees every interaction they have as marketing.  Can you imagine  a company as big as Intel that has that kind of an approach?  That’s when we will have a real shift. Business models based on that idea are emerging.  I run a company that is involved in that. I don’t try to push it on the stage ..but I do it is right there Federated Media!  And now I am pushing it [laughs]

Its an idea that comes from all this staring at this. I do think marketing is going to shift quite dramatically. So we may see in 10 yrs that we don’t have a big media budget  pushing adds at people. But will there always be budgets for creation of value exchange between consumers and producers? yes! There will just be new models for how that money is distributed and spent and new services and intermediaries for that value exchange.

Tish Shute: But who controls definition of data will remain key right?

John Battelle: There is a reason why Yahoo, AOL, Facebook, MySpace, all of whom are here, and Google, are all about the data….all about the data….sorry I have to go!

Jennifer Pahlka: I think in addition to the enormous changes that John was just talking about in marketing, and I think these are very significant – the way marketing will be seen completely differently 5 years from now.  There is also the shift in Web 2 away from an over identification with  Web 2.0 as being primarily about and driven by advertising  because of these models that are emerging for Web 2 that are driven by data, driven by services, subscription.  There are a whole bunch of other business models for Web 2 start ups and for enterprise that really don’t have anything to do with that at all.

Tish Shute: And in terms of participatory culture and sharing data?

Jennifer Pahlka: And even on a simpler level than the data.  Think of a company like Threadless [see screenshot above]. Their co-founders are keynoting at our Spring event.  They have taken some of the other principles of the architecture of participation and the creativity of the community and built a whole difference around that.  And all they do is sell T-Shirts.

“A Billion One-Person Enterprises”

New York Times writer, Saul Hansell, in his article, “Web 2.0 Gets Big and Corporate,” writes, “the best minds of our generation are turning to the Web for solutions.”  …..the big companies that make very complicated systems are reworking them using the principles of Web 2.0 companies.”

But “big companies” themselves may soon be a thing of the past.  One of the potential futures many my friends in virtual worlds have been looking at is, “if the future consisted of a billion one-person enterprises.”

Tony O’Driscoll described some of his thinking re the role virtual worlds will play in this potential future.  See Tony’s presentation, “A brief history of a potential future” on SlideShare. Tony’s research provides a window onto the new participatory architecture of business, government and the economy and the ways the individual and the collective will have new dynamic relationships based on “co-creation.”

Second Life and Wikipedia are the two great experiments in collaborative co-creation. They show us how co-creation can be one of the keys to a participatory global culture and sustainable living – part of creating an alternative to this economy of escalating consumption that has us in its death grip today.