Singly’s appearance at the startup showcase at Strata 2011 this week has excited thought leaders across the web since the story got out. Singly is a new startup that exists to provide oxygen and commercial support to the open source Locker Project, and new protocol TeleHash.

With some wonderful serendipity I met Singly on my first night at Strata.  The next day, I talked in depth to Jeremie Miller and Simon Murtha-Smith, two of the three Singly co-founders (see later in this post). I also had the opportunity to ask Tim O’Reilly, Alistair Croll and Roger Magoulas for some of their thoughts on the significance of this project (see below for their comments).

It was a real – pinch myself in case I need to wake up from a dream experience – for me, to stumble across Jeremie Miller with Simon Murtha-Smith sitting behind a hand written sign demoing Singly at Strata (see my  pic opening this post). As Marshall Kirkpatrick notes:

“Jeremie Miller is a revered figure among developers, best known for building XMPP, the open source protocol that powers most of the Instant Messaging apps in the world. Now Miller has raised funds and is building a team that will develop software aimed directly at the future of the web.”

Singly’s appearance at Strata began auspiciously when they won the judges choice award in the startup showcase. And following Marshall Kirkpatrick’s post, Creator of Instant Messaging Protocol to Launch App Platform for Your Life , and The Locker Project: Why Leave Data Tracking to Others? Do It Yourself, Singly have been burning up Twitter.

Singly, by giving people the ability to do things with their own data, has the potential to change our world.  And, as Marshall Kirkpatrick notes, this won’t be the first time Jeremie has done that.

“Pop-cultural instruments for data expression and exploration,” by Bloom

I was drawn over to the Singly table when an awesome app they were demonstrating caught my eye. Fizz, which is running on a locker with data aggregated from three different places is a first glimpse of one of Bloom’s, “pop-cultural instruments for data expression and exploration.”

Fizz is an intriguing early manifestation of capabilities never seen before on the web – the ability for us to control, aggregate, share and play with our own data streams, and bring together the bits and pieces of our digital selves scattered about the web (for more about Bloom and Singly, see Tim O’Reilly’s comments below). The picture below is my Fizz. In Fizz, large circles represent people and small circles represent their status updates. Bloom says:

“Clicking a circle will reveal its contents. Typing in the search box will highlight matching statuses.
This is an early preview of our work and we’ll be adding more features in the next few weeks. We’d love to hear your feedback and suggestions.”

If you are not already familiar with The Bloom team, Ben Cerveny, Tom Carden, and Jesper Sparre Andersen – go directly to their about page and you will understand why the match of Bloom and The Locker Project is a cause for great delight.

The Locker Project – a whole new way to connect from the protocol up

As Jeremie began explaining the depth and breadth of what The Locker Project is facilitating, I was utterly gob smacked. And when the penny dropped and I realized this is the whole 9 yards, bringing awesomeness to people with a whole new way to connect, from the protocol up, all I could think was, OMG finally!

Luckily I have had time to catch up with the whole team since then, and recovered my composure enough to ask some coherent questions. But I can still barely contain my enthusiasm for this project.

Singly, The Locker Project and TeleHash take on, and deliver a simple, elegant, and open solution to some of the holy grails of the next generation of networked communications. I have written on, and been nibbling at the edges of some of these grails in various projects myself for quite a while now. Even if you haven’t been reading Ugotrade, just a glance at the monster mash of my pre Strata post will give you an idea of how important I think Singly is.

My previous post raised the question of how to invert the search pyramid and to transform search into a social, democratic act. But if you are really interested in social search, I suggest staying keyed into what Singly is doing with The Locker Project!

One of Singly’s three founders, Simon Murtha-Smith, was building a company called Introspectr, a social aggregator and search product. Singly’s other founder Jason Cavnar was working on another similar project. And they came together as Singly because social aggregation and search is a very hard problem for one company to solve, and they realized that the basic infrastructure needs to be open source and built on an open protocol.

As Jeremie puts it, “We shouldn’t…(every startup that wants to do something interesting) have to spend this much time aggregating the data, building robust aggregators.”

To me what is so important about the Locker Project is that it is built on a new open protocol, TeleHash. And having the Singly team focused on supplying tools and the trust/security layer for the Locker Project will mean that developers have the whole stack they need to do some interesting stuff very soon.

I asked Jeremy to explain the relationship between TeleHash, The Locker Project and Singly.

Tish Shute: So TeleHash…

Jeremie Miller: It’s a peer-to-peer protocol to move bits of data for applications around. Not file sharing, but it’s for actual applications to find each other and connect. So if you had an app and I had an app, whenever we’re running that app on our devices, we can actually find those other devices from each other and then connect. Our applications can connect and do something.

For the entire edge of the network, basically, out there in the wild, and let those things mesh together.

And TeleHash is actually what has led to the Locker project itself.

Tish Shute: So TeleHash led to the The Locker Project and the Locker Project led to Singly?

Jeremie Miller: Singly is a company who is sponsoring the open source Locker Project…the three of us as founders, [left to right in pic below - Jeremie Miller, Jason Cavnar, Simon Murtha-Smith, ]

I took the pic above of all three founders being interviewed by Marshall Kirkpatrick of Read Write Web for his post, “Creator of Instant Messaging Protocol to Launch App Platform for Your Life.“  I think we will look back on this moment and say it was an inflection point for the web. At least I tweeted that!

Jeremie Miller: TeleHash is a protocol that lets the lockers connect with each other and share things. The locker is like all of your data. So it’s sort of like a digital person…

Tish Shute: A locker for bits and pieces your digital self?

Jeremie Miller: Yes. So TeleHash lets the lockers connect and directly peer-to-peer connect with each other and share things. Singly, as a company, is going to be hosting lockers first and foremost. But the Locker Project is an open source project. You can have a locker in your machine or you can install it wherever you want…

Tish Shute: Yes it’s often too difficult for a lot of people to set up something locally…so Singly makes it easy to have a locker right?

Jeremie Miller: A lot of people see this cool app or this cool thing they want to do, it’s something that you run in your locker that they need to be able to turn on a locker somewhere very easily.

Tish Shute: So Singly will provide the trust layer and hosting?

Jeremie Miller: Yeah, Singly is a company that will host lockers, as well as…when people build applications that run inside your lockers or use your data, you need to be able to trust them. Maybe it’s like social data and you don’t care that much. But especially once you start to get any of your transactions in there, your browsing history, your health data, like your running logs or sleeping…fit-bit stuff…then it’s much more important to be careful about what you’re running inside your locker and sharing.

So Singly will also look at the applications that are available that you can install and actually run them and look at what data they access, and look at who created them, and be able to come back and either certify or vouch for them.

And I hope in the long-run, as this grows and builds, that power users may actually be able to buy a little device that they can plug into their home network that is their locker. Wouldn’t that be cool? This little hard drive or whatever that you plug in.

Tish Shute: Wow – that would be very cool! Architecturally is TeleHash and the Locker Project related to your work on XMPP?

Jeremie Miller: Architecturally, some of the stuff I’ve learned, XMPP, in Jabber it was designed for the specific purpose of instant messaging, but it was still a federated model, in that you still had to go through sort of a central point so you could…a server that lived somewhere. So it was really optimized for like businesses and small groups, teams, as well as big companies out there; ISPs can use it.

So it was designed with that in mind—for the communication path to be routed through somewhere. And where I’ve sort of evolved over the years since then is really fascinated with truly distributed protocols that are completely sort of decentralized so that things are going peer-to-peer instead of actually through any server.

The last 10 years, peer-to-peer has gotten a pretty bad rap with file sharing.

Tish Shute: A really bad rap, yes.

Jeremie Miller: Yeah. And almost because of that, and because it’s really hard to do, that it hasn’t gotten…the potential for it’s awesome. There’s so many really good things that can be done peer-to-peer. And it hasn’t gotten used very much.

But the other side of the peer-to-peer thing that I think is critically important, look at the explosion of the computing devices around a person anymore, both in the home and on our person. We have one, two, three, four even. And the number of devices that are online for you that are yours…I look at my home network router and I’ve got 30 devices in my house on Wi-Fi. What the heck? That’s a lot of devices.

But right now, all of those devices, for me to work with them, I’m almost always going through a server somewhere, through a data center somewhere, which is ridiculous at face value. You go five, 10 years out from now, there’s probably going to be 300 devices on me in some form.

Tish Shute: So we need a peer-to-peer network just to manage our own devices?

Jeremie Miller: A peer-to-peer, yes. You know, my phone should be talking straight to my computer, or to the iPad, or to the washing machine, or refrigerator. The applications in my TV, or whatever, they should all be talking peer-to-peer. And it should be easy to do that. It shouldn’t be that the only way you can do that is to go through a data center somewhere.

[Our conversation continued, but to sum things up, for now, here is the final question I asked Jeremie which pretty much packs in everything I would like to do with TeleHash, the Locker project, and Singly tools/trust layer all in one!]

Tish Shute: How can TeleHash, the Locker, and Singly help people combine personal data from different sources – web and mobile for example, so the data locked up in our social graph on the web can be integrated with, for example, the location data and “the data wake” from our cell phone sensors, to know not only where we have been but to give us more ways to know where we are going?

Jeremie Miller: That’s a pretty packed question, but here’s my simple answer, hopefully just seeds the right discussion

Telehash is the protocol that lets the apps (mobile, sensor, or anywhere) talk to a locker as well as lockers talk to each other, it’s the chatter, moving the bits around the network. The locker is the storage for a person’s data and the crunching ability to analyze it or trigger actions from it. Singly is the company sponsoring the project(s) and helping anyone dev apps atop it. We’re going to build the platform and looking to the world to create some amazing things on top of it (we have lots of our own personal ideas we already want to create, hah!).

The Locker Project is not just “one more rebel army trying to undo these big data aggregations,” Tim O’Reilly

@lockerproject: “We’ll be posting updates on the Locker Project (https://github.com/quartzjer/Locker) here as we make progress, very awesome stuff “

During the Strata Media Conversation I asked Roger Magoulas about Singly and The Locker Project because Roger played Yentl and brought Singly and Bloom together!  Although there was not much time to discuss it, the relationship of TeleHash, The Locker project and Singly to the social network encumbents, came up, and Roger Magoulis and Tim O’Reilly gave some very insightful comments on this when I talked to them afterwards (see below).

Roger Magoulas pointed out:

“I think Singly has Facebook like aspects, but I think a better description is an app platform that integrates your personal and social network data – including data from Facebook. Sing.ly is likely to have challenges with some of their data sources, particularly if Sing.ly gains traction with users.

I like the app platform business model, although they face risks getting critical mass and app developer attention, and I like how they plan on using open source connectors to keep up with changing social network platforms. Jeremie has credibility with the open source community and is likely to find cooperating developers. The team seems to bring complementary strengths to the project and you can tell they all work well together. ”

And Tim O’Reilly went on to elaborate the awesome potential of this platform to bring something new to the ecosystem, and to comment on just how interesting Bloom’s insight into, “data visualization as a means of input and control” is.

Talking with Tim O’Reilly

Tish Shute: So will the Locker Project be able to break the lock of Facebook’s and other big sites’ control of everyone’s data. Sometimes I feel we are stuck in the era of Zyngification, where you have to do what Zynga did and leverage the system in order to gain traction or do anything with social data?

Tim O’Reilly: I don’t think that is the objective of the Locker Project —to break the Facebook lock, because I tend to agree, the value of Facebook is having your data there with other people’s data. What Singly may be able to accomplish is to give people better tools for managing their data. Because if you can actually start to abstract the data from various sites and you can set it and manage it yourself, then you can potentially make better decisions about what you’re going to allow and not allow. Because right now, the interfaces on a lot of these sites make it very, very difficult to understand exactly what the implications are.

And I think all this done right will create a marketplace where people will build better interfaces that will give people more control over their data. They’ll still want to put it on those sites, because why do you put your money in the bank? You know, because it’s more valuable being with other people’s money.

And I think that to conceive of it as one more rebel army trying to undo these big data aggregations is just the wrong way to frame it.

Tish Shute: Yes and framing the question the way you just did – that this is not just one more rebel army, might mean that the stage at Strata will be filled with new startups next year! That’s what I thought when I found out what The Locker Project and Singly was about – that we are about to see an explosion of creativity with personal and social data.

Tim O’Reilly: Yeah, sure. I mean, because at the end of the day, if you can start to extract your personal data in ways that make it more useful, you can potentially create the ability for people to build better interfaces. It’s not just Facebook.

You know, you think, “Oh, wow, I’d really like to have a management console for all my contacts.” And you go, “Well, I’m stuck with, I can use Facebook, I can use LinkedIn, I can use my address book in Outlook or Gmail or whatever, or on my local machine.” The tools are pretty primitive. And if we get a better set of tools, I think we’ll see a lot of innovation.

Now, some of those startups might well be acquired by a Facebook or a Google. But it if moves the ball forward in giving people better visibility and control over their data, that’s a good thing.

Bloom’s insight, “data visualization will become a means of input and control.”

Tish Shute: I loved the marriage with Bloom, which is interesting, because Ben and the Bloom team haven’t really talked a lot about Bloom yet, but I gather Bloom is moving to consumer facing work with data?

Tim O’Reilly: What’s really interesting about Bloom is the notion…You know people think of data visualization as output. And the insight that I think Ben has had with Bloom is that data visualization will become a means of input and control.

Tish Shute: Right, very cool.

Tim O’Reilly: So I’ve started to feel like data visualization as a way of making sense of complex data is kind of a dead-end. Because what you really want to do is to build these feedback loops where you actually figure something out, some particular atomic action well enough that you can create an application that let’s somebody actually do something with it. But the idea of visualization as a way of manipulating the data in real-time, data visualization as interface rather than as a report, it’s a small but subtle shift that I think becomes kinda cool.

Talking with Alistair Croll

Sing.ly – Join or Die from Singly Inc on Vimeo.

Alistair Croll: So I’m a big fan of Singly. They were my choice for the Startup Showcase. I think it’s certainly the right time – the team can execute on it. But the thing I like the most is I thought back to the early days of Photoshop.

So, Photoshop was a neat application that could take data in the form of an image and manipulate it. But the real value from Photoshop came from these amazing plugins. Like, there’s a company called Kai’s Power Tools that made these things that would allow you to do manipulations. Today, commonplace things that are built in. But at the time, they were things like building bubbles, and spheres, and drop shadows and stuff like that, cutouts, in amazing ways.

Another company, I think, called Alien Skin that made these things. There’s whole ecosystems of plugins. So you could go and get a plugin and transform that original data in ways you hadn’t thought of. And eventually, there was a macro language for scripting how you could do those things, and that found its way into the Photoshop environment.

But you think about the transformation of digital design from Photoshop, I think if you can take that same pattern of you create the basic ecosystem of a few tools, and then you allow people an open system on top of that, that’s unprecedented. I think it really does allow you to take ownership of that.

And then when you allow people the proper tools to federate that information. I was actually thinking of starting a company a couple of years ago based on data federation like that. But what you really want to say is I’ve got a pattern…It’s almost like a multi-channel mixer. You’ve got a band that is your health, your weight, your blood pressure, family photos, words you’ve used. You know, the more data I record when I carry my phone around with a headset of whatever, all of that stuff goes in, all my searches, everything.

And then I say, “Ah, I want to federate height, weight, and blood pressure with my doctor. I want to federate sleep cycles and nutrition with my child’s teacher,” and so on and so on. And you start to create these federated sources of data where now you have a teacher data mining, in a safe manner, the sleep and health habits of all the students along with report card information. And you suddenly realize that Johnny is borderline diabetic and falls asleep at recess.

That’s something that never would have happened. And that happens when you have tools to federate data and then compute on top of them. So this idea of, like, lifestreaming or life logging, this is a logical consequence of the whole lifestreaming movement; that whole recorded future stuff.

Tish Shute: Yes it really is a wonderful fruition to the visions of the lifestreaming movement [see this interview with David Galernter]. And best of all it sits on a new open protocol, TeleHash and the open source Locker Project that will give tools to everyone to work with these data streams.

Alistair Croll: Exactly. This is the toolset that sits on top of that stuff. Because once I’ve life-streamed everything, great, I have this bucket of stuff that I did that I never look at again. But if I can suddenly unlock that with data mining tools and analyze patterns, all of a sudden that life logging has a reason to have existed.

The biggest problem we have with data right now is we don’t have apriori knowledge of what will be useful. We could have been recording crime reports in the city of Chicago, and a year later it turns out that data is really useful for predicting diabetes in the city, but we didn’t know it was related.

So the problem, and one of the things I think that distinguishes big data from traditional data, traditional data is collected to some apriori knowledge of how it will be used. Big data tends to be collected for the sake of…it’s almost collected on faith that later on it will be useful for something.

Tish Shute: I am very interested in this idea of federation, I actually went as far as to deep dive into Wave servers because of this….

Alistair Croll: Yeah, Wave was a great example of federation, just too complicated. When it was canceled, both users [and developers] were furious.

Tish Shute: Yeah, I suppose you could see Google Wave as a bit of an Icarus project, right? I am so excited by Singly because it is coming sort of bottom-up – a very different approach…

Alistair Croll: And remember, Facebook didn’t work before Friendster. The only difference between being wrong and being too early is that too early costs a lot of money. So it may be that this is an idea that works now, but a couple years ago didn’t work.

Alistair Croll, co-chair of Strata 2011, in his post, reframed the question, “Who Owns Your Data?” as, “It’s not who owns the data, it’s about who can put the data to work.”

And I am sure there will be many more people able to put data to work, and into play, in a multitude of  interesting ways, now we have TeleHash, the Locker Project, and Singly.

Photo by Duncan Davidson.
Strata 2011 is presented by O’Reilly Media. Produced by Good Company Communications.

I think the photo above gives a good idea of how I felt on the last day at the Strata conference. Yup – like the cat who got the cream!

And in case you are wondering where AR is in this story – it is everywhere!  Below is a pic of the AR concept designs that were omnipresent in the media communications at Strata.  The one below I snapped off the job board.  But as Sophia Parafina noted,  “AR is maturing from displaying last year’s text bubbles and dinosaurs to big data overlaid on the world.”

We are in the age of unearthing and uncovering data, and only just at the beginning of the age of processing data and dealing with it (see my interview with Anselm Hook, Part 2 upcoming).  O’Reilly’s Strata Confernence 2011, will explore, “the change brought to technology and business by data science, pervasive computing, and new interfaces.” It is, perhaps, one of the most important events of 2011.

Data is driving a revolution much as coal, oil, and steel powered the industrial revolution.  And the world changing insight from Karl Marx that “the industrial revolution polarized the world into two groups: those who own the means of production and those who work on them,” is taking on on new life, as  Alistair Croll, co-chair of Strata 2011, points out in his post,  “Who Owns Your Data?”

“The important question isn’t who owns the data. Ultimately, we all do. A better question is, who owns the means of analysis? Because that’s how, as Brand suggests, you get the right information in the right place. The digital divide isn’t about who owns data — it’s about who can put that data to work.”

Strata is where a vanguard will be meet, not only to discuss this revolution’s futures, but to define how to create, handle, and build the platforms and experiences that will harness the data. My flight is booked!  (Also check out BigDataCamp which takes place the night before Strata.)

The picture opening this post is from Michael EdgeCumbe’s  Fall 2010: ITP Winter Show Project.  A project exploring ways to intuitively get the feel of what it going on with big data sets using “the gestural manipulation and stereoscopic visualization of complex data to create a meditative state for data analysis.”  Michael project will be part of the Science Fair at Strata.  For more on Michael’s work see Noise Derived. I also have a number of the  interesting new interface sessions at Strata in my schedule.

The daily Strata Gems on O’Reilly Radar are great place to get a gestalt of some of the Strata themes, and this post by Edd Dumbill, program chair for Strata, Three key data trends for 2011, looks at the year ahead.  This week, I got the chance to ask Edd a few of the questions that I will have on mind at Strata – see his responses below.

If you have been reading Ugotrade, you will know I am interested in our mobile social augmented futures and there is no question in my mind that these will be unleashed by our new capacities to work with data (see my post here).

Data is the how.

The pic above is from “Secrets of BackType’s Data Engineers.” This post on ReadWriteHack by Pete Warden, an ex-Apple engineer, and founder of OpenHeatMap, really lives up to its title.  Check it out if you want to know how  “three guys (the BackType team ) with only seed funding process a hundred million messages a day?”

I asked on Quora, “What would be the most important developments for Augmented Reality in 2011,” Michal Avny, Strategist & Real Time search expert, wrote:

“AR strongly relies on localized personalized real time information.

Having a stream of tweets based on keyword search, location or circle of friends doesn’t really make the AR experience; it is the processed real time relevant information that will make AR useful and intensify the experience.”

In 2011 Real Time search and Social Search will drastically change to provide the infrastructure required.”

I followed up on Michal’s Quora answer with some more questions – see below in this post.

Also note the response from David Molnar, here is an excerpt:

“2. A wave of actionable, important data APIs opened up, enabling useful non-gimmicky AR apps for the first time. Think geoloqi.com , or the work Max Ogden has done with Portland civic data. Plus of course face.com , email providers and calendar providers, etc.”

Amber Case, one of the founders of Geoloqi, is on the programming committee of Strata and will be speaking. Be sure to catch her session! Posthumans, Big Data and New Interfaces, and if you haven’t already seen it, Amber’s TED talk is a must see.

Geographic proximity is a powerful filter, as is route, and time. But clearly social proximity, social relevance, and shared tastes are also key dimensions for location based experiences, (see my convo with Schuyler of Simple Geo, upcoming).

While the whole business of location based search and curation of augmented mobile social experiences is still, for the most part, uncharted terrain, the danger of key points of control being only really accessible to elite players looms large. I asked Sophia Parafina, a pioneer in the open geo space for some thoughts on real-time local /geosearch and geomessaging, and the future of openess & big data (see Sophia’s response below).

Is the market ready yet for P2P cloud computing?

This is another question I’m following,  Is the market ready yet for P2P cloud computing? It is one of those questions that we seem to have been asking in various forms for a very long while now, but without a  major shift in sight.  The pic above is from, The Cloud Made Open Source “Invisible” This Year.  But, perhaps, we are at the point when open p2p clouds will find a place in the market because of their potential importance in real time social search and discovery. Borislav Agapiev, Search Entrepreneur and founder of Vast.com, writes on Quora:

“I believe a P2P cloud is ideally suited for social & real-time search and discovery.

Consider MapReduce, a very interesting and popular paradigm for distributed computing. MapReduce is very much about bringing computation to data i.e. doing computation at nodes (map) and then aggregating results through network (reduce).

It is very clear now that user attention data (what they click on) is very valuable for search and discovery, yet a centralized model relies upon uploading all that to a single location and then doing a supposed local MapReduce. Clearly, MapReduce could be done across the network, without any centralized uploads.

In addition to the efficiency argument raised here, it is even more important to consider privacy issues. Uploading massive amounts of user attention data to a centralized location is not something that is going to make users warm and fuzzy as we are increasingly seeing.

In a P2P cloud, there is no big brother watching over anyone, all computation and data storage is done in the cloud, fragmented in many, many small encrypted pieces ala BitTorrent.”

Picture above from Brynn Marie Evans, “It takes two to tango: review of my social search panel“

The Delta of Now – Transforming Search into a Social Democratic Act

Picture of Maneki Neko “beckoning” cats from Journeyetc

New ecologies of human and machine intelligence are beginning to change basic social structures – see the Future of Work (Biewald and Chirayath Janah 2010). And projects like Swift River, using search and machine mining to filter out streams on topics of interest that can then be subsequently curated by human beings. This may be extended to the curation of real-time data streams and employment of machine learning algorithms based upon the explicit relationships.

Augmented mobile social experiences are a new frontier in which ideas and practices from a number of fields collide, including: ambient findability (Morville 2005), urban psychogeography, narrative structures, ambient games and devices, 4d (time-space), explorations of place and memory, enchanted objects and people (Kuniavsky 2010), and designed animism (Laurel 2010), to mention just a few.

Mobile local interaction presents an opportunity to invert the search pyramid and to transform search into a social, democratic act (see my interview with Anselm Hook upcoming).  Up until now search has been predicated around a very narrow revenue model. Google has an implicit model of a B2C – business to consumer brokerage. We are only just beginning to get a glimpse of the disruptive potential of C2C – consumer to consumer brokerages. Mobile local C2C brokerages that allow us to transact in a trustworthy way over our local geography in close to real time (Hook 2010) have the potential to enable new forms of social organization. Bruce Sterling’s short story about a networked gift economy, Maneko Neki, is a brilliant glimpse at the disruptive potential of such re-imaginings.

Augmented experiences that shift or change a person’s situated geolocal experience of social reality, and change our relationship to the people and the place by augmenting engagement in, and reputation through, socially driven consumer tie ins and game dynamics, like Four Square, & Gowalla are beginning to emerge, as Kati London pointed out in her excellent keynote at Web 2.0 Expo. And, while the integration of mobile local interaction and an augmented view that shifts our geolocal experience visually will involve creative solutions to some well churned mobile, tracking, mapping and registration challenges, the exploration and development of new dimensions through which we can filter and create trusted and meaningful augmented mobile social experiences is vital, whether you are considering a mobile screen, map, camera view, or futuristic HUDs and gestural interfaces.

Talking with Edd Dumbill

Picture from O’Reilly Community.

Tish Shute: First congratulations on Strata! On the Strata homepage there is a quote from Jason Hoffman:

“My gut feeling is that we’re going to look back at the upcoming Strata Conference like we do at the Web 2.0 Conference in 2004/2005.”
—Jason Hoffman, CTO/Founder, Joyent, Inc.

Why do you think Jason’s comparison might be prescient?

Edd Dumbill: Web 2.0 is a development that ran through every brand that has a web presence and radically changed the way business is done for many companies and brands.

Strata will have a similar impact: every business has data, every business collects an increasing amount of data. This data is the new oil – a valuable raw material that when refined or combined creates value and opportunity.

Tish Shute: The rise of real time was one of your three key data trends for 2011. Hadoop is bringing the capacity to work with big data to more than just a few elite players. But the challenge is still real time. You mention we will be seeing a hybrid approach to real time and batch MapReduce processing. Will we hear more about these approaches to real time at Strata? And, what do you see as the most important conversations on real time data analytics emerging at Strata?

You point out “open source projects and cloud infrastructure means developers can evaluate and learn to love technologies without requiring support or approval from above.” What are the most exciting developments on the horizon for open source tools?

Edd Dumbill: Here are some projects worth watching, in the key areas of real time, cluster management and Hadoop.

* Cassandra and MongoDB — NoSQL databases that will prove vital for anybody with real time big data needs

* Mesos — a compute cluster management tool, modeled after that which powers Google

* Hadoop ecosystem’s continuing maturation, especially HBase and Hive.

Tish Shute: Do you think the market is ready for p2p cloud computing?

Edd Dumbill: The market is emerging for decentralized and distributed cloud computing, and P2P technologies are one way of achieving that. They key trends will be moving computation nearer the data sets or nearer the point of user consumption of the result.

P2P is a difficult model for anybody wanting to commercialize a service, so I think it will tend to form part of a hybrid solution.

Tish Shute: We have seen enormous strides in our ability to work with giant unstructured databases recently. Do you think, perhaps, that the dream of a web of linked data – “a web of data that can be processed directly and indirectly by machines,” will be attained through brute force – i.e. through our ability to harness the power of massively parallel processing, as much as by Semantic Web approaches focused on machine readable metadata? [Also see my question on Quora, “Is this a good approach (www.dist-systems.bbn.com/people/…) to use Hadoop to build a scalable, distributed triple store?”]

Edd Dumbill: I’ve been an observer of the SW for over a decade and I tend to believe that on the web, data means to you whatever meaning you give it as the consumer. With that model, the links are made by the consumer rather than sitting out there explicitly. Some links become de facto standards, and some very few become web standards.

I think the actuality will be a mix of both explicitly stated metadata and that which is inferred. The Semantic Web is a great framework for certain operations, especially interoperable exchange of metadata. A great many more private meanings, never intended to be shared, will be created by consuming software.

There’s no question that machines will learn how to process most of the Web. Furthermore, machines will learn how to process most of the physical world we’re in. And that by the end of this decade.

Talking with Sophia Parafina

Picture of Sophia at Where 2.0

Tish Shute: Sophia you have worked in the trenches for a long time now to support the growth of open geo data. What do you hope to see emerge in 2011 in the field of geo-data?

Sophia Parafina: Better support for displaying and handling location data across multiple apps. Fred Wilson recently blogged about content-shifting, he talks about overcoming content silos across devices. We’ve worked very hard to reduce data silos via formats, but devices are creating their own silos. I would like to see a standard method for sending geo data and geo information to mobile devices.

Producing content for mobile is different from producing content for a computer browser. Web 2.0 produced a lot of infrastructure for browser based interfaces, but in mobile devices that gap has been filled with apps which is fragmenting how data is handled by various devices. What is even more interesting in the mobile space is that devices can push data back that contains location, user updates, photos and even sensor data.  If mobile data standardizes, it could lead to browser based applications and stem the continued fragmentation of the mobile application market.

Tish Shute: Simple Geo and Factual are startups emerging in the geodata space. What do you see on the horizon in terms of both the growth of business opportunities and an open geo data community?

Sophia Parafina: In the near future think we’ll see startups providing curated data + API and in response we will also see companies that provide a single interface across multiple data providers. We saw this when everyone released a mapping API and companies such as Mapufacture provided a single interface across multiple APIs.

We will see a resurgence in data providers repackaging the the 2010 US Census data in different ways to respond to market segments, some of this will be open data but all of it will be provided through an API instead of file. Additionally, we’ll see more data from outside the US.

Tish Shute: What are the biggest obstacles to having the open geodata sets available that we need to enable mobile local interactions and social augmented experiences?

Sophia Parafina: Licensing for both crowd sourced data and private curated open data will become an issue. We recently seen VLC, the open source video player, pulled from the Apple app store because of licensing issues. Also, licensing of content by geography will be problematic, limiting searches by geographical location. In addition, how will licensing of data that is updated by crowd sourcing work?

Multiple APIs for accessing data sources. The current trend for each provider to create an API for their data sets will result in data silos – there needs to be a single sign-on equivalent for requesting data.

Size of data on the wire, the current models for delivering data is based on broadband connections. However, as mobiles increasingly become the way people use the web, the data needs to be sized accordingly. This also goes for mobile interfaces. Have you tried to shop on a mobile device, or buy a train or plane ticket? It’s frustrating and error prone. There is a large untapped market of people who only use the Internet on mobile devices.

Tish Shute: You pointed me to this link in Strata Gems re “an interesting and pertinent (also a competitor to GeoLoqi),” – the Android Tasker app. What do these emerging services bring to the table in terms of the next generation of location based services?

Sophia Parafina: This app let’s your device interact with the environment. I think that this is a great way of using the sensors on existing platforms to increase interaction and to implement ambient findability. The basic premise of Tasker is that some action happens in response to an event in an application, time, date, location, event, or gesture. Tasker has defined 180 actions that can occur based and number or combination of events. This can provide a basic vocabulary for interaction between the user and the device and more importantly between users. Tasker also can use Android script plugins, which lowers the bar to creating your own ambient application.

Programs such as Tasker can provide a way for people to interact with social networks beyond sending messages. People can use their mobile devices to interact with their surroundings with out having to interact with the device.

Tish Shute: We have had many conversations about emerging ideas of geo-search, geo-messaging and geo-fencing. What are the most interesting developments in these areas and what do you see on the horizon for 2011?

Sophia Parafina: The map will fade into the background and become less important. Display of information will be context aware, that includes location. For example, let’s say I make a grocery list, when I’m at the grocery story, the list will just pop-up without the need for me to find the app that has the list. Or reminders or offers pop-up when you are near a place at a certain time, let’s say you need to buy a present for a birthday party for a child, you could send out a request that you are looking for an item and retailers could offer “on the spot” discounts if you are in the area.

Geo-search, geo-messaging, and geo-fencing are geared to towards mobile devices, so I expect to see them soon as part of apps. Building generic applications that implement geo* will fail because that sort of information is useful only within a context. Geo* apps are solutions looking for an problem. The killer mobile app will use these functions transparently to reduce the cognitive load of the user who is busy moving around in the world.

User data gathered from multiple web applications will become consolidated profiles that will used for context aware applications. For example, there could be a service which matches prices of items that you have shopped for on the web, so for example the service would have access to your cookies, know your favorite retailers, things you have shopped for, your location and activity patters (when you are at home, work, restaurant). When you are in the vicinity of a brick and mortar retailer with the same or similar items, the service can send you alert to match the price of the item you found on line. So your digital life will become more closely linked with your day to day activities.

Talking with Michal Avny

Tish Shute: At Web 2.0 Summit, one of the highlights for me was the, Q&A:The New Search Insurgents lunch where Charlie Cheever of Quora, IMO, stole the show. I tweeted:

“One of my takeaways from #w2s is that #quora points to future of augmented mobile social experiences – a search filter for experience! #AR”

In your view what are the biggest challenges for location Q&A to emerge as a search filter for location based experiences?

Michal Avny: The biggest location Q&A challenges yet to be conquered are immediacy (real time dynamic data), relevancy (strong personalized filters) and user experience (simplified interface).

Location Q&A enables different use cases. The most prominent are Follow (follow places, topics and friends to learn about a location), Interact (meet new people based on common interests), Plan ahead (plan a trip, night out or a shopping day by asking and searching for local information) and On-site (check for recommendations, friends, deals, events and traffic nearby).

Unlike Follow, Interact and Plan ahead that can be added to existing Q&A platforms (such as Quora) by attending location specifics as they share similar characteristics, the on-site mode introduces a completely different experience, first and foremost it requires immediate attention. It is real time based and the nature of the data is dynamic. Traffic updates, current events, nearby friends, all that changes constantly. Posting a location question on-site implies the response should be in real time (e.g. best kid friendly restaurant), the normal Q&A response latency wouldn’t work.

Strong relevancy filters are required to accommodate for the overwhelming flood of information. Moreover, some of the data should be filtered by user behavior and preferences, check in notifications (type of relation), restaurant recommendations (type of food, price level, etc), shopping deals (commercial categories) and more.

Mobile experience requires ease of use and simplicity. A new Q&A interface and query language that allows for posting questions should be defined as well as coherent summarized response interface. User on the go should not have to post lengthy questions, browse through tens of results or search for the right service, but instead use a simple intuitive tool.

Tish Shute: Real- time location based search is in its infancy. Real time questions can be answered using different services such as Yelp, TripAdvisor, Waze, Foursquare, IMDb and more. But what are the challenges to moving forward with aggregating these sources and then into “locals” that are able to process and deal with vast amounts of information?

Michal Avny: Using some of the leading location services to answer question is sufficient to start with.

In order to provide broad coverage (worldwide) and reliable information, aggregation of the different services is required for instance to normalize product and service rank, aggregate classified, and more. This is quite challenging as there is no one standard available.

When location Q&A user base is big enough, I foresee a tendency to rely more on ‘locals’ input as the base of information. As the platform grows, communities will be formed with different cultures, relationships and trust levels, making the information more valuable and customizable. Some of the challenges I already mentioned are implementing filters, query language and interfaces to enable using the vast amounts of real time data in a mobile environment. More of the challenges lying ahead are integrating the ‘locals’ data with location based services as they are integral components of the Q&A ecosystem. Merging trust levels and relationships while adhering to different privacy guidelines is a challenge yet to be explored. (This should be discussed in more detail under the protocols topic).

It is quite evident that Quora is now facing growing pains and is struggling to maintain its character. Same as with Quora, it will also be a challenge to support and maintain the ecosystem while allowing for massive scale-up.

Tish Shute: I have been very interested in exploring protocols that will be enablers to micro local interaction and mobile social interaction for AR – particularly the XMPP extensions and operational transform work of Google Wave (now Apache Wave), and PubSub protocols like PubHubSubbub and Erlang based RabbitMQ. We are beginning to see protocols emerging that could enable new real time local services. What do you think are some of the most valuable use cases for “locals” that this new generation of real time protocols can enable?

Michal Avny: AR is about interacting with digital information; the AR ecosystem is composed of layers and components such as devices, platforms, browsers, applications and content. For the different components to interact new protocols, security guidelines, and privacy policies must be in place. A standard will enable local vendors and service providers to publish specials, deals, updates and events for any application to broadcast, identify people and places by proximity (without having to use the same application or device), local recommendations will be shared by services, devices will be able to interact, location based platforms, such as Q&A, will have access to vast breadth of information, geo aware devices will provide consistent experience globally, and much more.

Tish Shute: What do you think are the biggest challenges to going mainstream for this emerging field of real time social discovery?

Michal Avny: The biggest challenge is building towards real time, geo-aware, localized, personalized ambient data. Discovery is in its infancy, location social based Best, Top, and Trending lists with some basic filtering options are available, and this is great as people are getting accustomed to information surrounding them. To some degree it can intensify the AR experience, for instance suggest the most popular dish in a restaurant, or map the best coffee shops nearby, but it is customized at best by friend recommendations and depends on the coverage and broadness of the specific discovery service.

There is a need for the next generation of discovery, customized geo social aware discovery that filters the vast amount of real time data by learning user preferences and behavior (built on top of the much needed local social real time open protocol)

Tish Shute: Who are your favorite startups/upstarts in the the field of real time search and why?

Micha Avny: My6Sense - My6sense provides a sharper and better way to experience your information from feeds you subscribe to (Social Networks, News, RSS feeds, etc.). It’s personal – Content is ranked based on what’s relevant to you. It learns what’s valuable to you by translating your consumption behavior into a personalized ranking function.
My6Sense – because it is a personalized prediction filter, a critical foundation for AR

Topsy – Topsy is realtime search powered by the social web that finds the most relevant conversations happening online. The site’s underlying technology examines popular links as well as the influence of each person citing a link. Topsy augments traditional search engines by finding information that people are talking about.
Topsy – because its ranking is based on retweets and influencers, a great social experience

Collecta – Collecta is a real-time search engine for the social web. It monitors the update streams of popular realtime blogs and sites like Twitter, WordPress, and Flickr, and shows results as they happen. Results can be filtered by status updates, comments, stories, or photos. The entire engine is built around the XMPP standard, which pushes out data on a continual basis, so that for every search you end up watching a stream that keeps updating itself.
Collecta – because it is built around XMPP, a real time experience