This story is moving so quickly that it is hard to keep a handle on all of the developments, not to mention trying to follow the denials and non-denials from those who are allegedly involved, and the threads that tie this particular story to the long and sordid history of the U.S. government’s surveillance of its own citizens. So we thought it would be useful to try and collect what we know so far in a single post, which will be updated as often as possible with new information.

The Guardian leak

Guardian blogger and former lawyer Glenn Greenwald reports that the NSA has gotten a secret order from the Foreign Intelligence Surveillance Court that allows it to collect data about phone calls made by “millions of customers” on the Verizon network: location data, time and other identifying info about the call — everything except the actual content of the calls themselves (the Guardian has a background piece about what kind of metadata is available with such an order).

“The National Security Agency is currently collecting the telephone records of millions of US customers of Verizon, one of America’s largest telecoms providers, under a top secret court order issued in April. The order… requires Verizon on an ‘ongoing, daily basis’ to give the NSA information on all telephone calls in its systems, both within the US and between the US and other countries.”

The leak widens

Other stories that follow the Guardian report quote anonymous sources saying the Verizon court order is a renewal of an order that has been in place for some time, and add that other telecom companies such as AT&T are also involved in similar programs. Greenwald notes in his story that the NSA started a program of bulk collection of telephone, internet and email records in 2001 under President Bush and this later caused controversy when it was reported in 2006 that the NSA had been saving all of this information and was analyzing it to try and detect terrorism.

Information-security experts and other industry watchers note after Greenwald’s story is published that the NSA and other government agencies have had these kinds of abilities for years thanks to laws such as the Protect America Act and the FISA Amendments Act. ProPublica has a roundup of what the government can find out about you and your behavior without a search warrant, and security expert Bruce Schneier says that what we don’t know about the government’s surveillance programs is even more frightening than what we do know.

Meanwhile, our Stacey Higginbotham wonders whether the NSA story will be a wakeup call about the power of big data, while Derrick Harris looks at how the security agency and other government entities analyze the vast amounts of information that come from such programs.

Freelance journalist Joshua Foust argues that the NSA revelations won’t cause most people to change their behavior — including their habit of voting for politicians who enact the kind of legislation that permits such surveillance — because they simply don’t care enough about the issue. Some experts said the kind of data the NSA is getting can be very powerful when it comes to finding patterns of behavior, but research from the Cato Institute says that even mining large amounts of data can turn out to be not that helpful when it comes to catching terrorists.

The Wall Street Journal, meanwhile, said that the NSA’s surveillance program was “legal and necessary” and the furor over the disclosure of this program was misplaced:

“Nobody’s civil liberties are violated by tech companies or banks that constantly run the same kinds of data analysis. We bow to no one in our desire to limit government power, but data-mining is less intrusive on individuals than routine airport security. The data sweep is worth it if it prevents terror attacks that would lead politicians to endorse far greater harm to civil liberties.”

The Washington Post leak

Within hours of the Guardian story appearing, the Washington Post reports that it has been leaked an internal slide presentation from the NSA that describes a program it calls PRISM — which involves the collection of email and other personal data from internet companies including Google, Microsoft, Facebook, Apple and Yahoo. According to the Post report (and a subsequent Guardian report based on a similar leak), this program has been underway since at least 2007, and involves what one NSA slide refers to as “data collected directly from the servers” of the companies named.

All of the companies who are reportedly involved in PRISM (which refers to them as “partners”) deny any knowledge of such a program, and say they only provide data when forced to do so by court order, and that they have no “back door” systems that would allow the NSA to do what it claims to be doing. These denials are met by widespread skepticism, and many observers — including TechCrunch founder turned VC Michael Arrington — wonder why insiders working at the tech giants allegedly involved in the program wouldn’t have leaked the information earlier.

The ongoing fallout

Some tech-industry observers say the denials from internet companies may be true, because they aren’t convinced the companies in question would even have to know about the NSA’s collection practices in order for them to work. The original Washington Post story is updated early Friday to note that it’s not clear whether “direct access” to the servers of those companies would be required, and quotes from another leaked document that says the program allows NSA officers to send “content tasking instructions directly to equipment installed at company-controlled locations,” which could mean boxes installed at ISP switches.

Several sources note that former AT&T employee Mark Klein revealed in 2007 that he had come across documents that showed the telecom company installed equipment — using glass prisms as “splitters” — that allowed the NSA to make a copy of the data stream coming from the AT&T network and send it to data-storage centers operated by the security agency. This was alleged to be part of a larger program that stored telephone calls, emails and other internet activity for the government and had been underway for years.

Some network analysts speculate that the NSA may be making use of equipment installed at CDNs (content delivery networks), which handle much of the data traffic for companies like Google and Yahoo. Laws passed in the U.S. require equipment makers such as Cisco to build into their products a way for law enforcement officials to tap into the streams they carry, and the NSA could be searching those streams directly instead of copying or storing all the data itself (since the cost of the program is a relatively cheap-sounding $20 million, according to the Post leak).

In a statement about the leaks, the Office of the Director of National Intelligence said that it does its best to work “within the constraints of the law” to collect information related to national security, and that unauthorized leaks such as those to the Guardian and Post “threatens potentially long-lasting and irreversible harm to our ability to identify and respond to the many threats facing our nation.”

Trying to track down PRISM

A search for entities that might be involved in the NSA program turns up software from a relatively secretive startup called Palantir — which has been funded by the CIA through its investment arm — that happens to be named PRISM. According to descriptions of the software, it allows clients of Palantir to sift through massive amounts of data and find patterns quickly.

Others are skeptical, however, that the software described could be used to do what the NSA appears to be doing, and security-industry sources say the NSA usually builds its own products and doesn’t like to use those from third parties. On Friday afternoon, Palantir told The Verge: “Palantir’s Prism platform is completely unrelated to any US government program of the same name.”

Former Reuters social-media editor Matthew Keys said on Twitter that he had found several references to the PRISM program in classified job listings dating back to 2007:

Not wanting to be left out, the secretive activist group Anonymous released some classified documents that refer to Defense Department information technology — but they appear to be mostly jargon-filled descriptions of the department’s IT infrastructure, with little or no connection to PRISM or any NSA-related data collection practices.

The ripples spread outside the U.S.

As our man in Europe — David Meyer — noted in a couple of posts Friday morning, the repercussions from the PRISM and NSA revelations are being felt in Europe as well, with some critics calling for changes to the so-called “Safe Harbor” program, which allows data about EU citizens to be stored by non-EU companies. And the Guardian has reported that the U.K. government appears to have been getting information via the PRISM program, which was designed to focus on the communication activity of non-U.S. residents (since U.S. law still technically prevents the government from spying on its own citizens without a warrant).

Meanwhile, President Obama — whom many critics have accused of carrying on with surveillance programs started by his Republican predecessor, despite his disavowal of such methods while campaigning — said through a spokesman that he “welcomes the discussion” about privacy and security:

“The president welcomes the discussion of the trade-off between security and civil liberties. The close examination of some of these complicated issues could cause people to arrive at differing opinions… The president welcomes that debate.”

Late Friday, the Guardian posted another security-related scoop, publishing what it called a “secret presidential directive” that orders the U.S. government’s top national security and intelligence officials to draw up a list of potential overseas targets that the U.S. could hit with cyber-attacks. The story goes on to say this operation:

“can offer unique and unconventional capabilities to advance US national objectives around the world with little or no warning to the adversary or target and with potential effects ranging from subtle to severely damaging”

Google denial and Sir Tim Berners-Lee

The creator of the world wide web, Sir Tim Berners-Lee, posted a statement at the Web Foundation blog saying:

“Today’s revelations are deeply concerning. Unwarranted government surveillance is an intrusion on basic human rights that threatens the very foundations of a democratic society. I call on all Web users to demand better legal protection and due process safeguards for the privacy of their online communications, including their right to be informed when someone requests or stores their data.”

And Google co-founder Larry Page posted a response Friday afternoon to the accusations in the Guardian and Post stories, written with Chief Legal Officer David Drummond, saying the company does not provide the government with “back door” access to its servers, and had never heard of the PRISM program until Thursday:

“Press reports that suggest that Google is providing open-ended access to our users’ data are false, period… Any suggestion that Google is disclosing information about our users’ Internet activity on such a scale is completely false.”

Zuckerberg denial

Facebook co-founder and CEO Mark Zuckerberg posted a statement about PRISM on his Facebook page late Friday, saying he wanted to respond personally to the “outrageous press reports” about his company’s involvement in the surveillance scheme. In language very similar to the Google denial, Zuckerberg said the network has not been part of any program to give the U.S. government “direct access” to its servers.

“Facebook is not and has never been part of any program to give the US or any other government direct access to our servers. We have never received a blanket request or court order from any government agency asking for information or metadata in bulk, like the one Verizon reportedly received. And if we did, we would fight it aggressively. We hadn’t even heard of PRISM before yesterday.”

Does the NSA even need a back door?

Christopher Mims at the Atlantic business site Quartz quotes NSA veteran and whistle-blower William Binney — who was part of a group that asked the Defense Department to investigate the NSA in 2002 — saying the security agency could probably get its hands on about 80 percent of the web traffic that passes through the U.S. without even having direct access to the servers of companies like Google. That’s because the NSA has access to at least one of the largest communications hubs on the continent, as described by the Electronic Frontier Foundation.

The Wall Street Journal posted a story that quoted unnamed security experts who said the tech companies mentioned in the PRISM presentation could be telling the truth about not providing “direct access” to their servers, but still have their data collected by the NSA. The Journal said U.S. officials told the paper that the NSA “receives copies of the data through a system they set up with a court order.”

“One industry executive familiar with the handling of data requests from U.S. intelligence agencies said companies have set up ways to cope with the volume of data by automating parts of the process. This method would allow data to be funneled to intelligence agencies without the need for manual steps by company employees.”

At The Daily Beast, writer Megan McArdle looked at the issue of whether tech company denials should be believed or not, and quoted privacy expert Julian Sanchez from the Cato Institute saying there are a number of ways that the NSA could get the data it wants without requiring direct access, including the “secret room” with splitter equipment that Mark Klein described at AT&T (mentioned above):

“Most likely… is that they’ve got something akin to the “Secret Room” that Mark Klein disclosed in AT&T hubs where traffic is being cloned (the companies would need to provide the relevant SSL encryption keys) split off into NSA’s own machines. It would be literally true, in that case, that the NSA does not have direct access to Google’s servers.”

How PRISM might work in practice

Late Friday, the New York Times posted a story that said some tech companies resisted the NSA’s demands to provide easier ways to get access to user data — including Twitter — but that some consented, opened up discussions with the security agency about developing methods to share that data, and even “changed their computer systems to do so.”

“In at least two cases, at Google and Facebook, one of the plans discussed was to build separate, secure portals, like a digital version of the secure physical rooms that have long existed for classified information, in some instances on company servers. Through these online rooms, the government would request data, companies would deposit it and the government would retrieve it.”

In other words, “companies were essentially asked to erect a locked mailbox and give the government the key” and Facebook actually built such a system, the NYT story said. Declan McCullagh at CNET explained in a post that according to his sources, all that the PRISM process does is automate something that is required under FISA (the Foreign Intelligence Surveillance Act) — so court orders are given to the tech companies and they have simply made the process of handing over that information easier.

Marc Ambinder, a security expert who writes for The Week, also described his understanding of how PRISM functions — in a nutshell, PRISM is just a piece of software that allows the NSA to collect and interpret data that is handed over under FISA. The actual software itself isn’t classified, which is why mentions of it show up online and in job postings. In McCullagh’s piece, a former NSA lawyer says that the slide presentation the Washington Post published is “suffused with a kind of hype that makes it sound more like a marketing pitch than a briefing.”

Meanwhile, for those trying to keep track at home, the Electronic Frontier Foundation has put together a comprehensive timeline of events related to NSA surveillance activity over the past decade:

It was for your own good

First tech companies claimed they didn’t know anything about PRISM and weren’t supplying data (or at least not direct access), and now the story some sources close to those companies are telling is that they set up portals or some other method of complying with FISA requests in order to “protect the innocent,” according to a post at TechCrunch.

“The NSA may have wanted full firehoses of data from Google, Facebook and other tech giants, but the companies attempted to protect innocent users from monitoring via compliance systems that created segregated data before securely handing it over as required by law.”

The Guardian has responded to criticisms of its original description of PRISM and the whole notion of “direct access” — as well as the repeated denials from Google executives and others that this has been taking place — by posting another slide from the leaked NSA presentation. While some have speculated (as mentioned above) that PRISM could mean simply sucking data from ISP equipment, the NSA slide contrasts this method of getting data with PRISM’s, which it describes again as “collection directly from the servers” of the companies mentioned.

The Director of National Intelligence released another statement on Saturday, calling the disclosures by the Guardian and Washington Post about NSA data collection “reckless” and filled with “significant misimpressions.” So DNI James Clapper said he had declassified some details about the program, published in a fact sheet (PDF link). Among other things, it says:

“PRISM is not an undisclosed collection or data mining program. It is an internal government computer system used to facilitate the government’s statutorily authorized collection of foreign intelligence information from electronic communication service providers under court supervision, as authorized by Section 702 of the Foreign Intelligence Surveillance Act.”

The Washington Post published a follow-up story on Saturday that described the PRISM process in much the same way as earlier stories from the Guardian and the New York Times: as a system or software that allowed the NSA to process FISA requests for information more quickly — and the paper reiterated earlier statements that because the program was top secret, only a few individuals within those companies would even know about it, let alone be able to discuss it. According to the Post:

“Executives at some of the participating companies, who spoke on the condition of anonymity, acknowledged the system’s existence and said it was used to share information about foreign customers with the NSA and other parts of the nation’s intelligence community.”

Much of the criticism about the original Post story and the Guardian story has focused on the description of PRISM as allowing “direct access” to the servers of companies like Google, Facebook and Yahoo — something the leaders of those companies have strenuously denied providing. The most recent Post story suggests that at least some of the debate over this term is semantic, and that its sources say PRISM did allow the NSA to get data from those companies directly:

“Intelligence community sources said that this description, although inaccurate from a technical perspective, matches the experience of analysts at the NSA. From their workstations anywhere in the world, government employees cleared for PRISM access may ‘task’ the system and receive results from an Internet company without further interaction with the company’s staff.”

NSA whistle-blower reveals his identity

In another bombshell, the Guardian revealed the identity of the whistle-blower who sent them the leaked documents about PRISM and the NSA surveillance program: he is Edward Snowden, a 29-year-old former technical assistant at the Central Intelligence Agency, and he is now living in Hong Kong and expects he will “never see home again.” He said his family doesn’t know about his activities, and that he fully expects to be charged and potentially face jail time for his actions.

In an interview with the Guardian, Snowden says that he gradually became frustrated with what the NSA was doing and believed it was wrong — but originally held off on leaking anything because he thought Barack Obama would change those policies when he was elected president. But Snowden says the president continued with “the policies of his predecessor” and so he decided to come forward and let the American public know what was happening behind closed doors:

“I don’t want to live in a society that does these sort of things … I do not want to live in a world where everything I do and say is recorded. That is not something I am willing to support or live under.”

Snowden also said the documents he leaked clearly show that “the NSA routinely lies in response to Congressional inquiries about the scope of surveillance in America” and that the abilities that he had as a contractor with the CIA were beyond what most people can even imagine:

“You are not even aware of what is possible. The extent of their capabilities is horrifying. We can plant bugs in machines. Once you go on the network, I can identify your machine. You will never be safe whatever protections you put in place.”

The reaction

In a post written for The Atlantic magazine, James Fallows said that the most frightening and important part about PRISM and the rest of the NSA surveillance activity revealed by Snowden is that it is all legal under the Foreign Intelligence Surveillance Act and other legislation.

“That these programs are legal — unlike the Nixon “Plumbers” operation, unlike various CIA assassination programs, unlike other objects of whistle-blower revelations over the years — is the most important fact about them. They’re being carried out in “our” name, ours as Americans, even though most of us have had no idea of what they entailed.”

Fallows — and others such as Talking Points Memo founder Josh Marshall — raised some question marks about the wisdom of Snowden’s choice of Hong Kong, which is still part of China and therefore not particularly open to harboring whistle-blowers. However, according to some experts in the law, Hong Kong might be a good place to seek asylum because of a loophole that could allow Snowden to remain there indefinitely.

Icelandic MP Birgitta Jonsdottir, an early supporter of WikiLeaks and of freedom-of-information laws in general, told Forbes magazine that she plans to try and get her country to offer Snowden political asylum. But observers of the political scene in Iceland say this might be more difficult than it would have been in the past, since the new Conservative government is seen as more friendly to the Obama administration.

Daniel Ellsberg — the man who leaked the famous “Pentagon Papers” in 1971 and revealed that the government had been lying about the Vietnam War — said in a piece written for the Guardian that Snowden’s leaks give the United States a chance to “roll back what is tantamount to an executive coup against the U.S. constitution.” Ellsberg said that Snowden’s revelations were the most important leak in the history of the United States, including his own.

“Since 9/11, there has been, at first secretly but increasingly openly, a revocation of the bill of rights for which this country fought over 200 years ago. In particular, the fourth and fifth amendments of the US constitution, which safeguard citizens from unwarranted intrusion by the government into their private lives, have been virtually suspended.”

Meanwhile, David Kirkpatrick — author of the book “The Facebook Effect” — asked whether the secrecy and privacy invasions involved in the PRISM program might impair the growth of social networks and cloud services like Facebook.

“Do we really want to impair such powerful tools for spreading dialogue, political discourse, and U.S. values? Is it worthwhile to impair the extraordinary financial and commercial success of these great flagships for the American economy? Does Obama want Facebook et al just to be seen as tools of American power?”

Politico took a look at some of the things that we still don’t know about PRISM and the activity involved in the NSA’s surveillance program — including how much data the spy agency has been collecting from phone companies as well as tech companies like Google, whether this data collection has actually thwarted any specific terrorist attempts or not (something that is the subject of much debate) and how exactly the PRISM program works in practice.

Meanwhile, the Daily Beast has a piece that looks at the group within the U.S. intelligence apparatus that hunt down leakers like Snowden, a kind of internal police force called the Associate Directorate for Security and Counterintelligence — or the Q Group for short. And Salon magazine has a feature and interview with Laura Poitras, the documentary film-maker who was contacted by Snowden and later helped both the Post and the Guardian write their stories about the leak.

Got anything I am missing? Let me know at mathew@gigaom.com

Post and thumbnail images courtesy of Shutterstock / Lightspring and the Washington Post

But how exactly ebook publishers reach that audience and how the industry tracks who’s interested in reading what is less clear. A series of ebook publishers who spoke at our PaidContent Live conference in New York on Wednesday talked about the critical importance of gathering data on readership and consumption, and using it to transform the industry:

“The old eveolution of the book publishers used to be very allergic to data. And what you just heard is a very different approach from that. For us it’s about metadata and surfacing. And then rinse and repeat,” said Dominique Raccah, the publisher and CEO of Sourcebooks. “Metadata is a new term in our industry, but it really is the key.”

Raccah pointed out that unless publishers know who is reading the content, it’s hard to craft specific marketing messages or know what people respond to:

“It’s really important to know that book publishers know a lot about what touches readers,” she said. “So it’s important to help craft those messages in interesting ways.”

Rachel Chou, the CMO for Open Road Integrated Media, said they’ve seen a lot of success working with Twitter, as well as sponsored stories in Facebook, to drive traffic and understand where customers are coming from.

“Then after a while, you start understanding what the best partners are,” she said.

Evan Ratliff, founder and CEO of Atavist, said they have a small team but because they’re especially focused on finding customers by building up the Atavist brand, understanding data on the company’s products is important.

“We’re also on a very small level, we’re experiment with different ways of reaching people and social media,” he said.

Check out the rest of our paidContent Live 2013 coverage here, and a video embed of the session follows below:

A transcription of the video follows on the next page

Go to page 2 (of 2) on paidContent .

As AdAge reports, Facebook may obtain the phone numbers and email addresses that customers supply to drug stores in return for discounts and loyalty points. The company will then match the numbers and emails with user Facebook profiles in order to create new marketing opportunities. This means, for instance, that I might buy ex-lax at the store and then see ads for All-Bran in my Facebook feed.

This merging of online and offline data creates powerful marketing opportunities and, as ad industry people like to point out, will let people see ads they’re theoretically interested in seeing. In this perfect world of data-based marketing, no company will waste their money showing baby food or make-up ads to the likes of me — and I will be happier because I don’t have to see those ads.

The flip side, of course, is the creepy factor. Facebook and the data companies (in this case Epsilon, Acxiom and Datalogix) do take care to protect so-called “PPI” (protected personal information) by scrubbing out names and just using other techniques to make the data anonymous. But one still shudders about the unencrypted information escaping into the wild through carelessness or hacking. In my case, it just increases my resolve to provide retailers with fictitious names and numbers.

For now, we may still be a ways of from the onslaught of fiber ads in our Facebook. According to AdAge, the new “custom audience” options are still winding their way through nervous corporate legal departments.

(Image by gcpics via Shutterstock)

In his post, Battelle describes how his thinking has been influenced by some of the recent offerings Twitter has come up with around broadcast events — such as the “Oscars Index,” (a partnership with Topsy) which tracks the sentiment around the Oscar-nominated movies and personalities leading up to the Academy Awards by analyzing tweets about them. Although he doesn’t mention it, Twitter also recently announced an even more ambitious effort to create a verified “Twitter TV Rating” for TV shows as part of a partnership with media-metrics company Nielsen.

Having lots of data is great — relevance is better

The Federated Media founder points out quite rightly that one of the things that makes Twitter a potential gold mine — both for media companies and for advertisers — is the number of signals that the network can accumulate about users, their behavior and their interests. More than half a billion tweets a day is a lot of data, and somewhere in the midst of that are the keys to delivering better content, and better advertising (which is increasingly becoming the same thing, a topic we’ll be discussing at paidContent Live in New York on April 17). As Battelle puts it in his post:

“Twitter presents a massive search problem/opportunity. For example, Twitter’s gotten better and better at what’s called “entity extraction” – identifying a person, place, or thing, then associating behaviors and attributes around that thing… Real time entity extraction crossed with signals like those described above is the Holy Grail.”

This is fundamentally the same goal that both Google and Facebook are focused on as well: how do you show users only things that are relevant to them, and hide those that aren’t — in real time? Facebook has gotten criticism for the way it tweaks the news feed based on its algorithms, but the reality is that most users don’t want to see everything that streams through their networks. And Google started its Google+ social network, and built it into everything it does, in part because it needs more data signals about its users.

The problem for all of these companies is that doing this is really, really hard — every user’s stream consists of billions of data signals, and deciphering which are meaningful and which aren’t is a complicated business. To get a sense of how difficult it is, all you have to do is look at Twitter’s “Discover” tab, or the “Trends” listings, or look at the promoted tweets and promoted trends that show up in your stream (of course, Facebook is almost as bad with its sponsored pages, and it has orders of magnitude more data).

Relevance is the key to digital media

As Battelle notes, Twitter has gotten better at discovery, and the revamped version of its Discover tab is better than it used to be — and so are the suggestions Twitter sends to users for other accounts they should follow. But the Discover tab in particular is still light-years away from where it needs to be in order for it to be a compelling content-discovery mechanism for users, and the curated email newsletter Twitter sends out is even worse: it shows me Canadian news because I live in Toronto, even though it knows (or should) that I rarely ever tweet about that topic.

Simply put, relevance is the key attribute for any digital-media entity in the 21st century. Newspapers and other traditional sources of content are terrible at suggesting or curating relevant content for individual readers, but no one really expects them to be any good at it — they have zero experience in doing that. Twitter, however, has enough data that it arguably should be much better than it is. And it needs to get there quickly, before Google or Facebook (or god forbid, even Yahoo) get much better.

Coming up with visualizations about the Oscars or highlighting tweets about NASCAR may be useful for reeling in big media brands, but users are going to need a little more than that before they trust Twitter to curate content for them.

Post and thumbnail images courtesy of Shutterstock / noporn and John Battelle

The site, which lets people share their favorite music track-by-track, has proven an underground hit online less than a year after launching publicly: more than 100,000 users have signed up, sharing over 900,000 songs. But the London-based service was started as a pet project inside music data company The Echo Nest — and it’s now exploring what happens next.

“Up until this point we’ve been incubated by The EchoNest, but now we’re looking at options for spinning out in our own right,” creator Matthew Ogle told me.

It’s been an unusual course for This Is My Jam so far — in fact, Ogle says that the site “wasn’t supposed to happen for a whole bunch of reasons”. Chief among them? The fact he’d decided to move out of the online music industry after leaving his role as head of web product at Last.fm back in 2010.

“Despite swearing off online music forever, in less than a year I’d been convinced by the awesome folks at The Echo Nest that we could do some cool stuff together,” he says. “They didn’t hire me to make Jam, they hired me in a dual role to be their man in Europe — evangelizing at hackdays, talking to developers — and also to be a kind of internal product skunkworks, prototyping stuff based on new APIs and using that to spark new direction that The Echo Nest could be going with their data.”

But when it turned out that Jam, an idea he’d been throwing for a while, had more going for it than the other skunkworks proposals, Ogle’s focus switched and the parent company funded development. Jam now has four full-time staff, as Ogle brought on former Last.fm refugee Hannah Donovan, and engineers Ralph Cowling and Andreas Jansson.

The failure of frictionless

The site is one of my favorite services to have launched in the last year or two, and it’s one I’ve found myself going back to more often than I expected. Posting your favorite tracks and browsing the tracks of others turns the site into a curious mixture of status update and radio station. It’s a great tool for telling people about the music you like — lots of people use it to showcase their mood, for example. But it’s also a treasure trove of music, allowing you to dig around in the tastes of others. Because it’s almost exclusively focused on what people are listening to now, it has a real-time quality to it… yet the conscious decision that goes into making your choice means that the end result is more personal than Pandora but way more curated than Spotify’s frictionless sharing.

In fact, says Ogle, the failure of frictionless sharing to provide anything more than a fleeting dip into a raging river of data left an interesting gap for Jam to fill.

“I couldn’t quite believe that in 2011 that social song sharing wasn’t just a solved problem,” he explains. “One thing that we talked about at last.fm a lot — you know that amazing moment in real life when someone grabs you and say ‘you have to hear this song, check it out’ and they put the headphones in your ears or put a record on. Anyone who cares about music even a little bit has had that moment.”

“We’re in this golden age of social media, we’re always connected to all of our friends at all times, and there’s big data around music — Spotify and Facebook have basically taken scrobbling to the mainstream — [so] there should be more ways than ever for me to go ‘I want to hear some new tunes, what are my friends listening to?’ and get good stuff… not just whatever they happened to accidentally listen to on Spotify. Conversely there was no way for me to share a song that people would still see five hours later. Everything was being forced in real-time.”

Curation/exploration

Donovan, who heads up the site’s design, points out that isn’t just vanity that drives — the performative aspect of social media where you are showing off your taste to others — but a sort of shared curation where users collaborate to uncover interesting tracks, point to classics or dig up forgotten material.

“This was actually really cool when we discovered this happening, because there’s no other music service on the internet where it’s OK to have old stuff mixed in too,” she says. “Either everything is organized around the music data thing — singles live inside albums that live inside artists — or it’s promotional in some respect, in which case they’re always pushing the latest album or the latest single that just came out.”

“On top of that I think there’s something around our culture today that’s driven by newness, and how things on the internet always have to be the latest and the newest. I heard somebody say when you’re curating something, you don’t necessarily want just the latest or the newest, you want to dig up old things that were really great and put them back in context alongside newer things, or mixed in with other stuff. That’s the job of the curator and that’s what makes it really enjoyable for the user or the observer.”

“We thought about curation a lot when we were starting Jam. The overall effect is that our users became the curators of this music, and we wound up with this lovely space where you could get Prince and Fleetwood Mac right next to the latest trendy pop band.”

That is, in turn, helping the product develop further forward. Coming very soon are some new additions to the site’s Explore pages, which will offer up some new ways for users to browse the site. Explore categories will include “Breaking” (songs that have been jammed for the first time recently, and subsequently shared), “Rare” (songs that have been jammed only once), and another one that lists just the first jams of newly-registered users. Although it could act as a way for users to introduce themselves to the service, the first jams could also become a sort of Greatest Hits package.

“A lot of people sign up and start with their favorite song of all time,” says Ogle.

D’oh! This is precisely the sort of thing that retailers who have just laid out a million bucks for Facebook or Twitter ads don’t want to hear. But here’s the study, showing that Facebook ads only led to 0.68 percent of shoppers making an online purchase on Black Friday (and Twitter ads produced no sales at all!) :

Fortunately for retailers (and Facebook), there is more to the story. According to a media intelligence service, Aggregated Knowledge (AK), Facebook ads have a big effect on online purchase decisions — leading to a 51 percent increase in conversions (the term for when people buy or do something in response to an ad). AK also says Facebook ads are a way for brands to reach big target audiences they wouldn’t reach elsewhere online, and that “conversion rates” are 72 percent cheaper than for other online channels.

AK knows this because it works with large social media advertisers to collect data about Facebook user behavior. This data collection is possible because Facebook recently provided e-tailers with tools that will let them track users to see who responded to an ad. (Yes, there’s a creepy factor here though major sites like Amazon and Google have long used similar tools).

AK’s numbers are good news for Facebook but the social network will still have to persuade advertisers and the media to accept them. The challenge lies in getting ad types to accept the notion of “multitouch attribution” rather than “last touch.” In plain English, this means that conversions should be counted even if the occur hours or days after you leave Facebook — for instance, you see a bath salts ad on Facebook, surf dozens of other websites, and then buy those bath salts the next day. (TechCrunch’s Josh Constine has made the same point well here).

Seen from this lens, Facebook’s “party” problem isn’t so bad. After all, we discover lots of new desires at a party — that girl’s dress, that guy’s cowboy hat, those delicious cocktail wienies — that we go out and buy for ourselves later on.

(Image by Ariwasabi via Shutterstock)

This, anyways, is the promise of BlueKai, a company that helps publishers and brands collect information and marketing insights about their audiences. BlueKai’s primary customers have always been advertisers but now it’s turning more attention to publishers, offering tools that help them create reader profiles and develop data-driven ad sale strategies.

“The world before for publishers was ‘I have these great audiences, I have these places’ please buy them’,” said BlueKai CEO Omar Tawakol in a recent phone interview. Now, though, ad buyers want publishers to supply information that lets them reach very specific groups of people — soccer dads, not soccer moms for instance. Publishers can offer this information by mining their audience data (including registration info and log-in location) and combining it with data BlueKai has about those users’ web-surfing activities. Such tools can also help publishers track when the same customer is coming to the site through a desktop, tablet or cell phone device.

The data tools are fine in theory but are they a practical investment for publishers who could spend the money on writers or developers instead? According to one executive, some publishers might not have a choice.

Vikram Somaya, a former VP of Operations and Audience at Reuters, says that audience data has become “table stakes” for premium publishers if they want to be considered for major ad purchases by national brands. In practice, this means that a company like Ford might only consider buying online ads for a given campaign if the publisher can provide a target audience of car buyers.

“Anyone who hasn’t picked an [audience data] product, is behind… it’s not an option to ignore anymore,” said Somaya in a phone interview. He acknowledged though that, for now, BlueKai-type tools may only be practical for big publishers that are in Comscore’s list of top 500 web sites. (Somaya, who is now a VP at the Weather Company, used to work at BlueKai but said his opinion is impartial).

A publisher’s size, however, may not be the only consideration about whether it should invest in audience data. The nature of its readers is a consideration too. The more general a publisher’s audience, the greater its need for data. As the FT reported last week (reg. req’d), Time Inc has just launched a major new ad strategy based on mining customer information so as to offer better-placed ads.

On the other hand, audience data is less critical for more specialized publications. As an executive for a major New York based publication, who did not want to be named, explained by email:

While we are a BlueKai customer, our experience is that most advertisers do not require audience segmentation data as part of their buy. It certainly sometimes comes up, but it’s the relatively rare exception, not the rule. My guess is that it’s probably more important for publishers with a more mass audience. Our audience is relatively targeted (and valuable) already and so segmentation isn’t needed to get the audience advertisers are seeking.

What this means in practice is that niche publishers can likely forgo fancy data tools for now.

(Image by Dusit via Shutterstock)

As a bill attempting to lower the royalty rates paid to artists by streaming music services such as Pandora works its way through Congress, Keating took to her Tumblr blog last week and offered a solution that both sides should listen to, but won’t. You might have read about her stance in Billboard or ITworld already, or perhaps on Slashdot. If you haven’t, here it is in a nutshell, from Keating’s blog: “The law only demands I be paid in money, which at this point in my career is not as valuable as information. I’d rather be paid in data.”

Leaving aside the entire issue about royalties and copyright (and privacy policies), her statement is still powerful. Keating understands that in order to prosper in a world of digital music — just like in the world of e-commerce, digital publishing, you name it — information is power. The names, email address and perhaps mobile numbers of individuals listening to her music are nice, clean data that Keating could use with little to no analytic effort by reaching out to fans when a new tour is coming to town or a new album drops.

Actually, Keating noted in a subsequent blog post on Tuesday that even less-personal data can have a material impact on a performer’s bottom line. Using postal code data provide to her from iTunes sales, she’s able to plan tours more efficiently because she knows, or can make a safe assumption, that she has paying fans in certain cities.

Touring and merchandise sales remain most artists’ primary means of income, and the current royalty rate of $.0011 per play doesn’t add up fast (at least according to Keating’s math), so it’s easy to see why she — and probably many other up-and-coming or niche performers — would rather have the data that properties like Pandora almost certainly have.

And whether Keating knows it or not, the idea of using data as a substitute for money extends beyond web radio stations and musicians arguing about royalties. A couple weeks ago, I highlighted a handful of attempts to convince consumers to hand over, in exchange for cash rewards or product discounts, valuable data that advertisers can’t collect by tracking their online activity. This is data such as recent and future purchases, personal interests, your web-surfing habits and where you shop in the physical world.

Just like Keating is willing to forgo one-tenth of one cent per play (real money, even if not a lot) in exchange for data, these brands are willing to trade cash or something like it for data they don’t have to run through a Hadoop cluster and seven segmentation algorithms before they can tie it to a real person. They know they have to give a little bit in order to improve upon the status quo that’s good, but not nearly good enough for their purposes.

Previously, the notion that data is the currency of the web meant users gave away their behavior data to web sites in exchange for free services. Slowly but surely, however, that notion seems to be evolving. Maybe Zoë Keating wants data in lieu of royalties for the privilege of streaming her music, and maybe a web site wants my offline location data enough to give me a gift card worth enough that I’d hand it over. Either way, it’s all about the realization that some data is worth its weight — and then some — in cold, hard cash.

Feature image courtesy of Flickr user eschipul.

These dreams may be difficult to see when you look at the cramped office Cross and his five co-workers inhabit in San Francisco (a converted apartment that had a mattress on the floor the last time I visited) but they come to life when you talk to him. Of course, that passion also means it can be hard to get a word in edgewise sometimes, as the Prismatic founder holds forth on his vision for the future, or how current recommendation services are failing users, or the state of affairs in the Twitterverse. Like a lot of startup founders, Cross can be a bit of a whirling dervish of ideas — but listening to him is almost always worth it.

Prismatic is interesting in part because it isn’t run by media experts or anyone with journalism training; in fact, Cross has no media background of any kind. His specialty is data science and machine learning, and he says he chose to focus initially on news recommendation because it is an obvious problem — the deluge of information that we are all submerged in requires smart filters — and solving it will help lead the way to other similar problems. And one of the reasons why media players of all kinds would be wise to pay attention is that this data-powered filtering ability could be the key to whatever success the media industry has in the future, as traditional gatekeepers are replaced by crowd-powered and algorithm-driven sources.

“Brad is probably one of the leaders right now in using big data and machine learning to provide consumer services,” says Jason Freedman, his former co-founder at Flightcaster, the startup that the two worked on for several years before selling it last year. Using that knowledge of data science and machine learning, Cross says that he hopes to eventually learn so much about his users via Prismatic that he can start making smart recommendations about almost everything, from movies or books to varieties of scotch or even people:

“The idea is that we become this trusted agent that you rely on to show you things, and over time we can really start to learn a lot about you. We do care a lot about [news recommendation], but we’ve also thought through how it’s a stepping stone to something much bigger. And a lot of what we do in the background, and how we slice and dice data and so on… is relevant across a really wide range of problems.”

Lessons learned from failure

Cross started out in computer science at Virginia Polytechnic Institute, but later switched to finance after he grew bored with his computer courses. Even then, the now-32-year-old founder was interested in large-scale machine learning projects and what they were capable of — and that desire would cost him dearly, at least in the short term. After university, he and a friend started their own investment fund and did fairly well (a 30-percent return in their first year, Cross says) and he parlayed that into a job with a small hedge fund called O’Higgins Asset Management in Miami.

Given his own fund to run, he made even better returns and generated a substantial income for himself, even though he was just 23 at the time. But Cross says he was obsessed with starting his own trading firm, based on his own software. Unfortunately, he admits now that he didn’t really know what he was doing, and the job he took on — building his own data-driven trading algorithms — was far beyond what he was capable of at the time, and he says he learned a painful lesson about entrepreneurialism:

“I basically failed at it, and lost all the money I had made through actual trading, which is ironic. I had no idea what I was doing. It was a pretty hard-core problem, one of the more complicated things in the world to do. It was a very interesting experience that lasted for about 12 months. I knew it wasn’t going to work but I didn’t want to let it go. And eventually I realized I just wasn’t very good at building software.”

Humbled by that experience, the Prismatic founder says he realized that he needed a lot more than just some stock-trading skills and a computer-science background in order to do what he had in mind, so he decided to try and get a job that would allow him to broaden his skill set. Freedman says this is a classic Cross-type move: when he realizes he needs to learn certain things in order to advance, the Prismatic founder figures out a way to get that knowledge. “Brad has this thing where he realizes something is holding him back and he just goes into a 100-percent all-out focused sprint to get better at it,” says his Flightcaster co-founder.

Not long ago, Freedman says he told Cross that if he wanted to hire engineers, he would have to become better known in the hacker community — but his blog was old and stale, and he never spent any time on Hacker News or other popular sites. “So he just went all out, and built a new profile, and wrote all these really smart blog posts, got a new blog design and in seven days had learned everything you need to know about using social media,” says Freedman, who is now running a startup called 42floors.

The building blocks of a smart recommendation engine

After blowing most of his savings on his failed attempt to build a trading system, Cross wound up getting a job with ThoughtWorks, a consulting company that he contacted when his venture was falling apart. The company’s specialty was getting parachuted in to fix things at other companies when they were coming off the rails, and one of the companies Cross spent a lot of time working with was Google. The team spent much of its time on the operating system that powered Google’s storage clusters, but he also worked on consumer products such as Books, Cross says (and met his wife, a Slovak software engineer).

“I didn’t especially enjoy it, but it taught me a lot — it taught me a lot about what not to do. You get an understanding of really gnarly technical issues [and] at the same time, you develop this sixth sense where you can see early on where something bad will lead you. That was a huge experience for me.”

After the Google stint came Flightcaster, a Y Combinator-funded startup. Freedman and his partner Evan Konwiser needed a more technical co-founder in order to get their funding from the incubator (“someone who could actually build what they were talking about,” Cross says) and so they brought Cross in to run the technical side. He designed and built the systems that sucked in terabytes worth of data from airlines, weather services and other sources and made recommendations about routes or flights. “Flightcaster was a very early permutation of this kind of thing [social recommendations], which is the whole reason why I joined,” Cross says.

Once Flightcaster was sold, Cross finally had enough time, knowledge and resources to start his own venture, and Prismatic was born in early 2011 and initially known as Woven. The idea was to use the machine-learning techniques that Flightcaster was based on to filter through and understand social data, and then use that to make recommendations about news and other content. In other words, to create a kind of customized newspaper for users based on their activity on social networks like Twitter and Facebook.

Can a mad scientist reinvent the way media works?

Facebook may have close to a billion users and vast storehouses of knowledge — as does Google — and Twitter may be focusing more on the kind of smart curation that Prismatic is talking about as the service evolves, but Cross says he is convinced that his small company (which was seed funded with $1.2 million from investors including Battery Ventures and Javelin Venture Partners) has a better chance of solving this problem than any of its much larger competitors — even though that might seem to some like an excess of hubris:

“Google won’t get this right, Twitter won’t get this right, Facebook won’t get this quite right, Amazon won’t even get this right — the company that gets it right needs to have it in its DNA. We think this is a Trojan horse into a much bigger thing… in five years time or 10 years time, AI will be all over our daily lives, everything we interact with will be intelligent, and the interfaces to it will be completely different. Backtracking from that very distant kind of vision led us to start in this place.”

There’s no question that the odds of Prismatic succeeding are almost astronomical — not only are there other older, better-funded players going after a similar goal, including News360 and Zite and even Flipboard, but Twitter and Google and Facebook are not likely to let an upstart take away what could be the future of their business.

If sheer devotion to wrestling with big problems is the secret, then Cross has a good head-start: Freedman says he remembers how Cross was always taking the engineers away for 18-hour hackathons to solve pressing issues with the code base. “One day I looked around the office and it was just my co-founder and I,” he says. “Everyone else was at Brad’s house, in his living room, working these crazy hours, but also just having fun. He loves a big problem.” Cross is definitely still a “mad-scientist type,” Freedman says, “but he’s also a lot more than that.”

When we spoke recently, Cross was busy trying to manage a hundred different aspects of being a small six-person startup: a deluge of interest based on the launch of Prismatic’s new iPhone app (an iPad app is coming soon), calls from potential acquirers, and the need to raise financing to fund the growth that he expects — or wants — to see in the future. Is it worth it, I asked? “It’s not worth it unless you have something really important that you’re trying to do,” he said. “And I think we do.”

The study, which MIT’s Technology Review highlighted in a recent post by the Physics arXiv blog, was done by a pair of researchers in Virginia, Hany SalahEldeen and Michael Nelson. They took a number of recent major news events over the past three years — including the Egyptian revolution, Michael Jackson’s death, the elections and related protests in Iran and the outbreak of the H1N1 virus — and tracked the links that were shared on Twitter about each. Following the links to their ultimate source showed that an alarming number of them had simply vanished.

After two and a half years, 30 percent had disappeared

In fact, the researchers said that within a year of these events, an average of 11 percent of the material that was linked to had disappeared completely (and another 20 percent had been archived), and after two-and-a-half years, close to 30 percent had been lost altogether and 41 percent had been archived. Based on this rate of information decay, the authors predicted that more than 10 percent of the information about a major news event will likely be gone within a year, and the remainder will continue to vanish at the rate of .02 percent per day.

It’s not clear from the research why the missing information disappeared, but it’s likely that in many cases blogs have simply shut down or moved, or news stories have been archived by providers who charge for access (something that many newspapers and other media outlets do to generate revenue). But as the Technology Review post points out, this kind of information can be extremely valuable in tracking how historical events developed, such as the Arab Spring revolutions — which the researchers note was the original impetus for their study, since they were trying to collect as much data as possible for the one-year anniversary of the uprisings.

Other scientists, and particularly librarians, have also raised red flags in the past about the rate at which digital data is disappearing. The National Library of Scotland, for example, recently warned that key elements of Scottish digital life were vanishing into a “black hole,” and asked the government to fast-track legislation that would allow libraries to store copies of websites. Web pioneer Brewster Kahle is probably the best known digital archivist as a result of his Internet Archive project, which keeps copies of websites dating back to the early days of the web (Kahle also has a related project called the Open Library).

Getting access to social data is not easy

Although the Virginia researchers didn’t deal with it as part of their study, a related problem is that much of the content that gets distributed through Twitter — not just websites that are linked to in Twitter posts, but the content of the posts themselves — is difficult and/or expensive to get to. Twitter’s search is notoriously unreliable for anything older than about a week, and access to the complete archive of your tweets is only provided to those who can make a special case for needing it, such as Andy Carvin of National Public Radio (who is writing a book about the way he chronicled the Arab Spring revolutions).

As my colleague Eliza Kern noted in a recent post, an external service called Gnip now has access to the full archive of Twitter content, which it will provide to companies for a fee. And Twitter-based search and discovery engine Topsy also has an archive of most of the full “firehose” of tweets — although it focuses primarily on content that is retweeted a lot — and provides that to companies for analytical purposes. But neither can be linked to easily for research or historical archiving purposes. The Library of Congress also has an archive of Twitter’s content, but it isn’t easily accessible and it’s not clear whether new content is being added or not.

Twitter has talked about providing a service that would let users download their tweets at some point, but it hasn’t said when such a thing would be available — and even if users did create their own archive in this way (or by using tools like Thinkup from former Lifehacker editor Gina Trapani) it would be difficult to link those in a way that would provide the kind of connected historical information the Virginia study is describing. And it’s not just Twitter: there is no easy way to get access to an archive of Facebook posts either, although users in Europe can request access to their own archive as a result of a legal ruling there.

For better or worse, much of the content flowing around us seems to be just as insubstantial as the clouds that it is hosted in, and the existing tools we have for trying to capture and make sense of it simply aren’t up to the task. The long-term social effects of this digital amnesia remain to be seen.

Post and thumbnail images courtesy of Shutterstock user Ribah and Flickr user See-ming Lee