MLB is also planning online ad exchanges in which brands can bid in real time to show ads to a specialty audience — say affluent female car buyers — on sites they own such as the New York Yankees team page. According to a Thursday press release:

“Advertisers now can identify affluent audiences based on a range of demographic, behavioral and purchasing attributes and target them across all MLB properties.” The ad initiative reflects MLB’s role as the most tech-savvy sports league; to learn more, come see MLB Advanced Media CEO Bob Bowman join us at paidContent Live on April 17.

The release was put out by BlueKai, a company that collects and analyzes data from consumers as they move across different websites. In a phone interview, BlueKai Director of Business Development, Gina Kim, said the company is providing MLB with a data platform but that the league will not use it sell customer information to other publishers.

MLB’s plans to create more granular advertising segments also reflect a broader trend across major websites. Facebook, for instance, announced this week that it’s now using third party data companies to offer super-specific audience segments like “children’s cereals” or “full-size sedan buyers.”

The BlueKai executive said the data deal for now just covers desktop browsers and not mobile devices.

If you’re not familiar, Upworthy adds splashy headlines to photos and videos it culls from across the internet, and encourages viewers to share them on Facebook and other social media sites. This is akin to what sites like BuzzFeed do but with two major differences: Upworthy doesn’t have advertising and it focuses exclusively on political and social issues like gender equality and climate change.

So far, the site has made a splash with fare like “If this video makes you uncomfortable, then you make me uncomfortable” (advocating for gay marriage) and “Bully Calls News Anchor Fat, News Anchor Destroys Him On Live TV.” Upworthy also stands out for its editorial process: curators prepare 25 versions of each headline and engages in extensive A/B testing to find out which version is most likely to go viral.

“When we look at the media landscape, we see there being more of a demand problem than a supply problem – how do you get people to care about important stuff amidst the avalanche of content we all face each day?” said co-founder Peter Koechley.

So far, Upworthy is off to a roaring start and not just thanks to its millions of visitors. The press has praised Upworthy for using viral tricks to promote content unrelated to cats, while high-profile media figures like BuzzFeed co-founder John Johnson and Facebook co-founder Chris Hughes have put their own money into it. The site has also received $4 million from venture capital firm NEA.

All of this has made Upworthy a darling of the startup scene. But what is the company’s business model? As noted, Upworthy has no advertising income, nor does it plan to have any. Meanwhile, the company is in the midst of a mini-hiring spree, while also maintaining a high-gloss website and social media operation.

Upworthy says it earns money by connecting “readers with non-profits and other organizations who are looking to grow their memberships via the sign-up boxes” on its site. In other words, the company is collecting email addresses and social media profiles for “lead generation.”

The company adds it will not work with just any organization — only those that “create positive social change.” In response to an email query, Upworthy co-founder Peter Koechley declined to provide financial figures but did say the site has been taking in revenue since its third month of operation.

It seems far-fetched, however, to build a major media venture on the backs of the Sierra Club, the American Worker or other social-change groups. Unless, that is, Upworthy’s primary goal is instead to build a political operation aimed at gathering voter data and boost the Democratic party in upcoming elections.

Recall how the Obama administration won the 2012 race by using big data to identify and energize individual voters. The Democratic Party’s campaign, which relied heavily on Facebook connections and custom email messages, ran circles around Mitt Romney’s TV-based campaign. Now, with the help of Upworthy, the Democrats could be in a position to do it all over again — the site could not only help identify passionate supporters of liberal issues, but also be a laboratory to experiment with headlines and marketing messages like the ones used in an election.

Some members of the Upworthy team certainly have the pedigree for it. Koechley’s co-founder is Eli Pariser, the former head of Moveon.org, a liberal activist group closely tied to Democratic presidential campaigns. Meanwhile, Upworthy staff also include Zane Shelby and Ryan Resell, who worked on analytics and tech for the Obama campaign. And, according to Wired, Koechley is closely connected to Obama’s chief digital strategist, who gained fame for focus-tested emails like “I will be outspent” and “Do this for Michelle.”

Koechley told me: ”We don’t view ourselves as a political organization, although some of us do have backgrounds in politics,” he said. “Some of the most popular stuff on Upworthy is about the wonders of science, building women’s self esteem, or feel-good stories about overcoming adversity.” He also pointed to the site’s own description of itself as a “mission-driven media company.”

There is no reason to doubt Koechley and Upworthy’s sincerity about using viral media to advance social change. And it’s hard not to support much of what they’re doing; I don’t like homophobia or bullying either. But it’s also pretty easy to look at the company and see the seeds of something far more potent than just another viral media site.

Correction: This story was updated at 12:08pm to say that BuzzFeed co-founder John Johnson is an investor in Upworthy; an earlier version listed BuzzFeed co-founder Jonah Peretti.

(Image by SoulCurry via Shutterstock)

Gravity’s recommendation system is based on its interest graph technology, which we detailed last year. Here’s how I described it then:

[T]he gist is that humans first serve as guides for machine-learning algorithms by determining connections between terms within large data sets, then the algorithms take over to complete the job faster than humans ever could. When they’re done, the humans step in one more time to kill any bad connections between terms. The result is a system that can determine with high accuracy that a person tweeting about Vanessa Laine (Los Angeles Laker Kobe Bryant’s ex-wife), for example, is probably more interested in basketball than about Laine’s date of birth or other accurate but irrelevant information.

As new content streams into Gravity’s system, it’s analyzed and categorized in real time, then presented to users accordingly based on their interests and behavioral history.

How Gravity’s platform works

Graph processing and graph databases — which store and analyze data based on their relationship to one another — are critical to our onlines lives, powering everything from online recommendations to social search to knowledge discovery. Graph technologies are also the focal point of some impressive life sciences work from companies such as Syapse and Ayasdi, which will be presenting at Structure: Data in New York next month.

But publishers struggling to stand out on a noisy web might have the most to gain from graphs and personalization, generally. At our PaidContent Live conference (April 17 in New York), executives from Prismatic, Zite and Bluefin Labs will take the stage to talk about the importance of personalization for helping consumers filter through the deluge of content online so they can find what they really want. It’s arguable that the trick to keeping readers happy is knowing what they want to read — possibly better than they do themselves.

According to Gravity, its platform currently “delivers more than 25 million personalized content recommendations per day to more than 200 million users. Beta partners have reported click through rates two to three times above previous levels, return visitation increases of 300 percent and session length increases up to 40 percent.”

The experiment’s techniques actually point to a future where researchers are spared the grunt work of poring through thousands of pages of news or watching hundreds of hours of programming, and can actually focus their energy of explaining. As the researchers note in their paper, the real ramifications of this research lie more in what it accomplished than in what it found.

Namely, they demonstrated that with new big data techniques such as machine learning and natural-language processing, it’s possible to accurately analyze millions of pieces of content spanning almost a year without requiring humans to read and score it all. Choosing hypotheses with predictable results meant it was easier to verify their accuracy.

Here’s how how they explain the promise of their work and some potential use cases, the latter of which they go into far more detail about in the paper:

“[I]t allows researchers to focus their attention on a scale far beyond the sample sizes of traditional forms of content analysis. Rather than spending precious labour on the coding phase of raw data, analysts could focus on designing experiments and comparisons to test their hypotheses, leaving to computers the task of finding all articles of a given topic, measuring various features of their content such as their readability, use of certain forms of language, sources etc. (just a few of the tasks that can now be automated).

… Our approach — apart from freeing scholars from more mundane tasks — allows researchers to turn their attention to higher level properties of global news content, and to begin to explore the features of what has become a vast, multi-dimensional communications system.”

Put more simply: This research underscores the common big data maxim that knowing the right questions to ask is now the biggest challenge in gleaning insights from data. It’s increasingly easy to get data, analyze it and visualize it, so humans really just need to hypothesize and be able to explain the results. (This also seems like a good place to plug ScraperWiki as a great source for gathering potential research data from websites.)

Creating the workflows for gathering and analyzing the data as the authors suggest still isn’t child’s play (it might take some assistance from the computer science department), but it’s a lot better than the alternative.

Feature image courtesy of Shutterstock user Ruggiero Scardigno.

As a bill attempting to lower the royalty rates paid to artists by streaming music services such as Pandora works its way through Congress, Keating took to her Tumblr blog last week and offered a solution that both sides should listen to, but won’t. You might have read about her stance in Billboard or ITworld already, or perhaps on Slashdot. If you haven’t, here it is in a nutshell, from Keating’s blog: “The law only demands I be paid in money, which at this point in my career is not as valuable as information. I’d rather be paid in data.”

Leaving aside the entire issue about royalties and copyright (and privacy policies), her statement is still powerful. Keating understands that in order to prosper in a world of digital music — just like in the world of e-commerce, digital publishing, you name it — information is power. The names, email address and perhaps mobile numbers of individuals listening to her music are nice, clean data that Keating could use with little to no analytic effort by reaching out to fans when a new tour is coming to town or a new album drops.

Actually, Keating noted in a subsequent blog post on Tuesday that even less-personal data can have a material impact on a performer’s bottom line. Using postal code data provide to her from iTunes sales, she’s able to plan tours more efficiently because she knows, or can make a safe assumption, that she has paying fans in certain cities.

Touring and merchandise sales remain most artists’ primary means of income, and the current royalty rate of $.0011 per play doesn’t add up fast (at least according to Keating’s math), so it’s easy to see why she — and probably many other up-and-coming or niche performers — would rather have the data that properties like Pandora almost certainly have.

And whether Keating knows it or not, the idea of using data as a substitute for money extends beyond web radio stations and musicians arguing about royalties. A couple weeks ago, I highlighted a handful of attempts to convince consumers to hand over, in exchange for cash rewards or product discounts, valuable data that advertisers can’t collect by tracking their online activity. This is data such as recent and future purchases, personal interests, your web-surfing habits and where you shop in the physical world.

Just like Keating is willing to forgo one-tenth of one cent per play (real money, even if not a lot) in exchange for data, these brands are willing to trade cash or something like it for data they don’t have to run through a Hadoop cluster and seven segmentation algorithms before they can tie it to a real person. They know they have to give a little bit in order to improve upon the status quo that’s good, but not nearly good enough for their purposes.

Previously, the notion that data is the currency of the web meant users gave away their behavior data to web sites in exchange for free services. Slowly but surely, however, that notion seems to be evolving. Maybe Zoë Keating wants data in lieu of royalties for the privilege of streaming her music, and maybe a web site wants my offline location data enough to give me a gift card worth enough that I’d hand it over. Either way, it’s all about the realization that some data is worth its weight — and then some — in cold, hard cash.

Feature image courtesy of Flickr user eschipul.

In a new study, two University of Arizona researchers use Twitter’s emergence as a “serious newswire” to compare the reach and longevity of news stories tweeted by organizations like Reuters, NPR and the Washington Post. Over a three-week period last winter, the researchers looked at tweets containing story links and found that stories from the BBC and the New York Times were the most widely retweeted.

The study’s authors, Sudha Ram and Devi Bhattachary, also looked at metrics like articles’ half-life to determine the popularity and longevity of a news story. They found that articles from BBC, Mashable and the NYT had the longest life span, while the BBC, Mashable and Wired were most likely to publish popular articles — stories on Twitter that exceeded the average article half-life of 5.5 hours (“half-life” is based on a bitly definition that says it’s the amount of time at which a link receives half of the clicks it will ever receive after it’s reached its peak).

The study also looked at rates of engagement — how often a Twitter user is likely to tweet a given news source. On this front, financial publications like the FT and Forbes scored lowest while the NYT, NPR and the BBC scored highest.

The BBC’s prominence can be explained in large part by the fact that is has three major Twitter spigots that frequently retweet each other: “bbcnews,” “bbcbreaking” and “bbcworld.” This means that the BBC has far more of what the study calls “Maximum Level” retweets — Level I is an initial retweet, Level II is a retweet of Level I and so on.

The study, which draws on methods used for epidemics and network analysis, also uses intriguing graphics to display news organizations’ influence. This picture, for example, shows how the NYT and the Washington Post stories produce similar network effects, but the NYT stories are retweeted by more people in isolation:

So what to make of all this? One obvious observation is that the pool of data tied to Twitter gives news agencies unprecedented tools to measure their influence and shape strategy. But, as the study notes, one size may not fit all:

This leads to the question of what constitutes successful news diffusion on Twitter. Bursts of 1st level tweets within the first hour of diffusion (corresponding to instant reach to a large audience) or a high network diameter indicating multiple levels of exchange of news over a period of time (longer lifespan)? This depends on the objective of the news media source.

Another takeaway is that these are still early days for data and news analysis. While the Twitter study is intriguing, it is presented (appropriately) in the language of science — “edge/node ratios,” “ego network details” and so on. This means it may take time for the study’s implications to be translated into everyday guidance for publishers and editors.

Not only was this Carter’s first win, it was also his first contest. You can read the detailed explanation of his victory on his blog, but the gist is that he didn’t get too involved with complex social graphing to determine relationships or natural language processing to determine topics readers liked. He figured out that most of what people liked came from blogs they’ve already read, and that the vast majority of posts people liked fell within a three-node radius on a simple social graph.

Statistically speaking, he did a generalized linear regression model, followed by a random forest model and averaged the results. “I’m not sure it’s a very unique technique,” he told me, “but it’s certainly a very powerful one.”

Source: Overkill Analytics

And therein lies the beauty of overkill analytics, a term that Carter might have coined, but that appears to be catching on — especially in the world of web companies and big data. Carter says he doesn’t want to spend a lot of time fine-tuning models, writing complex algorithms or pre-analyzing data to make it work for his purposes. Rather, he wants to utilize some simple models, reduce things to numbers and process the heck out of the data set on as much hardware as is possible.

It’s not about big data so much as it is about big computing power, he said. There’s still work to be done on smaller data sets like the majority of the world deals with, but Hadoop clusters and other architectural advances let you do more to that data in a faster time than was previously possible. Now, Carter said, as long as you account for the effects of overprocessing data, you can create a black-box-like system and run every combination of simple techniques on data until you get the most-accurate answer.

I wrote about the same general theory recently in explaining why Sparked.com’s Daniel Wiesenthal believes that big data (i.e., lots and lots of data combined with new storage and processing technologies) improves the practice of data science (i.e., the application of statistical techniques to data). The gist of his theory is that although complex models are great for small data sets, simple models can close the accuracy gap when applied to large data sets. Combine that with infrastructure that can process a lot of data relatively fast and support a wide variety of jobs, and you have a simpler, faster equally effective method.

Still, Carter said he didn’t get involved in Kaggle just to prove the effectiveness of overkill analytics. He does hope to get exposed to new data science techniques that haven’t yet caught on in the insurance industry, and he also wants to make a name for himself. When you work for a company with little turnover, he said, your professional network doesn’t grow too much, but doing Kaggle competitions is a great way to meet other data scientists — and winning is a great way to earn respect.

Ali Ahmad (username Xali) won the separate Splunk Innovation portion of the contest. According to a statement from Splunk, he “used Splunk’s built in statistical and visualization features to map out the relationship between blogs containing YouTube videos with those that are most likely to be viral, as measured by likes and shares. As a bonus, he fed the data into an app to view the YouTube videos most commonly liked and shared via WordPress blogs!”

Disclosure: Automattic, maker of WordPress.com, is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, GigaOm. Om Malik, founder of GigaOm, is also a venture partner at True.

Feature image courtesy of Shutterstock user nasirkhan.

Aside from Khan’s star power, the show is so popular because of the types of issues it tackles — female feticide, caste discrimination, dowry deaths, child abuse and medical practice among them. According to one of the show’s producers, the amount of engagement and the number of responses from viewers is “completely unprecedented.” Here’s a sample of what we’re talking about, just 13 episodes into the show’s existence:

• 400 million viewers on Indian television and across the world on YouTube.
• More than 1.2 billion people have connected with Satyamev Jayate across its website, Facebook, Twitter, YouTube and mobile devices.
• More than 8 million people have contributed a total of more than 14 million responses to the show’s content via Facebook, web comments, text-message votes and a telephone hotline. More than 100,000 new people respond each week.

The responses take all sorts of forms, from votes on a weekly poll question to long, heartfelt letters explaining a viewer’s experience with an issue or how the show has changed their thinking on an issue. And although 95 percent of responses come from India, the show has received them from 5,000 locations in 165 countries, including as far away as northern Canada and Alaska. The show’s topics regularly rank among the top trends on Twitter shortly after each episode airs.

Surprisingly, the producer said, the India-created Satyamev Jayate has not received a single piece of hate mail from bitter geopolitical rival Pakistan. In fact, there have been numerous requests for an episode on India-Pakistan unity. (If you have 90 minutes, here’s an episode on human dignity.)

Parsing through millions of messages

In order keep up with all the messages, Satyamev Jayate turned to Persistent Systems, an Indian IT consultancy with offices around the world, which created a system for automating their analysis. Here’s how the process works.

About a day-and-a-half before each show, Satyamev Jayate’s production company tells Persistent what the issue will be and the two groups come up with a taxonomy that will help the system sort through messages based on what topics will be brought up during Sunday’s show. But it’s not by any means the definitive list. As activity ramps up on Twitter while the show airs (tweet rates are highest during commercials and immediately after it ends, by the way), the team gets a sense of what topics are resonating with viewers and what themes they can expect in the nearly million responses that will follow.

When the responses actually do start pouring in after lunch, they hit a system designed by Persistent to automatically tag them and score them based on interest level and sentiment. So, as Mukund Deshpande, head of business intelligence and analytics at Persistent, told me, a long message with an interesting story will be marked as higher quality, while a short, congratulatory note will be scored lower. Because so many viewers write in “Hinglish,” a combination of Hindi and English, an off-the-shelf system wouldn’t have been as accurate for processing these messages.

In the future, he’d like to train the system to recognize various gradients of emotion, too, beyond just simple sentiment. That means not just “positive” or “negative,” but also “happy,” “sad,” “angry” and any other way a viewer might be feeling.

The best messages are then sent to a team of trained analysts — often college students and graduates, along with some Persistent employees — who decide which ones are worth following up on for a Friday radio show Khan does, and for placement on Satyamev Jayate’s web site. These analysts try to ensure that the stories shared are truthful and that the messages don’t contain personal information that could get viewers in trouble or affect their privacy. Data visualizations about how many people have responded and where they come from is available on the Impact section of the show’s site, as well as on separate Impact pages for each episode.

Making a difference with data

Aamir Khan

All this feedback has an impact, both on the show itself and on India. Satyamev Jayate’s voting process, in particular, has yielded some impressive results. After the first episode about female feticide, or the selective abortion of female fetuses, 99.8 percent of viewers said they agreed with the idea of a fast-track court to prosecute doctors who perform such operations. When Khan presented the results to the Indian government, officials agreed almost immediately to amend the court system accordingly, the producer told me.

Sometimes, though, the results simply present an interesting — if not troubling — view into the Indian subconscious. Almost 32 percent of respondents, for example, voted in favor of the right of families to use force preventing the marriage of two willing adults (subsequent analysis uncovered some reasons why, including continuing opposition to inter-caste marriage), while almost 14 percent of respondents one week said that beating a woman is a sign of masculinity. And although women comprise only about 32 percent of the show’s audience, they have accounted for the majority of responses on shows addressing issues important to them.

The producer said his team also uses the data to inspire ideas for future shows and to populate a weekly radio show that Khan does with a local journalist. The Satyamev Jayate team analyzes the week’s messages in order to pick the most powerful and determine trends in viewers’ feelings, and Khan shares them during the interview. The second season, he said, will be shaped in part by how viewers responded to the format during the first season and the issues they want covered next.

Beyond just the next season, though — and the occasional political victory — the hope is that all the data Satyamev Jayate generates will have continuing utility. Deshpande said he’d like to see it used for ethnographic and social science research, because the dataset is larger than most academic studies could generate (something that’s already happening with crowdsourced medical research) and it’s very high quality because of the demographic and geographic information attached to it.

However, the producer with whom I spoke seems perfectly content right now with the way Satyamev Jayate is resonating with the public. For example, he said, viewers are reporting crimes they previously might not have considered too big a deal and are reaching out to disabled citizens. This is the first time many people are speaking openly about these issues, he said, and they’re able to track the effects because they’re able to ensure no message is left behind.

The company isn’t willing to share such data with the Wall Street Journal, but Barnes & Noble and Kobo talk a bit about the types of data collection they’re doing in this piece. For instance, they can track where a reader stops reading an ebook. The article notes a few non-surprising results from Barnes & Noble: readers of genre fiction (romance, mystery, science fiction) read fast and finish books; “nonfiction books, particularly long ones, tend to get dropped earlier.”

Jim Hilt, the company’s VP ebooks, says Barnes & Noble is in “the earliest stages of deep analytics” and has “more data than we can use,” but in some cases it’s making decisions based on the data:

Mr. Hilt says that when the data showed that Nook readers routinely quit long works of nonfiction, the company began looking for ways to engage readers in nonfiction and long-form journalism. They decided to launch “Nook Snaps,” short works on topics ranging from weight loss and religion to the Occupy Wall Street movement.

For now, Nook Snaps, Barnes & Noble’s e-singles section, remains B&N’s unimpressive competitor to Kindle Singles. Apparently, though, it’s an area Barnes & Noble wants to focus on.

Amazon, which is likely doing the most interesting and large-scale data collection and analysis of e-book readers, “declined to comment on how it analyzes and uses the Kindle data it gathers.” And from this piece, we learn that interactive fiction startup Coliloquy — the first app to be chosen for Amazon’s Kindle data developer program — can’t disclose sales figures because of an nondisclosure agreement with Amazon. That suggests that as Amazon starts adding more interactive reading apps to Kindle, it will keep the most interesting data it gleans close to the vest — while doling out “stats” like the non-surprise that people really like the first sentence of Pride and Prejudice, but even more than that, they like the Hunger Games.

We’re hosting a competition on Kaggle’s data science platform to find the best models around likely readership across the WordPress (see disclosure) ecosystem of blogs. Here are the details:

The challenge is to predict whether a particular user will like a particular WordPress blog post.  The data consists of eight weeks of posts collected by WordPress, along with anonymized user responses to each post.  This challenge is an interesting mix of natural language processing (the raw blog posts) and metadata on the blogs and users. Contestants can download the data and submit prediction through the Kaggle platform, but a new feature for this competition is that they will also have free access to a Splunk server containing all the data, which they can employ for data exploration, visualization, feature extraction and modeling.

Aside from offering resources to work on the data, Splunk is also putting up $25,000 in prize money. The winning model will receive $10,000, second place $5,000, third place $3,000 and fourth place $2,000.

There’s also a $5,000 Splunk Innovation Prize for the most innovative use of data science, whether that comes in the form of a visualization, app, business model, you name it. Submissions for the latter track can be submitted through Kaggle’s new Prospect platform. Winners for both competitions will be announced at GigaOM Mobilize in September.

You can find out more about the competition here. Good luck!

Disclosure: Automattic, maker of WordPress.com, is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, GigaOM. Om Malik, founder of GigaOM, is also a venture partner at True.

Image courtesy of Shutterstock user sukiyaki.