It’s part of a bigger plan to figure out how to address consumers’ base needs first and foremost, before then adding the nice-to-have features for a price. Aereo sees the future of television content as being what Kanojia calls “skinny live, deep library,” so the live parts are only for the content people really need in real time — stuff like news and sports.

“(The consumer is) the one constituent in this industry that’s unserved,” Kanojia said. “Everyone’s businesses are stacked to take advantage of the consumer, not to serve the consumer.”

If on the other hand, the value-add of a movie channel (oh, Aereo’s probably going to add one of those, too) is to watch stuff on your own time, people will probably willing to pay 50 cents or a dollar a month, he said. The same thing goes for programming from, hypothetically, a content provider like Viacom has a broad range of shows that people don’t really need or want to see only while they’re airing.

The only way to do this correctly, though, is to avoid traditional licensing models that have jacked cable prices through the roof and have led to a lot bloat because consumers are getting way more channels than they ever would want to watch. Kanojia wants Aereo to provide 50 percent of the value for 10 percent of the cost of cable, and then let partners and services like Netflix or Amazon Prime fill in the rest.

“The last time I checked,” he joked, “there’s no need to have Desperate Housewives or the Real Housewives of Orange County running on four channels at the same time.”

As for those lawsuits that have plagued the company since its inception, Kanojia said he’s not surprised but he’s disappointed by threats from companies such as Fox and CBS to pull their stations off the public airwaves (the spectrum on which is provided for free because stations are supposed to operate in part in the public interest).

“I just don’t understand the logic behind that,” he said. “I think it’s disappointing to say the least.”

But with significant legal victories already behind it, the future looks a little clearer. He expects the company model could realistically net the company 20 percent of the American television market, and the company is expanding fast outside of New York. It’s supposed to be in 22 more cities by July.

“The one thing that would float by boat more than anything else,” Kanojia said, “is I get a chance to put my product in front of consumers and be judged by the consumers.”

Check out the rest of our paidContent Live 2013 coverage here, and a video embed of the session follows below:

A transcription of the video follows on the next page

Go to page 2 (of 2) on paidContent .

Gravity’s recommendation system is based on its interest graph technology, which we detailed last year. Here’s how I described it then:

[T]he gist is that humans first serve as guides for machine-learning algorithms by determining connections between terms within large data sets, then the algorithms take over to complete the job faster than humans ever could. When they’re done, the humans step in one more time to kill any bad connections between terms. The result is a system that can determine with high accuracy that a person tweeting about Vanessa Laine (Los Angeles Laker Kobe Bryant’s ex-wife), for example, is probably more interested in basketball than about Laine’s date of birth or other accurate but irrelevant information.

As new content streams into Gravity’s system, it’s analyzed and categorized in real time, then presented to users accordingly based on their interests and behavioral history.

How Gravity’s platform works

Graph processing and graph databases — which store and analyze data based on their relationship to one another — are critical to our onlines lives, powering everything from online recommendations to social search to knowledge discovery. Graph technologies are also the focal point of some impressive life sciences work from companies such as Syapse and Ayasdi, which will be presenting at Structure: Data in New York next month.

But publishers struggling to stand out on a noisy web might have the most to gain from graphs and personalization, generally. At our PaidContent Live conference (April 17 in New York), executives from Prismatic, Zite and Bluefin Labs will take the stage to talk about the importance of personalization for helping consumers filter through the deluge of content online so they can find what they really want. It’s arguable that the trick to keeping readers happy is knowing what they want to read — possibly better than they do themselves.

According to Gravity, its platform currently “delivers more than 25 million personalized content recommendations per day to more than 200 million users. Beta partners have reported click through rates two to three times above previous levels, return visitation increases of 300 percent and session length increases up to 40 percent.”

The experiment’s techniques actually point to a future where researchers are spared the grunt work of poring through thousands of pages of news or watching hundreds of hours of programming, and can actually focus their energy of explaining. As the researchers note in their paper, the real ramifications of this research lie more in what it accomplished than in what it found.

Namely, they demonstrated that with new big data techniques such as machine learning and natural-language processing, it’s possible to accurately analyze millions of pieces of content spanning almost a year without requiring humans to read and score it all. Choosing hypotheses with predictable results meant it was easier to verify their accuracy.

Here’s how how they explain the promise of their work and some potential use cases, the latter of which they go into far more detail about in the paper:

“[I]t allows researchers to focus their attention on a scale far beyond the sample sizes of traditional forms of content analysis. Rather than spending precious labour on the coding phase of raw data, analysts could focus on designing experiments and comparisons to test their hypotheses, leaving to computers the task of finding all articles of a given topic, measuring various features of their content such as their readability, use of certain forms of language, sources etc. (just a few of the tasks that can now be automated).

… Our approach — apart from freeing scholars from more mundane tasks — allows researchers to turn their attention to higher level properties of global news content, and to begin to explore the features of what has become a vast, multi-dimensional communications system.”

Put more simply: This research underscores the common big data maxim that knowing the right questions to ask is now the biggest challenge in gleaning insights from data. It’s increasingly easy to get data, analyze it and visualize it, so humans really just need to hypothesize and be able to explain the results. (This also seems like a good place to plug ScraperWiki as a great source for gathering potential research data from websites.)

Creating the workflows for gathering and analyzing the data as the authors suggest still isn’t child’s play (it might take some assistance from the computer science department), but it’s a lot better than the alternative.

Feature image courtesy of Shutterstock user Ruggiero Scardigno.

As a bill attempting to lower the royalty rates paid to artists by streaming music services such as Pandora works its way through Congress, Keating took to her Tumblr blog last week and offered a solution that both sides should listen to, but won’t. You might have read about her stance in Billboard or ITworld already, or perhaps on Slashdot. If you haven’t, here it is in a nutshell, from Keating’s blog: “The law only demands I be paid in money, which at this point in my career is not as valuable as information. I’d rather be paid in data.”

Leaving aside the entire issue about royalties and copyright (and privacy policies), her statement is still powerful. Keating understands that in order to prosper in a world of digital music — just like in the world of e-commerce, digital publishing, you name it — information is power. The names, email address and perhaps mobile numbers of individuals listening to her music are nice, clean data that Keating could use with little to no analytic effort by reaching out to fans when a new tour is coming to town or a new album drops.

Actually, Keating noted in a subsequent blog post on Tuesday that even less-personal data can have a material impact on a performer’s bottom line. Using postal code data provide to her from iTunes sales, she’s able to plan tours more efficiently because she knows, or can make a safe assumption, that she has paying fans in certain cities.

Touring and merchandise sales remain most artists’ primary means of income, and the current royalty rate of $.0011 per play doesn’t add up fast (at least according to Keating’s math), so it’s easy to see why she — and probably many other up-and-coming or niche performers — would rather have the data that properties like Pandora almost certainly have.

And whether Keating knows it or not, the idea of using data as a substitute for money extends beyond web radio stations and musicians arguing about royalties. A couple weeks ago, I highlighted a handful of attempts to convince consumers to hand over, in exchange for cash rewards or product discounts, valuable data that advertisers can’t collect by tracking their online activity. This is data such as recent and future purchases, personal interests, your web-surfing habits and where you shop in the physical world.

Just like Keating is willing to forgo one-tenth of one cent per play (real money, even if not a lot) in exchange for data, these brands are willing to trade cash or something like it for data they don’t have to run through a Hadoop cluster and seven segmentation algorithms before they can tie it to a real person. They know they have to give a little bit in order to improve upon the status quo that’s good, but not nearly good enough for their purposes.

Previously, the notion that data is the currency of the web meant users gave away their behavior data to web sites in exchange for free services. Slowly but surely, however, that notion seems to be evolving. Maybe Zoë Keating wants data in lieu of royalties for the privilege of streaming her music, and maybe a web site wants my offline location data enough to give me a gift card worth enough that I’d hand it over. Either way, it’s all about the realization that some data is worth its weight — and then some — in cold, hard cash.

Feature image courtesy of Flickr user eschipul.

According to a release from the MIT News Office, Associate Professor Devavrat Shah says his model has been 95 percent accurate during testing and has been predicting trends hours before they appear on Twitter’s list. The algorithm incorporates a new approach to machine learning that compares real-time data with historical data and predicts outcomes based on past events that most closely align with the current situation. So, rather than analyzing a topic’s chances of trending equally against the entire historical corpus of topics, it will assign more weight to topics whose paths followed similar trajectories up the ranks of top trends.

And Twitter is certainly interested in the research. A company spokesperson emailed me to point out that Shah’s graduate research assistant, Stanislav Nikolov, is a Twitter employee.

Imagine knowing these topics before Twitter does.

However, the algorithm’s level of accuracy and speed would have to translate to a much-larger and more-complex stage — Twitter’s real-life firehose and stockpile of historical tweets — if the company were to use its predictions to charge premiums for ads associated with certain topics, as Shah suggests. Advertisers might not be happy to pay premium rates for topics that fizzle out before ever becoming top trends (although a tiered rate system based on the model’s confidence or, perhaps, projected ranking among top trends could work). Thus far, the algorithm has been trained using a set of 400 topics, half of which trended and half of which did not.

Shah thinks it’s a great fit for Twitter data because the data is relatively clean and he has found a strong correlation between past and future activity. Other historical data sets might be more messy or have more noise than does Twitter’s data set, which would make it much more difficult to filter out extraneous data and discern the real factors that lead to a particular result. However, even Twitter has presented research showing, in the case of its search engine at least, how the sheer volume of data it receives and the speed at which it comes in can make it difficult to accurately predict what someone wants to see.

The good news, though, for anyone willing to give Shah’s algorithm a try is that it’s designed to process data in parallel across scale-out systems like those used by large web companies. Therefore, training it and then running it in production across a voluminous data set won’t run into the same obstacles traditionally faced by machine learning algorithms as data sizes increase. And there are potentially more lucrative and rewarding endeavors that could benefit from this type of predictive power: Shah suggests stock markets, movie ticket sales and public transportation as possibilities, but others might include combating cybercrime by identifying threats earlier or predicting the severity of disease outbreaks.

Feature image courtesy of Shutterstock user turtleteeth.

Not only was this Carter’s first win, it was also his first contest. You can read the detailed explanation of his victory on his blog, but the gist is that he didn’t get too involved with complex social graphing to determine relationships or natural language processing to determine topics readers liked. He figured out that most of what people liked came from blogs they’ve already read, and that the vast majority of posts people liked fell within a three-node radius on a simple social graph.

Statistically speaking, he did a generalized linear regression model, followed by a random forest model and averaged the results. “I’m not sure it’s a very unique technique,” he told me, “but it’s certainly a very powerful one.”

Source: Overkill Analytics

And therein lies the beauty of overkill analytics, a term that Carter might have coined, but that appears to be catching on — especially in the world of web companies and big data. Carter says he doesn’t want to spend a lot of time fine-tuning models, writing complex algorithms or pre-analyzing data to make it work for his purposes. Rather, he wants to utilize some simple models, reduce things to numbers and process the heck out of the data set on as much hardware as is possible.

It’s not about big data so much as it is about big computing power, he said. There’s still work to be done on smaller data sets like the majority of the world deals with, but Hadoop clusters and other architectural advances let you do more to that data in a faster time than was previously possible. Now, Carter said, as long as you account for the effects of overprocessing data, you can create a black-box-like system and run every combination of simple techniques on data until you get the most-accurate answer.

I wrote about the same general theory recently in explaining why Sparked.com’s Daniel Wiesenthal believes that big data (i.e., lots and lots of data combined with new storage and processing technologies) improves the practice of data science (i.e., the application of statistical techniques to data). The gist of his theory is that although complex models are great for small data sets, simple models can close the accuracy gap when applied to large data sets. Combine that with infrastructure that can process a lot of data relatively fast and support a wide variety of jobs, and you have a simpler, faster equally effective method.

Still, Carter said he didn’t get involved in Kaggle just to prove the effectiveness of overkill analytics. He does hope to get exposed to new data science techniques that haven’t yet caught on in the insurance industry, and he also wants to make a name for himself. When you work for a company with little turnover, he said, your professional network doesn’t grow too much, but doing Kaggle competitions is a great way to meet other data scientists — and winning is a great way to earn respect.

Ali Ahmad (username Xali) won the separate Splunk Innovation portion of the contest. According to a statement from Splunk, he “used Splunk’s built in statistical and visualization features to map out the relationship between blogs containing YouTube videos with those that are most likely to be viral, as measured by likes and shares. As a bonus, he fed the data into an app to view the YouTube videos most commonly liked and shared via WordPress blogs!”

Disclosure: Automattic, maker of WordPress.com, is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, GigaOm. Om Malik, founder of GigaOm, is also a venture partner at True.

Feature image courtesy of Shutterstock user nasirkhan.

Aside from Khan’s star power, the show is so popular because of the types of issues it tackles — female feticide, caste discrimination, dowry deaths, child abuse and medical practice among them. According to one of the show’s producers, the amount of engagement and the number of responses from viewers is “completely unprecedented.” Here’s a sample of what we’re talking about, just 13 episodes into the show’s existence:

• 400 million viewers on Indian television and across the world on YouTube.
• More than 1.2 billion people have connected with Satyamev Jayate across its website, Facebook, Twitter, YouTube and mobile devices.
• More than 8 million people have contributed a total of more than 14 million responses to the show’s content via Facebook, web comments, text-message votes and a telephone hotline. More than 100,000 new people respond each week.

The responses take all sorts of forms, from votes on a weekly poll question to long, heartfelt letters explaining a viewer’s experience with an issue or how the show has changed their thinking on an issue. And although 95 percent of responses come from India, the show has received them from 5,000 locations in 165 countries, including as far away as northern Canada and Alaska. The show’s topics regularly rank among the top trends on Twitter shortly after each episode airs.

Surprisingly, the producer said, the India-created Satyamev Jayate has not received a single piece of hate mail from bitter geopolitical rival Pakistan. In fact, there have been numerous requests for an episode on India-Pakistan unity. (If you have 90 minutes, here’s an episode on human dignity.)

Parsing through millions of messages

In order keep up with all the messages, Satyamev Jayate turned to Persistent Systems, an Indian IT consultancy with offices around the world, which created a system for automating their analysis. Here’s how the process works.

About a day-and-a-half before each show, Satyamev Jayate’s production company tells Persistent what the issue will be and the two groups come up with a taxonomy that will help the system sort through messages based on what topics will be brought up during Sunday’s show. But it’s not by any means the definitive list. As activity ramps up on Twitter while the show airs (tweet rates are highest during commercials and immediately after it ends, by the way), the team gets a sense of what topics are resonating with viewers and what themes they can expect in the nearly million responses that will follow.

When the responses actually do start pouring in after lunch, they hit a system designed by Persistent to automatically tag them and score them based on interest level and sentiment. So, as Mukund Deshpande, head of business intelligence and analytics at Persistent, told me, a long message with an interesting story will be marked as higher quality, while a short, congratulatory note will be scored lower. Because so many viewers write in “Hinglish,” a combination of Hindi and English, an off-the-shelf system wouldn’t have been as accurate for processing these messages.

In the future, he’d like to train the system to recognize various gradients of emotion, too, beyond just simple sentiment. That means not just “positive” or “negative,” but also “happy,” “sad,” “angry” and any other way a viewer might be feeling.

The best messages are then sent to a team of trained analysts — often college students and graduates, along with some Persistent employees — who decide which ones are worth following up on for a Friday radio show Khan does, and for placement on Satyamev Jayate’s web site. These analysts try to ensure that the stories shared are truthful and that the messages don’t contain personal information that could get viewers in trouble or affect their privacy. Data visualizations about how many people have responded and where they come from is available on the Impact section of the show’s site, as well as on separate Impact pages for each episode.

Making a difference with data

Aamir Khan

All this feedback has an impact, both on the show itself and on India. Satyamev Jayate’s voting process, in particular, has yielded some impressive results. After the first episode about female feticide, or the selective abortion of female fetuses, 99.8 percent of viewers said they agreed with the idea of a fast-track court to prosecute doctors who perform such operations. When Khan presented the results to the Indian government, officials agreed almost immediately to amend the court system accordingly, the producer told me.

Sometimes, though, the results simply present an interesting — if not troubling — view into the Indian subconscious. Almost 32 percent of respondents, for example, voted in favor of the right of families to use force preventing the marriage of two willing adults (subsequent analysis uncovered some reasons why, including continuing opposition to inter-caste marriage), while almost 14 percent of respondents one week said that beating a woman is a sign of masculinity. And although women comprise only about 32 percent of the show’s audience, they have accounted for the majority of responses on shows addressing issues important to them.

The producer said his team also uses the data to inspire ideas for future shows and to populate a weekly radio show that Khan does with a local journalist. The Satyamev Jayate team analyzes the week’s messages in order to pick the most powerful and determine trends in viewers’ feelings, and Khan shares them during the interview. The second season, he said, will be shaped in part by how viewers responded to the format during the first season and the issues they want covered next.

Beyond just the next season, though — and the occasional political victory — the hope is that all the data Satyamev Jayate generates will have continuing utility. Deshpande said he’d like to see it used for ethnographic and social science research, because the dataset is larger than most academic studies could generate (something that’s already happening with crowdsourced medical research) and it’s very high quality because of the demographic and geographic information attached to it.

However, the producer with whom I spoke seems perfectly content right now with the way Satyamev Jayate is resonating with the public. For example, he said, viewers are reporting crimes they previously might not have considered too big a deal and are reaching out to disabled citizens. This is the first time many people are speaking openly about these issues, he said, and they’re able to track the effects because they’re able to ensure no message is left behind.