Discoverability is a problem for publishers, maybe not so much for readers

This post on why online book discovery is broken and how to fix it got the most comments of any post I’ve ever written, and a couple commenters complained that the solutions I offered in that post were aimed at publishers, not readers. That might be because discovery is more of a problem for publishers than readers: It is in publishers’ best interest to help readers find a not-so-well-known book, but it is not necessarily in readers’ best interest to read that book. It’s also unclear whether the average reader is really having all that much trouble finding the next book he or she wants to read.

A Book^Camp session led by Jeff O’Neal and Rebecca Schinsky of BookRiot focused on the “average” reader, a person who reads at most a few books per year. (Recent Pew data shows that of the 75 percent of Americans who read at least one book in 2012, the median number of books read was six.) This session was, not surprisingly, filled with bookish people who read at least a book a week, so I suggested that we think about areas of media consumption in which we, ourselves, are average.

For me that’s music and movies. I’m an avid reader — I spend a lot of time thinking about what I will read next and searching for books and talking to people about books — but I don’t put that level of effort into finding which songs to listen to next or which movie to watch. Instead, I kind of wait for things to rise to the surface. When something finally breaks through to the point where I’ve heard about it enough, through various internet and non-internet sources, I consume it.

This is why I saw Argo three months after it was released and will maybe get around to watching Zero Dark Thirty some time in 2014. It’s why I mostly listen to the radio on Spotify. I’m not really proud of this, but I’m not that embarrassed by it either. If I put as much effort into consuming movies and music as I do into reading books, I would have way less time to read. I’d rather read, so something’s gotta give.

There are a lot of people like me — big readers who spend a lot of time thinking about what they are going to read next. Book publishers do not have to worry about these people. At the same time, getting average readers to be interested in book discovery — getting average readers to visit Bookish, for instance — is going to be difficult, because you are also going to have to require these people to make big shifts in their behavior and in their media consumption patterns.

Are these people really not reading more because they don’t know what they should read? Maybe. But it’s more likely that they have plenty of things they’d like to read, and just don’t have time, or, like me, there are other forms of media that they care about more than books, and if they were to shift into reading more books, they would have to give up things they really like instead.

As Brett Sandusky points out, “Most people who read books read for pleasure. They will have gaps in their reading before they pick up something else. Yet somehow, we’ve decided, implicitly, that the normative reading behavior, which discoverability facilitates, is shotgun style where readers are reading book after book after book after book.”

It’s hard to change people’s behavior patterns — that’s a challenge for any industry, not just for book publishing. Book publishers have to continue to focus on getting their books into new readers’ hands, but it is unclear whether algorithmic solutions like Bookish are going to be of interest to anyone but the people who are the most avid readers already. Since publishers can’t physically enter people’s living rooms, turn off their TVs and shove books into their hands, they may instead have to focus on retail and, as Guy LeCharles Gonzales writes, work on their direct relationships with readers.

Walled gardens are permeable

At Tools of Change on Wednesday, Goodreads CEO Otis Chandler presented the results of a survey of 1,500 U.S. Goodreads users. (His full presentation is here.) This is, of course, a survey of those avid readers I mentioned above — not only are they on Goodreads but they are willing to actually sit down and take a survey about their ebook reading behavior. Nevertheless, check out this slide:

Goodreads / “What’s Going on with Readers Today” http://www.slideshare.net/GoodreadsPresentations/whats-going-on-with-readers-today-16508449

There are way more questions than answers here, but the results appear to suggest that readers don’t see platform lock-in as an insurmountable problem — or in fact as something that’s actually locking them in. Instead, they’re reading across different retail platforms.

These results “made us scratch our head,” Chandler said. The company didn’t delve further into which devices readers are using to read ebooks across platforms, and so it’s unclear how exactly this experimentation is taking place. For example: Are people confusing “iBooks” with iPad — so that someone reading ebooks on a Kindle is also reading them on an iPad Kindle app, but somehow counts that as reading on iBooks? Or are readers using multiple retailers’ tablet apps, and also buying ebooks from multiple retailers? Or are they actually breaking DRM so that they can buy a Nook book and read it on a Kindle? It seems possible that tablets actually break down walled gardens because readers can have multiple ebook vendors’ apps on a single device.

Disclosure: Goodreads is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, founder of Giga Omni Media, is also a venture partner at True Ventures.

I got a demo of Bookish at the company’s trendy, book-filled offices in Manhattan’s Flatiron District last week, and had a chance to use the site further on Monday when it was prematurely available online for several hours as it was being tested. Overall, I think the long-delayed Bookish is off to a promising start.

Bookish has the opportunity to shape book discovery and offers publishers a chance to directly engage with readers. It also allows them to tiptoe into direct sales. I’m less intrigued by the original editorial content: I’m not sure it differentiates itself enough from other book-related content on the web to draw users to the site for the first time. Once those users make their way to the site, though, they’ll find a clean, easy-to-use design, and an algorithm that may well find them their next book — even though it’s limited to less than a quarter of the books on the site for now. Here’s my overview of the site.

 The basics: Books and authors

While only three of the big-six publishers are financially backing the site, the other three — Random House, HarperCollins and Macmillan — are making their books available through it, along with 10 other publishers including Scholastic and Houghton Mifflin. In total, that’s 1.2 million unique titles spanning 18 genres (fiction and literature, children’s, cookbooks, and so on), and 400,000 authors have profile pages. The book pages include basic information, a preview of the first chapter, related news and videos, and a roundup of any “must-read” lists that the book has appeared on (for more on those lists, see below). Each book page also includes purchase links (more on that below, too).

Algorithm-generated book recommendations

Online book discovery is a huge problem for publishers, and Bookish tackles it with a recommendation algorithm that lets users input up to four titles to find what to read next. “We’re very much a technology company,” Karen Sun, an MIT grad (and book blogger) who is heading the company’s recommendation engine, told me. “This is probably the largest venture in the book space, in terms of data.” Sun explained that while Amazon and Goodreads primarily deliver book recommendations based on “collaborative filtering” — namely, a user’s purchasing or rating and reviewing history as well as those of other users — Bookish doesn’t have that user or purchase data yet. Instead, it relies on “deep, introspective” data: “Recommendations are based on the books and understanding of the books.” The recommendation looks at features like the authors, editors and illustrators who contributed to a book, the awards a book has won, and genre and publication date, then layers on a machine-learning component that parses user and professional reviews to try to distill themes, concepts and sentiments. Insights from the editorial team are included, too.

A user who liked The Help, for instance, receives recommendations for Hotel on the Corner of Bitter and Sweet by Jamie Ford — another women’s fiction title that features race relations — and The Guernsey Literary and Potato Peel Pie Society, a book that, like The Help, includes an aspiring female author. Type in Malcolm Gladwell’s The Tipping Point and the engine pulled up four similar “big ideas” books, but also two Spanish-language titles that were out of place even if the subject matter was similar (and you’ll see a Spanish-language edition of The Room in the recommendations for The Help above).

For now, Bookish’s recommendation engine works with only about 250,000 of the 1.2 million books on the site. Sun says the engine will improve over time, and will eventually integrate reader reviews and user actions — other books users have looked at and rated on the site.

E-commerce: Essential, but…

Each book on the site can be purchased in print or digital formats directly through Bookish or from another retailer — there are affiliate links to Amazon, Barnes & Noble, Books-A-Million, IndieBound, Apple and Kobo.

Distributor Baker & Taylor is handling all of Bookish’s direct sales. For now, ebooks purchased through Bookish are only available in EPUB and PDF formats, for reading on iPad, Android, Nook and desktop — no Kindle.

Bookish seems to want to stress that it’s not cutting into other retailers’ sales, even though a serious direct-sales outlet is something that book publishers desperately need.

“We want to be able to say you can buy [a book] here and it’s reasonably priced. We’re not trying to steal sales away from other places,” CEO Khazaei told me. Publishers probably don’t care about taking sales from Amazon, but they may not want to sour relationships with retailers like Barnes & Noble and the independent bookstores represented by IndieBound.

Bookish’s print and ebook prices appeared to match those offered by Amazon, though I wasn’t able to test many titles. Khazaei told me that “I don’t know how the pricing decisions are made, really,” Khazaei said. “I assume [Baker & Taylor] is tracking [prices on other sites] but we just leave it in their hands.” While the site seems like an obvious place for publishers to run special sales on both print and digital books, that doesn’t seem to be a priority for now. Update: Khazaei stressed to me that his lack of involvement with pricing is required by the Department of Justice in order to be compliant with antitrust regulations. (The DOJ sued Hachette, Penguin and Simon & Schuster, along with Macmillan and HarperCollins, last year for allegedly colluding to set ebook prices; Hachette, Penguin and S&S all settled.)

Original editorial content along with the algorithm

Bookish has seven full-time editors who each manage different genres and update those sections daily with original book coverage. The site is also soliciting pieces from well-known authors and other public figures. In one ongoing feature, for instance, editors from The Onion review books. Other editorial features at launch include a column by Eat, Pray, Love author Elizabeth Gilbert and an interview between bestselling thriller authors Michael Connelly and Michael Kortya. In addition to that content, the site’s editors are curating columns and lists of books like “The Biggest BFF Breakups in YA Books” and “Big Ideas.”

Advertising, revenue and partnerships

Bookish is collaborating with USA Today’s books website. Its original editorial content will be syndicated on USA Today’s website, and the technology that Bookish uses to let readers view the first chapter of a book and to offer book recommendations will also be included on USA Today’s site. In exchange, Bookish will feature USA Today’s book bestseller lists on bookish.com.

In addition to book sales, Bookish will get revenue from advertising. For now the site’s ad slots are taken up with books from the three launch partners, but eventually the company will expand advertising to other publishers and to companies from outside the book business. Prior to its launch two years ago, Bookish had announced an advertising and content syndication deal with AOL Huffington Post, but that’s off the drawing board for now. A company spokeswoman told me Bookish is “in discussions about continuing to work with AOL in the future.”

Not a focus: Social, self-publishing

Other publishers can sign an agreement with Bookish to add their titles to the site. (Khazaei told me Bookish doesn’t charge publishers anything to join, but they presumably have to fulfill a number of requirements to be included.) However, self-published authors can’t add their books. “The focus right now is on traditionally published titles,” Khazaei said.

Also at launch, the social features that are a key part of Goodreads’ mission are absent from Bookish. Users can’t friend or follow each other — the focus is on a reader’s individual interests. I found that refreshing: Just because you’re Facebook friends with someone doesn’t mean that he or she shares your book preferences, and I prefer the algorithm-driven approach.

In an email interview about the funding, Cross said that the financing will allow Prismatic to grow from a group of just six, and allow it to take on more of the recommendation and filtering tasks it is trying to build into the product:

“This financing finally allows us to have the team that we need to tackle the problems we have ahead in the next 24 months. To date, we’ve run Prismatic with only 6 people, and built it with only 5… the active customer base is growing fast, and we need the resources to keep up and move on to our new product, distribution, research, and revenue ideas.”

As we described in a profile of Cross and his startup earlier this year, Prismatic is one of a number of tools that are trying to fix the problem that media theorist Clay Shirky once described by saying “It’s not information overload, it’s filter failure.” The service is based on machine-learning algorithms developed by Cross — a data scientist who used to run a hedge fund and later worked as a consultant on a number of Google projects — and his co-founder Aria Haghighi, and is designed to learn from users what they like or dislike about the content they are reading through the service.

The news-filtering business is a crowded field

In that sense, Prismatic is similar to Zite, the Canadian-born startup that was acquired last year by CNN and recently came out with a revamped version of its service, and other smart filters such as News360 and Pulse. But only Prismatic has explicitly said that it wants to use news-recommendation as a kind of Trojan horse to get a foot in the door with users, so that it can eventually learn enough about them to recommend all kinds of things to them — including purchases. As Cross explained it recently:

“The idea is that we become this trusted agent that you rely on to show you things, and over time we can really start to learn a lot about you. We do care a lot about [news recommendation], but we’ve also thought through how it’s a stepping stone to something much bigger. And a lot of what we do in the background, and how we slice and dice data and so on… is relevant across a really wide range of problems.”

Prismatic, which has about six staff — many of whom work out of a small office in San Francisco’s SoMa district that looks a lot like a university dorm room — also recently launched a mobile version of the app, which Cross said was designed to take advantage of the down time that many phone users have while waiting for the bus or standing in line at the airport. The information needs of mobile users are difficult to satisfy in part because they have so little time, he said, but Prismatic managed to build what amounts to a specialized mobile browser that makes the process far more painless than the usual mobile web experience.

I experiment with almost every news-recommendation app or service that comes along, from News.me (which eventually merged with what was left of Digg) and Summify — which was acquired by Twitter and became the foundation of its daily emails pointing users to worthwhile content — to News360, Pulse and Zite. So far, Prismatic is one of the few that has been able to capture my attention and keep me coming back to the app, although the new Digg has also quickly become almost as important in my daily browsing habits as the old version of the service was.

That’s the central risk for Prismatic: the world of news recommendations is a harsh one, and users are always looking for whoever can give them the best fix. Twitter is clearly focused on doing this, as Cross has acknowleged, and so are plenty of others with fairly deep pockets. Hence the need to raise a $15-million Series A round — which came from Accel Partners, along with a personal investment from partner Jim Breyer, and from Russian oligarch and Facebook investor Yuri Milner.

According to a release from the MIT News Office, Associate Professor Devavrat Shah says his model has been 95 percent accurate during testing and has been predicting trends hours before they appear on Twitter’s list. The algorithm incorporates a new approach to machine learning that compares real-time data with historical data and predicts outcomes based on past events that most closely align with the current situation. So, rather than analyzing a topic’s chances of trending equally against the entire historical corpus of topics, it will assign more weight to topics whose paths followed similar trajectories up the ranks of top trends.

And Twitter is certainly interested in the research. A company spokesperson emailed me to point out that Shah’s graduate research assistant, Stanislav Nikolov, is a Twitter employee.

Imagine knowing these topics before Twitter does.

However, the algorithm’s level of accuracy and speed would have to translate to a much-larger and more-complex stage — Twitter’s real-life firehose and stockpile of historical tweets — if the company were to use its predictions to charge premiums for ads associated with certain topics, as Shah suggests. Advertisers might not be happy to pay premium rates for topics that fizzle out before ever becoming top trends (although a tiered rate system based on the model’s confidence or, perhaps, projected ranking among top trends could work). Thus far, the algorithm has been trained using a set of 400 topics, half of which trended and half of which did not.

Shah thinks it’s a great fit for Twitter data because the data is relatively clean and he has found a strong correlation between past and future activity. Other historical data sets might be more messy or have more noise than does Twitter’s data set, which would make it much more difficult to filter out extraneous data and discern the real factors that lead to a particular result. However, even Twitter has presented research showing, in the case of its search engine at least, how the sheer volume of data it receives and the speed at which it comes in can make it difficult to accurately predict what someone wants to see.

The good news, though, for anyone willing to give Shah’s algorithm a try is that it’s designed to process data in parallel across scale-out systems like those used by large web companies. Therefore, training it and then running it in production across a voluminous data set won’t run into the same obstacles traditionally faced by machine learning algorithms as data sizes increase. And there are potentially more lucrative and rewarding endeavors that could benefit from this type of predictive power: Shah suggests stock markets, movie ticket sales and public transportation as possibilities, but others might include combating cybercrime by identifying threats earlier or predicting the severity of disease outbreaks.

Feature image courtesy of Shutterstock user turtleteeth.

In a Twitter Engineering blog post explaining its findings, analytics research scientist Jimmy Lin explains the problem of ranking tweets by relevance as partly being a problem of time. In the case of breaking news, the system is simply overwhelmed by tweets and queries on that topic, which means Twitter’s relevancy models can’t always keep up to determine which ones you probably want to see. While it’s relatively easy to build a simple search algorithm utilizing the concept of “term frequency-inverse document frequency weight” when the overall corpus of documents is fairly static, it’s a lot harder when terms suddenly surge in popularity and a system has to constantly re-process the dataset in real time.

These numbers from Twitter’s research help explain the problem:

• Examining all search queries from October 2011, we see that, on average, about 17% of the top 1000 query terms from one hour are no longer in the top 1000 during the next hour. In other words, 17% of the top 1000 query terms “churn over” on an hourly basis.
• Repeating this at a granularity of days instead of hours, we still find that about 13% of the top 1000 query terms from one day are no longer in the top 1000 during the next day.
• During major events, the frequency of queries spike dramatically. For example, on October 5, immediately following news of the death of Apple co-founder and CEO Steve Jobs, the query “steve jobs” spiked from a negligible fraction of query volume to 15% of the query stream — almost one in six of all queries issued! Check it out: the query volume is literally off the charts! Notice that related queries such as “apple” and “stay foolish” spiked as well.

Of course, this particular phenomenon doesn’t explain why Twitter’s search doesn’t go back further in time, or why its algorithms for ranking tweets based on source or the number of time they’ve been retweeted don’t appear too accurate. Even if relevancy improves, there’s still a lot to be desired in terms of getting Twitter to return the types of results users have come to expect.

Lin’s post also highlights another piece of research from Twitter that’s less noteworthy to individual users but probably more telling about the world as a whole. A visualization of Twitter usage patterns in New York City, Tokyo, Sao Paulo and Istanbul creates a picture of cultural and seasonal differences at play.

Twitter users in Tokyo, we see, tweet a lot less during the work day and also go to bed and wake up at about the same times throughout the year. Elsewhere, users show pretty distinct differences in activity as the seasons change. Lin also points out the afternoon lull in Sao Paulo. It’s difficult to discern the exact reason from looking at this chart, but the lull does coincide with Sao Paulo’s winter season and a generally later beginning to the tweeting day.

I’d love to see these results analyzed against other cultural datasets, or even just against a knowledge base of local customs and behaviors, to see how Twitter  use — and web use, generally — comports or doesn’t comport with a region’s typical norms.