How can we best measure how good people think our recommendations are?

Recently a friend was describing how her friend had persuaded her to watch a DVD that she thought she’d absolutely hate, but ended up loving. The DVD was the 2004 BBC TV series ‘Blackpool’ (described as “part musical, part thriller, part drama”) – and my friend was initially sceptical because, as a general rule, she really doesn’t like musicals. She is a big fan however of two of the main actors, David Morrissey and David Tennant, which slightly warmed her to the idea, but it was really the insistence of her friend that made her sit down and watch it – and thoroughly enjoy it!

I think this story highlights the problem with TV programme recommendations based solely on genre: given my friend’s dislike of the ‘musicals’ genre, she would never have been recommended the programme in the first place by a genre-based recommendation system, or watched it based on genre information alone.

With NoTube TV recommendations, it has always been our intention to generate personalised recommendations based on preferences that include genres, but are not limited to them. By using linked data techniques, preferences for specific programmes can theoretically be based on any metadata associated with a programme; such as directors, writers, actors, presenters, contributors, locations, and other people (both friends and/or experts) who are watching, or have watched, the programme. As danbri has suggested, we could also make quirky connections such as “you were recommended this programme because the star, Cary Grant, used to live in your street”.

Using this approach, my friend might have been recommended ‘Blackpool’ based on the fact that two of her favourite actors are in it, and that her friend (whose taste she admires) has watched it. We’ve always believed in the importance of explaining to users why a programme has been recommended to them, but would the evidence have been strong enough to persuade my friend to try watching it? I think this issue of persuasion is important: after all, one of the things we want from a good recommendation system is for it to help us broaden our horizons and discover new, enjoyable things that we would probably not have discovered on our own.

If my friend hadn’t been persuaded to try the programme, she would never have discovered that she really loved it. And she would never have found out that it was actually a great recommendation for her, which could have led to other new and interesting programme suggestions.

This leads to the challenge of how you get people to subjectively evaluate the quality of the programme recommendations suggested to them by a recommendation system when:

  • Firstly, they need to be persuaded to try out some new things, which might at first seem rather too obscure or too far removed from their comfort zone.
  • They need to have the motivation to provide the system with some feedback, and the patience and commitment to give the system time to make adjustments.
  • Ideally, they also need to give reflective thought to the question of whether a programme recommendation really was ‘relevant’ or ‘useful’ – how would you really know unless you tried watching it?

In all, this requires a substantial amount of effort and dedication on behalf of the user – it’s not quite the same thing as asking people if a shopping cart transaction was confusing or not.

Further, evidence suggests that TV viewing is still commonly a social activity in which negotiation and compromise are inevitably part of the decision-making process of the group.  Group recommendations therefore have to take into account the preferences of multiple users, making these sorts of evaluations even more challenging.

In the next phase of the NoTube project we want to test the quality of our recommendations on users. To get really useful feedback we will need to tackle these various issues along the way.

This entry was posted in Recommendations, Thinking Out Loud. Bookmark the permalink.

3 Responses to How can we best measure how good people think our recommendations are?

  1. Pingback: On serendipity | NoTube

  2. Pingback: Finding interesting new programmes – trial results | NoTube

  3. Tim says:

    Well, if you can avoid releasing enough data that you compromise privacy, there’s the equivalent of the netflix prize: http://www.netflixprize.com/index

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s