‘ “Show me the money”: Podcasting, measurement and ROI’

Admittedly, this is being posted a little later than I had hoped, as I have been away on holiday, but what follows is a précis of a presentation/discussion we gave at PodcampUK earlier in September entitled :’ “Show me the money”: Podcasting, measurement and ROI”

Why is there a need to measure podcasts?
As for measuring podcasting and its ROI, it’s a very important issue, especially in relation to marketers, advertising agencies and publishers. How to track the number of times an advertisement is heard, how to track the PR/brand value of being associated with a podcast or of reaching an influential audience and also in terms of impact in terms of outcomes :e.g.; increased task completion rates/ improved customer satisfaction numbers/ increased depth of visit over time on site/ increased lead generation/ increased sales etc.

Why are podcasts so difficult to measure?
In essence, this is because there are no real agreed standards of measurement. There are several methods being used (more of which below) some of which attempt to measure ‘hard stats’ - or not so hard stats as we shall see!; i.e; numbers of downloads and subscribers - and others which attempt to measure ‘engagement’.
None of these are perfect, by any stretch, nor arguably to be used in isolation.
As will be made clear further below, some even argue that attempting to use and apply metrics in such a potentially hard and fast way could be detrimental to the ongoing development of podcasting as a communication medium (note: not as an advertising tool)
The below goes through some of the various methods used, all of which are far from perfect. Broadly speaking, this can be carved up into 2 camps:
1/measurement of downloads/subscribers
2/ measurement of ‘engagement’ which try to infer user behaviour and intent

What methods are being used to try to measure downloads / subscribers?

1. Number of hits to file:
Whilst tracking file requests is a good place to start, it’s not a perfect method of measurement. For one thing, can you determine how much of the file was actually transmitted? Also, there’s the issue of repeat requests - same ip address, same file. How should you count them? If they’re within a few seconds of each other, it may be an artifact of the way some podcatchers work and the duplicates should be discarded. On the other hand, e.g, if you get popular at a university halls of residence that’s behind a firewall, it may actually be legitimate downloads. Some have argued that a way to work around this difficulty is to adopt some form of arbitrary cut-off and keep it consistent, for example, not to count two file requests for the same file from the same ip if they occur within 5 minutes of each other. Clearly, though, not ideal.

2. Bandwidth transferred
This file had x bandwidth transferred / average download of that file.
Using number of bytes downloaded vs. file requests is seen as preferable by some since there are so many spiders/crawlers out there that hit mp3 files daily.

3. Measuring downloads
Another approach is to compare the number of downloads that various podcasts get. However, as accurate data on downloads is not available for all podcasts, this makes it impossible to rank podcasts in terms of actual downloads or listens.

4. Podcatcher Software Subscriber Statistics
Podcatcher software records the ip addresses of people who click on the RSS or subscribe button. Some users might take advantage of the “subscribe” feature to receive a podcast series, while others use it to sample a single episode. While subscriber numbers can provide counts of users who clicked a subscribe button within various podcatcher software, subscriber numbers derived from “subscribe” button-clicks are not entirely perfect, e.g. a subsequent decision by the user to cancel the subscription, or to interrupt the download, may not be successfully matched back to the original subscription.

5. RSS subscription reports
RSS subscription reports provide data to podcasters regarding recurring subscriber access to RSS hosting feeds. This measures access to the feed itself, but does not provide any confirmation that subscribers actually downloaded episodes reported in the RSS feed. ( For more on the issue of RSS feeds and the complexity of their measurement, see Avinash Kaushik “Web Analytics:An hour a day” - pg 185ff )

6. Podcast Directory User Statistics
Podcast directories offer users the ability to sample a wide variety of podcasts. Some directories collect data and provide rankings based on the number of users clicking a subscribe button for a particular podcast series, within the directory, whilst other directories base their rankings on some measure of users’ opinions of the podcast expressed as the number of users voting for a specific podcast. However, the problem with this is that it could be self-selective and biased towards those who are either willing to vote or for those who use this for more nefarious reasons, much as in the same way some have tried to ‘game’ Digg by paying users to submit content or,as could be the case in this instance, pay for votes. (Clearly, also, this latter aspect of rankings reliability very much applies to evaluating theories of engagement metrics - see below)

7. The iTunes top 100 lists
iTunes publishes a top 100 list for all of the podcasts in its directory and separate top 100 lists for each podcast category.
However, the iTunes top 100 lists are based primarily on new subscriptions initiated in the iTunes podcast directory during the past week, with the more recent subscriptions weighted more heavily. So, in this context, iTunes is useful to identify the up-and-comers and growing podcasts, but again, not an absolute definitive as it also doesn’t count the proportion of the audience that doesn’t use iTunes. Again, even this form of measurement is regarded by some as controversial, as is evidenced by debate on this issue on various blogs such as the Astronomy Blog

Measures of ‘engagement’
These metrics have already been set out very comprehensively by my co-presenter, Marianina Chaplin, a.k.a. Web Analytics Princess, on her blog here

How does this all work in real life? (*discussion)
This was perhaps the most controversial aspect of our presentation as it was suggested that by trying to apply metrics to podcasts, or even discussing what types of metrics are being used not very successfully / which should or could be used, this was tantamount to holding podcasters to the same standards in other advertising formats and could ‘kill off’ podcasting in its infancy.
Far from it!!!! This was not our intention at all !!!! But clearly, for potential sponsors, advertisers or corporates looking to create and host their own podcast, there is the issue of monetisation: what kind of bang you get for your buck: why should I pay you X to sponsor your podcast or pay for an advert in the podcast? Am I going to see any kind of increase in my online sales or leads generated? If I create a podcast, what impact will this have on my business goals?
This was the issue which we tried to draw out by highlighting the different methods used to track downloads / subscribers and also other ‘engagement metrics’ and use this to try to create a discussion.

On a personal viewpoint - whilst I am a big podcasting fan and am resolutely in favour of podcasts being used to communicate ideas to/with often small,fragmented audiences (cf the excellent Slam Idol poetry oriented podcast) , I do see that the need arises where there is a necessity to monetise your actions, particularly if you are using podcast sponsorship/advertising or creation as part of your advertising strategy.

This may be less of an issue (-is it?) with larger enterprise sized corporates who have time, energy and resources to devote to a brand-building exercise, and are ?happy? to see it in isolation as simply that and use engagement metrics in combination with a rough and not-entirely-reliable form of attempt at measurement of downloads/subscribers, but what if you want to try to tie your podcast hosting/sponsorship/advert into some form of direct correlation with your overall business strategy outcomes :eg, increased task completion rates/ improved customer satisfaction numbers/ increased depth of visit over time on site/ lead generation/ increased sales etc.
What,then, is “success”? How do you define your Return on Investment?

If so, how to do this? How to package the advert so that it can be tracked back to the podcast in question?
Does this mean resorting back to conventional methods such as custom telephone numbers or offer codes so that you have some way of definitively tracking, for example, ‘this user listened to podcast episode #3 (whether 2 days/2 months/2years hence) and purchased goods on my site/made firm enquiry”?

Furthermore, particularly if you are a company and have chosen to create and host a podcast yourself, how do you monetise your actions taken to do all this? Although the barriers to entry to podcasting are low and relatively inexpensive (*eg, you can record your interviews via Skype using a conference call facility, edit using open source Audacity software and use a simple to use podpress plugin on a wordpress blog, all of which are very, very straightforward ) , there are still the issues of time and (wo)manpower to consider. ( : here, apply Avinash Kaushik’s method of measuring blog ownership to that of podcast ownership to work out how much all this time and effort is costing you: see Avinash Kaushik “Web Analytics:An hour a day” - pg 189ff )

How do you determine when or indeed if the podcast isn’t working for you in terms of your envisaged business goals and outcomes? For example, you might be getting lots of fantastic feedback on the content of your podcast (i.e, you have great engagement metrics in place which tell you that everyone loves your podcast - but as we know KPI’s aren’t the same as business goals), but if you can’t discern a correlation over time between that great *crunchy* qualitative user opinion to any discernible increase in eg, orders/leads generated over time or any other outcomes - is it worth it? At which point can you definitively say - this is not working for us.

I don’t claim to have the answers to this but I would love to hear about your experience or thoughts on the matter.

Shinkansen Snack Queen: we could all learn a thing or two from her

A fascinating article recently in The Times featured the story of Kumiko Mogi, who has risen to become the most successful snack saleswoman ever on the Japanese Shinkansen (Bullet train) network.
Whilst there may not appear to be an immediately obvious link between this and the world of internet marketing and web analytics, there are actually more than a few overlaps.

For one thing, Shinkansen sales(wo)menship is retail at its most raw: each trolley girl is in charge of her own trolley and completely in control of determining what the 800 passengers will be able to snack on. In short, their role is simply to keep on selling until the doors open. A poor choice in terms of which snacks to bring on board is clear: a dent in profits. A case in point being dried fish snacks on a train packed full of school kids.

Mogi manages to collect 5x what her fellow trolley girls collect, by virtue of “absorbing data at lightening speed: the ambient temperature, the size of the passengers’ baggage, the books and newspapers tucked under their arms, what the youngsters are wearing and a bewildering range of other variables.”
As Mogi goes on to explain in the article, she “works constantly to read and engage with the customers…and switches between regional accents to suit the person ”
As she willingly admits herself, her greatest innovation was to pull, rather than push, the trolley through the train “By pulling the trolley, and never turning my back towards the customers, I can constantly look at their faces and work out what they are going to want”

To me, this is the epitome of what internet marketing and web analytics should ideally be all about: analysing your data and using that to find out as much as you possibly can about your target customers and delivering what they want, all with a clear objective in mind.