Category Archives: Measurement

06.14.11
by

Direct visits: A referral data black hole

“Facebook drives more traffic than Twitter” ran the headline in May, after a Pew study seemed to show that Twitter just wasn’t as good for traffic numbers as people had thought. But there were problems with the study’s methodology, as many people, including Steve Buttry said:

The PEJ report acknowledges that the Nielsen Co., the source of all the data studied, relies “mainly on home-based traffic rather than work-based,” without adding that most use of news sites comes during the workday.

and

The study uses strongly dismissive language about Twitter’s contribution to traffic to news sites. But it never notes that many – probably most – Twitter users come from TweetDeck, HootSuite, mobile apps or some other source than Twitter.com. Twitter “barely registers as a referring source,” the report concludes, ignoring or ignorant of the fact that the data counted only traffic from Twitter.com and did not count most visits from Twitter users.

As the web evolves, so the tools that we use to measure and assess activity need to evolve, but this hasn’t really happened. We might have managed to ditch the misleading idea of ‘hits’, but web traffic measurement is still immature, with many of the tools remaining basic and unevolved. But this problem is only going to get worse, as Steve’s second point hints at.

As I mentioned in this post, earlier this year I did some work looking at referrer logs for a client, OldWeather.org, a citizen science project that is transcribing weather and other data from old ships logs. One of the things that I noticed was how messy Google Analytics’ data is when it comes to finding out which social networks people have visited from. Many social networks have multiple possible URLs which show up in the stats as separate referrers. For example, Facebook has:

  • facebook.com
  • m.facebook.com
  • touch.facebook.com

And Twitter has:

  • twitter.com
  • mobile.twitter.com

So in order to get a better picture of activity from Facebook and Twitter, we need to add the numbers for these subdomains together. But that alone doesn’t provide the full picture. A list compiled by Twitstat.com in August of last year showed that only 13.9% of its users were using the Twitter.com website, with another ~1% using Twitter’s mobile website. That means around 85% of Twitter users are not going to show up in the twitter.com referrals because they haven’t come from twitter.com or mobile.twitter.com.

It is possible to get some other hints about Twitter traffic as some web-based clients do provide referral data, e.g. twittergadget.com, brizzly.com, hootsuite.com or seesmic.com. But the big problem is that much of the traffic from Twitter clients will simply show up in your stats as direct visits, essentially becoming completely invisible. And when direct visits make up 40% of your traffic, that’s a huge black hole in your data.

It used to be assumed that direct visits were people who had your website bookmarked in their browser or who were typing your URL directly into their browser’s address bar. The advent of desktop Twitter clients has undermined this assumption completely, and we need to update our thinking about what a ‘direct visit’ is.

This obfuscation of traffic origins is only going to get worse as clients provide access to other tools. Tweetdeck, for example, can no longer be assumed to be a Twitter-only client, because it also allows you to access your LinkedIn, Facebook, MySpace, Google Buzz and Foursquare accounts. So even if you can spot that a referral has come via Tweetdeck, you have no idea whether the user clicked on a link from their Twitter stream, or via Facebook, LinkedIn, etc.

This makes understanding the success of your social media strategy and, in particular, understanding which tools/networks are performing most strongly, nigh on impossible. What if 20% of your traffic is coming from invisible Twitter clients and only 1% comes from Twitter.com? Because the majority of your Twitter traffic is hidden as direct traffic you might end up sensibly but wrongly focusing on the 5% that has come via Facebook.com, thus reworking your strategy to put more effort into Facebook despite the fact it is actually performing poorly in comparison to Twitter.

I recommend to all my clients that they keep an eye on their statistics and that if a tool isn’t working out well for them, that they should ditch it and move on to another. There are so many social networks around that you just can’t be everywhere, you must prioritise your efforts and focus on the networks where you are most likely to reach your target audience. But we need to have clarity in the stats in order to do this.

The scale of this problem is really only becoming clear to me as I type this. For sites with low direct traffic, a bit of fuzziness in the stats isn’t a big deal, but for sites with a lot of direct traffic – and I see some sites with over 40% direct traffic – this is a serious issue. You could potentially have a single referring source making up a huge part of your total traffic, and you’d never know. And as more services provide APIs that can feed more desktop clients, which themselves provide more functionality than the original service itself, the growth of wrongly attributed ‘direct visits’ is only going to accelerate.

Without meaningful numbers, we’re back to the bad old days of gut feeling about whether one strategy is working better than another. I already see people making huge assumptions about how well Facebook is going to work for them, based on the faulty logic that everyone’s in Facebook, ergo by being in Facebook they will reach everyone.

Now, more than ever, we need reliable web stats so that we can make informed decisions, but these numbers are turning out to be like ghosts: our brains see what they want to see, not what is actually there. Even established research institutions like Pew are suffering pareidolia, seeing a phantom Facebook in their flawed numbers.

05.04.11
by

LinkedIn as a source of traffic

Earlier this year I did some work for OldWeather.org, a citizen science project that is transcribing weather and other data from old ships logs. As part of their website progress assessment, I hand-analysed their web traffic referrers to see where people were coming from and whether we were reaching our core communities. One of the things I found was that whilst Facebook sent over two orders of magnitude more visitors than LinkedIn, LinkedIn was responsible for much higher quality visitors. Visitors from LinkedIn visited an average of 17 pages per visit, staying for 34 minutes with a bounce rate of 33%, compared to Facebook’s 1.8 pages per visit, 1:41 minutes on site, and 79% bounce.

The quality difference is stark and indicates that for OldWeather.org, perhaps a bit more promotion in LinkedIn might be in order. But is LinkedIn capable of the same volume of visitors that Facebook can provide? Facebook still provides a far higher overall share of time on site compared to LinkedIn, although on some sites (this one included) a single page view isn’t all that useful in terms of the site being able to fulfil its remit. Lots of single-page-view visitors aren’t as valuable as fewer multi-page-view visitors.

According to Business Insider, recent changes to LinkedIn has upped their ante quite significantly.

Out of nowhere, Business Insider started seeing real referral traffic from LinkedIn last month. […]

LinkedIn product manager Liz Walker tells us the traffic is coming from a bunch of sources – mostly new products like LinkedIn.com/Today, newsletters, and LinkedIn News.

It seems to me that, if these visitor quality stats and this new trend in volume hold true, then LinkedIn is successfully shifting from being a site often marginalised in social media outreach strategies to one that should be central. After all, with traffic it’s not just the volume you should be interested in but the quality of visitors as well.

06.25.10
by

NewsRewired: Marc Reeves, TheBusinessDesk.com

I had to dash out for a lunch meeting, but I was happy to make it back for Marc Reeves keynote. He is the editor of TheBusinessDesk.com West Midlands. I found myself applauding over my morning coffee when I read his recent speech to the CBI. His frankness and lack of sentimentality was refreshing. He started that speech with this statement:

Journalism has no God-given right to exist and journalists are owed a living by nobody.

Here is summary (not word perfect and and many places paraphrase) of his keynote at NewsRewired. It was nice to hear his lack of sentimentality in person:

Three main points, in a niche not enough to be a journalist, have to provide other stuff too. Need to provide more than just information. Niche audience needs a niche approach not a mass market approach. Why has this subject risen up the agenda?

The internet didn’t create this. We publishers forced them into buckets because it was more profitable for us. By lassoing 100 interests, we deluded ourselves into thinking they were a unified whole. It actually created some ineffectual and inefficient advertising models.

Not all niches created equal. Football readers deliver half of the audience. Most of the advertising wasn’t on those football pages. Many readers online bypassed the home page and went straight to football pages.

Now, the internet has revealed the niches in the mass. Financial model is there. Can’t talk about journalistic approaches to serving audiences without talking about how the businesses are organised and how they relate to their audiences, and I include advertises as an audiences.

Are you going to hope that by sitting back and writing about what they do in those niches? No. You have to produce other content that is relevant to their lives. In business terms, it’s always been about the relationship you have with your audiences. You turn acquaintances to transactions. Once you have attracted them, you need to think of other ways to interact with them. Events, get them to sell things to each other, get them to interact with each and sell services to them.

To people who say that I’m just a journalist and I don’t do events, he said: “Tough. That’s the way it is now.” By putting the noisy people called advertising in one room and the studious in another room was a big mistake. Just giving the audience journalism alone is not enough in the long term.

They now have 40,000 registered users across their regions. They hit their target of registered users in Birmingham in four and a half months. They have a daily email in the morning that drives 80% of their traffic.

We ignore the CPM race and refuse to become SEO tarts

At the centre of their business is the intelligence that they have about their users. He tries to personally review every new sign-up. He calls up some and thanks them for registering.

As a small business, we avoid waste. I reject things that don’t deliver audience, revenue or attention.

He says that this might be a way to support journalism in the future.

Q: Do newspapers not get the ‘net?

A: Yes, I’m afraid so. That is not because there aren’t brilliant individuals and editors who do get it. Structurally, I don’t think they can turn themselves around to make money on the internet.

I asked him to follow up with that. It is said you loose a pound in print for a penny online. That’s often true, but he said that the high fixed costs of newspapers – print plant, pensions, staff costs – make it almost impossible to ‘turn that super-tanker around’ and sustain their business with a digital revenue.

He uses Google Analytics to monitor his traffic and track the performance of their morning email.

Q: What is there that helps you re-engineer the cost base?

A: I’m not dragging behind a 100 ton press behind me or having to manufacture the most perishable product daily. Taking that cost out of the business makes all the difference. When we launched in February, it was me and my deputy in a serviced office with two laptops. In terms of when we scale up, we’ll keep on that trajectory.

One thing that stands out is their focus on keeping costs low and developing multiple revenue streams. They hold physical events. They do mail shots for promotion.

05.27.10
by

Life is not a marathon, it’s a series of sprints

If you can get past the slightly rambling intro, this conversation between Jonathan Fields and Tony Schwartz is a fascinating look at what’s wrong with the way we currently tend to work. It really starts to get interesting about 8 minutes in.

Although very focused on American business and culture, pretty much everything they say relates to British and European work culture.
One important idea they discuss, and something I’ve found essential myself, is the idea of pulsing or sprinting when working: to focus for a while and then relax for a bit. This idea is common in athletics, where it’s called the work-rest ratio: “It’s as important to renew energy as it is to spend energy if [you] want to be a consistently great [athletics] performer.”
We forget too easily that the brain is an organ that requires periods of replenishment as much as muscles do. If you work your muscles too hard, they ache, so we learn very early on not to overdo it. Yet we expect our brain to perform at maximum capacity, consistently, throughout our workday. It’s just not possible, yet we don’t allow for this fact in the way that we work.
Schwartz also says, “It’s not the number of hours people work that matters, it’s the value they produce during the hours they work, so stop worrying about how many hours that person spends at their desk, and start figuring out, What can I do to help this person design his life so that when he’s working or she’s working, she’s really working?”
To me this is the essence of what social media is in business is all about. We, as humans, work better when we are socially connected. It fulfils a fundamental human need to be part of a group whose whole is bigger than the sum of its parts. Social media also provides ways to communicate and collaborate more effectively and more easily, to benefit from the wisdom in the crowd. As we become more enmeshed in our community, so our ability to solve problems by drawing upon the resources of that community increases.
Social media is, at the moment, only doing a fraction of what it could for business. It’s an area full of potential and as we start to marry technology, psychology, business and human nature together, we are beginning to find ways to unlock our potential, not just as individuals but as members of a huge social gestalt.
Most businesses using social media at the moment are dabbling, going for the easy, obvious wins like marketing or some internal Wikipedia clone. We need more business executives to be brave, to think about their business as a multi-human organism that has its own needs and that isn’t being properly fed by current business practices and cultures.
When I look at what could be done, how we could use social media to really change our work environments in to something more effective, more enjoyable, I really do think we have a long, long road ahead of us. Change is often slow and incremental. We need some businesses to take a deep breath and leap, to remake their internal culture, to be more human, using social media as the agent of change.
But ultimately, I think what we’ll see is the old cultures dying off as new, nimble, socially aware businesses rise up in their stead. This new era of socially capable business is only just now dawning.

05.26.10
by

Conflict of interest: Success vs the user

I’m very wary of what sort of metrics and definitions of success are used to decide whether a project is working or not. To often, the wrong metrics and definitions are used, resulting in bad managerial decisions that are based on flawed assumptions.
A couple of good posts about how metrics and definitions of success (and, therefore, business models) can work against the user: OKCupid talks about why you should never pay for online dating, and Joshua Porter points out a paragraph in one of Mike Davidson’s posts which explains why companies’ iPhone/iPad apps are often better than their websites. In short, on a mobile app they don’t have the opportunity to finagle the user experience to artificially bump up their metrics.
In both cases, you have a situation where the metrics and definitions of success upon which the business model relies distort the user experience by forcing them to take actions which are not necessarily in their best interests. Indeed in these cases, a swift and satisfying experience for the user is damaging to the business providing it.
When you’re putting together a social media project, think first about what the most beneficial outcome for your users would be. Then figure out it can form the basis of a business model (hint: your income/ROI may be orthogonal to your desired user outcome) and then how that can be measured.
Do not start with a metric, build a business model on top of it, and then force the user to have a shoddy experience for the sake of your bottom line. And yes, this applies just as much to enterprise social media as any other sort. Don’t start thinking that ‘number of edits’ on a wiki is a definition of success, because that just means you’ll push people into more pointless editing and will take your focus of signs of real success, e.g. people being able to achieve their goals more quickly and more efficiently.

05.20.10
by

Fun makes for passionate users

How much enterprise software is truly fun to use? Aarron Walter discusses the importance of fun in his article Emotional Interface Design: The Gateway to Passionate Users. It’s a very interesting read with some enlightening examples.

But to take the ball and run with it a bit, I think ‘fun’ is one reason that people who use social media can get so passionate about it. We engage much more with tasks that are fun and enjoyable, and we work better on projects where we are working with people who are fun. Just think about the tasks on your to-do list, and think about the ones that you find fun. I bet they’re the ones you actually want to do!

For me, blogging is fun. Working on a wiki is fun. Setting up a Kickstarter project is fun. Heaven forfend, but I even like playing with numbers in spreadsheets on Google Docs. (Don’t tell anyone, but I love setting up spreadsheets with formulas that suck data from one cell, transform it in some way and then spit out a number in another.)

Putting my numbergeekiness aside, the one thing those tools have in common is the presence of other people. The fun to be had in writing a blog increases the more other people engage with it. Wikis are both productive and fun when you’re working with other people on achieving a shared goal. Kickstarter is fun not just because it offers the opportunity to do cool projects, but because you’re doing that cool project with the support of other people. GoogleDocs allow me to collaborate with other people and even discuss the document in real time whilst we’re working on it.

Other people make things fun. Fun things are things we want to do, and keep on doing. The more we want to do something, the better we get at doing it. The more we enjoy a task, the better we get at doing it, the more efficient and productive we becomes.

Which begs the question: Can we make work more fun? Of course we can. And we should.

04.01.10
by

The Tyranny of the Explicit

Johnnie Moore has a great podcast episode talking with Viv McWaters and Roland Harwood on how an undue focus on metrics can get in the way of real thought and understanding. I see this frequently myself, too, when people want to focus on ‘return on investment’ or ‘success metrics’ for social media at the cost of understanding the intangible results, which are actually more important than the measurable ones. There are some great nuggets, so well worth listening to. I particularly liked Johnnie’s discussion of how learning has become codified in unrealistic ways and how that relates to best practice documents that don’t get practised.

04.01.10
by

The lure of the partial post

Friend and colleague Stephanie Booth writes about the blogazine, which I’ve covered here already, and the frustration she feels when faced with blogs that only post excerpts to their front page (and, I’d add, RSS feeds). I want to pick up on the point about partial posts and want to say in no uncertain terms:

Partial posts or excerpts are bad practice.

They are bad practice for media outlets, but they are especially bad practice for business blogs. As Steph says, partial posts put a barrier between your content and your readers and although it’s a low barrier, just a click high, it’s still a barrier. Trying to artificially inflate page views by forcing people to click through from the front page, or from RSS, is nothing more than an attempt to fake greater popularity. It doesn’t mean that you actually have more readers, just that they have to click twice. Like Steph, I seriously doubt that it makes any difference to SEO, and if you’re willing to sacrifice user experience for a potentially tiny bump in your search engine ranking, what does that say about how you treat your customers?

12.24.09
by

Metrics, Part 4: Subjective measurements

(If you haven’t already read them, you might like to take a look at Part 1: The webstats legacy, Metrics, Part 2: Are we measuring the right things?) and Metrics, Part 3: What are your success criteria?)

In the last instalment of this series I mentioned that sometimes there just aren’t objective metrics that we can use to help us understand the repercussions of our actions. Yet much of what we try to achieve with social media projects is exactly this sort of unmeasurable thing.

No amount of understanding of page views, for example, is going to tell us how the people who have viewed that page feel about it. Did they come because they were interested? Or because they were outraged? Is your comment community a healthy one or a pit of raging hatred? Are your staff better able to collaborate now you have a wiki or are they finding it difficult to keep another datastore up to date?

There are two ways round this:

  • Surveys
  • Subjective measurement scales

Surveys are sometimes the only way you can get a sense for how well a social media project is going. All the metrics in the world won’t tell you if your staff are finding their internal blogs useful or burdensome. Random anecdotes are liable to mislead as you’ll end up relying on either the vocal evangelists who will give you an overly rosy picture, or the vocal naysayers who will give you an overly pessimistic picture. The truth is likely to be in the middle somewhere, and the only way that you can find out where is to ask people.

Survey questions need to be very carefully constructed, however, to ensure that they are not leading people to answer a certain way. At the very least, make sure that questions are worded in a neutral way and that you cover all bases for the answer options you give. Test and retest surveys as it’s so easy to get something crucial wrong!

The second way to try and measure subjective metrics is to create a scale and regularly assess activity against that scale. If you were assessing the comments in your customer-facing community, for example, you might consider a scale like this:

?????…..Lively discussion, readers are replying to each other, tone is polite, constructive information is shared

????………Moderate amount of discussion, readers replying to each other, tone is polite, some useful information shared

???………….Little discussion, readers reply only to author, tone is mainly polite, not much information shared

??……………..Discussion is moribund OR Tone of discussion negative, tone is impolite, no information shared

?…………………Abusive discussion OR Discussion is just a torrent of “me too” comments

?…………………No discussion

The idea here isn’t to create an enormous cognitive load but to try and have a consistent understanding of what we mean when we rate something 3 out of 5. This means keeping scales clear and simple, and avoiding any ambiguity such as language which could be misunderstood or which has an inherent value judgement that could sway an assessment.

I would also suggest that valuable data would be compiled by having a varied group of people rating on a regular basis and then averaging scores. That would hopefully smooth out any variation in interpretation of the scale or personal opinion.

Again, I’m going to stress that both these methods need to be put in place and measurement started before a project begins. Thinking ahead is just so worth the effort.

In all honesty, I’ve never had a client do either surveys or subjective scales. Mainly because none of them have ever really given enough thought to metrics before they start a project. It’s a shame because with services like Survey Monkey, it’s really not hard to do.

12.21.09
by

Metrics, Part 3: What are your success criteria?

(If you haven’t already read them, you might like to take a look at Part 1: The webstats legacy and Metrics, Part 2: Are we measuring the right things?)

It’s never been more true to say that just because we can measure something it doesn’t mean we should. The temptation to amass as many stats as possible about our social media projects, in the hope that somewhere in the numbers lies enlightenment, is almost irresistible. Instead, we need to do the opposite: Measure only the things that can tell us something useful. And some of those measurements may not actually come from social media at all.

To know what to measure, we first need to understand the strategic goals of the project. This is the 60,000 ft view, the “We want increased profitability” or “We want to be more productive” view. These aren’t easily measured directly. Profitability, for example, may be improved by a whole host of actions taken by the company as well as by market forces, so teasing out which bit is down to a specific social media project could be very difficult.

Instead, strategic goals provide us with a context for tactical goals. Increased productivity, for example, may mean decreasing email use, decreasing hours spent in meetings, improving collaboration, improving communication, decreasing duplicated projects, and improving employee engagement.

Of these tactical goals, some are easier to measure than others. Leisa Reichelt has written a great post on the importance of measurement and criteria for success in which she says:

Some success criteria are immediately apparent and easy to measure, for example return visitors, increased membership, activity or sales. Ideally you want to put some numbers around what you’d consider would define this project as ‘successful’, but even just identifying the metrics that you will use to judge the success of the project is a good start.

Some success criteria are less easy to ‘measure’ but don’t let that discourage you. Often for these kinds of criteria I’ll use a round of research to determine whether or not we’ve been successful – those things that are difficult to quantify are often quite easy to examine using qualitative research. I find myself more and more using a last round of research to ‘check off’ the less quantifiable success criteria for projects.

I think of these two types of success criteria as objective and subjective:

  • Objective criteria map fairly cleanly to something you can measure. For example, you can measure how many emails are sent and received and so can see if your social media project is reducing email flow.
  • Subjective criteria do not map cleanly to any metric. For example, it’s hard to define, let alone measure, collaboration.

Sometimes one can get creative around subjective criteria and create a new metric that can shed light on matters, but often there isn’t much more than gut feeling to go on. In that case, it is worth asking our gut how it feels on a regular basis so that we can at least look back dispassionately rather than trying to remember how things felt six months ago. (More on this in a later post.)

For all measures, it’s important to understand what the numbers are really telling you and to discard any measurements that could be in any way misleading (cf Part 2).

A good workflow for this whole process might be:

  • Set out strategic and tactical goals
  • List objective and subjective criteria for success
  • Map criteria to measurable metrics
  • Discard misleading metrics
  • Discard unimportant metrics
  • Identify desired trends
  • Start measuring

One word of warning: Beware numerical targets. It’s often not possible to know how big of a change you need to create in order to meet your goals. And in many cases, social tools scale best when they scale slowly. Rapid change can even destroy the very thing you’re trying to create (especially when you’re looking at community building). Numerical targets are often nothing better than fairytales that may or may not one day resemble reality.

The final thing to remember is to start taking measurements before the project launches. It might seem like a no-brainer, but in my experience it’s common for companies to forget that without information on starting conditions, there’ll be nothing to compare to.