(If you haven’t already read them, you might like to take a look at Part 1: The webstats legacy and Metrics, Part 2: Are we measuring the right things?)

It’s never been more true to say that just because we can measure something it doesn’t mean we should. The temptation to amass as many stats as possible about our social media projects, in the hope that somewhere in the numbers lies enlightenment, is almost irresistible. Instead, we need to do the opposite: Measure only the things that can tell us something useful. And some of those measurements may not actually come from social media at all.

To know what to measure, we first need to understand the strategic goals of the project. This is the 60,000 ft view, the “We want increased profitability” or “We want to be more productive” view. These aren’t easily measured directly. Profitability, for example, may be improved by a whole host of actions taken by the company as well as by market forces, so teasing out which bit is down to a specific social media project could be very difficult.

Instead, strategic goals provide us with a context for tactical goals. Increased productivity, for example, may mean decreasing email use, decreasing hours spent in meetings, improving collaboration, improving communication, decreasing duplicated projects, and improving employee engagement.

Of these tactical goals, some are easier to measure than others. Leisa Reichelt has written a great post on the importance of measurement and criteria for success in which she says:

Some success criteria are immediately apparent and easy to measure, for example return visitors, increased membership, activity or sales. Ideally you want to put some numbers around what you’d consider would define this project as ‘successful’, but even just identifying the metrics that you will use to judge the success of the project is a good start.

Some success criteria are less easy to ‘measure’ but don’t let that discourage you. Often for these kinds of criteria I’ll use a round of research to determine whether or not we’ve been successful – those things that are difficult to quantify are often quite easy to examine using qualitative research. I find myself more and more using a last round of research to ‘check off’ the less quantifiable success criteria for projects.

I think of these two types of success criteria as objective and subjective:

• Objective criteria map fairly cleanly to something you can measure. For example, you can measure how many emails are sent and received and so can see if your social media project is reducing email flow.
• Subjective criteria do not map cleanly to any metric. For example, it’s hard to define, let alone measure, collaboration.

Sometimes one can get creative around subjective criteria and create a new metric that can shed light on matters, but often there isn’t much more than gut feeling to go on. In that case, it is worth asking our gut how it feels on a regular basis so that we can at least look back dispassionately rather than trying to remember how things felt six months ago. (More on this in a later post.)

For all measures, it’s important to understand what the numbers are really telling you and to discard any measurements that could be in any way misleading (cf Part 2).

A good workflow for this whole process might be:

• Set out strategic and tactical goals
• List objective and subjective criteria for success
• Map criteria to measurable metrics
• Discard misleading metrics
• Discard unimportant metrics
• Identify desired trends
• Start measuring

One word of warning: Beware numerical targets. It’s often not possible to know how big of a change you need to create in order to meet your goals. And in many cases, social tools scale best when they scale slowly. Rapid change can even destroy the very thing you’re trying to create (especially when you’re looking at community building). Numerical targets are often nothing better than fairytales that may or may not one day resemble reality.

The final thing to remember is to start taking measurements before the project launches. It might seem like a no-brainer, but in my experience it’s common for companies to forget that without information on starting conditions, there’ll be nothing to compare to.

(If you haven’t already read it, you might like to take a look at Part 1: The Webstats Legacy.)

Anand Giridharadas asks in the New York Times, Are metrics blinding our perception?. Giridharadas begins by talking about the Trixie Telemetry company which takes data about a baby’s naps, nappy changes and feed times and turns it into charts, graphs and analyses to “help parents make data-based decisions”. He then goes on to say:

Self-quantification of the Trixie Telemetry kind is everywhere now. Bedposted.com quantifies your sexual encounters. Kibotzer.com quantifies your progress toward goals like losing weight. Withings, a French firm, makes a Wi-Fi-enabled weighing scale that sends readings to your computer to be graphed. There are tools to measure and analyze the steps you take in a day; the abundance and ideological orientation of your friends; the influence of your Twitter utterances; what you eat; the words you most use; your happiness; your success in spurning cigarettes.

Welcome to the Age of Metrics — or to the End of Instinct. Metrics are everywhere. It is increasingly with them that we decide what to read, what stocks to buy, which poor people to feed, which athletes to recruit, which films and restaurants to try. World Metrics Day was declared for the first time this year.

But measure the wrong thing and you end up doing the wrong thing:

Will metrics encourage charities to work toward the metric (acres reforested), not the underlying goal (sustainability)? […] Trees are killed because the sales from paper are countable, while a forest’s worth is not.

The same is true in social media. Count the wrong thing and you’ll do the wrong thing. As Stephanie Booth says, in the second video in this post:

As soon as you start converting behaviours into numbers then people adapt their behaviour to have good numbers.

She goes on to say that some of her clients believe that the number of comments they have on a blog post is a measure of success, but because of this they become obsessed with getting people to comment:

So you’re going to write posts which make people react or you’re going to encourage people to have chatty conversations in your comments. That’s really great, you get lots of comments, but does it mean that what you’re providing is really more valuable? […] I don’t believe that more is always better, that more conversation is always better. It’s “Is it relevant?” And that’s something that we do not know how to measure in numbers.

If the key metric for assessing success is a simplistic one like ‘page views’ or ‘unique users’ or ‘comments’, the emphasis in your web 2.0 strategy will be on creating something populist instead of something that meets a business need.

Let’s say you’re in eCommerce and you sell pet supplies. Your business goal is not ‘get more people onto our website’, it is ‘get more people buying pet supplies from our website’. The two are very different indeed. A company that believes that they need to just lots and lots of people through the virtual door will focus on anything that might get them more attention and traffic. A company that understands they need to attract the right people will focus on communicating with passionate pet lovers who arrive at the site primed to buy.

This is why niche blogs can command higher advertising rates than general news sites. Advertisers can see that more of the people who click their ads will actually buy their products and are willing to pay more for these higher quality visitors.

Equally, let’s say you want to ‘improve collaboration’ internally and to that end you start a wiki. You start measuring activity on the wiki and focus on ‘edits per user’ as a key metric. You encourage people to edit more, but the quality and amount of collaboration doesn’t increase as you expected. Why? Because people learnt that changing a single typo boosts their ‘edits per user’ count and took a lot less effort than creating a new page, engaging with a co-worker or making a substantive change. Focusing on the wrong numbers changes the wrong behaviour.

In order to think about metrics, you need to know exactly what you’re using social media for. Figure that out and you’re halfway there.