(If you haven’t already read them, you might like to take a look at Part 1: The webstats legacy, Metrics, Part 2: Are we measuring the right things?) and Metrics, Part 3: What are your success criteria?)

In the last instalment of this series I mentioned that sometimes there just aren’t objective metrics that we can use to help us understand the repercussions of our actions. Yet much of what we try to achieve with social media projects is exactly this sort of unmeasurable thing.

No amount of understanding of page views, for example, is going to tell us how the people who have viewed that page feel about it. Did they come because they were interested? Or because they were outraged? Is your comment community a healthy one or a pit of raging hatred? Are your staff better able to collaborate now you have a wiki or are they finding it difficult to keep another datastore up to date?

There are two ways round this:

• Surveys
• Subjective measurement scales

Surveys are sometimes the only way you can get a sense for how well a social media project is going. All the metrics in the world won’t tell you if your staff are finding their internal blogs useful or burdensome. Random anecdotes are liable to mislead as you’ll end up relying on either the vocal evangelists who will give you an overly rosy picture, or the vocal naysayers who will give you an overly pessimistic picture. The truth is likely to be in the middle somewhere, and the only way that you can find out where is to ask people.

Survey questions need to be very carefully constructed, however, to ensure that they are not leading people to answer a certain way. At the very least, make sure that questions are worded in a neutral way and that you cover all bases for the answer options you give. Test and retest surveys as it’s so easy to get something crucial wrong!

The second way to try and measure subjective metrics is to create a scale and regularly assess activity against that scale. If you were assessing the comments in your customer-facing community, for example, you might consider a scale like this:

?????…..Lively discussion, readers are replying to each other, tone is polite, constructive information is shared

????………Moderate amount of discussion, readers replying to each other, tone is polite, some useful information shared

???………….Little discussion, readers reply only to author, tone is mainly polite, not much information shared

??……………..Discussion is moribund OR Tone of discussion negative, tone is impolite, no information shared

?…………………Abusive discussion OR Discussion is just a torrent of “me too” comments

?…………………No discussion

The idea here isn’t to create an enormous cognitive load but to try and have a consistent understanding of what we mean when we rate something 3 out of 5. This means keeping scales clear and simple, and avoiding any ambiguity such as language which could be misunderstood or which has an inherent value judgement that could sway an assessment.

I would also suggest that valuable data would be compiled by having a varied group of people rating on a regular basis and then averaging scores. That would hopefully smooth out any variation in interpretation of the scale or personal opinion.

Again, I’m going to stress that both these methods need to be put in place and measurement started before a project begins. Thinking ahead is just so worth the effort.

In all honesty, I’ve never had a client do either surveys or subjective scales. Mainly because none of them have ever really given enough thought to metrics before they start a project. It’s a shame because with services like Survey Monkey, it’s really not hard to do.

(If you haven’t already read them, you might like to take a look at Part 1: The webstats legacy and Metrics, Part 2: Are we measuring the right things?)

It’s never been more true to say that just because we can measure something it doesn’t mean we should. The temptation to amass as many stats as possible about our social media projects, in the hope that somewhere in the numbers lies enlightenment, is almost irresistible. Instead, we need to do the opposite: Measure only the things that can tell us something useful. And some of those measurements may not actually come from social media at all.

To know what to measure, we first need to understand the strategic goals of the project. This is the 60,000 ft view, the “We want increased profitability” or “We want to be more productive” view. These aren’t easily measured directly. Profitability, for example, may be improved by a whole host of actions taken by the company as well as by market forces, so teasing out which bit is down to a specific social media project could be very difficult.

Instead, strategic goals provide us with a context for tactical goals. Increased productivity, for example, may mean decreasing email use, decreasing hours spent in meetings, improving collaboration, improving communication, decreasing duplicated projects, and improving employee engagement.

Of these tactical goals, some are easier to measure than others. Leisa Reichelt has written a great post on the importance of measurement and criteria for success in which she says:

Some success criteria are immediately apparent and easy to measure, for example return visitors, increased membership, activity or sales. Ideally you want to put some numbers around what you’d consider would define this project as ‘successful’, but even just identifying the metrics that you will use to judge the success of the project is a good start.

Some success criteria are less easy to ‘measure’ but don’t let that discourage you. Often for these kinds of criteria I’ll use a round of research to determine whether or not we’ve been successful – those things that are difficult to quantify are often quite easy to examine using qualitative research. I find myself more and more using a last round of research to ‘check off’ the less quantifiable success criteria for projects.

I think of these two types of success criteria as objective and subjective:

• Objective criteria map fairly cleanly to something you can measure. For example, you can measure how many emails are sent and received and so can see if your social media project is reducing email flow.
• Subjective criteria do not map cleanly to any metric. For example, it’s hard to define, let alone measure, collaboration.

Sometimes one can get creative around subjective criteria and create a new metric that can shed light on matters, but often there isn’t much more than gut feeling to go on. In that case, it is worth asking our gut how it feels on a regular basis so that we can at least look back dispassionately rather than trying to remember how things felt six months ago. (More on this in a later post.)

For all measures, it’s important to understand what the numbers are really telling you and to discard any measurements that could be in any way misleading (cf Part 2).

A good workflow for this whole process might be:

• Set out strategic and tactical goals
• List objective and subjective criteria for success
• Map criteria to measurable metrics
• Discard misleading metrics
• Discard unimportant metrics
• Identify desired trends
• Start measuring

One word of warning: Beware numerical targets. It’s often not possible to know how big of a change you need to create in order to meet your goals. And in many cases, social tools scale best when they scale slowly. Rapid change can even destroy the very thing you’re trying to create (especially when you’re looking at community building). Numerical targets are often nothing better than fairytales that may or may not one day resemble reality.

The final thing to remember is to start taking measurements before the project launches. It might seem like a no-brainer, but in my experience it’s common for companies to forget that without information on starting conditions, there’ll be nothing to compare to.