Metrics, Part 4: Subjective measurements

(If you haven’t already read them, you might like to take a look at Part 1: The webstats legacy, Metrics, Part 2: Are we measuring the right things?) and Metrics, Part 3: What are your success criteria?)

In the last instalment of this series I mentioned that sometimes there just aren’t objective metrics that we can use to help us understand the repercussions of our actions. Yet much of what we try to achieve with social media projects is exactly this sort of unmeasurable thing.

No amount of understanding of page views, for example, is going to tell us how the people who have viewed that page feel about it. Did they come because they were interested? Or because they were outraged? Is your comment community a healthy one or a pit of raging hatred? Are your staff better able to collaborate now you have a wiki or are they finding it difficult to keep another datastore up to date?

There are two ways round this:

  • Surveys
  • Subjective measurement scales

Surveys are sometimes the only way you can get a sense for how well a social media project is going. All the metrics in the world won’t tell you if your staff are finding their internal blogs useful or burdensome. Random anecdotes are liable to mislead as you’ll end up relying on either the vocal evangelists who will give you an overly rosy picture, or the vocal naysayers who will give you an overly pessimistic picture. The truth is likely to be in the middle somewhere, and the only way that you can find out where is to ask people.

Survey questions need to be very carefully constructed, however, to ensure that they are not leading people to answer a certain way. At the very least, make sure that questions are worded in a neutral way and that you cover all bases for the answer options you give. Test and retest surveys as it’s so easy to get something crucial wrong!

The second way to try and measure subjective metrics is to create a scale and regularly assess activity against that scale. If you were assessing the comments in your customer-facing community, for example, you might consider a scale like this:

?????…..Lively discussion, readers are replying to each other, tone is polite, constructive information is shared

????………Moderate amount of discussion, readers replying to each other, tone is polite, some useful information shared

???………….Little discussion, readers reply only to author, tone is mainly polite, not much information shared

??……………..Discussion is moribund OR Tone of discussion negative, tone is impolite, no information shared

?…………………Abusive discussion OR Discussion is just a torrent of “me too” comments

?…………………No discussion

The idea here isn’t to create an enormous cognitive load but to try and have a consistent understanding of what we mean when we rate something 3 out of 5. This means keeping scales clear and simple, and avoiding any ambiguity such as language which could be misunderstood or which has an inherent value judgement that could sway an assessment.

I would also suggest that valuable data would be compiled by having a varied group of people rating on a regular basis and then averaging scores. That would hopefully smooth out any variation in interpretation of the scale or personal opinion.

Again, I’m going to stress that both these methods need to be put in place and measurement started before a project begins. Thinking ahead is just so worth the effort.

In all honesty, I’ve never had a client do either surveys or subjective scales. Mainly because none of them have ever really given enough thought to metrics before they start a project. It’s a shame because with services like Survey Monkey, it’s really not hard to do.

links for 2009-12-23

Developing etiquette

Technological change is outpacing our ability as a society to negotiate and agree upon acceptable behaviour sets for each new tool type. The mobile is a great example of this: Some of us think that it’s rude to sit in a cinema yapping away on your mobile, whilst others feel that it’s not only acceptable but also their right.

Where social media steals a march on mobile phones is that we can use the very tools we are discussing to negotiate what acceptable behaviour means. What is rude on a social network? What is expected on a wiki? And what is good etiquette for comments?

Justine Larbalestier has a great post outlining what people engaging in comment threads should do before plunging in, including:

  • Read the entire post before commenting. Nothing is more annoying to a blogger than to have someone say “But why did you not mention French beanbags?” when you have just spent six paragraphs doing exactly that.


  • Do not explode on to a comment thread in a whirl of fire and outrage. Particularly don’t do this if all the discourse up to that point has been calm and measured.

Maybe it’s because I’ve been online for so long, but it does seem to me that most of these are no-brainers, yet they still appear to be news to some! Will the day ever come where online etiquette is pervasive or are manners a thing of the past?

The power of ecosystems

It used to be that I judged a new social tool, service or application purely on intrinsic qualities such as functionality, usability, design or utility. In an increasingly competitive social media landscape, where even niches are getting crowded, what is it that makes one tool stand out above the others?

Ecosystem. Plain and simple. What kind of API does the tool provide? How vibrant is the developer community? What other tools and services are building up around it?

Twitter, for example, has a really vibrant ecosystem, full of tools that allow you to post pictures, track compressed links, analyse statistics, manage your accounts, send Tweets longer than 140 characters. The ecosystem is so vibrant that a service devoted just to cataloguing it has sprung up:

Wordpress is another tool with a fabulous ecosystem, not just in terms of plug-ins and themes, but also in terms of developers. I’ve never had a problem finding WP developers to help me out when I need a hand wrangling the finer points of WP admin.

The Delicious ecosystem, on the other hand, appears to be moribund. The Firefox add-on that I used to rely on, Delicious Complete, is now defunct (does not function in FF3) and I can’t find an alternative that allows me to manage multiple accounts. There are a handful of dedicated add-ons for Firefox, but beyond that the wider developer community is either invisible or just doesn’t exist. New tools, like Instapaper, that could tie in with Delicious don’t. That’s a real shame because Delicious is one of my favourite tools, but it simply isn’t that easy to embed it in my working processes.

I know that there may be some very good reasons why some tools have better ecosystems than others, but as I’m not a developer and have never looked at the barriers to entry for working with these different tools, I can’t speak to that side of the debate.

But ecosystem is very important to users not just in deciding which tools to use, but whether to be loyal. The cost of switching from Twitter or WordPress to a competitor is quite high. It’s not just a matter of swapping one service for another, but of having to start again with the community, whether that’s moving your own social network over or finding new people to worth with on the new platform. The cost of moving from Delicious, however, would be relatively low. If someone offered a better service with better tools and even a small ecosystem that showed promise, I wouldn’t hesitate to migrate my data.

Metrics, Part 3: What are your success criteria?

(If you haven’t already read them, you might like to take a look at Part 1: The webstats legacy and Metrics, Part 2: Are we measuring the right things?)

It’s never been more true to say that just because we can measure something it doesn’t mean we should. The temptation to amass as many stats as possible about our social media projects, in the hope that somewhere in the numbers lies enlightenment, is almost irresistible. Instead, we need to do the opposite: Measure only the things that can tell us something useful. And some of those measurements may not actually come from social media at all.

To know what to measure, we first need to understand the strategic goals of the project. This is the 60,000 ft view, the “We want increased profitability” or “We want to be more productive” view. These aren’t easily measured directly. Profitability, for example, may be improved by a whole host of actions taken by the company as well as by market forces, so teasing out which bit is down to a specific social media project could be very difficult.

Instead, strategic goals provide us with a context for tactical goals. Increased productivity, for example, may mean decreasing email use, decreasing hours spent in meetings, improving collaboration, improving communication, decreasing duplicated projects, and improving employee engagement.

Of these tactical goals, some are easier to measure than others. Leisa Reichelt has written a great post on the importance of measurement and criteria for success in which she says:

Some success criteria are immediately apparent and easy to measure, for example return visitors, increased membership, activity or sales. Ideally you want to put some numbers around what you’d consider would define this project as ‘successful’, but even just identifying the metrics that you will use to judge the success of the project is a good start.

Some success criteria are less easy to ‘measure’ but don’t let that discourage you. Often for these kinds of criteria I’ll use a round of research to determine whether or not we’ve been successful – those things that are difficult to quantify are often quite easy to examine using qualitative research. I find myself more and more using a last round of research to ‘check off’ the less quantifiable success criteria for projects.

I think of these two types of success criteria as objective and subjective:

  • Objective criteria map fairly cleanly to something you can measure. For example, you can measure how many emails are sent and received and so can see if your social media project is reducing email flow.
  • Subjective criteria do not map cleanly to any metric. For example, it’s hard to define, let alone measure, collaboration.

Sometimes one can get creative around subjective criteria and create a new metric that can shed light on matters, but often there isn’t much more than gut feeling to go on. In that case, it is worth asking our gut how it feels on a regular basis so that we can at least look back dispassionately rather than trying to remember how things felt six months ago. (More on this in a later post.)

For all measures, it’s important to understand what the numbers are really telling you and to discard any measurements that could be in any way misleading (cf Part 2).

A good workflow for this whole process might be:

  • Set out strategic and tactical goals
  • List objective and subjective criteria for success
  • Map criteria to measurable metrics
  • Discard misleading metrics
  • Discard unimportant metrics
  • Identify desired trends
  • Start measuring

One word of warning: Beware numerical targets. It’s often not possible to know how big of a change you need to create in order to meet your goals. And in many cases, social tools scale best when they scale slowly. Rapid change can even destroy the very thing you’re trying to create (especially when you’re looking at community building). Numerical targets are often nothing better than fairytales that may or may not one day resemble reality.

The final thing to remember is to start taking measurements before the project launches. It might seem like a no-brainer, but in my experience it’s common for companies to forget that without information on starting conditions, there’ll be nothing to compare to.

links for 2009-12-19

links for 2009-12-18

Let’s just not build teams

Robert Brook writes an impassioned post about his distaste for artificial games and the overuse of competition as a motivator. I’ll write more on that later, but I wanted to pick up on one thing that Robert says at the bottom of his post:

Lindsay Marshall, of Bifurcated Rivets fame – I’m a long-time reader – reminds me of another grim manifestation: team building. Real teams come together organically, or emerge – they are rarely, if ever, built. That false application and bonhomie is dreadfully thin stuff, especially in comparison to emergent groups.

This reminds me of a story a friend of mine once told me about how he was chastised by his boss for not being a ‘team player’ because he didn’t join in office conversations about football.

Team building is not about creating groupthink. The kind of mutual respect and understanding that underpins the best teams is something that can’t be forced.

So here’s a thought: How about we use social media to build internal communities from which teams emerge spontaneously? How about providing people with the means and opportunity to get to know each other, understand each others’ skills and ways of working, and then let teams coalesce around projects? Turn Google’s 20% Time on its head: Instead of giving staff one day a week to work on whatever project they want, give them four days to work on projects that they get to choose from a list of things that the business needs doing (which they also get to contribute to), and one day where everyone has to do unavoidable unpopular tasks. If you share the good, you have to share the bad, after all!

What kind of company might that create? I would hazard a guess that it would be highly creative, innovative, productive and successful. It would be a company that retained its best staff because they are happier there than they could ever be in a traditional management structure. And if you only have one day to do chores, then the needless administrivia that gets created out of nowhere and which serves no purpose other than to feed the bureaucracy will just die off.

Who’s going to give it a go, then?