The plural of anecdote is not data

The City, and sections of the media, are getting a touch overexcited by a “research note” written for Morgan Stanley by Matthew Robson, a 15 year old on work experience. The Guardian said:

The US investment bank’s European media analysts asked Matthew Robson, an intern from a London school, to write a report on teenagers’ likes and dislikes, which made the Financial Times’ front page today.

His report, that dismissed Twitter and described online advertising as pointless, proved to be “one of the clearest and most thought-provoking insights we have seen ā€“ so we published it”, said Edward Hill-Wood, executive director of Morgan Stanley’s European media team.

“We’ve had dozens and dozens of fund managers, and several CEOs, e-mailing and calling all day.” He said the note had generated five or six times more responses than the team’s usual research.

The research note itself can be read on The Guardian’s site.

I’m going to start by giving Robson the kudos that he deserves. He has written a very well thought out piece which describes the media habits of him and his friends. In no way do I want to criticise a teenager for being thoughtful, engaged and articulate.

But one has to put this research note into context: This is one teen describing his experience. It is not a reliable description of all teens’ attitudes and behaviours, yet both Morgan Stanley and the media seem to be treating it as if Robson has Spoken The One Great Truth. “Twitter is not for teens, Morgan Stanley told by 15-year-old expert” coos The Guardian. “Note by ‘teenage scribbler’ causes sensation” says the FT in astonishment.

Neither Morgan Stanley nor the media seem to be able to tell the difference between anecdote and data. This “research note” is more note than research, and it should not be taken to be representative of all teens. A teenager in a rural setting, or in an inner city estate, or one who feels socially excluded from web culture will have a very different experience than a teen who’s well-connected enough to get himself an internship at Morgan Stanley.

What is worrying about this is not Robson’s note: He’s simply doing what most teens (and most adults) do, which is to extrapolate from his own and his friends’ experience to form generalisations about the world around him. It’s a very human thing to do, but the important thing about businesses like Morgan Stanley, and the journalists who write about them, is that they are supposed to be able to tell the difference between data and generalisations. Yet they don’t seem able to sort the wheat from the chaff. It seems yet another symptom of the group-think in the media and financial sector that led to the Great Recession, rather than an indication that we have learned anything from it.

Sarah Perez on ReadWriteWeb says:

Matthew Robson, a 15-year-old intern at analyst firm Morgan Stanley recently helped compile a report about teenage media habits. Overnight, his findings have become a sensation…which goes to show that people are either obsessed with what “the kids” are into or there’s a distinctive lack of research being done on this demographics’ media use. Robson’s report isn’t even based on any sort of statistical analysis, just good ol’ fashioned teenage honesty. And what was it that he said to cause all this attention? Only that teens aren’t into traditional media (think TV, radio, newspapers) and yet they’re eschewing some new media, too, including sites like Twitter.

Well, research has been done. danah boyd has done some excellent research into the use of the web by teens – it’s her speciality and she’s one of the foremost experts in this area. Her research would help Morgan Stanley understand the teen demographic much more clearly than any single anecdote, however well written, ever can. The fact that they haven’t ever had a clear insight into the teen demographic would seem to imply that their existing researchers and analysts aren’t doing their jobs properly. The information is out there, a lot of it is freely available, and all that remains is for someone to read it and write the report.

This story also feeds into the concept of the ‘digital native’ which, as I’ve blogged before, is a very poor way to talk about a very diverse section of the population. But because this report fits in with widely-held assumptions about teens and technology – not only does it describe ‘digital natives’, it’s written by one too – it’s immediately accepted without query or question. Morgan Stanley and the media both seem to be more interested in having their biases validated than they are in exploring the evidence to see where it leads them. Sadly, it seems that neither have been spending enough time watching CSI and drawing from it key lessons about assumptions, evidence and how to draw conclusions.

If I had been Matthew Robson’s boss at Morgan Stanley, on receiving his report I would have praised him on his good work and then asked him to look for evidence to either support or refute his points. That would have been an interesting exercise for Robson, and would have led to a research note that actually had some research in it. Instead, Morgan Stanley seem to have taken his work as gospel. I wonder why. Perhaps it was because they thought that, as a 15 year old, he’s privy to the inner workings of mysterious teen minds, a High Priest in the Digital Native Mythology?

If I relied on Morgan Stanley for anything, I’d be rather concerned right now regarding their lack of critical thinking.

38 thoughts on “The plural of anecdote is not data

  1. Actually, what I was thinking was, what a brilliant bit of PR. Get an intern to write what everyone knows, write a press release and you’re all over the press for free.

  2. Absolutely.

    Matthew Robson’s thoughts are actually not that far away from current research, but the idea of basing corporate decisions based on a sample-size of one is scary.

  3. I read it very much as a story about the reaction of financiers to a report written by a 15 year-old – that they’d clutch at anything to find out what the kids think. That’s certainly the sense of the FT piece. A story about a teen writing ‘with authority’ on media habits isn’t much of a story.

    As far as I can tell, this is an agency piece, probably from PA, given that the same quote about “dozens and dozens of fund managers” turns up in all the daily stories I’ve looked at on this, and they all, apart from the FT, lead on Twitter.

  4. Suw, as far as I can see (based on actually owning teenage kids and observing their friends too) he is largely correct for the UK and as Mark above notes he is not that far off current research. A lot of the US stuff you allude to is US only, and well, a bit old. The US digital past is definitley another country.

    Also, he has written clearly and concisely, and has had the courage to identify a few naked emperors that everyone suspected were there anyway (Teen Twitter usage for example) which is why I think it has resonated so far.

    Being put out by Morgan Stanley doesn’t hurt either.

    As far as not trusting Morgan Stanley’s analysis, bearing in mind their inability to predict the cataclysms in their own industry I think this piece is a darn site more accurate šŸ˜‰

  5. I think I may have to start marketing a seminar to investment banks on the subject “How to Generalise”.

    You’ve reminded me that my favourite explanation of the sub-prime mortgage crisis is that it was as though the banks had said “The most weight I’ve ever put on is half a stone in a year, so I’m just going to eat cheesecake for breakfast every day”.

  6. I read this blog and thought it made a lot of sense. Then I read the original memo and thought this blog was too kind. Robson’s note is rubbish, it reads like it was written by a 15 year old, showing no particular insight and all the world assumptions one expects of a teenager (I especially like the assumption that most teenagers are unaffected by most advertising – yes, go on believing that, but I bet you are wearing shoes that had a good ad campaign, buying gaming consoles with good ad campaigns, etc. I thought my generation was immune to advertising too, and was clearly wrong.)

  7. I’m not sure what’s more worrying – Morgan Stanley making a big deal out of a teen’s anecdotal observations of his peers’ media usage; or that any of it is a surprise to anyone in the media business.

  8. To be fair to Morgan Stanley they say in the preamble to the note “Without claiming representation or statistical accuracy” at the start of the sentence containing the lovely “so we published it” line that everyone is quoting.

    The Guardian conveniently omit this preamble from their “full copy” of the note, but it can be found from the Evening Standard’s site: http://i.thisislondon.co.uk/i/pix/2009/07/TEENAGERS1.pdf

  9. I’d treat it like a teen barometer for the purposes of driving research on the subject. Anecdotal, yes, but worthy of further actual investigation. In addition, there has been no predictions made for future trends (whether they would adopt certain social media or not). I’d be interested to hear about that.

  10. @alan p – just devil’s advocating for a moment, surely all that proves is that your kids (and thus, their friends) are in the same social niche as the “report author”, and that two anecodes still doesn’t equal data? šŸ™‚

  11. Wall Streets response to your charge that “The plural of anecdote is not data” would be – “See the huge pile of money? – yah I made that taking anecdotal reports just like this one and trading on it – its what I do. Oh yah. Where is your pile of money?”

  12. My 16 year old daughter had just told me over the weekend that she doesn’t Tweet and has no interest in Twitter. Furthermore, she said, the majority of her friends (with the exception of perhaps one or two) don’t use Twitter either.

    So I found Robson’s essay, which is a window into his peer group, quite interesting.

    It is what it is. I don’t think anyone is going to base any big decisions on it. But I’m sure it will spur additional research in the area.

  13. @adam – oh, sure – but if Robson and my kids are not near the norm on this one I’d be dumbfounded to know where the “norm” is.

    My reaction was more “yep, most of this is about right – why didn’t these banker types know this?”, and that seems to be a strong theme on the comment boards of the various articles, which actually is corroborative data rather than pure anecdote.

    If I was being provocative (as if…) I’d say this gefuffle is more about a 15 yr old kid nailing this better than all the old media analysts and new media experts put together šŸ˜‰

  14. I think it’s really hard to generalize “what teenagers do”. It’s like asking the question “what music is popular with teenagers”… you’d get a very varied answer from those into mainstream pop through to the extremes where they’ve discovered rave culture or taken up rock guitar.

    Knowing danah and her research and pairing it with my observations of general tech use in US and my native UK, I would say that it’s entirely likely that danah paints a v North American perspective compared to a British one.

    Sure, the Morgan Stanley report is poor because it’s doesn’t stand up to research quality but I think it’s indicative that some greater British-orientated research would be useful, perhaps by the BBC or IPPR.

  15. @PRoales Wall Steet’s big pile of money is mine. I’m an American taxpayer. And as for Morgan Stanley, had the US government not ‘facilitated’ their purchase by Bank of America, they would be on the deadpile with Lehman.

    The fact that this is news to the media explains why one of the few sectors which is underperforming investmnent banks is the media.

  16. Just got back from dinner (In Prague! Lovely evening!) to find all these comments. Some further thoughts:

    @benedict If one’s best attempt at PR is to make one look like an idiot for acting surprised when a 15 year old states the obvious, well, I wouldn’t want that PR department.

    @mark My post wasn’t so much about the validity or not of Matthew’s comments. He’s described – accurately I’m sure – his and his friends’ experience. But that still doesn’t mean that his experience is descriptive of all teens’. And current research, particularly that done by danah boyd and colleagues, paints a much more nuanced picture than we see in this report. As with many things, the devil is in the detail, and taking sweeping generalisations as gospel is something we should all be wary of.

    @chris If a teen has an authority on being a teen, doesn’t that mean that all teens are authorities? The problem is not that MS want to know what teens think, it’s that they assume that one teen is representative.

    @alan Again, this wasn’t a post about Matthew’s observations. Just as a stopped clock is right twice a day, so a teen’s observations will ring true to anyone inhabiting the same demographic segment. Many of the conclusions of US research are as valid here as they are in the US with regard to the fact that teens are not a homogeneous group and that some of them will not have the kind of access to technology that Matthew has. And again, I give him kudos for good writing, but *his* competency is not the issue and never has been. It’s the reaction to his report that’s the problem.

    @gordon ha ha ha ha! I love that!

    @gus I thought the note was well written given that it was written by a 15 year old. I used to have to read essays written by 18 year olds, and the vast majority of them were not as lucid, clear or well structured as Matthew’s piece. (Hell, half of them couldn’t even spell ‘technology’ — ‘teknowledgey’ anyone?) So I’m not going to criticise him on that. But the point is that he is 15, and his outlook comes with all those limitations that all 15 year olds (and clearly a lot of people older than 15) face. You’re dead right on the advertising thing though. There’s a reason that advertisers bank on ‘pester power’.

    @hugh it’s frankly terrifying.

    @nigel Don’t you think that preamble is full of weasel words? It’s like saying “I don’t want to be mean but…” and then promptly saying something unkind. Declaiming what you’re publishing and then publishing it anyway is equivalent to not declaiming it in the first place. It’s not like they don’t know that everyone’s going to ignore their preamble.

    @andy more research is always good! As Charlie says in his post which links to this, anyone who can predict teen behaviour accurately could become very wealthy, very quickly!

    @adam very true. if one anecdote doesn’t make data, neither does two.

    @proales Wall Street lost their big pile of money, or did the US government’s bailouts pass you by? I suspect Wall Street made such a mess of the economy precisely because of the sort of shoddy thinking that Morgan Stanley are exhibiting now.

    @Robert Undoubtedly some teens don’t use Twitter, but some do (see the comments on the Guardian’s piece). Teens, like adults, tend to move in packs and they go where their friends go. That’s why I’m on Twitter rather than Facebook – it’s where my friends are and where they went first. I wouldn’t extrapolate from even three anecdotes as to the state of play of teen adoption of Twitter or Facebook, particularly not over the long term. Indeed, ReadWriteWeb was writing not very long ago that teens and older students were fleeing Facebook… Which I think goes to prove that it’s all just a bit more complicated.

    @alan What is the norm? Is it what middle class kids do? Where do rural kids sit? Or inner city kids? The thing is, you can use all the common sense in the world to express your idea of ‘the norm’ but until you have hard data to back it up, it doesn’t actually tell us anything. This kerfuffle, in my opinion, is not about a 15 year old’s opinion. His description of his experience is as valid as anyone else’s. It’s about the reaction to his report. And no, he hasn’t nailed it better than “all the … experts put together”, because he hasn’t nailed it better than danah boyd, who’s done actual real statistically valid research into the area.

    @ben British oriented research would be wonderful, yes. I’d very much like to see that happen, not just to examine the use of the internet and technology by teens, but by everyone else too. Ofcom does some, and the Office for National Statistics does some, but no one really seems to address the questions that really need answering.

    @Eli No, there’s no version of Pew here, although I do suggest to the government that we should have one in my report (out soon but you can read most of it here already) on the use of social media by civil society. (See the Carnegie category on the right.)

  17. What I don’t understand is this:

    Robson writes

    “Outdoor advertising usually does not trigger a reaction in teenagers, but sometimes they will oppose it (the Benetton baby adverts)”

    The Benetton baby advert appeared in 1991. That would mean that when the ad came out he would have to wait three years until he was born.

  18. The only data I see here is suggestive that Morgan Stanley execs are a bunch of absent parents.
    You’re right to quote danah boyd, as an example of good science on youth and social media.
    There’s nothing wrong about the Robson report, except that for anyone who has bothered to speak to a teenager recently for more than 5 minutes, its plain common sense.
    Like I said, imagine Morgan Stanley’s shock when they hear that teens have sex.

Comments are closed.