How many news outlet staff actually read their own RSS feeds?

I don’t have a TV. I also don’t have a radio. I get my news the same way any self-respecting geek does, via the intarwebthingy. It used to be that I would pop along to news websites and see what was going on, but then Dave Winer invented RSS and that saved me all the fuss and bother of having to figure out whether a site had been updated or not by conveniently feeding new articles into my aggregator. Wonderful.

Blogs, you see, have been using RSS for almost as long as it and they have been around, because blog software is written by geeks, and geeks do like to save themselves some effort whenever they can. RSS was invented in 1997 when Winer created an XML syndication format for use on his blog. Now no self-respecting blog is without a feed. Yay us.

News outlets, on the other hand, suffer from Chronic 90s Web Buzzword Syndrome, which means that they are still thinking about ‘stickiness’ and ‘eyeballs’. I don’t know about you, but the thought of sticky eyeballs quite makes my stomach churn. However, they have – slowly, painfully, and with no small amount of looking over their shoulder to see if the Big Nasty Sticky Eyeball Eating Monster was creeping up behind them – adopted RSS. Despite the fact that bloggers saw RSS as a no-brainer, the media had to think long and hard before they committed to using a technology which made it easier for people to find out what they had published on their websites and which could, therefore, drive lots of traffic their way.

But they’ve got there. Sort of.

I’m glad that The Times, The Guardian, The Independent, the BBC et al are using RSS. I am, at heart, a lazy wossit, and I much prefer my news to come to me, rather than for me to have to go out and find it. However, I am afeared that the media has not quite paid enough attention to RSS, making the consumption of news via my aggregator a painful and unpleasant experience.

Firstly, no one seems to have figured out that when you change a story, the changes show up in some aggregators. I use NetNewsWire, and it’s set to show me the differences between old and new versions of an RSS feed. It’s true to say that sometimes NNW misinterprets what constitutes a change, but it also exposes all the real changes made to news stories.

The BBC seems to have the biggest problem with constantly changing RSS feeds. I brought this up once with a meeting of senior BBC news execs, and they failed to understand why this is a problem. It’s not just that it’s irritating – changing a story even a little bit causes it to be republished which then flags it up as ‘unread’ in my aggregator, even though I have actually read it. It’s also that changing stories after they have been published is unprofessional and damages the news source’s credibility. When I link to a news story, I want it to say the same thing next week as it said when I linked to it.

When I explained this to the BBC’s news execs, they cried in exasperation that they couldn’t possibly be expected to be right all the time, and where do you draw the line between a major update, which gives the story a new URL, and a minor update? Well, that is a good question. Another good question is, why do you even do minor updates? Perhaps better sub-editing, along with not rushing too fast to publish, would help get rid of the need for minor updates, and any major changes to the story are dealt with by a new article? Or perhaps there is an even better way to deal with additional facts coming in, such as saying ‘Update’ in the article, or some other methodology that I haven’t thought of that doesn’t screw with the integrity of the original.

To be fair, not all of the BBC’s RSS output is affected. Out of nearly 40 items from the BBC in my aggregator at the moment, five have changes. That’s only ~13%, which you might thing is negligible, but I think that figure should be zero.

You’ll also find, when you click through to the site, that the first paragraph is different from the excerpt that’s published in the RSS feed. Considering how concise some of these first paragraphs on the site are, it makes you wonder why the BBC are writing separate excerpts at all, and particularly makes me question why those excerpts get edited. Seems like make-work to me.

Here are a few examples from today, picked in sequence from headlines published around the 15:30 mark and including the copy from the website as well. I have replicated the additions (in italics) and the deletions (strikethrough) exactly as they show up in NetNewsWire, so you have to take into account its inherent over-enthusiasm for marking things as changed.

Virus-hit cruise firm apologises

Five hundred UK holidaymakers are sent home after their Hundreds of passengers whose cruise ship was detained because of holidays were ruined by a severe virus.virus outbreak are to be offered refunds.

Virus-hit cruise firm apologises

A travel company has apologised and offered a refund to hundreds of passengers whose cruise holidays were ruined by a virus outbreak.

US crash sparks Afghanistan riot

At least seven people are killed in the Afghan capital, Kabul, Violent disturbances rock Kabul after a deadly traffic accident involving a US convoy crashes, triggering a riot.military convoy.

US crash sparks Afghanistan riot

At least seven people have been killed in the Afghan capital Kabul after a traffic accident involving a US military convoy sparked mass rioting.

Race against time in Java quake

The United Nations warns that the task of bringing taking aid to the survivors of the earthquake in Indonesia is “enormous”.“enormous”, the United Nations warns.

Race against time in Java quake

The task of helping survivors of Saturday’s earthquake on the Indonesian island of Java is “a race against the clock”, the United Nations has warned.

Worse than the BBC is Google News. By it’s very nature, Google News is all about change, but by god that screws with your RSS feeds. Out of about 80 items, 68 had changes. Now, Google News aims to track news stories from multiple sources, so it is inevitable that their items should change frequently, but it makes it completely useless in an RSS aggregator, because every time I refresh, the items that I had read become marked as unread again because Google News have either done something as minor as changed the timestamp from “5 hours ago” to “6 hours ago”, which is not hugely useful, or added a new source, or substantively changed the copy.

This breaks Google News’ RSS feed in terms of usability. There’s just no way I can continue to have Google News in my RSS reader.

Now, what the BBC does get very right is its timestamps. Items published today have the time published as their timestamps, and items published yesterday and before have the date.

Would that The Times could learn that timestamps are important. Instead, RSS items from The Times are timestamped with the time that I refresh my aggregator, not the time that they are published. I have my news feeds grouped in one folder and I read them en masse. News is highly time-sensitive, and I want to read stuff as it comes in, so having an accurate timestamp is essential. The Times, however, hasn’t figured this out yet. Instead, I get a cluster of items grouped around a single timestamp, and when I refresh, I not only get new items, I get repeats of old items with the new ‘timestamp’. This is not helpful.

For example, I learnt that ‘3,000 UK troops are Awol since war began’ both at 11:09 and at 13:58; and that ‘Abbas threatens Hamas with referendum over blueprint’ both on 25th and 26th of May. These items are in the same feed, and appear to be identical, yet they are showing up twice.

The Guardian is pretty good, compared to The Times, Google News, and the BBC, in the way they treat RSS, as I would expect considering they have people like Neil McIntosh and Ben Hammersley to advise. They get timestamping right – the clusters of articles all being published at once is more to do with their editorial time-table than bugs in their RSS feed.

But there is still room for improvement. Whilst they do edit their RSS excerpts, sometimes just as pointlessly as the BBC, they do it a lot less often, so my main criticism would be that they are inconsistent in their excerpt writing habits. Some articles get a sentence, others get two bullet points; and sometimes the excerpt (and headline) is the same as on the site, and sometimes it isn’t. I have to say, I’d prefer a single sentence excerpt and headline which was the same as the site.

A few examples of what I mean.

Ghost ship washes up in Barbados

· 11 petrified corpses found in cabin· Letter left by dying man gives clue

After four months at sea, ghost ship with 11 petrified corpses washes up in Barbados

· Letter left by dying man gives clue to investigators

· Dozens of others thought to have perished en route

Climber left for dead rescued from Everest

A climber who was left for dead on Mount Everest has been found alive.

Climber left for dead rescued from Everest

· Team forced to leave Australian at 8,800 metres

· ‘I imagine you’re surprised to see me,’ he tells rescuer

Cage swaps Malibu for own desert island

Nicolas Cage has bought a 40-acre undeveloped island in the Bahamas for $3m (£1.6m)

Cage swaps Malibu for own desert island

Dan Glaister in Los Angeles

Monday May 29, 2006

The Guardian

Wherever you go, people stare at you. Paparazzi take pictures, fans ask for autographs, absolute strangers wonder aloud if they once met you at a party. For the hard-pressed celebrity there’s only one way to get away from it all: hide on your own desert island.

The surprise in all this is that the one newspaper who gets it spot on is The Independent. I really can’t fault their RSS feed at all. Timestamps are reliable, and again reveal their editorial timetable with many articles being published in the small hours and few being published during the day. Their excerpts vary in length, but some of the longer ones are more useful than those of other news outlets. Personally, I like longer excerpts because I would rather skim a two sentences that give me a better feel for whether I want to read the article than have just one short sentence that doesn’t tell me much.

Some examples, again with the copy from the RSS and the copy from the site:

British journalists killed in Iraq

Two British television journalists were killed in Iraq by a roadside bomb today.

British journalists killed in Iraq


Published: 29 May 2006

Two British television journalists were killed in Iraq by a roadside bomb today.

Bush ‘planted fake news stories on American TV’

Federal authorities are actively investigating dozens of American television stations for broadcasting items produced by the Bush administration and major corporations, and passing them off as normal news. Some of the fake news segments talked up success in the war in Iraq, or promoted the companies’ products.

Bush ‘planted fake news stories on American TV’

By Andrew Buncombe in Washington

Published: 29 May 2006

Federal authorities are actively investigating dozens of American television stations for broadcasting items produced by the Bush administration and major corporations, and passing them off as normal news. Some of the fake news segments talked up success in the war in Iraq, or promoted the companies’ products.

Indonesia Earthquake: As a people, they already had little – now they are left with nothing

In the morning, Salim retrieved the lifeless body of his three-year-old son, Sihman, from the ruins of their brick and bamboo hut. In the afternoon, he buried him, digging the grave himself. As night fell, he searched through the rubble of his former home for scraps of food. “I have lost everything,” he said.

Indonesia Earthquake: As a people, they already had little – now they are left with nothing

By Kathy Marks in Bantul, Indonesia

Published: 29 May 2006

In the morning, Salim retrieved the lifeless body of his three-year-old son, Sihman, from the ruins of their brick and bamboo hut. In the afternoon, he buried him, digging the grave himself. As night fell, he searched through the rubble of his former home for scraps of food. “I “I have lost everything,” he said.

I like the fact that the RSS feed and the website copy are identical. To me, that’s ideal – what I see in RSS is what I get on the site. I also can’t see any evidence of changes, although I will say that this RSS feed is new in my aggregator so maybe this point will clarify itself over time.

Now, all this criticism may seem like pointless nit-picking. Perhaps some it is down to my inner editor screaming for consistency and my inner blogger begging for honesty, but certainly some of this has a direct impact on the usability of RSS feeds for the reading of news.

I want news. I have no problem with the idea of clicking on a link in my aggregator and reading the full article on the news outlet’s website – this is not a plea for full posts (although hell, that’d be great and if Corante can put advertising in their RSS feed, so can anyone, but that’s not the point I want to make).

It’s a plea for journalists and media IT staff to think a little harder about how news is being read these days. RSS is not a fad, and it’s not going to go away. It is going to flourish, with more and more people using it to get their news from many disparate sources. It is in your best interests to ensure that your RSS feeds work, that your editorial policies take into account the effect of new technology on the transparency of your medium, and that you strive at all times for honesty even if that means owning up to your updates (note, ‘update’ does not necessarily mean ‘mistake’).

Google News is revealing your reliance on syndicated content, and RSS is revealing your edits. If you want to remain credible, you must adapt. In an increasingly competitive world, where people choose which news sources to read not just based on content but also on usability and accessibility, can you really afford not to?

13 thoughts on “How many news outlet staff actually read their own RSS feeds?

  1. I think you need to bear in mind three things: most newspapers are still running the CMS that they installed in 1998: Vignette, et al, are gloriously tricky to get RSS feeds out of in the way you want. This is why we use blogging platforms for things like Comment is free: they’re modern gear, and especially so compared to the installed base.

    Second, newspapers are paid for by advertising, so you’ll never ever have anything but excerpts. Full stories in feeds will come if adverts come with them, but not before.

    Third, RSS isn’t mainstream. Really, it’s not. The figures are tiny compared to anything else, and they’re not growing that fast, if at all. I love RSS, have written two books on it, was a member of the RSS 1 working group, have coded for it for years, and so on and so on, and although it pains me to say so, it appears like it’s not crossing the chasm at all.

    But having said that, your wider point is true: we do need to think about how people are reading news these days. It’s just that when we do, we find behaviour very different to the behaviour that we alpha geeks enact. The public, in general, do not care about RSS. (nor validation, nor the evil of flash sites, nor anything else I could rant about) This is a problem. But I’m not sure who’s.

  2. Suw, they could just use identifiers in their feeds to keep the news readers from seeing the articles as new. There’s an element in RSS 2.0 called “guid” that’s there for just this purpose. NetNewsWire supports it, as do many of the professional publications. The BBC didn’t used to, but they do now. It would solve the problem you describe here.

    I totally agree about the use of descriptions consistently. I think the NY Times does this best. I often get enough information from the description, saves me a click and a context change to get the information. But I appreciate it when the publications provide RSS, like Ben I remember a time when they didn’t have the feeds.

    As to whether it’s mainstream or not, what’s the point. I think Suw’s premise is right on, reporters should not only read their own feeds, they should read all feeds of all publications, professional or amateur in their areas of expertise.

  3. Whilst this blog post is entirely my own personal opinion I have worked as a software engineer on the BBC News Website content management system that Suw discusses

    I think it’s correct for the minor updates to occur in the existing RSS feed. (But then as Gabe says in his post about this post: “1. Find good advisors familiar with the technologies (duh),” – so maybe he has a point and the BBC has been short changed! ho ho)

    I’m a bit unsure as to what you want to see instead? A different feed item each time a change is made – however small…? You would just get inundated with changes – yes, the BBC tweaks it’s stories loads.

    At least with the curret situation, whilst your newsreader might say the post is now unread again, a new item isn’t spawned for each change (causing a complete mess).

    I would also point out that the unique id isn’t changed, so if your news reader is set to ignore major changes in the description tag and just set id’s to read/unread (as mine does) then the behaviour is actually correct.

    I acutally wonder whether your newsreader, which I don’t deny is a popular one, is actually “super-serving” you by providing strike throughs, etc. Tracking the changes to RSS items in this way is not actually a function I would say is not actually a normal use of your browser. It’s a _really_ geeky thing!

    The editing of RSS entries (which is actually editing the headline and summary that appears on the index the story is linked from) is great editorial manangement and really great SEO’ing. The story can have a depthy, descriptive headline + summary, but on the index can be given a more punchy, teasing headline/summary.

    You have to remember that under different contexts different types of headline/summary work. Plus it’s important to look at what draws your users in and entices them to click through and visit the site (rather than view the short summary in the newreader). BTW I would personally love to see full-feed content in the RSS from the BBC, but that’s just my opinion and would negate the above need!

    As someone who only consumes their news online, and indeed via RSS I would add one final point: Newsreaders still don’t accurately reflect the weighting that occurs on a site. IE what’s the “top story”, what’s the “second story”, etc.

    If you visit the BBC, or any other news site, you can see this clearly. But it’s not refelcted in the RSS feeds, nor am I aware of any of the newsreaders currently support MS Simple List Extensions or any other technology that would enable news providers to include this information.

  4. “Tracking the changes to RSS items in this way is not actually a function I would say is not actually a normal use of your browser.”

    … I meant of your newsreader, of course.

  5. Ben M, NNW showing the diff’s is a major benefit, as otherwise you can get an updated item showing as unread again. It does have a preference setting for both behaviours, so you can avoid seeing updated items as new, but that can be counterproductive if you read bloggers who often update entries with further material.

  6. Kevin Marks said:

    “NNW showing the diff’s is a major benefit, as otherwise you can get an updated item showing as unread again”

    See, I would expect that read guid to remain read, regardless of whether it was updated or not.

    That’s what my newsreader does (I used Newz Crawler). Maybe I’m living in ignorence of all those updates, but I have too many feeds to read – let alone re-read ones I’ve already looked at once!

  7. Hmm, you would think that by now someone would have a ‘smart’ news reader that can read in articles and tell if 90 percent of the story is the same and not feed it to you again.

    Then again its not THAT big of a deal just to ignore it a second time around…

  8. Ben writes that online news sources are hampered by outdated CMS that don’t easily generate RSS feeds.

    That makes me wonder: is there a CMS for newspapers that’s best for generating RSS feeds?

  9. Ben H: Having worked with Documentum myself in the past, I can sympathise with anyone trying to wring RSS out of such a system. However technological problems usually have solutions somewhere. I’m not a techie, so I’m not going to attempt to posit a solution here, but I see the tech problems as a hurdle, not a barrier.

    Paid-for advertising in RSS is already being done – right here on Corante, as it happens. If Corante can do it, I don’t see why everyone else can’t do it too, but like I said in my post that’s not really my focus here.

    As for RSS being mainstream… Well yes, geeks using specialised RSS readers with diffs turned on is edge case behaviour. I know I am an edge case, but the thing with edge case behaviours is that they often don’t stay on the edge, particularly if there is demonstrable utility. Remember when using a mobile phone was edge case behaviour?

    RSS is being increasingly used by information sources that people care about. It’s not just blogs anymore. Via RSS, or RSS-like feeds, I can get my news, email, weather, photographs, bookmarks, stock market information, video, audio… I can track sellers on eBay, comparison shop… I can get all sorts of information, and the trend is towards more types of information being made available using RSS, not fewer.

    Equally, the increasing numbers of blog writers and readers means increasing numbers of people being introduced to RSS by name.

    But what I suspect will happen is that people will start using RSS without realising it. Homepages such as MyYahoo! or NetVibes, allow you to customise what you see by adding or removing feeds. All it is going to take is for one major web service to offer an easily customised RSS-based start page like theirs, and for information providers to create compelling RSS feeds to use in such a service, and you’ll see RSS cross the chasm in a single bound. They may never actually use the acronym RSS, but that won’t matter.

    How long before everyone is using RSS as a matter of course? I have no idea. But I think that it’s inevitable, and that media companies should be working on the assumption that it’s going to happen sooner rather than later. By preparing for mass adoption they may even help encourage it, not to mention serving current RSS users better in the meantime.

    Dave: GUIDs to keep the reader from seeing an article as new when minor changes are made, such as corrections to grammar, would be a good start. The issue then becomes when is something a minor change, and when is it a major change that needs a new GUID, or a new article. That’s an editorial decision that needs more thought from the media.

    I think the reason that the BBC does so many updates to its RSS feed is that it’s working on a broadcasting paradigm where the bulletins are updated with latest info every time they are broadcast. The newspapers, on the other hand, are working within a ‘publish once’ paradigm, where you have a deadline for copy and after that, frankly, it’s too late.

    I’m not sure quite how to deal with updates gracefully in the current paradigms. I know that frequently updated articles that keep changing annoy the hell out of me. I shall have to give this question more thought.

    I have to add, though, that like you and Ben H, I really do appreciate the spread of RSS. I think it’s fantastically useful.

    Ben M: What I don’t want to see is a different feed item every time a change is made. But I also don’t want an update every time a change is made. I think what I want is for fewer changes to be made, full stop. I have been told by a reliable source (one of the BBC execs, as it happens) that some stories get changed up to 100 times. That is, to my mind, excessive and a waste of time. Like I said before, I want the story I link to to stay the same – I honestly believe that that is good practice.

    I don’t want to turn diffs off in my aggregator, because sometimes it *is* useful. For example, most bloggers I read rarely make changes to their posts, but when then do I want to see what they are. And as I said before, I know that this behaviour is edge case, but this is not really about the behaviour of my aggregator as it is about editorial policy, and seeing it revealed via RSS.

    Also, I don’t want punchy and teasing, I want information, and I want enough of it to assess whether or not I want to read an article, because if I’m not sure, I am just not going to bother at all, rather than click through out of curiosity.

    Your right about the fact that RSS doesn’t reflect the weighting of articles as per the site. I don’t find this a problem, actually, as I don’t need anyone to tell me what’s important – I can usually figure it out for myself based on the headline and summary.

    So, I guess, the issue that underpins the diffs issue is ‘When is a change important enough that I want to read about it, and how do I want to be informed of that change?’.

    What I really do not want to see are updates that deal with grammar, spelling or punctuation errors, or which are nothing more than rewriting the same information in a slightly different way. These sorts of updates just should not be happening at all – there should be an effective sub-editing system that catches these errors before publication. A news story is not a painting – it does not need endless tinkering to make it right. It needs to be researched, written, subbed and published and then left alone.

    Major updates to a story should clearly be dealt with by the creation of a new article, leaving the old one to stand as archival material. I think that one is a no brainer.

    Minor updates are trickier. Again, I think the problem here is that if you have a single new fact, or if the facts change (such as a death toll), should you add it in to the article or wait til you have a more substantive change to make which would count as a major edit? The line between major and minor edits is not always going to be a clear one, but my feeling is that what we are trying to do is shoehorn old-school broadcast thinking into a medium that doesn’t work that way.

    So, instead of trying to figure out what is a minor change, and what to do with it, perhaps we need to think of more imaginative ways to deal with constantly updating information. A bit of freeform thinking out loud here…

    The great thing about websites is that you can aggregate on one page a number of different items – think of transclusion in wikis. What would happen if each new additional paragraph of information was a new item which was transcluded into a main article, and marked in the RSS as an update? That means I can mentally filter out updates on stories that don’t interest me, and follow more closely stories that do. One of the problems with the current method of editing up stories is that half the time, which bit of information is new is not made clear and this would fix that.

  10. Uh, Suw, when I just had a look at my aggregator, this item in the “Strange Attractor” folder — not the newest: the second newest — showed up as updated. Just saying.

    (Yes, I agree with you. And S.A. isn’t the BBC.)

  11. That’s because it was updated. I accidentally hit ‘post’ instead of ‘preview’ in Ecto before I had finished adding in links and tweaking it. Shit like that happens.

    But you see, this is a blog. Bloggers seem to rarely change their posts, and when they do they usually have a full feed so I can see what they’ve changed at a glance. I never said I minded that.

    But you’re right. Strange is not the BBC. No sub-editors here. Not even any editorial process. No posts getting changed multiple times – usually just the once if ever. No brand, or contrived survey saying how trusted we are. No claims to authority. Just two people thinking out aloud and occasionally fucking up.

    So what exactly is your point?

Comments are closed.