I have blogged before about RSS overload, the problem of simply having too many feeds in your aggregator to be able to read them all. Now Bill Burnham gives it a name, Feed Overload Syndrome, and discusses how “RSS threatens to sow the seeds of its own failure by creating such a wealth of data sources that it becomes increasingly difficult for users to sift through all the “noise” to find the information that they actually need.”

He then describes the problem in detail and discusses possible solutions. Syndicating the results of keyword searches instead of actual blogs, he says, is not an ideal approach for three reasons: many RSS feeds are excerpt not full post, thus preventing comprehensive indexing; keyword searches become less effective the more data you index; keywords can have multiple meanings which produce noise in the results.

The new Technorati tag system is also ‘fundamentally flawed’ in his view:

The problem at the core of tagging is the same problem that has bedeviled almost all efforts at collective categorization: semantics. In order to assign a tag to a post, one must make some inherently subjective determinations including: 1) what’s the subject matter of the post and 2) what topics or keywords best represent that subject matter. In the information retrieval world, this process is known as categorization. The problem with tagging is that there is no assurance that two people will assign the same tag to the same content. This is especially true in the diverse “blogsphere” where one person’s “futbol” is undoubtedly another’s “football” or another’s “soccer”.

I agree that this is a big problem with tagging, if what you are aiming to achieve is a flawless, cross-referenced database of blog posts. In an ideal world, that would be nice, but this is not an ideal world and people are used to the internet not working quite right. Users learn how to rephrase their search terms to improve results and once Technorati allow for more complex tag searches or starts to produce clustered search results then semantic issue becomes less important. (Although I doubt they will ever become irrelevant regardless of what is done.)

Instead, Bill Burnham believes that the way to RSS nirvana is through the use of metafeeds – “RSS feeds comprised solely of metadata about other feeds”.

Combining meta-feeds with the original source feeds enables RSS readers to display consistently categorized posts within rich and logically consistent taxonomies. The process of creating a meta-data feed looks a lot like that needed to create a search index. First, crawlers must scour RSS feeds for new posts. Once they have located new posts, the posts are categorized and placed into a taxonomy using advanced statistical processes such as Bayesian analysis and natural language processing. This metadata is then appended to the URL of the original post and put into its own RSS meta-feed. In addition to the categorization data, the meta-feed can also contain taxonomy information, as well as information about such things as exact/near duplicates and related posts.

RSS readers can then request both the original raw feeds and the meta-feeds. They then use the meta-feed to appropriately and consistently categorize and relate each raw post.

The benefits of using metafeeds as outlined by Bill look great. You would be able to find related documents, eliminate duplicates, create custom taxonomies, combine metafeeds and have your information “consistently sorted and grouped into meaningful categories”.

I have to admit, that sounds great. It would be wonderful to be able to create complex search strings and to get a feed back from the web that would contain only relevant posts and no duplicates. It would indeed be a form of RSS bliss.

It won’t, however, solve the problem of RSS overload – it is likely that it will just make it worse. Bill’s fix is a technical solution to a non-technical problem, and as such it is only half a fix.

We have always lived in a world where there was more information available than any one person can comprehend, but before email, the internet, blogs and RSS feeds, the limiting factor was not the existence of the information but gaining access to it. The form of the information limited the speed with which it could be accessed: having to go to a library, find the right book or journal, turn the pages, reading them one by one; gaining an introduction to an expert, persuading them to sit down with you and discuss the matter at hand; or doing empirical studies in order to reveal the information sought. It all took time.

Now the data we seek is easily accessible and the problem has shifted – it’s not finding information that’s the issue, it’s finding the right amount of the right information. The limiting factor is no longer access but discrimination. There is so much information available that it’s hard to know which bits to trust.

Anyone who paid attention at university learnt that the way you do library research is to cross reference your sources – you can’t trust one single source to be telling the truth so you learn to triangulate. The more sources that tell you that zebras are black and white, the more you believe it. Then you learn to weight your sources by credibility and reputation. If Learnéd Academic Journal tells you that zebras are black and white, then you feel confident that all other sources are going to agree with that, and it’s easier then to discount the Tabloid Freakshow Magazine article that claims to have discovered a purple zebra.

That’s basic research methodology. Cross reference. Consider the source. Keep a bibliography. And it’s a hard, hard habit to break, even for people who didn’t know that they were doing it.

RSS overload is partly to do with trying to triangulate the ‘truth’ from too many sources. There are many blogs devoted to Macs, for example, and the urge is to read them all to see what each one is saying, to compare the information in order to draw some conclusion as to what is most likely to be true. In blogging, there really aren’t any Learnéd Academic Journal-type sources with the sort of standing that allows you to immediately trust them. There are many reliable blogs written by many well-informed people, but it is difficult to tell which they are until you have completed your triangulation, reached your own conclusion and found that it syncs with what your now trusted blog tells you.

Of course, this is not necessarily a bad thing, as many previously trusted data sources are being shown to be less than trustworthy, but we do have to recognise that this whole process of building up a list of trusted blogs takes time and effort. Although to some degree trust can be passed on to other readers through word of mouth recommendations, we are still doing more work to locate trusted sources than we used to.

Another problem not solved by Bill’s metafeeds is that of completism. If you’ve ever met a rabid collector of stuff then you have probably met a completist, someone who just can’t bear not to have every last Star Wars toy, or every last scrap of Elliott Smith memorabilia. That’s what makes collectors collectors.

Many bloggers are completists too – information completists. To go back to the Mac example, you may rapidly decide which feeds are most reliable and which are mainly talking rubbish, but that doesn’t mean you are going to delete the rubbish feeds from your aggregator because there is the possibility, however slim, that they might just break the rumour of the G5 PowerBook that you’ve been desperately waiting for all these months.

Then there are the long link trails left for us to follow when we are researching our next post. You come across an interesting post, it contains links, which you follow, and then that contains more links which seem relevant so you follow those too… and then you check Technorati and read the posts you find there, and they lead to more and more posts and before you know it you’ve spent a day researching a blog post that is only two paragraphs long.

Information completism is dangerous – it leads to chronic information overload and can turn into a form of ‘legitimate procrastination’. Because link trails are convoluted and potentially exceedingly long, it’s easy to over-research instead of actually get on with the post.

The only cure is to accept that we are human and flawed and we cannot possibly know everything about everything. We can’t even know everything about one thing, because there is too much to know, too many perspectives to take on board, too many angles to look at it from. We cannot and should not attempt to read every post and comprehend everyone’s point of view on a subject.

Instead we should refine our lists of sources down to a few trusted writers, and let the rest go. Is the Mac idiot whose blog makes you fume really going to break news about a new G5 PowerBook? No. Ditch it. Is reading every post about RSS really going to make your post about RSS overload any better? No. Read what you need then get on with the writing.

If anything, Bill’s metafeeds could well add to the problem of RSS overload by adding more sources to the mix. Instead of cutting down the number of feeds people try to read, it will add to them by providing alternative concretions of data which supplement existing sources rather than supplant them. This is because of the third flaw in his plan – blogs are social, and his fix is technological.

Most of the blog feeds I read on a daily basis I read for social reasons rather than informational reasons. I have 56 feeds in my ‘friends/dailies’ group in NetNewsWire, another ten under ‘acquaintances’. None of these feeds have anything to do with information per se. They could not be replaced by any sort of keyword search and metafeeds would be simply irrelevant in this context. I read them because I want to know what these people are up to – they are friends or people I wish were friends.

But even here, where you would think that the territory is fairly well defined, there is a problem of bloat. Social networking is great, it allows you to meet a whole bunch of interesting people you would never otherwise have met, but widening your social circle also means you have more friends and acquaintances to keep up to date with. Whilst individuals may not expect you to read their blog, (indeed, I remain in a state of permanent surprise that anyone reads any of my blogs at all), there remains a nebulous feeling that one really ought to. I’m now connected to a ludicrous number of people, and in all honesty there is no way I can read everyone’s blog.

The problem of RSS overload is not completely technological and a technological fix will not work. Instead it is partly technological, partly cultural, partly social, and partly down to our own personality quirks and habits. Metafeeds may help us find more relevant information more easily, but they won’t cure the information overload problem. Only we can do that, by cutting down on the number of feeds we read, the number of tabs we leave open in Firefox, and the number of people whose blogs we follow.

RSS, metadata, tags, blogs, weblogs, blogging, feeds, metafeeds, meta-feeds, information overload

I’ve been gathering feeds in my Bloglines aggregator for some time now, hoarding them like a bower bird in a tinsel shop, weaving them together into one unholy unread mess. A few months ago I had a flurry of half-hearted search activity for the perfect aggregator, and although then I think I concluded that the RSS plug-in for Firefox was nifty and that BlogMatrix Jäger was also worth a look, my nomadic non-laptop owning lifestyle of the time meant that a web-based aggregator was the only serious option, so I stayed with Bloglines.

At the beginning of this week I had 310 feeds showing around 25,000 unread posts. I had toyed with the idea of declaring RSS bankruptcy and just starting again, but I was getting increasingly unhappy with chaotic state of my feeds and deep down I knew that hitting ‘mark all posts read’ would do nothing to solve the problem in the long run.

There were two issues. Firstly, I never had enough time to sit and read all my feeds, or even to work out which ones I could safely mark as read whilst actually leaving them unread. Thus I would pick which feeds to read based on which had the lowest number of unread posts (anything in double figures was likely to get ignored, triple figures ensured I wasn’t gonna touch it for a goodly long time). Secondly, although I had made a stab at categorising them through the use of folders, they really were all over the place and utterly chaotic. This meant that ever time I glanced at Bloglines I was confronted with one fugly mess.

Aggregator crisis point had been reached.

The advantage of hanging out with well informed blog-geek Mac-obsessives is that when I whine about needing a new aggregator, I am given advice and I happily make the assumption that whatever is recommended is going to be good. So over the last couple of days I have migrated my OPML (someone, please sort out some sort of OPML standard so that I can export/import without having to manually to fix crappy, import-snafuing code) from Bloglines to NetNewsWire.

Immediately my unread headlines list diminished to less than 4500, just because NNW only pulls down the latest 30 headlines, instead of the maximum of 200 that Bloglines marks as unread before it stops counting. I managed to quickly delete 25 blogs I knew I didn’t need anymore, and easily sorted the rest into folders. Sitting now on the train back to Dorset, I’ve read through around 500 posts, because NNW caches them locally so I don’t need to be connected in order to read.

At last, I feel like I am in control of my aggregator again. Instead of feeling overwhelmed by the amount of information that I ought to be absorbing, instead of feeling scared to open my aggregator because the unread posts are gonna overtop any second and flood my poor little brain, I feel like I have a nice, tidy resource that I can dip into any time I want. Of course, much of this is an illusion, facilitated by a folder cunningly called ‘blogs/tech/stuff’ which contains pretty much everything that’s currently uncategorised, but I can cope with that act of wilful self-deception.

All this, the offline reading, the chilling out with my friends’ feeds, the feeling of regained control, has been reinvigorating. There have been blogs of friends that I’ve not read in ages because I felt like I ‘should’ be reading blogs related to work, even though frequently those are some of the least interesting blogs to read. No one can begrudge me spending a train journey reading through non-work stuff, not even me and I’m the worst workaholic I know.

Thing is, it’s reading the unrelated stuff, the fun stuff, that is important. It’s through picking up on a random comment by someone else that some how fits in just so with something that someone else said and something that I was thinking that pokes my brain and gives me that a-ha! moment that I constantly seek. It’s through faffing and playing around on the edges of things and allowing my brain to synthesise ideas without the imposition of expectation or structure that I stand the greatest chance of coming to some new understanding. It’s through finding a gem of a post that I regain/retain my love for blogging – and, doing what I do, maintaining a love for blogging is essential.