Opportunities from the data deluge

There are huge opportunities for journalism and data. However, to take advantage of these opportunities, it will take ?not only a major rethinking in the editorial and commercial strategies that underpin current journalism organisations, but it will take a major retooling. Apart from a few business news organisations such as Dow Jones, The Economist and Thomson-Reuters, there really aren’t that many general interest news organisations that have this competency. Most smaller organisations won’t be able to afford it on an individual level, but it leaves room for a number of companies to provide services for this space.

Neil Perkin outlines the challenge and the opportunity in a wonderful column that he’s cross-posted from Marketing Week. (Tip of the blogging hat to Adam Tinworth, who flagged this up on Twitter and on his blog.) In our advanced information economies, we’re generating exabytes of data. While we’re just getting used to terabyte disk drives, this is an exabyte:

1 EB = 1,000,000,000,000,000,000 B = 1018 bytes = 1 billion gigabytes = 1 million terabytes

To put this in perspective, I’ll use an oft-quoted practical example from Caltech researcher Roy Williams. All the words ever spoken by human beings could be stored in about 5 exabytes. Neil quotes Google CEO Eric Schmidt to show the challenge (and opportunity) that the data deluge is creating:

Between the dawn of civilisation and 2003, five exabytes of information were created. In the last two days, five exabytes of information have been created, and that rate is accelerating.

All the words spoken since the dawn of language in 5 exabytes or the amount of information created in the last two days helps illustrate the acceleration of information creation. Those mind-melting numbers wash over most people, especially in our arithmophobic societies. However, there is a huge opportunity here, which Neil states as this:

The upside of the data explosion is that the more of it there is, the better digital based services can get at delivering personal value.

And journalists can and definitely should play a role in helping make sense of this. However, we’re going to have to overcome not only the tyranny of chronology but also the tyranny of narrative, especially narratives that prejudice anecdote over data. Too often to sell stories, we focus on outliers because they shock, not because outliers are in any way representative of reality.

From a process point of view, journalists are going to need to start getting smarter about data. I think data crunching services will be one way that journalism organisations can subsidise the public service mission that they fulfil, but as I have said, it’s a capacity that will need to be built up.

Helping journalists ‘scale up what they do’

It’s not just raw data-crunching that needs to improve, but we’re starting to see a lot of early semantic tools that will help more traditional narrative-driven journalists do their jobs. In talking about how he wanted to help journalists at AOL overcome their technophobia, CEO Tim Armstrong talked about why these tools were necessary. Journalists have not been included in corporate technology upgrades (and often not included in creation of tools for their work). Armstrong said at a conference in June:

Journalists I met were often the only people in the room who never had access to a lot of info, except what they already knew.

It’s not technology for technology’s sake but tools to open up more information and help them make sense of it. Other industries have often implemented data tools to help them do their jobs, but it’s rare in journalism (outside of computer-assisted reporting or database journalism circles). Armstrong said:

You can pretty much go to any professional industry, and there’s some piece of data system that helps people scale what they do.

Journalists are being asked to do more with less as cuts go deep in newsrooms, and we’re going to have to work smarter because I know that there are some journalists now working to the breaking point.

There have been times in the last few years when I testing the limits of my endurance. Last summer, filling in behind my colleague Jemima Kiss, I was working from 7 am until 11 pm five days a week and then usually five or six hours on the weekends. I could do it for a while because it was a limited 10-week assignment. Even for 10 weeks, it was limiting the amount of time I had with my wife and was negatively affecting my health.

I’m doing a lot of thinking about services that can help journalists deal with masses of information and also help audiences more easily put stories into context. We’re going to need new tools and techniques for this new period in the age of information. The opportunities are there. Linked data and tools to analyse, sort and contextualise will lead to a new revolution in news and information services. Several companies are already in this space, but we’re just at the beginning of this revolution. We live in exciting times.

New news business models can’t ignore new economics

Normally, I would just add this to our (almost) daily collection of links, but Vin Crosbie has said something so succinctly and clearly that it deserves a post and a full reading. At ClickZ, Vin says:

…today with newspapers, magazines, and broadcasters who clamor for the “missing” business model that will allow them to stay in business doing what they’ve always done. It will never be found because continuing to do what they’ve done no longer makes sense. There are more quick and efficient ways to produce and disseminate information.

Anyone looking for the silver bullet business model to save their old business needs to read what Vin has to say.

The internet has fundamentally changed the economics of information. Digital distribution has ended information scarcity, and much of the new talk of paywalls isn’t about making money but attempting to recreate scarcity. I seriously doubt this will work, and I seriously doubt that trying to squeeze revenue out of much of the existing information output will work. There is no business model that will allow journalists to simply continue doing what they have done. Journalists, editors and publishers need to accept this and re-make their businesses.

Chris Anderson of Wired points out that the journalism businesses of the 20th Century was built on scarcity and monopoly rents. Newspapers were once the most efficient ways for advertisers to get their messages to the public. This created media empires that could fund huge staffs of journalists. Howeveer, beginning in the 1970s and accelerating with satellite television and the internet, people had more choices for entertainment and information. As I’ve often said, information isn’t the scarce resource now. We’re fighting for attention.

This leads to a host of questions. These are just a few.

  • Accepting that information is no longer scarce, what value can journalists add for our audiences?
  • If we’re not adding value, why are we doing it? What are we going to have to stop doing?
  • What new services can we create that will support journalism?

We really need to be thinking beyond business models to support our existing business and our existing ways of doing journalism. I used to think that the efficiencies of digital production would help existing journalism organisations to jump the chasm. I’m no longer confident that this is possible.

After a very busy summer, I’ve got a backlog of blogging here on Strange Attractor and a backlog of thoughts. In addition to considering the issues of over-supply, I agree with Dan Gillmor, we’ve got a problem with the demand for news. As per usual, Dan is asking some very important questions. I am starting to think of ways that we can stimulate demand by actively working to engage our audiences. I’m excited to be plugging back into the discussion about what we journalists do next, and Suw and I are looking to move this discussion beyond the talking and into doing.