Digital Directions 11: Josh Hatch of Sunlight Foundation

Josh Hatch, until recently an interactive director at USAToday.com and now with the Sunlight Foundation, talked about how the organisation loves data. The transparency organisation uses data to show context and relationships. He highlighted how much money Google gave to candidates. Sunlight’s Influence Explorer showed that contributions from the organisation’s employees, their family members, and its political action committee went overwhelmingly to Barack Obama.

Sunlight Foundation Influence Explorer Google

The Influence Explorer is also part of another tool that Sunlight has, Poligraft. It is an astoundingly interesting tool in terms of surfacing information about political contributions in the US. You can enter the URL of any political story or just the text, and Poligraft will analyse the story and show you the donors for every member of Congress mentioned in the story. They will highlight details about the donors, donations from organisations and US government agencies. It’s an amazingly powerful application, and I think that it points the way to easily add context to stories. It does rely on the gigabytes of data that the US government publishes, but it’s a great argument for government data publishing and a demonstration for how to use that data. Poligraft is powerful and it scales well.

Josh showed a huge range of data visualisations, and he’ll post the presentation online. I’ll link to it once he’s done.

Digital Directions 11: Fairfax’s digital business

I’m in Sydney to speak at Digital Directions 11. I’ll post my talk to Slide Share in a bit. The conference is hosted by Fairfax, and yesterday, we got a look at their digital business. There are a lot of news and media organisations that have built credible digital offerings over the last decade without building sustainable digital businesses. Fairfax is one of the exceptions. Fairfax CEO Greg Hywood said that digital is its third largest division by revenue and soon to take over the number two spot. Yesterday, we were told that transactions were 60% of digital revenue. Transactions? They have businesses such as a dating service called RSVP and a holiday home rental service, Stayz.com. They are seeing phenomenal growth in that business. Many of these businesses have been acquisitions, not businesses built in house.

Moreover, the revenue being generated by digital is now driving the ascendancy of digital in the organisation. Recently, Jack Matthews, who had been the CEO of digital, was made the CEO of their metro division overseeing both print and digital. He will drive integration. They are going to integrate their print and digital editorial operations, but the current thinking (and this might change) is that while journalism resources will become a common pool, digital will retain its own editorial independence, Matthews told me. Fairfax has found that print and digital offerings don’t share the same audience. Market research has found that their digital audience is slightly more upmarket than their print audience, and they have decided that they need to maintain digital and print independence to best serve those audiences.

They were ready to admit that integration has often meant print divisions taking over digital. If digital didn’t have such a strong revenue position, I doubt, actually, I’m almost certain, that digital wouldn’t be driving integration at Fairfax. That is not to say that it hasn’t been a battle at Fairfax. I know there has been a battle, and Matthews admitted as much. It’s not to say that the battle is over. However, when digital brings revenue, instead of being pushed aside during print and digital integration, they can actually be in the driver’s seat.

Live blogging evolved: Context and curation not just collection

When I was the blogs editor at The Guardian, I was a big booster of live blogging. We now think of Twitter and Facebook as “real-time”, but in terms of news, we’ve been living in a real-time environment now for decades. With the advent of radio, we stopped waiting for the news to arrive when we got the newspaper. Sometimes rolling news can descend into self-parordy, but after working for the BBC for eight years, I know that live and continuous news can be done well. For me live blogging and real-time curation allow newspapers via their websites and via mobile to compete against broadcasters in rolling news.

Seeing this post via Martin Belam that the Guardian Newsblog might somehow be “the Death of Journalism” by John at The Louse & the Flea* gives me a chuckle but it also raises some valid points. Martin responds to some of those points, and I know that he and others at The Guardian are actively working to address some of the deficiencies in the format. Martin says:

Nevertheless, John does identify some of the issues that concern me from an information structure point of view of the way we do live blogging now – notably it is very difficult within our templates to display a summary prominently enough, and the strict reverse chronology of entries whilst a live blog is “active” can lead to the more important chunks of the content getting buried. We could also probably do an improved job of permanently sign-posting packages of more conventionally formatted stories from within the live blog itself.

As anyone who works in online news knows, some of this is just down to the limits of technology, as Martin admits, but they are limits that can be addressed both technically and editorially. Live blogging began with sports coverage at The Guardian and moved on to media and tech conference coverage and also live blogging TV shows. The length of a post was limited by the event – 90 minutes of football or an hour episode of Big Brother, but I’m not sure that this format is really well suited for events that carry on all day for several days.

Drowning in a ‘River of News’

However, John does raise some issues that I think are worth addressing. John says that The Guardian’s live blog is:

a mish-mash of baffling tweets, irrelevant musings from the Guardian’s comments, contact details for those who want to find out about loved ones or make donations (including one from the New Zealand Red Cross, who actually says it doesn’t want donations just yet, and another from the Auckland University Students’ Union, the relevance of which escapes me), musings from a boffin at that world renowned centre of earthquake research, Bristol University, and speculation on how the tragedy might affect the Rugby World Cup, due to kick-off in NZ in seven months’ time. Scattered meagrely throughout, like sixpences in a Christmas pudding, are bits of what you and I might call “hard news”.

I really do worry that some of the aggregation that we’re doing is really difficult to navigate unless you’re a news junkie. We have to make sure that a stream of news aggregation doesn’t feel like a maddening stream of consciousness. I have the utmost respect for Guardian live blog masters (and friends) Matthew Weaver and Andrew Sparrow, but I can’t help but think there has to be a better way to package their prodigious and highly professional output. Andy said that some days during the election last year he was sometimes producing CORRECTION: 14,000 40,000 words a day. (Andy corrected me on Twitter. I thought 40k words in a day was ambitious, but he is prolific!) How does the average reader easily navigate this? The Guardian did a lot of work during the election to improve the format. They added better formatting for different elements such as blockquotes and contributions from other members of Guardian staff, but the reader still has to rely too much on searching within a page.

I know that Martin and Co will come evolve the format, but I still can’t help but wondering if simply breaking up the posts at major inflection points might be a good interim solution. I agree with Martin that there is a lot that can be done with better packaging.

Martin flags up the prodigious output of The Guardian yesterday, much of it in more traditional formats. However, looking at the headlines, I have to admit that I’m overwhelmed. With some of the headlines, I’m not entirely clear how the stories are different. In saying that, I don’t want to pick on The Guardian. Frankly, I’m really think that over-commissioning is part of a problem that newspapers are suffering from right now. They can publish continuously, but I know there is a better way to mix slow and flow news coverage.

Curation and context not just collection

I also think that John has hit on an issue that has become a real problem in real-time news coverage in the last couple of years. I’m a journalist. I’m a news junkie. I keep tabs on a wide range of stories in some considerable depth, but even with the background knowledge that I have, I’m sometimes lost. If I’m lost and overwhelmed in stories that I know really well I know that our audiences don’t even know where to start.

Whether it’s live blogging or new tools like Storify, I worry that sometimes we’re training a fire hose of news on our audiences. We’re not curating. We’re not making editorial choices and adding context. Instead I do fear that we’re causing information overload rather than helping people make sense of the world. Storify and live blogging are great tools and techniques, but they work when a journalist makes editorial choices and adds value through context. Who is this person on Twitter? What is their role in the story?

On Twitter, I occasionally hear the claim that to edit out information is some form of censorship. If people want the fire hose, they can use Google Realtime. People have a choice to swim in the waterfall, and our editorial choices don’t preclude people from digging deeper and in different ways than we have. Journalists report and choose what they think are the most important bits of information. That’s one of the services that we provide, and in the deluge of real-time news, that service is actually more important than before.

I guess to do that, to be trusted guides, we have to win rather than assume trust. That’s another change in terms of people’s relationship to journalism, but we can do that because we don’t have the walls that separate us from audiences.

Real-time in motion

Some of these super-long live blogs are also is a terrible experience on mobile even on light-weight mobile templates. The downloads are huge, and they don’t work well on small screens. As we increasingly move to mobile consumption, we’re going to have to rethink this format or more likely than not think of a new format entirely.

I hope that my friends and former colleagues at The Guardian don’t think I’m picking on them. These are more general observations than The Guardian’s live blogging, and I know that Martin and the great live bloggers on staff there don’t rest on their much deserved laurels. It’s a big challenge. We can relay so much more information than in the past when we had a few sources and the wires, but that means that we have to find new ways to help audiences make sense of that information.

*Note to John
This isn’t a snarky comment but honest advice. If you’re an unemployed journalist, I’d really suggest adding some links on your blog to your past work and an up-to-date CV. Suw and I use LinkedIn and have a link to our work histories there. Hopefully your blogging won’t just keep you sane while you look for another job but actually help you land it. It’s working really well for me, and I hope it helps you find a job soon.

Finding the signal in the flow

Suw and I had been noticing a bit of an economic uptick on our local high street here in London last autumn and into the winter. Shop fronts that had been empty were getting new tenants, and just in our corner of the Big Smoke, we could see green shoots of recovery.

[blackbirdpie url=”http://twitter.com/JonathanLloyd/statuses/40012699773575168″]

This morning Jonathan Lloyd who has a hyper-local content and commerce platform, Media Street Apps (story about the platform here on journalism.co.uk), tweeted this observation that he was seeing gutted shop units. It’s not the first rather grim economic observation that I’ve seen on Twitter as the UK economy seems to be softening again.

As a journalist, I immediately started to think of how we might find the signal in this flow of updates. Jonathan is very close to local retailers, and he might be flagging up an indicator of the direction of the economy. Google is already mining its own online shopping data to map local inflation trends in the US to create its own price index, and while I was writing this, Techmeme editor Mahendra Palsule says that financial market analysts are already doing this and flagged up the Stocktwits service.

[blackbirdpie url=”http://twitter.com/ScepticGeek/statuses/40017866702983168″]

I wonder what other signals we could find in the flow of social media and how this might be used for journalism.

Digital journalists and the battle over newsroom integration

I’ve been meaning to write about newsroom integration for quite a while and so I’ve written about it for journalism.co.uk. The article is based on conversations that I’ve had with journalists in newsrooms around the world and also from some of the well known examples in the industry, including the experience at the Washington Post. A lot of the quotes are unattributed, but I can say that there is a remarkable consistency to the comments I’ve heard.

Last summer, I was speaking to an award-winning digital journalist, and in terms of the fight for integration at his organisation, he asked: “Was there a battle that we lost?”

I want to say up front that I’m not opposed to newsroom integration. In many ways, I am a big booster of bringing digital newsrooms and traditional print or broadcast newsrooms together. I have worked at organisations where the newsrooms have been physically and organisationally separate, and it’s never been productive. I started working in an integrated newsroom in 1998 when I joined the BBC working in their Washington bureau. Not only did I work closely with radio and television correspondents and producers, but I also covered stories for radio and television. Working together was really positive for me and the broadcast staff. My esteemed former colleague Paul Reynolds used to tell me on a regular basis that I was the future of journalism, and I had all the support that I needed and more from bureau chiefs Andrew Roy and Martin Turner. It was a collaboration of mutual respect.

I think that Jim Brady has it spot on when he said that digital editors still need the autonomy to push news organisations in directions that they might not naturally head. Digital innovations are still often counter-intuitive to leaders in legacy media. We still need people who think different at the table.

However, based on conversations with fellow digital journalists and editors, newsroom integration has been very difficult for them, especially for those organisations that have tried to integrate organisationally as well as at the platform level. Many digital editors and sadly far too many digital journalists have been pushed aside or in some cases completely pushed out. From a business standpoint, especially for those news groups suffering financially, the motivation has been efficiency. As Francois Nel said in the piece, integration has to be about efficiency and effectiveness. Francois is the director of the Journalism Leaders Programme at the University of Central Lancashire, and he’s worked with WAN-IFRA to help newspapers groups including the Johnston Press in the UK with their integration efforts.

I’m surprised that news groups continue to pursue the ‘pure’ integration model because for those organisations that moved early in that direction, such as the FT, many have since pulled back. There is a realisation that print and digital often serve difference audiences, and I think this is especially true as digital has continued to develop. Editors now understand (what I’ve known for years) digital journalism is a practice with its own skills and proficiencies just as print reporting or broadcasting. The dream of the super journalist equally proficient at everything was always more myth rather than reality. Few journalists excel at all roles, and even if they did, the demands of any one role, especially in the age of shrinking staffs, would cause sacrifices to be made, corners to be cut.

One area I only touched on briefly in the piece for journalism.co.uk was the role of middle managers. Francois said, in a quote that didn’t make it in the final piece:

The most critical person in the any individual’s daily work life is his or her line manager. And, as such, the best examples of change management are those where courageous and visionary leaders empower and equip middle management to handle the fallout.

Right now, middle management is one of the key issues in terms of integration. The subject comes up again and again in the conversations that I have with digital staffs. Even at organisations with strong digital leaders at the top and willing staff, middle management can still stop change dead in tracks, and often this where the energy comes in terms of marginalising digital leaders. They have the most invested in the status quo and least motivation to change. Middle management can and should lead the charge ahead and create a constructive environment for collaboration. However, in a lot of organisations, this is where change lives or too often dies.

Comments as a premium service?

I’m often asked what are the metrics for success when it comes to blogging or community engagement on a website, and I always respond that it isn’t simply the number of comments. Chasing high comment counts can be a race to the bottom in terms of content as the most provocative content easily gets the most comments creating more of a bare-fisted brawl than a conversation. As time has gone on, more sophisticated community engagement systems and strategies have developed, although these have developed mostly outside of news organisations rather than by them.

One strategy that has started to develop is to view comments as a premium service. Everyone can read comments, but only those who pay can post comments. It’s not a new strategy. Metafilter has been asking people for $5 to comment since late 2004, and it’s actually quite successful. In terms of news sites, Civil Beat in Hawaii requires subscribers to pay to read most content and also to comment.

The BBC College of Journalism has a very interesting post by Tomáš Bella about different strategies in Slovakia and the Czech Republic to reduce the number of comments but improve other metrics such as quality and page views. Tomáš runs a start-up called Piano, a paid content system. Several of the most popular sites in Slovakia have started using Piano to charge €2.90 to comment and access “other premium services on the sites”. It is interesting to view comments as a premium service.

Something else caught my eye in the post though. To comment on the popular Czech site, Novinky.cz, they have instituted a process where you have to apply for a code using your real name and postal address before you can comment. Your real name and home town appear alongside your comment. The result?

This radical approach has worked. Readers’ comments have dropped from 50,000 to 4,000 a day. But the number of page views has risen by a third because the quality of the content has shot up.

Long ago (2005) when I was writing a blogging strategy for BBC News, I realised that the large audiences that major news sites can create for blogs or other participatory efforts might not scale. Open comments are fine on niche blogs such as here at Strange Attractor. People come here looking for specific content and wanting to take part in a specialist, professional conversation. The conversation is manageable because quite honestly, we rarely have that many comments, usually just a few if any and never more than 25. On major news sites, it’s easy to receive hundreds and sometimes thousands of comments. Depending on the content, they can become unmanageable for staff and commenters alike. Making people register or pay is one way to create a speed bump to commenting. That might not be a bad thing.

I know that participatory purists might cry foul saying that this is censorship by credit card, but I think that asking people for a little commitment before they participate might make participation better for everyone. Discuss.

Lesson from MySpace: “Never Stand Still”

MySpace has gone from fast rising social network to a fast falling has been, waiting to be sold by News Corp. In terms of mindshare, it’s already joined the deadpool. Chris Thorpe flagged up this great blog post this morning:

[blackbirdpie url=”http://twitter.com/jaggeree/status/32066764669980672″]

The blogger, John Willshire, is also a member of a band, and he managed to cobble together a new presence for his band, Gamages Model Train Club, by pasting together a number of other services: Tumblr, Soundcloud, iTunes, Facebook, Google Analytics and Feedburner.

Most articles written about the fall of MySpace focus on the lack of focus after News Corp acquired the social network. Rupert Murdoch turned his attention to other areas of his empire, and MySpace simply failed to keep pace with Facebook.

John’s big lesson from MySpace is that digital businesses must never stand still. As he says, MySpcace was never easy to use, which left open the huge opportunity that Facebook exploited. More than that, while MySpace had been improving incrementally before News Corp acquired it, those improvements stopped after it was bought by the Beast.

Don’t launch things small and often, let’s launch things infrequently, but talk about all the changes we make at once.  So don’t talk about anything in between that spoils this, please.  Oh, and we need a major review of what we’re doing with the site, platform, tech, so don’t do anything till that’s complete….

…and so and and so forth.  By trying to sort everything all at once, as old, established companies want to, and as you’d have to do with a newspaper redesign, Myspace stopped evolving.

Lessons for news organisations

When I started in digital journalism, I worked for small organisations, and we ran on shoestring budgets. It wasn’t until I joined the BBC that I worked for a well-resourced operation, and even then the BBC News website had (and still has) a tiny budget compared to BBC TV (but then my former colleagues in radio will say the same thing, and be right).

However, with integration, bringing together the print and digital or broadcast and digital newsrooms, resourcing is improving, but those digital organisations are now being brought into larger, sometimes less nimble, organisational structures. It’s a challenge that only a few news organisations have successfully met.

Prioritisation: How do news organisations decide what they must do?

Since leaving The Guardian last April and striking out on my own, I joke that I’ve become an occupational therapist. I’ve had the chance to speak to journalists, editors and media executives around the world and hear the issues and challenges facing them. Most of them are instantly familiar, but one issue that I first heard about in Norway in 2009 and heard with increasing frequency in 2010 was prioritisation. In digital, there are a myriad of things we could do, but in this era of transition and scarce resources, the real question is what we must do.

In the comments on my last post covering the opportunities for news organisation in location services and technologies, Reg Chua, the Editor-in-Chief of the South China Morning Post, had this insight:

Kevin’s point on prioritization is a critical one. We can’t do everything well; in fact, we can’t do everything, period. I’d argue that we should think about the product we want to come out with first – and then figure out what data is needed to make it work. I realize that leaves some value on the table, but I suspect we all need to specialize more if we’re to really create products that have real value.

For the news organisations that I’m working with now in developing their digital strategies, one of the things that I look at is where they have the most opportunity. I agree with Reg that there are a number of opportunities in terms of creating products using data, and I also think that data is important in determining which products to develop. News organisations need to get serious about looking at what their audiences find valuable, digging into their own metrics. Right now, we’ve got a lot of faith-based decision making in media. It’s critical that we begin to look at the data to help determine what new products we should deliver and how we can improve our existing offering.

For a good start on this kind of thinking, Jonathan Stray wrote an excellent post last year, Designing journalism to be used. He wrote:

Digital news product design has so far mostly been about emulation of previous media. Newspaper web sites and apps look like newspapers. “Multimedia” journalism has mostly been about clicking somewhere to get slideshows and videos. This is a little like the dawn of TV news, when anchors read wire copy on air. Digital media gives us an explosion of product design possibilities, but the envisioned interaction modes have so far stayed mostly the same.

This is not to say that the stories themselves don’t need to change. In fact, I think they do. But the question can’t be “how can we make better stories?” It must be “who are our users, what would we like to help them to do, and how can we build a system that helps them with that?”

Josh Benton of the Nieman Lab says that we have an opportunity to rethink the grammar of journalism during this period of transition in the business. Jonathan and Reg are definitely thinking about rethinking the grammar and rethinking the products that we create. Digital journalism is different, and the real opportunity is in thinking about how its different and how that creates new opportunities both to present journalism and support it financially.

Location: News organisations must seize this opportunity

Since I started geo-tagging content during my trip across the US for the 2008 elections, I’ve been interested in the possibilities of location-based services and news. Location is one way to deliver timely, relevant content to audiences. Smart news organisations such as TBD.com in Washington DC in the US are already leveraging geo-tagging to deliver their content, and now Examiner.com has struck a deal with Foursquare in 288 cities. MocoNews.net, part of the paidContent network, is reporting that:

In essence, Examiner’s 68,000 contributors, known as “Examiners,” will provide reviews and recommendations on nearby venues, restaurants, events, businesses and landmarks that will surface within the Foursquare mobile app when users following Examiner.com check in.

This is one of the opportunities that news organisations must not miss. Location allows for better delivery and discovery of content by readers, but it can also deliver new revenue streams to support journalism.

Hacking the BBC

I spent several months last year working with a team of people, including Kevin, on a retrospective of BBC Backstage, which has now been released as a free ebook. As I wrote for the introduction:

BBC Backstage was a five year initiative to radically open up the BBC, publishing information and data feeds, connecting people both inside and outside the organisation, and building a developer community. The call was to “use our stuff to make your stuff” and people did, to the tune of over 500 prototypes.

This ebook is a snapshot of some of the projects and events that Backstage was involved in, from its launch at Open Tech 2005, through the triumph of Hack Day 2007 and the shot-for-web R&DTV, to current visualisation project DataArt. We take a diversion to Bangladesh to see how a Backstage hacker helped the World Service keep reporting through the horrendous Cyclone Sidr, and look at the impact of the ‘playground’ servers, used inside the BBC.

Backstage’s mandate, throughout its history, was for change. It changed the way people think, the way the BBC interacted with external designers and developers, and the way that they worked together. So what remains, now Backstage is no more? The legacy isn’t just a few data feeds and some blog posts. Backstage brought about permanent change, for the people who worked there, for its community of external developers and for the BBC. What better legacy could one ask for?

Jemima Kiss has written a great piece on The Guardian about it:

Night has finally fallen on the visionary and quietly influential five-year project that was BBC Backstage, a collaboration of ideas, experiments and talent that informed and defined some of the corporation’s best technology work.

Now set to be replaced by a cross-industry developer network – a repository for data from many media organisations and tech companies – this special corner of the BBC devoted to developers has been wound down.

[…]

Woolard talks of Backstage in three phases: creating a space to make this kind of experimentation and open innovation possible; engaging the developer community; and a third stage that takes these findings and this attitude of openness further across the BBC and its output. He points to last year’s BBC 2 series Virtual Revolution, which explored the impact of the web, and was heavily influenced by the R&D TV project led by Rain Ashford, which also filmed wide-ranging interviews with high-profile technologists and allowed viewers to cut and shape footage for their own use.

Now, says Woolard, it is normal to talk about openness, innovation and working with external developers – and he claims the BBC is “fully technology conversant” in what it needs to do.

Backstage was born on the same day as the Open Rights Group, on 23 July 2005 at the Open Tech conference. I was rather busy bootstrapping ORG whilst Backstage was getting off the ground, but I kept my eye on it nonetheless.

Backstage and ORG had a lot in common beyond their birthday – a desire for more openness, a fervent dislike of DRM, and a strong DIY ethic. We also shared a community: many of our supporters were also Backstagers. In many ways, we were two sides of the same coin, with ORG campaigning publicly for change, especially around DRM, and Backstage working the inside line. I like to think we had a nice pincer movement going on.

Backstage had some amazing moments, most notably HackDay at Ally Pally, which I sadly missed due to being on a plane at the time. Spending time last year reading over old blog posts, it was great to be reminded of what a vibrant community Backstage built and what groundbreaking work they did, especially with events. Most importantly, they redefined expectations around open data and standards.

Congratulations to everyone who was involved in Backstage, and huge thanks to the Hacking the BBC team!