Data dot (dot, dot): the story of open government data

Don't judge a book by its cover: the open data study

When, earlier this year, the Ordnance Survey announced it would open up (most of) its mapping data for remix and reuse, my first thought was to check the calendar. Sure enough, as with EMI’s announcement in 2007 that it would drop DRM, it was 1 April. But – also as with the EMI announcement – this was no April Fool. After years of campaigning by a grassroots of the digitally-savvy and dedicated, a major organisation had agreed to change its business practice. This was for real.

William Heath and I had already agreed over a rather delicious lunch at the October Gallery that someone needed to tell the story of how open data had proved – as a campaign issue at least – such a success. I’ve been following the rise of mySociety since I interviewed mySociety’s Tom Steinberg about what all this “civic hacking” was about in 2004, the year TheyWorkForYou launched (with, ahem, “borrowed” data). Obama’s “data.gov” portal of reusable federal data was released in 2009, and data.gov.uk, the UK’s own (superior) open data portal, was publicly released this year. I’d be hard pressed to think of an idea that has permeated as quickly as open data has from the fringe to the centre. What did the open data people do so right?

Tim Berners Lee gets the audience chanting at TED 2009

I was delighted when the Transparency and Accountability Initiative (a consortium of funders and NGOs, including DfID, the Omidyar Network, Hewlett, Ford and OSI) approached me in April to answer just that question. Their motivation was to find out what the strategy was, and whether it could be repeated in middle income and developing countries. The result is this report, nattily titled Open Data Study (yes, the title sucks, all the ingenuity went into the text itself). I really enjoyed writing it (especially getting to interview Tim Berners-Lee over video phone) and I hope you’ll enjoy reading it.

Not all of it is easy reading. Digging into the subject meant confronting quite a few of my own misconceptions of the open data story. Here are just a few of the points that surprised me:

• This is not a story of the plucky grassroots winning out over all. “It has to start at the top, it has to start in the middle and it has to start at the bottom”, as Sir Tim puts it. Without a sleeper cell of dedicated and skilled civil servants who could see what open data was about and how it could help them, this project would never have gotten off the ground. And without a good political reason to open up government data, the project would never have soared to the heights it did.
• You can’t just fly Sir Tim around Africa and expect him to leave a trail of glistening open data portals in his wake. There are capacity issues. Not everything is computerised. Sometimes, data sets we would regard as core don’t even exist on paper, let alone online. And pockets of corruption are sometimes so intense that releasing data openly could be a life or death thing. There’s good news too – multilateral and bilateral donors could play a strong role in getting data open, and they can start by leading by example.
• The open data project was sold on its potential and not on its proven impact. I write “More often, it was the utility of applications (in contrast to the resources expended to produce them) and not their broad user bases, which seem to have inspired officials further up the line to engage with the open data agenda.” Of course, this is neither good nor bad: lots of things are sold on their potential, including the ID cards scheme (on second thoughts, maybe that was sold on fear), the NHS data spine…and probably some good things too.

As well as Sir Tim, the report contains interviews with the lovely Ory Okollah of Ushahidi, Jonathan Gray of the OKF, Tom Steinberg, Ethan Zuckerman, and many, many more. It’s already attracted comment from Glyn Moody, and Ory’s done a very kind write-up.

I hope you enjoy it, and do leave your thoughts in the comments.