Data dot (dot, dot): the story of open government data

The TAAI Open Data Study

Don't judge a book by its cover: the open data study

When, earlier this year, the Ordnance Survey announced it would open up (most of) its mapping data for remix and reuse, my first thought was to check the calendar. Sure enough, as with EMI’s announcement in 2007 that it would drop DRM, it was 1 April. But – also as with the EMI announcement – this was no April Fool. After years of campaigning by a grassroots of the digitally-savvy and dedicated, a major organisation had agreed to change its business practice. This was for real.

William Heath and I had already agreed over a rather delicious lunch at the October Gallery that someone needed to tell the story of how open data had proved – as a campaign issue at least – such a success. I’ve been following the rise of mySociety since I interviewed mySociety’s Tom Steinberg about what all this “civic hacking” was about in 2004, the year TheyWorkForYou launched (with, ahem, “borrowed” data). Obama’s “data.gov” portal of reusable federal data was released in 2009, and data.gov.uk, the UK’s own (superior) open data portal, was publicly released this year. I’d be hard pressed to think of an idea that has permeated as quickly as open data has from the fringe to the centre. What did the open data people do so right?

Tim Berners Lee gets the audience chanting at TED 2009

I was delighted when the Transparency and Accountability Initiative (a consortium of funders and NGOs, including DfID, the Omidyar Network, Hewlett, Ford and OSI) approached me in April to answer just that question. Their motivation was to find out what the strategy was, and whether it could be repeated in middle income and developing countries. The result is this report, nattily titled Open Data Study (yes, the title sucks, all the ingenuity went into the text itself). I really enjoyed writing it (especially getting to interview Tim Berners-Lee over video phone) and I hope you’ll enjoy reading it.

Not all of it is easy reading. Digging into the subject meant confronting quite a few of my own misconceptions of the open data story. Here are just a few of the points that surprised me:

  • This is not a story of the plucky grassroots winning out over all. “It has to start at the top, it has to start in the middle and it has to start at the bottom”, as Sir Tim puts it. Without a sleeper cell of dedicated and skilled civil servants who could see what open data was about and how it could help them, this project would never have gotten off the ground. And without a good political reason to open up government data, the project would never have soared to the heights it did.
  • You can’t just fly Sir Tim around Africa and expect him to leave a trail of glistening open data portals in his wake. There are capacity issues. Not everything is computerised. Sometimes, data sets we would regard as core don’t even exist on paper, let alone online. And pockets of corruption are sometimes so intense that releasing data openly could be a life or death thing. There’s good news too – multilateral and bilateral donors could play a strong role in getting data open, and they can start by leading by example.
  • The open data project was sold on its potential and not on its proven impact. I write “More often, it was the utility of applications (in contrast to the resources expended to produce them) and not their broad user bases, which seem to have inspired officials further up the line to engage with the open data agenda.” Of course, this is neither good nor bad: lots of things are sold on their potential, including the ID cards scheme (on second thoughts, maybe that was sold on fear), the NHS data spine…and probably some good things too.

As well as Sir Tim, the report contains interviews with the lovely Ory Okollah of Ushahidi, Jonathan Gray of the OKF, Tom Steinberg, Ethan Zuckerman, and many, many more. It’s already attracted comment from Glyn Moody, and Ory’s done a very kind write-up.

I hope you enjoy it, and do leave your thoughts in the comments.

Links for this week

Every week, I help compile a short mail out of interesting stories for the Open Society Institute’s Information Program, which aims to update their colleagues in the Soros network and friends further afield about the news, opinions and events the Program team have their eye on. Since the mailout is released Creative Commons, and usually contains a really excellent spectrum of information society issues, I’m going to start sharing the links on this blog. Here’s the first issue:

Why is there resistance to prize funds?
Prize funds are mechanisms for stimulating innovation. In healthcare, prize funds present an attractive alternative to patent-based mechanisms, and can be particularly useful in stimulating research around treatments for neglected diseases. Here, Jamie Love of Knowledge Ecology International tracks the history of prize funds, and examines why they still encounter such resistance from commercial organisations.

The death of fixed lines in Africa
Steve Song challenges the notion that Africa’s future communication infrastructure will be 100% mobile, and offers three reasons why fixed line communication should be viewed as a complementary technology with its own role to play.

The Partisan Internet and the Wider World
Is the internet making us ideologically isolated? Ethan Zuckerman dissects a recent study of how the internet affects our political views and associations, providing a good summary of the various arguments around this question so far, and raising some interesting questions about the most current research.

Access Controlled: The Shaping of Power, Rights, and Rule in Cyberspace
This new book from MIT Press, which features chapters from academics and practitioners associated with the Open Net Initiative, examines the state of privacy and free expression online, and the trends that got us here. Part II contains a series of informative regional profiles.

Deploying Ushahidi – Allocation of Time
Ushahidi community member Chris Blow shares his experiences of deploying Ushahidi, and reveals where Ushahidi-based projects most often fail.

Diasporas: A new sort of togetherness
This Economist feature posits that diasporas are being revived by social media.

The web’s first Cyrillic top level domain
This month, the first Cyrillic top level domain “.рф” (“.rf”) came online. This article examines take-up of the new domain, and questions whether this introduction of Cyrillic threatens the internet as a global and open space.

Facebook responds to privacy outcry
Facebook has announced a new simplified approach to user privacy, following growing concerns that the company was misusing personal data by making it harder to opt out of commercial data-sharing schemes. The new arrangement will provide users with a single page from which they can control all their privacy settings.

Tajikistan: President moves to ban cell-phones
President Rahmon of Tajikistan has “declared war” on mobile phones, citing dangers to health they are alleged to pose, and complaining that the revenues of Tajikistan’s burgeoning telecommunications sector are difficult to track. One source quoted in this report suspects that the revolution in neighbouring Kyrgyzstan might also have something to do with the move.

East African Community countries to register all SIM cards
The East Africa Communications Organisations (EACO) has agreed to register all mobile phone SIM cards across the region by June 2012. A representative from the Uganda Communications Commission said “The decision to register SIM cards was reached at for security purposes and for protection of our consumers,” claiming that registering SIM cards would stop anonymous, abusive phonecalls.

Iran’s Revolutionary Guards Claim “Iranian Cyber Army”
A spokesperson for Iran’s Revolutionary Guard has claimed that the “Iranian Cyber Army”, a group of hackers implicated in attacks on Twitter, Baidu and reformist Iranian web targets, “looks to” the Revolutionary Guard for direction.

Three Strikes… and you’re out?

Could it be the curse of Gowers? It’s been over 3 years since the former Financial Times editor made his seemingly doomed 39th recommendation in his otherwise excellent review of intellectual property:

Observe the industry agreement of protocols for sharing data between ISPs and rights holders to remove and disbar users engaged in ‘piracy’. If this has not proved operationally successful by the end of 2007, Government should consider whether to legislate.

Since then, it seems every minister tasked with overseeing implementing his idea has only just got in post before they are running for the door again. The news today that Sion Simon MP will step down as creative industries minister (and, indeed, get out of Westminster altogether) inspired me to write a list of all the other Ministers I could think of who’ve once been responsible for this policy, then legged it.

  • Baroness Delyth Morgan (reshuffled)
  • Baroness Shriti Vadera (stepped down to advise G20)
  • Andy Burnham MP (promoted)
  • Lord Stephen Carter (resigned)
  • Sion Simon MP(resigned)

I’m not sure this list is chronological, or indeed exhaustive – please help me revise it if you can. And of course, it may yet be incomplete. Last Summer, I predicted that the Digital Economy Bill – the latest zombie incarnation of this wretched policy – would be Peter Madelson’s undoing. I remain hopeful I was right.

Stewart Brand

I’ve got a review of Stewart Brand’s new book Whole Earth Discipline in this week’s New Statesman. It doesn’t appear to be on their website yet, so I’ve re-produced it below.

I became interested in Stewart Brand after reading Fred Turner’s excellent (if a little dry) book From Counterculture to Cyberculture last year (see this previous post). Like a secret time-traveller, Brand pops up at various moments that would go on to define the development of personal computing: filming the “mother of all demos” in the 60s; funding the Homebrew Computer Club in the 70s; coining the phrase “information wants to be free” in the 80s. You can hear me interviewing Stewart for my favourite independent radio show, Little Atoms, on Resonance FM next Friday.

Here’s my review (note this is unedited copy, but it doesn’t differ much from what you’ll find on p.54 of today’s Staggers):

Social entrepreneur and technology guru Stewart Brand’s first significant contribution to the environmental movement came to him as he sat on a rooftop in 1966. Tripping on LSD, he looked up at the stars and asked “Why haven’t we seen a picture of the whole earth yet”? Forty years later, Brand has given up the drugs and the mysticism of 1960s San Francisco, but he’s still thinking about the planet. This time, he doesn’t just want a photograph (that happened in 1968, leading to the first Earth Day in the US), he wants “a constant, real-time high-resolution video of the Earth turning in the sunlight” – the Deep Space Climate Observatory (DSCOVR), abandoned before launch by the Bush administration because it had been Al Gore’s idea. Much more than DSCOVR, though, Brand wants us to break free from our various ideological shackles and begin focussing on the task at hand – saving civilisation.

Whole Earth Discipline: an Ecopragmatist Manifesto is a rich, compelling guide to how old wisdom and new technologies can combine to help civilisation survive manmade climate change. But it should be read as much for its dissection of the way ideologies distort decision-making on science and technology. Why did the anti-state right oppose fluoridisation and the anti-corporate left oppose GM? “A political agenda is… poor at solving problems”, writes Brand. “Accustomed to saving natural systems from civilisation, Greens now have the unfamiliar task of saving civilisation from a natural system”. The ensuing ideological backflip will spread its own kind of chaos – a chaos budding ecopragmatists must learn to sidestep.

The book proposes three ideological heresies about to break on the shores of environmental consciousness: urbanisation; GM; nuclear. Earth’s population became mostly urban in 2007. The dream of going back to the land – an ideal Brand gained his fame by promoting in Whole Earth Catalog, a kind of Sears for hippy communes – is wrong-headed, because cities turn out to be the greener option. Urbanisation slows population growth (as women choose education and opportunity over large families), concentrates resource needs, and gradually empties rural areas of subsistence farmers, allowing planned approaches to agriculture that reduce environmental impact and leave more land surface to be gardened into “natural” ecosystems that will mitigate climate harm.

But urban populations demand grid electricity, and that means re-evaluating the nuclear option. The rejection of nuclear stems from our revulsion of nuclear weapons; the “absolute” nature of our other concerns – on grounds of safety, cost and waste storage – all flow from here. Brand dismisses each objection with a mix of hope and hard science. A trip to the experimental Yucca mountain 10,000-year storage facility leaves him agreeing with James Lovelock that “we need it about as much as we need a facility for imprisoning dangerous extraterrestrials”. We should divert the $28bn set aside to store waste from the nuclear we’ve already used towards research into new micro-reactors and the possibility of substituting uranium with thorium. Brand’s own ideological shift – here and elsewhere – is away from 60s individualism toward 21st century governance. Post-Copenhagen, we might wish that his proposed blend of the internet-inspired engineer/hacker frame with approaches to economic planning that might alarm the folks back home was a little less vague.

Most compelling is the book’s defence of GM agriculture. Here, Brand the trained biologist puts the leaders of Greenpeace and Friends of the Earth in the same dock as the leaders of Exxon Mobil for their crimes against science and humanity. Environmentalists who label GM “unnatural” have confused agriculture with nature, when agriculture itself is one of the world’s worst climate criminals. The work of Greenpeace and Friends of the Earth in demonising GM has left millions of Africans starving to defend a misguided European ideology. The organic and GM movements must converge around the shared goal of soil quality through no-till agriculture – the only thing stopping them from doing so is moral panic. Brand drolly reminds us that Frankenstein was the inventor of a creature that was mistaken by the public for a monster. “Of course, that’s a rhetorical argument, devoid of meaning. But so is the term Frankenfood.”

Urbanisation, nuclear, GM: all will happen whether the environmental movement adopts Brand’s manifesto or not. But if Greens take up his call, if they start working to “Green the Hell” out of the world’s new mega-cities, go “Glow-in-the-Dark” Green and make sure nuclear power adoption is directed in the right way, or encourage GM technology out of the patent portfolios of Monsanto and into the hands of local specialists (“Biotech wants to be free”), all three will happen better. Like adolescents emerging into adulthood, it’s time to put away our inner grudges and get used to the idea that we alone are masters of our own destiny. “We are as gods and have to get good at it”.

Stewart Brands campaigning buttons

Google in China, Pt. 3

I’m enjoying some of the early comment on Google’s announcement that it no longer intends to censor google.cn. When Google first went into China back in 2006, I predicted that both Chinese-sponsored cybercrime and user privacy were likely to feature in future twists in the tale of Google in China. The company appears to be performing much better than the subtext of this piece implied it would. So hats off, I guess.

One thing I’d like to know is what’s happened to Google’s 2.4% stake in Baidu – the search engine with majority market share in China – in the intervening years since I wrote this piece. Answers sent over https to my gmail account, please!

RATM for Chirstmas Number 1

This week’s Net Office went live this morning. Original copy here, final copy over on the NS website.

The Mirror is reporting today that RATM are 60,000 ahead of McElderry, but that CD sales may close that gap. I don’t mind saying that I haven’t been this excited by the charts since Blur vs Oasis.

And so, the decade ends with a clash of ideologies. In the blue corner, we have the individual, coiffed, tanned and flossed, battling against what life throws at him, relishing each challenge without questioning the system that created the obstacles he must overcome. In the red corner huddle the united masses, dreadlocked and disaffected, aware of the complex elites that govern their lives and ready to overthrow them through the simple act of violent rebellion. The fight for Christmas number one has never been so exciting.

Before Joe McElderry had been announced as X factor winner last Sunday, members of the 700,000-strong Facebook group “Rage Against the Machine for Christmas No. 1!” were already purchasing downloads of the US alt-metal outfit’s 1993 single “Killing in the Name”. Their aim? To boot McElderry’s disinfected country ballad “The Climb” off its almost guaranteed number one spot, in favour of a track whose lyrics contain the word “fuck” 17 times. If each of the 700,000 Facebook group members defy the economics of collective action and download the track this week, their victory is almost assured.

The Titanomachia of old and new media is compelling. Simon Cowell has branded the RATM campaign “cynical”. In reality it is anything but.

“Killing in the Name” was released back when the recording industry still seemed authentic – at least to the middle class kids who made up most of RATM’s fans then, and probably most of the Facebook group now. Yes, RATM are signed to Sony. But in 1993 Sony and the like hadn’t yet commenced their war against the internet – and by extension against all young people. It’s a war that is right now culminating in Westminster, as legislators debate a Bill with shady provisions for punitive action against illicit filesharers, that gives the Secretary of State carte blanche to devise enforcement measures in favour of record labels. What the RATM Facebook action recognises is that the music is ours, as well as theirs – that years after a track has been produced, hundreds of thousands of people can still be moved by it to take action, however trivial that action might seem.

So if you fancy your hope a little subversive this Christmas, join the Facebook group and get downloading. And as you’re pogo jumping to some of the best guitar riffs of the nineties, remember, the devil doesn’t have to have all the best tunes.

Spotted: me at Resonance FM’s Media Playground

I’ll be at The Foundry in London on Saturday the 19th, taking part in Resonance FM’s Media Playground event:

At 2pm: discussion for broadcast – Pathological Over-Sharing. Featuring Becky Hogge (New Statesman, Open Knowledge Foundation), Ken Hollings (Destroy All Monsters), Mark Rock (Audioboo, Best Before Media) and Paul May (The Whale in the Room). As social networks proliferate, newspapers retreat and self-exposure goes global, a panel of culturally savvy thinkers gather to address such questions as, Is the internet a tool for democratic change or economic repression? What is the individual’s impact on media? And what is the impact of the media on social space?

More details here.

Climate emails

My column went live at the New Statesman yesterday. Unedited draft here, final version here.

What does a journalist do when a metric conspiracy-load of private emails between scientists land on his desk which he has neither the time to read, nor the skills to dissect? As the Climate Research Unit hacked emails story has developed, we’ve seen several different answers to this question.

On Today last Monday John Humphreys and Ed Miliband, having both agreed with each other that neither of them was a scientist, proceeded to strip down to their metaphorical loin cloths and dance around the totem they had built together out of “the science”. The previous Saturday, Simon Heffer in the Telegraph had précised his denialist contribution to the debate with “I have not so much as an O-level in physics or chemistry”. The polemic was illustrated with a picture of a green-haired climate protester screwing her face to camera, sardonically captioned “the voice of reason”. Thus did it perpetrate that most common of crimes against the Enlightenment: confusing ideology with reality.

Like the scientific method – itself more cybernetic than democratic – the hacked emails debacle is very much an internet story. More, it is a story of the public web, whose high incidence of “flat-earthers”, sceptics and chat room mavericks has apparently helped dissuade the CRU from hitherto publishing their data and workings, versus the private web, across which those same scientists bounced email after email, amassing a decade-large corpus that would make Cardinal Richelieu giddy. Whoever sent the whole thing to Wikileaks less than a month before the Copenhagen negotiations knew what they were doing. The mainstream media mostly enjoys telling stories. Twists in the tale are likely to be evaluated less on merit and more on where they take the narrative.

The constant questioning and debate that indicate healthy scientific discourse look entirely different to a media obsessed with the U-turn, that nasty little concept that distils every public debate into something slightly less sophisticated than a football match. But whatever comes out of Copenhagen this week, it will be the beginning of science’s involvement in the public discourse, and not the end. For society to survive, we will have to make good choices, and they will be choices about science and technology. Totems, ideology and story-telling will not be useful. If the public is to have any kind of scrutiny over these choices, the media need to get used to an altogether different type of refereeing.

9/11 pager messages

This week’s column now live at the New Statesman. Unedited copy below, final version here.

If extraterrestrial life were to swing by, the first impression they got of our blue-green planet might well be cacophony. Last month’s Wikileaks release of pager messages sent on September 11 2001 is one testament to this. Over half a million messages, intercepted in New York and Washington DC for the 24 hours surrounding the World Trade Centre attacks, were released by Wikileaks, broadcast in a kind of sync with the day they documented. The 12MB file of text messages can still be downloaded from the Wikileaks website. They are credible and their provenance is undisclosed.

The messages paint a surprising, if chilling, picture. They are sent by both machines and humans, almost in equal proportion. Computers running major parts of the globe’s financial infrastructure deliver warnings about their faltering connectivity (“08:46:46 Market data inconsistent…Cantor API problem Trading system offline”). Humans deliver messages to employers and loved ones. Some call in sick (“06:50:48 THIS IS MIKE. I HAVE TO TAKE MY SON TO THE DOCTOR… ”), others send saucy greetings (“06:31:26 Got my zebra thongs on!!!”), a few talk about bagels, furniture deliveries. Those that have heard the news, send messages of panic – “10:07:46 Don’t leave the building… Please be careful. Love you – Tiffany”. The panic intensifies: “10:35:50 PLEASE PRAY…”

In the background, heard quietly but consistently, the Twin Towers fall. policemen and members of the secret services send messages to each other as the situation escalates: “08:50:50 BOMB DETINATED (sic) IN WORLD TRADE CTR. PLS GET BACK TO MIKE BRADY W/A QUICK ASSESSMENT OF YOUR AREAS AND CONTACT US IF ANYTHING IS NEEDED”; “09:21:44 US bombers are in the air in-route to Clasified (sic) targets waiting for strike orders.”; “10:24:31 TWINKLE AND TURQ (codenames for George Bush’s daughters) ARE ACCOUNTED FOR AND SAFE”.

Humans with their artefacts, crashing about to save themselves or else causing this destruction in the first place. The messages we and our machines sent that day are a new kind of news – raw data news, or sousveillance news, perhaps. Like spies, the community news portal Reddit dissected the raw material (tinyurl.com/y999cyj), sifting through the data by hand, and devising scripts to extract word frequencies (“please” came out top) or the pager numbers of all the Secret Service agents. Can we expect this window to open on all future news events? Perhaps not. But on this particular one, it bears a powerful aspect.

Dead Pirates

Currently enjoying this video: