A few random very personal thoughts about open data...

So this isn’t thought through at all and is going to be very random as a result. But also by design – I’d like this to be the start of a conversation. Too much to hope for? Well maybe but here you go. Maybe I’ll just keep chipping in a few thoughts and hope all you experts and enthusiasts will chip back. Chip back? Hmmm.

Well by background and training and for my whole working life I’ve worked with stats and data. I’ve done lots of quant analysis and lots of qual work, and I’ve managed surveys and managed other more intelligent people than myself to do analytical research. So I think we live in intriguing times. I won’t go over that here as I am sure if you are reading this (anyone anyone??) you understand the landscape. Suffice to say the Open Data movement is changing the way government works and thinks about the way it works, and is revealing the way in which government works (or doesn’t work) in very new ways.

This feels like deja vu to me because I’ve been working for a very long time now on releasing detailed local statistics. I think what is different at the moment is that the world of data has suddenly entered the mainstream. Researchers and analysts are of course still interested but developers and journalists and social media gurus and politicians..and well everyone really…is suddenly recognising the power of information and more importantly thinking of it in terms of data. And this is simply (I think) down to the internet – we’ve moved to a whole new way of working with and managing information.

But this makes working in this landscape incredibly complex because everyone comes at things from a different perspective, with different aims and with very differing experiences. It seems that literally in the last few weeks, maybe a few months at most, that the community of evangelists for opening up government data is now starting to say – ‘well great we are starting to see data but make sure it is accurate, make sure we understand the quality controls, the context, and make sure we can analyse and interpret in a meaningful fashion’. Erm…well yes. I’m not being funny but that should be obvious really shouldn’t it?

And of course they are right – to date government has responded to the call for open data by flinging open its systems and throwing it out into the ether. They have responded to developers calls and said ‘here is the data now do with it what you will – magically hold us to account and improve democratic engagement’.

Good luck with that.

So why deja vu? Well simply because anyone who comes from a stats/analytical background will tell you that information is beautiful but only when you understand that information. And they’ll tell you that there is a wealth of local information out there but that you have to deal with the context and the geography and the statistical hierarchies and the time points and the quality and the sheer quantity and and and… Damn lies and all that…

And of course in the context of local and central government the last 10 years has seen massive investment in statistics – in the collection and management of data and in the production of tools/websites etc so that anyone can undertake analysis of that information. And with the budget cuts most of that is being thrown away. And I won’t go into that here but I don’t necessarily think that is a bad thing. But it leaves a hole and that is what the new evangelists for open data seem to be realising. The Government line now seems to be very firmly – ‘we’ll make the data available but it is over to you to validate and analyse and democratise’. And I guess when it comes down to it the question is will this be enough or will it simply mean we have a level of openness (which I whole heartedly applaud) but little of substance to really inform and engage and empower citizens?  Will we simply drown in meaningless data, that provides a steady flow of reactionary headlines but little else? There is a clear analogy with The Big Society – yes we will do this but we are already and we only have so much time and at some point need to make a living. I love data, I love information and I love the new world of apps. It takes me back to my youth playing on my zxspectrum. The idea that one person can have a good idea and develop an app and maybe make some money too is incredibly powerful. But someone please tell me how to monetize the vast majority of government data? Yes TfL data because we’d all pay a little bit to be able to handle our commute more effectively, but what about deprivation data and complex financial information? A simple app that a large number of people will actually pay for?

Now again the government line on this is simple – ‘We in government are not creative (speak for yourselves buddies) but others are. We can’t even begin to imagine what amazing things will be done with the data we make available’

Well fine. Yes there are better minds out there. But you know what if you feed them partial data with little information on validation, little context and little comparability? Don’t the great minds out there have better business ideas to inspire them, more pressing issues they want to deal with. Yes holding government to account is vital but back in the real world people want to deal with education, and health care, and care for the elderly. Yes this is all tied up – if you take the view that information is critical to delivering better services and being accountable than everything can radiate out from data – but…but..but how can we do this without real support for developers and analysts and researchers?

So many threads I don’t know where to start. So over to you. Am I wrong? Are you all out there building fantastic apps based on government data. Do you know how to make a living from it? Maybe you don’t care, maybe you are happy to dedicate all your spare time to this?

Thanks!

ade