Open Data: Goin’ local, down in….wherever you live….

This post introduces my new demonstration “Local Data Discovery” dashboard, explaining my rationale and motivation for developing it.

I had the pleasure of attending an event on 24 March hosted by Francis Maude, to celebrate the fantastic work by Local Authorities leading the way on transparency and open data.

For me, this was a timely reminder of the sheer volume and variety of data that is now available about local areas.  The event also highlighted the truly innovative/inspirational applications that Councils have developed themselves, and in partnership with others – often despite significant lack of funding and resources.

I came away wondering three things:

1. how and where can I discover and access open data produced by the Local Authorities in and around where I live?
2. Once I’ve found what I’m looking for, how do I know this is the latest/authoritative information?  Has the Council or its neighbours published more up-to-date information on my topic of interest?  And are there other datasets relevant to that topic?
3. Has anyone else used the data I need, or asked for similar/alternative and as yet unpublished information?

OK, that’s more than 3 things, but you get my drift.  Looking at this from a user’s perspective,  I guess this boils down to ensuring that I can quickly find and use the right data in the right way, for my particular local issue or question.

Regarding accessing the data….broadly speaking, I have a choice of three main channels:

1. data.gov.uk and its growing collection of datasets registered by Local Authorities;
2. Datasets registered with the LGA’s OpenData service;
3. The Council’s own local data portal and/or website.

But how do I know whether my council/s of interest have made data available via one or more of these channels?  And, can I be sure that I’ll always get the latest version of a particular dataset regardless of which channel I use?

Regarding using the data…the great Mark Braggins has reminded me that users will want to incorporate our data and their analysis within their preferred communications and visualisation channel: this could be a local news and blogging site, or even a good old-fashioned letter/e-mail to the local council.  The obvious challenge for us (and I include myself here, as a publisher) is to be smarter at injecting our data into the various conversations and collaborations happening at local-level.   Within that, we need to be better at knowing what data folks are using and/or asking for.   Which got me wondering (again) – what do we already know about demand via things like FOI requests?

To help begin to unpick and answer these questions, I’ve develop a demonstration “Local Data Discovery” dashboard.   This has been inspired by a conversation with Mark Braggins, and also earlier work by Owen Boswarva – from whom I’ve re-used (“borrowed with pride”) the dataset underlying his UK Local Government Open Data Resources map.

At this stage, it is no more than an attempt to spark more debate around how to be smarter at providing, blending and using data from multiple sources about different places and topics of interest.   I’ll resist the temptation to rant about standards,  which I think are fundamental to achieving this web of integrated local data.    I’ll instead end by simply asking: how do we build on the great work by Local Authorities, and get our data used in other people’s websites and software tools?

Introducing the app

The app presents a table and map comprising four key facts about each Council:

1. The number of datasets registered on data.gov.uk;
2. The number of datasets registered with LGA’s Open Data schema service
3. The number of datasets registered with LGA’s Open Data Inventory service
4. The number of FOIs registered with whatdotheyknow.com

The table also includes links to the Councils Open Data landing page, and open data portal where I know about them –  thanks again to Owen Boswarva for curating this list.

Clicking on rows in this table should get you more detailed information about the selected Council.   Note that I’m retrieving contact information from DCLG’s OpenDataCommunities service in real-time.   We source updates to this dataset from the list maintained by LocalDirectGov.

The list of datasets registered with data.gov.uk is retrieved in real-time by querying their CKAN API.    Information on datasets registered with LGA’s Open Data service is again retrieved in real-time, using Yahoo’s Query Language.

How it works

Under the bonnet, the app is powered by this Google Docs sheet https://docs.google.com/spreadsheets/d/1pW0Hvw4OfEQJqlh7vaBVocm4H8ZLxHSwSQRC4xQvPXw/edit?usp=sharing

I started with Owen Boswarva’s Fusion Table powering his interactive map.    I then added additional information, both to update links to and counts of data.gov.uk entries, and to incorporate links to FOI requests registered with whatdotheyknow.

I get data on FOI requests by scraping whatdotheyknow.com using the brilliant service from KimonoLabs.  I’ve set up Kimono APIs which run weekly to get total numbers of requests registered with different types of public bodies (i.e. Local Authorities, National Parks, Police and Fire Authorities).   Data from those APIs is then retrieved directly into the Google sheet using Kimono’s Google Spreadsheets add-on.

Drop me a line if you want to know more, or have ideas on how this could be improved.

 

Advertisement