It is a new way for communities to decide the future of the places where they live and work.  They will be able to:

• choose where they want new homes, shops and offices to be built
• have their say on what those new buildings should look like and what infrastructure should be provided
• grant planning permission for the new buildings they want to see go ahead.

A Neighbourhood Planning Area (NPA) can cover any neighbourhood, as long as it is coherent, consistent and appropriate; they will differ for different areas across the country, and should be what is most appropriate for that area.   At the time of writing, around 500 local areas have either applied for or been officially designated by their Local Authority as a NPA.  Importantly, local residents in three areas (Upper Eden, St James in Exeter and Thame, South Oxfordshire) have now voted “yes” to formally adopting their Neighbourhood Plans

Opportunities for Open Data

As an open data enthusiast, I believe that Neighbourhood Planning  communities could benefit significantly from on-line tools and data sources that help them capture and share information, and work together on their plans and issues in their neighbourhoods.

Working with friends at DCLG, I’ve been exploring how to bring together various related data sources in fully open, accessible and re-usable ways.

The result is this new demonstration application

 The demonstration application

Currently, the application joins up three relevant, and (hopefully) useful and relevant sources.

The first is a new informal dataset that we’ve curated by scouring Local Authority and Neighbourhood Planning websites.   This captures basic information about each NPA; things like, where it is, key dates, and links to documents such as the original application and local authority designation papers.    Note that this data is not complete, not 100% accurate,  and I’d encourage you to get involved and help improve it – see Improving the map section below.

The opening map shows clusters of NPAs.  To view individual areas, simply zoom in (using the controls on the left of the map), or click on the numbers in the circles (which will be squares, if you’re using Internet Explorer).

Individual NPAs are then shown on the map as a blue balloon.  To view information about each area, hover your mouse over the balloon, and the information will display in the top right hand corner of the map.

Alongside the map, we have the second main data source: the Twitter conversation under the #NeighbourhoodPlanning hash tag.  You can join in the conversation by clicking on the box at the bottom of the list – note this may not work correctly if you’re using the Firefox browser.

The third data source is local photographs published via Panoramio.com.  To view these, click on the balloon icon for the area of interest.  Photos are then retrieved directly from Panoramio in real-time, and displayed in an interactive gallery beneath the map.   Of course, you could always add more photos by uploading them to Panoramio!

Improving the map: get involved!

The underlying data is held in OpenStreetMap.  This is a global initiative to “create and provide free geographic data, such as street maps, to anyone”.  The project is managed by the OpenStreetMap Foundation, “an international not-for-profit organization supporting, but not controlling, the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data and to providing geospatial data for anyone to use and share”.

This means that the data on the map is open and available to NPAs to edit and improve.   I’d value your help to do that, in two main areas.

Firstly, I’d like to extend the map beyond points, to show the boundaries of each NPA.   I have published a proposal for achieving that on this OpenStreetMap wiki.  Please get in touch if you’re interested in helping out.

Secondly, it would be great to see Local Communities mapping more detailed information within their NPA: e.g. to show areas targeted for housing development.    There are various free tools available for editing OpenStreetMap. For further information, please visit the Beginners Guide, and the list of editing tools.  I use JOSM.  Potlatch is great too for editing in the browser.

One of those is: testing how to grow and extend our open data store to incorporate more DCLG data.

Late last week, we released the first new, and (I think) exciting output: our Business Plan Indicators dashboard., and supporting page about DCLG’s priority actions.

If you’re unfamiliar with Departmental Business Plans, the indicators are designed to help you, the public, assess how we’re doing on our various policies and reforms. Our Business Plan includes further information about each indicator, and how they align with Departmental and the coalition government’s priorities. For ease of ref, I’ve repeated the list below.

Introducing the Business Plan Indicators Dashboard

You’ll see from the opening page that we’ve tried to present the latest headline data in a neat, engaging way. For each indicator, you’ll find a panel showing the latest figure with links to more detailed information.

Clicking on “more detail” provides additional information including: what the latest figures show; how the figure is calculated; and why the indicator is in our business plan. You’ll also find a time series line chart, and a link to embed the summary in your own web page.

Note too that each “more detail” page includes a link to the underlying source datasets. Click on the “latest source” or “more data” links, and you’ll jump across to our open data store, from where you can query, retrieve and re-use data in a range of open formats.

Where next?

I think of the dashboard as the top of DCLG’s data pyramid, beneath which are lots more detailed datasets.

Let’s take the Housing Starts impact indicator as an example. This is calculated from our official statistics on House building. We also routinely publish live-tables containing further detailed statistics; e.g. breaking down the figures by local authority and tenure. The same is true for most other indicators in the Business Plan.

So, we have a great opportunity here to join-up this lovely content. I want to make it possible for you to quickly and easily move up and down the data pyramid, discovering and exploring links within and between the myriad of related datasets at each layer. I also want you (and DCLG itself!) to be able to retrieve it, re-use it, and join it up over the web to related sources that we don’t hold: for example, to link to more detailed data in, say, local authorities on land for housing, completions and so on.

Clearly, this will take time to achieve. As outlined in my ODI blog post, a key first step is to establish our open data store as a permanent, sustainable resource for routinely releasing DCLG statistical outputs in 5-star, accessible formats. We’re on track to appoint a technology partner in February 2013.

As we move forward, I’m keen to work with our data users and third-party software innovators. I want to know that we’re releasing the right data, in the right formats, at the right time. I’d also like to explore and exploit opportunities for joining-up our datasets to external sources, over the web, using open re-usable standards.

Which leaves me with some questions for you:

1. Which DCLG datasets do you want us to release first in 5-star formats? If you’re not sure what we hold, please take a look around data.gov.uk, or search our statistics and our transparency data publications on gov.uk.

2. How can we help you to find and understand our datasets more quickly and easily? For instance, do you search for data using particular terms? Do you like (and use) visualization tools like the dashboard? Do you have any ideas for new tools and visualisations?

3. Do you use DCLG data alongside other third-party sources? If yes, which sources are you using? How can we make it easier for you to retrieve and combine multiple dataset? And, do you have ideas on how we could do that over the web, using open standards?

DCLG Business Plan priorities and Business Plan indicators

Decentralise power as far as possible

1. Percentage of local authority revenue expenditure funded by general government grant, broken down by class of authority (Input indicator)

2. Percentage of local authorities committed to identifying and beginning work with troubled families (Input indicator)

Reinvigorate local accountability, democracy and participation

3. Number of groups supported to submit an expression of interest as part of the Community Right to Challenge (Impact indicator)

4. Fire-related casualties (per 100,000 population) (Impact indicator)

Support and incentivise local growth

5. Business rates yield within Enterprise Zones (Impact indicator)

Meet people’s housing aspirations

6. Affordable Rent payment per dwelling by the Homes and Communities Agency (Input indicator)

7. Average New Homes Bonus grant payable per dwelling per year to different classes of authority (£) (Input indicator)

8. Total number of housing starts and completions (seasonally adjusted) (Impact indicator)

9. Number of affordable housing starts and completions delivered through the Homes and Communities Agency (Impact indicator)

10. Energy efficiency of new build housing (average Standard Assessment Procedure energy rating score) (Impact indicator)

11. Households in temporary accommodation (seasonally adjusted) (Impact indicator)

Put communities in charge of planning

12. Percentage of local planning authorities having an adopted local plan prepared under the 2004 Planning and Compulsory Purchase Act (Impact indicator)

13. The number of planning permissions granted as a percentage of all applications for major and minor schemes (Impact indicator)

This builds on the first set of new, experimental wellbeing statistics released in July by the Office for National Statistics  

As the ONS describes in its statistical release, the data provide new insight in to “how people think and feel about their own lives”.   When used alongside the more traditional economic measures, this data can help us to better understand the nation’s problems, issues and progress with resolving them.

 Since July, my analyst colleagues at DCLG have been busy modelling ONS’s data at neighbourhood-level.  I’ve provided further background below on how they did this.  We think the local modelled estimates are a useful resource for local authorities and councillors, and their local communities and residents. 

We think of the data as a sort of lens, through which anyone can view wellbeing in their local area, and compare/contrast our estimates against their own local data and knowledge.  We’d love to hear your views on if/how/where the data is useful to you, in your local area: feel free to e-mail wellbeing@communities.gsi.gov.uk or leave a comment on my blog.

We’re also very interested in any software tools that use our data alongside other sources, in new and innovative ways.   As an open data geek, I think the most exciting aspect is that the underlying estimates are also available as full, 5-star LinkedData.   You’ll find the datasets on our opendatacommunities site   

 If you’re a software developer or innovator, you can develop queries to retrieve the data via our API, and display it in your own application.  This extends to any dataset held in opendatacommunities.  That means you could write queries which blend and combine wellbeing with other data: for instance, to retrieve wellbeing and deprivation data for a particular postcode, or local authority.  

 To get started, please take a look at our developer documentation.   We’ve also provided some example queries, such as this one (http://opendatacommunities.org/datasets/wellbeing-lsoa-life-satisfaction-mean ), in the “API” tab on the dataset about ‘Life Satisfaction’.

Modelling the results – technical background

The Department for Communities and Local Government (DCLG) has estimated the expected wellbeing of residents at Lower-layer Super Output Area (LSOA) level. The purpose is to illustrate the likely degree of variation in wellbeing between neighbourhoods.

These are modelled estimates for local areas based on national findings from the ONS Annual Population Survey 2011-2012. DCLG analysts used CACI’s ACORN geo-demographic segmentation to estimate the likely wellbeing characteristics of each neighbourhood.

These estimates are not the actual survey responses of people living in those areas. This is because sample sizes from the APS do not permit reliable estimates of subjective wellbeing below the 90 unitary authorities and counties reported in the First ONS Annual Experimental Subjective Well-being Results. As such, DCLG encourage local areas to use these estimates alongside their own local knowledge and data.

The maps reflect the differences in wellbeing observed among the 56 ACORN Types and the unique ACORN profile of each neighbourhood. The method assumes the national profile of wellbeing for the ACORN types is broadly the same in each local authority. For all of the subjective wellbeing measures, DCLG tested this assumption broadly held across the nine regions.  As a result, we made a minimal number of adjustments to the profiles for life satisfaction, worthwhile, and happy yesterday, and determined that the method was not robust for modelling anxiety.

Requests for further details of the methodology can sent to wellbeing@communities.gsi.gov.uk.

Following today’s exciting announcements in the Open Data White Paper, I wanted to update you my recent work with open data at DCLG.

Despite the lack of posts to my blog since January,  I’ve been pushing forward with open data on two main fronts.

Firstly, I’ve been testing out ways of publishing more data in the full (5 star) LinkedData form.  The result is our proof-of-concept OpenDataCommunities triplestore, with a nice SPARQL endpoint and API for you to enjoy.

I’m really pleased, and proud that this gets a mention in the Open Data White Paper.

You’ll see that the triple store comprises a selection of DCLG’s housing, local government finance and deprivation statistics.  We think we’ve chosen a collection of data that is widely used – particularly by Local Authorities – and ripe for linking together in new and interesting ways; including linking up with related sources over the web.   We are currently trying to test and prove that by working with a small group of local authorities and voluntary organisations.  For me, this is an essential part of gathering evidence to help me scope, plan and cost how to move to routinely releasing all DCLG data in this way.  So, if you are a regular user of DCLG data, and want to find smarter/faster ways to acquire and blend it with other sources, I’d love to hear from you.

We’ve done some neat things with geographic data too.  One example is the new set of identifiers for local authorities of various types in England, with links to other schemes, such as those defining the geographic extent of each authority.  We think this is important because it now enables local authorities to be referenced in terms of the organisation itself, and the geographic areas they serve.  Let’s take Amber Valley as an example.  The new URI - http://opendatacommunities.org/id/district-council/amber-valley - essentially provides information about the council as a public body, or legal entity.  If you follow the link, you’ll see basic information such as the link to the council’s website, area codes defined by the Office for National Statistics, and DCLG’s code for local government financing purposes.  This link also provides connections through to data, from ONS and Ordnance Survey about the geographic area governed by Amber Valley.

The ambition is for local authorities to start adopting these URIs, include them in their own web sites and APIs, and start attaching additional information at source about their organisation – such as “contact us” type information; and details about local councillors and their senior management team.  Again, if you’re based in a local authority and agree this is worth pursuing, then please do get in touch.

The second strand of my work has been on building relatively simple demonstration applications which “show and tell” how open data, served from our triple store, can enable innovative new tools and insights.   The three recent developments are:

1.   The fantastic Local Authority Dashboard

2. My new application showcasing DCLG’s Household Projections Statistics

3. An updated version of my Index of Deprivation Explorer

I’ve also built a landing page for the various apps I’m working on, and will be extending this to APIs and data source’s I’ve used.

Hi.

I’m sure you’ve heard about the Government’s localism agenda.  It is at the core of shifting power into the hands of individuals, communities and local authorities.

For this to work, I think we’ll need new and smarter ways for citizens, communities and public service providers to work together and solve real-world problems.   I’ve been wondering about the potential role for on-line technologies here  - in particular social media.   This led me to one obvious key question, which is: to what extent is the debate and pro-active collaboration already happening in different localities, via tools like Twitter or  Facebook?

To help answer that, I’ve used some of my spare time over Christmas to build a new prototype application.   At its core, is a register of news feeds and twitter streams provided by all English local authorities, and several hundred hyperlocal sites.

At the time of writing, I’ve catalogued around 350 news and twitter feeds from local councils, and over 400 equivalent feeds from the hyper-local world.   To find out more about the sources I’ve used, please see the section below.

The application then uses this register to “read” individual feeds, highlight trends  and show you the latest tweets and news articles.  I’ve called the app “drumalism”.  You’ll find it here: http://dclgexamples.mywebcommunity.org/localities/drumalism.htm.

Where next for the app and data feeds register?

As with my previous prototypes, I’ve developed drumalism in my spare time, using my own equipment.  This really is a “one man and his dog” operation: where the dog is Lillie, my Bassett Hound, who is usually wrapped around my computer chair with a “when are we going out” expression on her face.

I’ve licenced the app and data registry under Creative Commons’ NonCommercial-ShareAlike (v3.0), so you’re free to re-use it on those terms.  I hope that what I’ve done serves as a good example of what can be done to join-up established sources of news and opinion in different localities.  It would be great to see similar sources and tools incorporated in Local Authority and hyperlocal sites.

Currently, I have no plans to develop drumalism as a commercial venture, but please do get in touch if you’re interested in exploring that further.

Drumalism – data sources

The feeds registry behind drumalism comprises four main as Google Fusion tables.

1.  A list of all English Local Authorities – fusion table ID 2456608 .  I’ve sourced this from Local DirectGov, adding in grid references by geo-coding the main office address.

2.  A list of twitter, RSS,  facebook and open data feeds provided by Local Authorities – fusion table ID 2455274.   I’ve developed this myself, by manually trawling through local authority websites.

3.  A list of hyperlocal sites – fusion table ID 2452252.  I’ve sourced this from the list at OpenlyLocal.com, and then extended it to incorporate additional sites I’ve uncovered as I developed the application.

4. A list of twitter and RSS feeds provided by hyper-local sites – fusion table ID 2455251.  Big thanks again here to OpenlyLocal for providing the initial list, which I then manually checked and extended.

Hi.

Those of you with longer memories will recall my post back in May, promoting the then new triple store for DCLG’s Indices of Deprivation.

I’d love to report that this has generated a flurry of new apps.  I think I’m not alone when I say: we need some more real-world examples which show us the true power and potential of blending together different data sources….over the web…..using open standards.

The reality is: no apps, and no more than a passing curiosity in all the nice linked data, triple store wizardry.

Now, please don’t get me wrong.  I’m not complaining.  I think there are many very good reasons why we are where we are.   I’ll spare you my thoughts on the possible causes.  I’m guessing you’ll be much more interested in what I’ve actually done about it…..(which I’m hoping will spark a bit of discussion about going further, to tackle the underlying causes).

What I’ve done is: create this small, but hopefully important demonstrator application, along with some slides.

Acknowledgements

Before I outline how the app works, and what I’m trying to test and prove,  I wanted to say a huge thank you to the various data aggregators, API providers,  and generous human beings without whom none of this would have been possible.  A round of applause please for, in no particular order:

• The good folk at MySociety, for theyWorkforYou - which my app calls directly to show information about MPs.  Also, for the fab MapIT API, which I’m using to get and display info about geographic areas (Wards, Districts, and Constituences)
• Chris Taggart and his OpenlyLocal crew – which I use to show data about local councillors, and get the links to a Local Authority’s twitter feed.
• Friends at data.gov.uk, for their fantastic work on LinkedData and Sparqly endpoints.   You must learn to love this stuff, as I have done.
• The Guardian Datastore, and Alasdair Rae, for opening my eyes to the possibilities if only the data were available in more open, re-usable forms.
• And last, but by no means least, to Bill and Ric Roberts at Swirrl – for their generosity in hosting the IMD dataset, and their patience in answering my various dim, newbie questions.

 About the Indices of Multiple Deprivation (IMD) explorer application

Where has this come from?

I’ll begin with an excuse/confession.  I am not a professional software developer.  My code is a bit like my waistline: functional, but a little bloated.  All I have is my trusty old PC,  a copy of Aptana studio that I’m not afraid to use,  and copious volumes of coffee and paracetamol.

I’m doing this stuff in my spare time, to help me understand what open data really means from a data users and publishers perspective.   My passion for this stuff comes from a long, sometimes bitter experience of managing data in its various guises. Over the years, I’ve spent many, many hours with my head inside a SQL database, wrestling with the madness of pulling in content from a mix of magnetic tapes,  floppy disks, pen drives, CDs (and yes, even good old handwritten input).   I even remember punch cards and ticker tape!

So, my main motivation for developing the app was to test and prove how and why open data really is different.  For instance, how can it help us to quickly bring together disparate sources, and tell us something we didn’t know? Can it really cut out some of the cost and complexity of publishing and integrating ever increasing volumes of information?

I went in to this knowing a little bit about the LinkedData theory –  with a head full of acronyms and phrases like APIs; ontology; RDF; triples; and Sparql.  I had zero experience of actually using this stuff in the wild.

The result….

Four weeks, and many late nights later, out pops the IMD explorer.

For those of you who aren’t familiar with the IMD, it is – as the Guardian Datastore puts it – “possibly the most significant research into poverty in England ever put together“.

The IMD is particularly significant because it measures deprivation at a fine-grained, geographic level which relates directly to people, and where they live.   For the map geeks, the IMD data is published at the Lower Layer Super Output Area  (LSOA) level, which the ONS defines as an area with

“a minimum population of 1,000, with an overall mean of 1,500. LSOAs are built from groups of Output Areas. There are around 34,000 LSOAs in England and Wales”

IMD is also significant because is measures deprivation from different perspectives: for example, to show how deprived an area is according to crime, education, health, and so on.

So what, I hear you cry.  This matters because we often see the IMD being used as a sort of lens for viewing other, related datasets.  For a really topical example of this, check out the Guardian’s article exploring whether deprivation was a factor  in the recent riots.

Enough of all this theory – show me how the app works!

On one level, the application is simply about helping ordinary people to understand deprivation in all its guises where they live, work and play.

On the opening screen, I’ve provided a national map showing the top 50 most and least deprived areas (LSOAs).  You can switch the view to different measures (or domains) of deprivation via the drop down list, just below the page title.

Alongside that, is a box to type in a postcode, street, or place name, which then zooms to a map showing deprivation in the immediate local area.  From there, you can then move around the map – e.g. to zoom out to see deprivation across the whole Local Authority, or explore Wards within that locality.

Now here’s the twist, and the reason why linked/linkable data matters.

When exploring data at Ward-level, the application hops over to OpenlyLocal to pull back data about the councillors who you’ve elected to represent your interests in that ward.   So, if you’ve found issues with deprivation in your local area and are motivated to do something about it, you now have information about some of the people to talk to!

It’s a similar deal when exploring the Parliamentary Constituency – where the app goes and gets data about your MP from TheyWorkForYou, with links back to that site if you want to know how your MP is working in your local interests.

And there’s another twist.  Take a look at the “Education, Skills and Training” Deprivation domain.  Provided you’re not using Internet Explorer, you’ll see a map overlaid with the location and other information about local schools and other educational establishments.    I’ve particularly enjoyed developing this bit. It’s interesting, but not suprising to see the proportion of kids attending schools in deprived areas, who receive free school meals.    The data also suggests that schools in deprived areas are, relatively speaking, running below capacity – which is just an observation on what we can see and learn by joining up disparate datasets, and not a party-political point.

Anyway, that’s probably quite enough by way of an introduction to what I’ve done and why I’ve done it.  I will be following this up with further posts, exploring some of the more technical, “under the bonnet” aspects.

In the mean time, enjoy the app.  And my previous challenge still stands.  Start creating, and show us what you can do!

Steve

Following my recent post, about trialling publication of statistical data as LinkedData, I wanted to update you on other things I’m doing to (hopefully) make it easier to find, understand and re-use DCLG’s data outputs.

I’ve been playing around with DCLG’s data on spending over £500, trying to find ways to:

• Make it more accessible – by amalgamating separate spreadsheets in to one long list of transactions, with the ability to search or filter that list in various ways; and
• Make it more useful – by including additional information about each transaction, to help the non-accountants amongst us begin to understand how  or why the money was spent.

The result is this Google Fusion table and – to illustrate what can be done – this prototype Spending explorer application.

Making the data more accessible

The Fusion table brings together all data for the 2010-11 financial year, as published in 10 or spreadsheets on the DCLG site.   Thanks to Google’s Fusion Table’s API, you can slice and dice this single list of transactions in a number of ways.

Here’s an example, which retrieves a Comma Separated list of all payments we’ve made to MITIE CATERING SERVICES LIMITED

http://tables.googlelabs.com/api/query?sql=SELECT%20*%20%20FROM%20334877%20where%20Supplier=’MITIE%20CATERING%20SERVICES%20LTD’

And another example, this time to get a CSV list of payments for Agency Staff – as recorded under the “Agency Staff” Expense Type.

http://tables.googlelabs.com/api/query?sql=SELECT%20*%20%20FROM%20334877%20where%20′Expense%20Type’='Agency%20Staff’

So much easier that wading through separate spreadsheets, and thousands of lines of data – and a nice illustration too of how and why APIs are an important alternative to publishing flat, largely disconnected files on a website.

Making the data more useful

The fusion table incorporates two additional columns – i.e. information that isn’t available in the published spreadsheets.

First, is data on the organisational “Units” in DCLG which made each payment.  The values here correspond to units in the organogram we published  in June 2010 - with some additional entries, reflecting more recent changes to the Department’s structure.

Second,  and with thanks to my colleagues in DCLG’s Finance Team, I’ve included a flag for each payment indicating whether it relates to: (a) money the Department spends on its own, day-to-day operations; or (b) money it has paid to external  organisations, typically as government grants.   You’ll see the Fusion table includes a new “FundingType” column, containing, with values of OVERHEADS or GRANT to indicate the type of payment.

So how does this make a difference?

In part as a personal learning exercise, I wanted to test whether and how offering data through an API with additional contextual information can make a difference.

I’m thinking about this from the perspectives of:

• Data owners and publishers – e.g. to begin to understand how APIs help us to remove some of the publishing burden, or  ”scale-up” to release more data, without getting sucked in to a file-based, content management quagmire.
• Data users and consumers – to determine how APIs can help you retrieve or data much more quickly and easily, understand it once its been retrieved,  and blend/combine it with other sources.

The prototype Spending Explorer application is my first stab at demonstrating why APIs are good.

For example,  its very easy to calculate headline trends in the data, then – using open free visualisation tools – compare, contrast and explore spending in various ways.  The application provides treemaps  for you to rummage through the data, starting at either total spend over the year under individual expense types, or – as in the example below – spending by DCLG Units.

Here, we can see that DCLG spent just over £311m running the organisation in 2010/11.  The treemap shows us how this breaks down across individual DCLG Units, which are “clickable” for you to drill in for more detail.

Because we’re calling data through the API, we can also quickly examine and explore it from different perspectives.  The application demonstrates this through its “By supplier” tab,  where the start point is now a list of payments to individual suppliers, ordered by total cash value, with the number of payments made, and the average value per-payment.

Here, we can see that DCLG paid a total of £15.5m to Land Securities, in 35 separate payments.  The smaller tables indicate how these payments break down across organisational units, and individual expense types.

Remember that, in these and all other cases, the application is simply asking the API to provide some data; which returns results in an open format; for the application to manipulate and display in any number of ways.

So what’s next then?

I’m hoping that I’m on to something here.  I think I have proved, in a modest way, that APIs can and do offer real benefits over publishing flat, disconnected files, especially when we incorporate additional contextual information.

I think too that we can see through these examples clear advantages for data publishers and data consumers – e.g. efficient, scaleable publishing in searchable, open formats, can and does enable re-use and innovation.

As we move through the next year, I can see a number of opportunities to improve and extend the volume and quality of data available via APIs.   Specific things I’m looking to do include:

1.  Incorporating links to the excellent Open Corporates site’s information about registered companies – so you can begin to join the dots between payments made by DCLG and other public bodies to the same individual suppliers.

2.   Aligning spending information with our soon-to-be-released, updated organogram – so you can see more clearly who in DCLG is accountable for different bits of departmental expenditure

3.   Providing additional context – such as postcodes for supplier addresses (to show where in the UK our suppliers are based), or to indicate whether the supplier is a private sector or voluntary/charity sector organisation.

4.   Joining up payment and budget information – so you can determine whether we’re on track to spend to agreed budget allocations.

I’m trying to move this forward in bite-sized chunks, as time and resource allows.   This work is running alongside a significant restructuring of the Department’s responsibilities, and downsizing of its resources.    In the mean time, please bear with me if things to be moving a little too slowly for your liking.

I would of course welcome any offers of help or support!  You know where I live.

Disclaimer: the views expressed in this article are personal, and do not represent official DCLG policy or opinion.

Introduction – why you should read this

Its been a while since I blogged about my work at Communities and Local Government to start opening up the Department’s public data.

I wanted to bring you up to speed with some things I’ve done in recent months. I think they are interesting and important first steps in the Department’s journey towards fully open, re-usable data. I hope you agree.

This article majors on work to test publishing of statistical data in a highly accessible, re-usable way. Thanks to friends at Swirrl Ltd, we have now built a full LinkedData version of the 2007 and 2010 English Indices of Deprivation (IMD). You’ll find it here.

The dataset sits on Swirrl’s “PublishMyData” platform, which means – if you’re a software developer – it can be directly queried and results obtained in a variety of formats.

Before I say more about that, I first wanted to recap on why open data matters – i.e. explain the context and rationale for our work with the IMD and other DCLG datasets. Feel free to skip over this, and go straight to find out more on the IMD open data experiment.

Open data – why it matters

Much has been written and said about why open data is important.  For me, there are two main reasons.

First, it should allow data owners – like me – to publish their data in a consistent, standardised form. This has the potential to save money (e.g. by rationalising or consolidating what we publish, and how we publish it), and reduce the burden of shunting data between different systems (e.g. by standardising on formats and definitions which let the data flow more easily and efficiently).

Second, open data should remove some of the pain for our data users (you!) in finding and understanding what we publish, discovering links to related external sources, and combining or blending it in lots of new and innovative ways.

We want to see data from various sources, including our own, brought together and actively used to tell us something we (or you) didn’t know, to help people understand and engage with public services, and to hold service providers to account if/where things aren’t going so well.

What we’re doing about it

I like to think that DCLG is already doing quite well with making its data more open and accessible. For instance, friends in our Finance and Procurement Teams are working really hard to give you better data on departmental spending over £500. We’ve recently incorporated columns which tell you whether a supplier is a Voluntary or Charity Sector organisation, and whether the payment was a grant to the organisation concerned, or money for DCLG’s own day-to-day activities.

As part of its Business Plan, the Department has also recently published information about the data it will use to track and measure its performance against key, so-called, impact and input indicators. DCLG has a good presence on data.gov.uk too.

Testing open data standards and approaches using the Indices of Deprivation

Whilst this is all very positive, the focus is primarily on helping you understand what data the Department has. On its own, it doesn’t really help you to find the data more quickly, or re-use it – alongside other information – in new, innovative ways.

I am trying to address that through our work to publish the recently released 2010 Indices of English Deprivation in a searchable, “LinkedData” form, along with data for 2007.

We’ve chosen the IMD because it provides rich, comparable figures about deprivation over time in different localities. The data measure a broad concept of ‘multiple deprivation’ across seven domains: Income, Employment, Health and Disability, Education Skills and Training, Barriers to Housing and Other Services, Crime and Living Environment.

We know that the IMD is widely used across the public sector, both to plan Government programmes, and to coordinate or target services at those most in need. Examples of previous uses include: determining optimal locations for the initial wave of Sure Start and Children’s Centres; and reducing Stamp duty for transactions on property and land based in deprived localities.

If you want to know more, please check out OCSI’s article on Why the Indices of Deprivation are still important in the open data era, or the Guardian data blog’s Indices of multiple deprivation: find the poorest places in England.

Given its broad appeal and importance, I believe that the IMD is ideal as a candidate for sharing and re-use in open formats. Perhaps more importantly, I also believe that IMD is ripe for blending and combining with other external sources. For example, it would be great to see IMD used as a sort of lens for comparing and contrasting other related information – such unemployment   housing benefit, or crime statistics.

Get involved!

My hope is that publishing this searchable, open and re-usable version of the IMD will spark a range of creative, innovative uses and applications.

Perhaps you work in a Local Authority or other public body, and have a bright idea around combining IMD statistics with your own local intelligence? Perhaps you are a software developer or innovator, and want to find a neat way of packaging and presenting the data in a powerful, citizen-friendly manner?

Whoever you are, I would love to see you making use of this data. Please let me know what you create, and give me some feedback on what works well and what could be better. What are you waiting for! Head on over to http://opendatacommunities.org/ and help us make a difference!

Technical stuff you need to know

We have utilised several standards from the excellent work by the data.gov.uk team. At the heart of the data representation is their Data Cube ontology.

Although we created new geographic identifiers for the IMD Lower Super Output Areas, these are cross-linked to the local authority identifiers from http://statistics.data.gov.uk and Ordnance Survey’s Linked Data for postcodes.

To help get you started, Swirrl friends have kindly posted this blog, outlining how to retrieve IMD data by starting with a postcode.

If you get stuck, or need some advice, feel free to drop me or Swirrl a line – we are contactable on hello@opendatacommunities.org , or stephen.peters1@sky.com

Other things in the pipe-line (separate blog posts to follow in the coming weeks)….

1. Improving access to and usability of DCLG data on payments to suppliers over £500.

2. Building a “dashboard” for DCLG’s impact and input indicators

3. A nice, new, interactive Departmental organogram

4. Experimenting with on-line policy engagement

5. Open data on business insolvency – a personal venture.

The SPARQL endpoint is available at TSO OpenUpLabs, a site that provides an area for TSO to showcase work that is new, being trialled, or hasn’t yet been released as an official product or service.

Why not pop along and try out the sample SPARQL queries (link to http://openuplabs.tso.co.uk/sparql/tso-dclg). To find out more about the LDEx API review the following blog post on data.gov.uk (link to http://data.gov.uk/blog/local-data-exchange-ldex-incubator).

Well by background and training and for my whole working life I’ve worked with stats and data. I’ve done lots of quant analysis and lots of qual work, and I’ve managed surveys and managed other more intelligent people than myself to do analytical research. So I think we live in intriguing times. I won’t go over that here as I am sure if you are reading this (anyone anyone??) you understand the landscape. Suffice to say the Open Data movement is changing the way government works and thinks about the way it works, and is revealing the way in which government works (or doesn’t work) in very new ways.

This feels like deja vu to me because I’ve been working for a very long time now on releasing detailed local statistics. I think what is different at the moment is that the world of data has suddenly entered the mainstream. Researchers and analysts are of course still interested but developers and journalists and social media gurus and politicians..and well everyone really…is suddenly recognising the power of information and more importantly thinking of it in terms of data. And this is simply (I think) down to the internet – we’ve moved to a whole new way of working with and managing information.

But this makes working in this landscape incredibly complex because everyone comes at things from a different perspective, with different aims and with very differing experiences. It seems that literally in the last few weeks, maybe a few months at most, that the community of evangelists for opening up government data is now starting to say – ‘well great we are starting to see data but make sure it is accurate, make sure we understand the quality controls, the context, and make sure we can analyse and interpret in a meaningful fashion’. Erm…well yes. I’m not being funny but that should be obvious really shouldn’t it?

And of course they are right – to date government has responded to the call for open data by flinging open its systems and throwing it out into the ether. They have responded to developers calls and said ‘here is the data now do with it what you will – magically hold us to account and improve democratic engagement’.

Good luck with that.

So why deja vu? Well simply because anyone who comes from a stats/analytical background will tell you that information is beautiful but only when you understand that information. And they’ll tell you that there is a wealth of local information out there but that you have to deal with the context and the geography and the statistical hierarchies and the time points and the quality and the sheer quantity and and and… Damn lies and all that…

And of course in the context of local and central government the last 10 years has seen massive investment in statistics – in the collection and management of data and in the production of tools/websites etc so that anyone can undertake analysis of that information. And with the budget cuts most of that is being thrown away. And I won’t go into that here but I don’t necessarily think that is a bad thing. But it leaves a hole and that is what the new evangelists for open data seem to be realising. The Government line now seems to be very firmly – ‘we’ll make the data available but it is over to you to validate and analyse and democratise’. And I guess when it comes down to it the question is will this be enough or will it simply mean we have a level of openness (which I whole heartedly applaud) but little of substance to really inform and engage and empower citizens?  Will we simply drown in meaningless data, that provides a steady flow of reactionary headlines but little else? There is a clear analogy with The Big Society – yes we will do this but we are already and we only have so much time and at some point need to make a living. I love data, I love information and I love the new world of apps. It takes me back to my youth playing on my zxspectrum. The idea that one person can have a good idea and develop an app and maybe make some money too is incredibly powerful. But someone please tell me how to monetize the vast majority of government data? Yes TfL data because we’d all pay a little bit to be able to handle our commute more effectively, but what about deprivation data and complex financial information? A simple app that a large number of people will actually pay for?

Now again the government line on this is simple – ‘We in government are not creative (speak for yourselves buddies) but others are. We can’t even begin to imagine what amazing things will be done with the data we make available’

Well fine. Yes there are better minds out there. But you know what if you feed them partial data with little information on validation, little context and little comparability? Don’t the great minds out there have better business ideas to inspire them, more pressing issues they want to deal with. Yes holding government to account is vital but back in the real world people want to deal with education, and health care, and care for the elderly. Yes this is all tied up – if you take the view that information is critical to delivering better services and being accountable than everything can radiate out from data – but…but..but how can we do this without real support for developers and analysts and researchers?

So many threads I don’t know where to start. So over to you. Am I wrong? Are you all out there building fantastic apps based on government data. Do you know how to make a living from it? Maybe you don’t care, maybe you are happy to dedicate all your spare time to this?

Thanks!

ade