Monday, 16 October 2017

Local House Price Growth in England and Wales, 1995 to 2016

On the BBC website and news today there's a great story about house price growth in England and Wales over the last decade. It uses ward-level house price data to track changes over the past decade. I wasn't aware until relatively recently that the ONS had done a lot of the leg-work with HM Land Registry price paid data and aggregated it all to smaller geographies. They publish it down to MSOA level as 'house price statistics for small areas', and very nice it is too. There's median price, lower quartile and a range of other indicators. Naturally, I wanted to see what this looked like, but I thought it would have to be simplified. What I did was take two market price points - £200k and £500k (as very rough approximations for national and London averages) and map the spread of house price growth since 1995. Here's what it looks like in a 30 second animated gif. The geography is the MSOA, with an average population of about 5,400.

Watch this on loop for best effect

I wonder if I should post all the individual frames of this below? Okay, go on then. And you should be able to click to open then in a larger size in a new tab/window. It's interesting that after 2008 there appears to be a geographical contraction and also a bit more volatility and less uniformity of spread - and by 2016 you see a big chunk of the London area in dark red (over £500k average).

Okay, so we can really tell that by 2016 in most MSOAs across England and Wales house prices are at £200k or above and in a good chunk of London and the South (and a few areas elsewhere, notably Cornwall and south Manchester) they are £500k or more. That's all for now. I'll leave you with the same gif as above, but this time at 1 second per frame, plus another earlier version with the year label in the top right.

Slightly different design, same data

Thursday, 5 October 2017

Measuring Land for Mountain Goats

Land is frequently contested and fought over. In fact, in the history of civilisation, you could say that land has been at the centre of some quite important disputes (if you really wanted to understate things, that is). Because I'm a) a bit of a nerd and b) technically some kind of planner or geographer, I decided to look into the question of how you measure it. More specifically, I looked at the question of how you measure land area in areas where land is not flat - as it mostly is not across the surface of the earth. I did this one Friday afternoon many months ago with colleagues Ed Ferrari and Ruth Hamilton, and then I wrote up the slide set below. I'm hoping someone out there finds this interesting and/or useful.

The basic problem: land is very rarely totally flat. A 50m x 50m square on a flat plane will cover an area of 2,500 sq metres. But that doesn't take into account slopes, bumps and so on. So when one of your friends asks you 'if you flattened Switzerland, how big would it really be?' - after reading this you should be able to tell them.

To find out more, click to follow the slides below. The slides explain things in more detail and also say a bit about how to do these calculations in a GIS.

If you're too busy to do this, just know that when you take into account the topography of the land, Switzerland would grow by about 7% and Liechtenstein would grow by about 8%. But of course this partly depends upon the resolution of the data you measure it with. These figures come courtesy of Ed Ferrari.

A picture of a mountain goat I found on the internet

To make it easy for you to remember, should you ever want to use these slides, will take you straight there. That's just followed by measure sloping land as one word, after the forward slash.

Notes: spot an error in all this? Let me know. Got better examples? Please get in touch. I believe land is measured most often as if it were on a flat plane. Mostly this probably doesn't matter but I did want to find out what impact there would be if you took slopes into account. There are quite a few notes and disclaimers in the slides, so take a look at those for more information. Mountain goat picture is from this blog.

Tuesday, 19 September 2017

Buildings of Great Britain

One of the great things about Great Britain is of course Ordnance Survey, our 225 year old national mapping agency. Since a lot of their data was made open in the last decade it has become easier than ever to explore and map the country (particularly if you're some kind of GIS boffin). I should point out before going any further that Ordnance Survey covers Great Britain only, not Northern Ireland. For that, you'll need to look at Ordnance Survey Northern Ireland, who also have a good selection of open data. In the past, I've created sets of building data for English cities, done some calculations on how much land is taken up by golf courses, and a variety of other things, including creating 3D building models using OS open data. Today I'm sharing some shapefiles of all buildings in Great Britain

London has lots of red buildings

Why have I done this? The main reason is that I want to share these complete files with others who might need them and don't want to - or know how to - patch together separate tiles of Ordnance Survey open data. If you need some building footprint data, you can just download the set you need and zoom to your chosen area, or extract what you need from an individual shapefile. I used the OS OpenMap - Local product for this, so the detail in the buildings is very good, as you can see below in the zoomed-in extract.

The reason I did this in the first place was because I wanted to come up with a number for the percentage of the land area of Great Britain that is covered by buildings. There has been some debate about this topic, and it was covered in an FT Fact Check piece in 2016 by Kate Allen partly in response to the claim that golf courses cover more land than housing. My calculations using new OS data revealed that Great Britain is 0.54% golf course (1,256 sq km, about the same area as Greater Manchester). But do buildings cover more than this? Yes they do. 

Click to enlarge - you can see lots of detail

Using the data I'm sharing here, I calculated that buildings in the UK cover 1.35% of the land. I reported this previously in a tweet that was quite widely shared. To my embarrassment, for the GB figure I used the UK area as the denominator so the figure reported there was a little low, though the England, Scotland and Wales figures were and are accurate. Here's the important information you need, if you're ever faced with an awkward silence at a party.

> Great Britain is 1.35% buildings

> England is 2.0% buildings
> Scotland is 0.4% buildings
> Wales is 0.9% buildings

The Great Britain figure equates to about twice the area covered by Greater London - 3,150 sq km. However, the OS OpenMap - Local product isn't the most detailed building-level data covering Great Britain. For that, you'd need to look at OS MasterMap, a much bigger job. That is, unless you are Mike Gale and Tom Armitage at Edina and you have all this information in a lightning-fast database ready to query. They very kindly went beyond the call of duty and did some calculations and confirmed that my figures are pretty much spot on. They did loads of other really cool stuff too, but more on that another time perhaps.

If you click on the link, you'll see the set of shapefiles I created in a Dropbox folder. It contains some licence information, a few sample images, plus the following shapefile sets:

  • All buildings in Wales in a single shapefile
  • All buildings in Scotland in a single shapefile
  • All buildings in the North of England in a single shapefile
  • All buildings in the Midlands in a single shapefile
  • All buildings in the South West of England in a single shapefile
  • All buildings in the South East of England in a single shapefile

The complete set, and individual files, are pretty big, since they cover large areas and have millions of individual polygon features in them. This isn't exactly the best way to view and map this data, I know. That's obviously an understatement. But I also know that it's the format many people know and love, and want to work with. So, if you want to play around with buildings or use this as background mapping, be my guest. 

Data notes
Giant shapefiles make the world go round, okay. More seriously, there are better ways to download and view this data, but that's for another blog post. On a related note, see this from Emu Analytics on a very cool project which utilises OS building data. If you download the files and look at the attribute table in your GIS of choice, you'll see that I've added an area column for each polygon showing the area in square metres. 

Using this to query the dataset can be interesting - e.g. to find the largest building. Be aware, however, that the OS OpenMap - Local data still has a degree of generalisation in it so sometimes separate buildings can be merged together - e.g. if they are very close together. But I know from what Mike and Tom did with MasterMap data that this doesn't affect the final calculation much at all, thankfully.

As for the golf courses vs. housing thing, I know that's not strictly-speaking solved definitively either. The reason for this is that we have no way of knowing exactly what area houses cover vs non-residential buildings. As far as I know, even Ordnance Survey don't know this, and probably can't know this from what data we have available to us. However, the vast majority of buildings are residential (not sure on the %) and I'm very confident that housing covers a much bigger area than golf courses, I just can't say how much.

Friday, 11 August 2017

US 2017 Total Solar Eclipse Animation

I've only just started paying attention to the fact that there's a total solar eclipse in the United States on 21 August this year. What made me really take notice were the imaginative, amusing and bizarre maps people have been posting online - all good fun and often pretty interesting too. What seems to me to be missing - and absolutely essential, of course - is an animated eclipse gif showing the path of totality, towns and cities, a bit of terrain and some roads. So, I grabbed the geodata from NASA and spent a couple of hours playing around with it to make this thing you see below. This is an extract of the full file, in gif format. The full gif is over 100MB, so I've created a video file instead for the final file at the end of this post, below - and also posted on twitter. Click an image to see it in full size.

Note the small part of the path of totality in Montana

Nashville looks like the place to be

I also extracted a series of animations covering the whole US, in separate parts, so that the whole of the path of totality has been giffed, as it were. These are pretty big files and if I add them all here not only will it may crash your browser but it will also probably be a bit too bamboozling with all the animations going on at one time. Here's part 1 and part 2 below. The rest I'll keep to myself for now.

Part 1 - at 50 frames per second
Part 2 - also at 50fps

I just did this out of curiosity and to learn a bit more about the event. I thought it would be interesting to see for myself where the eclipse will go. I remember here in the UK back in 1999 when we had a total eclipse, but I didn't manage to get down to Cornwall to see it properly. Just take a look at this video to see how long ago this now seems. I was in Glasgow at the time and although it didn't get fully dark it was pretty interesting and the birds were very confused. Also, if I didn't do an eclipse map now I might never remember again. The next total eclipse in the UK isn't until 2090 and I just get the feeling I won't be on here making animated gifs of it then.


Notes: mapping software was QGIS 2.14, using the Atlas function to extract the frames for the gif. I used the NASA shapefiles to show where the path of totality and centre line are. I added some place names, roads and US boundary and state lines from Natural Earth and I also added an ESRI shaded relief base layer to give more of sense of the underlying physical geography. I patched the gif together in GIMP and animated the movement of the sun across the land using the vertices of the eclipse centre line. I've added a little glowing corona round the black dot, which is of course supposed to represent the sun.

There were about 1800 points in the original NASA eclipse path file so I used every second one to generate the animation because it still creates a reasonably smooth image. For a proper set of scientific maps, I suggest you check out the NASA Total Eclipse pages, as they are full of great information and maps. They also have an animated gif.

Monday, 17 July 2017

What Percent Golf Course is Your Area?

There has been a fairly long-running debate in the UK media about the extent to which golf courses cover the landscape. I've been working on a related topic recently, so when Ordnance Survey last week released an open dataset containing golf course coverage I thought I'd have a go at answering the question myself. The answer I arrived at was that Great Britain is about 0.54% golf course - or about 1,256 square kilometres (about the same area as the whole of Greater Manchester). 

The local authority with the largest area of golf courses is Woking in Surrey, at 10.74%. I originally calculated it at just over 7% but I had to make some edits due to the number of courses that straddle the local authority boundary and weren't being included. I've done the calculation a couple of times and also manually so I'm sure it's pretty accurate.

Woking - 10% golf course

Edinburgh - about 4.2% golf course

Why am I doing this? Do I hate golf and golfers? No, not at all. I used to play golf (a fine game) and have some golf clubs in my shed so I couldn't say I'm part of any anti-golf lobby. The reason is that I like to know things like this, particularly when such figures are used to debate issues like housing and the availability of land in some parts of the country. The fact that Woking appears to be 10% golf course is not a criticism, it just appears to be a fact. But I do find these numbers interesting. Here's the top 20 by local authority - and remember that Ordnance Survey just covers Great Britain not the whole UK so there are no Northern Ireland figures here. I'm not aware of a similar dataset that would allow me to do these calculations for Northern Ireland.

Is this a lot? Well, that depends

I've also done some summary statistics for Great Britain and the different countries within it. From this, we can see that Great Britain is about 0.54% golf course, England 0.74%, Scotland 0.28% and Wales 0.34%. From my analysis of the data, this does seem to include golf driving ranges as well. I'm not sure about crazy golf - don't think that's included but it would probably be a tiny fraction of land. The same goes for 'pitch and putt' type areas

Golf courses take up about the same area as Greater Manchester

If you want to explore the data yourself, I've made it available in a Google sheet - and also added in area codes and region names. According to my calculations, only 10 local authority areas in Great Britain have no golf courses within them, as follows:

  • Tamworth
  • Adur
  • City of London
  • Camden
  • Hammersmith and Fulham
  • Islington
  • Kensington and Chelsea
  • Lambeth
  • Tower Hamlets
  • Westminster

Finally, at the suggestion of David O'Sullivan, I had a little look to see if there was a correlation between the percent of each area covered by golf courses and the percent voting for Brexit. There really isn't.

Not much of a relationship at all

Data credit: Contains Ordnance Survey data © Crown copyright and database right 2017

Data note: to deal with the issue of golf courses that straddle local authority boundaries, I cut them in two using a local authority boundary file. My original calculations didn't do this and it turns out it makes a bit of a difference in some areas, including Woking, where the courses on boundaries end up not being counted. I'm pretty confident these figures are accurate - based on the underlying data - but of course other people have calculated similar numbers in the past and arrived at different results. Some of these were based on estimates but my calculations are based on the new Ordnance Survey dataset so I'm confident they are at least close to the correct figure.

Monday, 12 June 2017

General Election 2017: some maps and data

It's 2017 and there has just been a General Election in the UK. In case you're reading this in the future, I'm talking about the first General Election of 2017. Today's post is a brief comment, plus a few maps, starting with one on the Labour win in Kensington in London. To summarise briefly: Labour won a seat that many people thought they couldn't, and won by 20 votes with Emma Dent Coad becoming the new MP in a gain from the Conservatives. 

But this is an immensely wealthy area, isn't it? Yes and no. There is a lot of money here but also a good deal of poverty, as you can see from the map below, where the red areas are among the 20% most deprived in England. As with most things, however, it's not a simple story and the result is perhaps not that shocking when you look at the map, even if some have named it the 'UK's richest constituency' (and there is an argument for that view). It's also definitely not 'London's Richest District', as one report puts it.

Kensington is a very different area north to south - click to enlarge

Some numbers, to help put things in context... If we rank the 533 English constituencies by deprivation, using the Indices of Deprivation 2015 measure calculated by the House of Commons Library, the Kensington constituency ranks 178 out of 533 - just on the edge of the most deprived third in all of England. One look at the map tells us that within the area there is considerable variation, with some parts much more deprived than the national average and some parts less so. In total, 22 of Kensington's small areas (LSOAs) are in the 20% most deprived in England - and none are in the 20% least deprived. The picture is very similar if you look at other indicators, particularly those related to income.

However, in London and beyond, the name 'Kensington' has it seems become synonymous with wealth, opulence and 'the elite'. This may be part of the Kensington story, but it's by no means the full story. I just wanted to take a closer look at what the data say in order to figure out if this is really such a surprise. As I did so, I also homed in on a few other constituencies. In addition, I attempted to look at the correlation between Conservative vote share and level of deprivation by constituency. I did this for 2015 and 2017 and although the results are not that surprising - i.e. the more deprived an areas is, the lower the Tory vote - some of the outliers are quite interesting. Here are the two charts - below. I've labelled some of the interesting ones as you can see below. 

A pretty strong relationship in 2015, as you'd expect

The correlation is a bit weaker here - everyone has ideas about why

Let's take a look at Sheffield Hallam because it's such an outlier - and also I live in Sheffield (but not there). It was Nick Clegg's constituency but is now Labour (Jared O'Mara MP). It's also the 9th least deprived constituency in England so is something of an anomaly, though of course not so much when you look at voting history and demographics. Still, the map is interesting in itself - it bucks the general pattern of the least deprived areas voting Conservative.

It has a high student population, among other factors

I then wanted to look at a couple of other places to see what things looked like on the ground. First up is Islington North (Jeremy Corbyn MP) and Islington South and Finsbury (Emily Thornberry MP). Note the little patch of blue around Highbury in an otherwise quite deprived area. This also contrasts with stereotypes of Islington as some kind of land of milk and honey. There is considerable wealth here but it sits beside large areas of deprivation.

Note the little blue patch near Highbury

Then I looked at the most deprived Conservative-voting constituency. This is currently Walsall North, which is ranked 31 out of 533 constituencies on the deprivation measure. Again, there are local and historic explanations for this but I don't want to dwell on those now.

Surprising? Maybe, maybe not.

Last of all, I wanted to look at a constituency where a Labour win truly would be a shock - for this, I looked at the least deprived constituency in all of England: North East Hampshire. When Labour win here I think we can all agree that it would be a shock, just as if the Conservatives won Walton in Liverpool (85.7% Labour, 8.6% Conservative in 2017). Actually, the latter might just be the biggest shock in the history of the world. Mind you, these days you never know what's going to happen next.

The Conservatives got 65.5% of the vote here.

Addendum: I saw a great histogram by Owen Boswarva looking at party by median age in each constituency so I attempted something vaguely similar for deprivation deciles and party. It's not at all surprising but I did find it interesting so am posting it here too.

Click to enlarge - the pattern is to be expected, but quite interesting

Notes: I do know, of course, that there will always be a strong linear relationship between deprivation and % voting Tory - or Labour for that matter. The point here is that because this is true, and because Kensington is actually quite deprived, the result there is less of a shock than some are claiming. Also, ranking deprivation at the scale of constituencies masks lots of underlying variation - but that's partly why I mapped it at LSOA. If we had LSOA General Election results that would be interesting. The scatterplots were interesting to me not because of the obvious linearity but because a) the relationship changed a good bit between 2015 and 2017 (UKIP effect?) and b) the big residuals - e.g. Sheffield Hallam. The ones which defy the general pattern are the ones I'm interested in. Basically, Kensington is right on the trend line and maybe it's because many more people from the deprived parts of Kensington voted this time - plus rich remainers. Finally, a lack of deprivation is not the same as affluence but on any measure you'll find they correlate strongly.

Wednesday, 24 May 2017

The Great Polish Map of Scotland (aka The Mapa Scotland)

Last week I was in Edinburgh to give a talk at the Edinburgh Earth Observatory seminar series at the University of Edinburgh, so I thought I'd try to see The Great Polish Map of Scotland - also known as the Mapa Scotland - before heading back down south. As you can see below, I did manage to go, but I am not quite tall enough to get a good view of it, so I have embedded a video below to give you a proper view of what it's like. I've also posted photos of the very informative signs that have been put up, in addition to a few more views. It's still being renovated so perhaps I didn't visit at the best of times but I'm really glad I got to see it - it's said to be the world's biggest topographical map.

The Great Polish Map of Scotland is located in Eddleston in the Scottish Borders, about 45 minutes south of Edinburgh. When I went on a Saturday morning with an old friend the roads were pretty quiet but it's definitely reachable in under an hour either way. You can see the location in the map below.

Just a short drive and you're there

The Map is actually in the grounds of Barony Castle Hotel, and when we went we parked up in the hotel car park. Above the front door of the hotel you'll see a very ominous message - "Prepare to Meet Thy God" - but since I've not got to the bottom of that yet, I'll leave it there for now and just show you what the Map looks like in the photos below.

Yes, welcome to our hotel!

You go round to the left side of the hotel and then follow the signs to Maczek's Map, as you can see from the next two images. Then it's through the gate and across the bridge over Dean Burn (in case you didn't know, in Scotland and some other parts of the UK we tend to call a stream a 'burn').

I think that's Maczek rather than MacZek

Almost there - you can see the bridge here too

Okay, once you're at the Map you'll see nice new green railings surrounding it. I'm reliably informed by Addy Pope - great Scottish adventurer, ESRI boffin and local person - that you used to be able to walk all over the map but given the new fence and ongoing restoration I thought that might now be frowned upon, so I stayed on the right site of the fence. I'm going to be a bit controversial now on two points. First, I was a little disappointed. Not by the map, but by the fact that I couldn't get a better view of it. I'm close to 2 metres tall but that's not enough even when you're on top of the newly constructed viewing platform. That brings me to the second point. I really wish the viewing platform was higher. These are kind of unfair things to say given the excellent restoration work going on but I do hope someone reads this and gives them tons of money to build a 50m high viewing tower. That would be amazing. Planning permission might be an issue.

This makes a big difference - I just wish it was higher
A couple of views of the map now follow. The first was taken from the viewing platform and the second from the west side of the map. At this point I should probably say that it's not technically The Great Polish Map of Scotland in the sense that Orkney and Shetland are missing. I'm from the north of Scotland so I notice these things... I can understand the omission though - what is there now took years to build.

This was as high as I could get my camera

Extra points if you recognise where this is
One of the great things about the Map is the information signs all the way round that give you the history of the map. I've taken photos of all of them so hopefully you can click the images to read the text but I've also done a few zoomed-in ones, just in case.

Inspired, of course, by a 1958 map of Belgium

I didn't know about this

A close up from the image above

A few images of construction

How the map was made - closer view above

Scotland and Poland have many connections - see above

As you can see, North Uist remains under cover (Uist = "you-ist")
I wasn't really complaining about the viewing platform itself - it's a great addition and allows you to get a nice overview of the Map - but I do think it would be a much better experience if the tower could be higher. I'm sure everyone thinks that and it's such an obvious, annoying thing to say. Anyway, I took this image to show how it was funded.

Lottery funding for the viewing platform

And that's it. I am very glad I went, but it would have been better if a) it wasn't raining - though this is always a risk in Scotland and b) I could fly. In that respect, I think the best views of this are to be had by the few drone videos on YouTube, one of which is posted above. Finally, despite the ominous sign at the entrance of the hotel, I can confirm that they actually serve a very good cup of tea and there wasn't even a hint of death. In fact, the staff were most welcoming.

Do widzenia (for now).