Wednesday 28 December 2016

Interactive Terrain Mapping in QGIS

During last year's winter break I did a little animation of Arctic sea ice, which eventually ended up being shown in slightly modified format in Svalbard Museum - and they even sent me a nice hat for my efforts. This year's spare time project was to look a bit more at terrain mapping in QGIS, with a purpose. Since I'm a Highlander in exile I dream of the hills quite a lot, and the mountains in particular. If you follow mountain weather reports for the Highlands, you'll hear of fairly regular avalanches. They may not have the ferocity of avalanches elsewhere in the world, but they can be deadly. The Scottish Avalanche Information Service do a great job of recording, documenting, mapping and educating here so I thought I'd add this data to some 3D interactives using Minoru Akagi's Qgis2threejs QGIS plugin and Ordnance Survey Open Data. If that sounds like gobbledygook to you, never mind - just look at the little animation below to see one of the outputs. The red dots are the locations of recorded avalanches since 1991. You can also play around with it in your web browser - it's quite good fun.

Avalanche data is from the Scottish Avalanche Information Service

I extracted the avalanche data from the SAIS webmap and just plotted the locations in QGIS. I decided to focus on an area I know relatively well - the Cairngorm mountain area. It's not high by world standards, at 1,245m/4,085 feet (6th highest in the UK), but at 57 degrees north and nothing much protecting it from the fierce winter weather, conditions up there can be extreme. In fact, I remember growing up sitting many times on the old chairlift as it swung in the wind! Not for the faint hearted. Anyway, as is the way with these things, the area I wanted to look at was split between map tiles, as you can see below in this QGIS screenshot.

I think there must be a name for the law of split map tiles

Methods-wise, I just patched together a few map tiles, clipped out the area I wanted to focus on, overlaid the avalanche location data, generated a hillshade with an azimuth of 180 and elevation of 27 (to simulate shadows at noon on 26 December), and then added some Ordnance Survey map tile data on top of it. I also did a version with Google satellite imagery, just to see what kind of result I could get. I've posted a little gif of the satellite version below, followed by more of the images. You can also play around with a smaller version of this in your web browser. It should also work on phones, but you might have to use three fingers to pan around.

This uses Google satellite imagery - interactive here

What you'll see below in the next five images are just screenshots from the final interactive from my web browser. You should see them in larger size if you click on them. I've modified some of the settings slightly but basically it's quite an easy thing to do in QGIS with the Qgis2threejs plugin. The updated document on this (18 Oct 2016) is also available in a single, handy pdf. If you want to use this tool, you'll find lots of useful tips in here.

Full view of the final interactive - looking north

Focused on Loch Avon (pronounced Loch A'an

This shows the main areas of avalanche activity in one view 

This is a view with the Lairig Ghru pass in the foreground

Another more vertical view of the southern part of the area

I like the effect here with the hillshade layer providing some sense of depth and shadow. There is no vertical exaggeration here and in the interactive version you can zoom in quite far before it becomes pixellated. I also wanted to see what kind of quality I could get by adding in a Google satellite layer and then exporting that. For this, I used the OpenLayers plugin in QGIS (though QuickMapServices is also really good for this). The final version was pretty big as I exported at a high resolution, but here are some snapshots.

The full view of the satellite version - click to enlarge

Loch Etchachan - watch this video!

A different view, showing a cluster of recorded avalanches

The resolution here is actually quite good

This is a view from 'over the back' of the ski area

One final overview image, just for completeness

As I said above, I also did a little extract of this focusing on a smaller area around Loch Avon. The satellite imagery changes part way along the loch but it's still quite pleasing. The interactive version works pretty well in the browser and the image quality here is also pretty good (I exported this at 200% in the Qgis2threejs settings).

Loch Avon - a nice little video from March 2016

It's not always winter here - take a look

Along the way, I almost forgot why I started to map this - the purpose was to plot some interesting and important data on top of the terrain in order to try to understand more about which areas are particularly prone to avalanches. Along the way, I learned more about Mike Spencer's snow hydrology research and PhD (more on that here) and found out some stuff that will help me update my teaching material on this topic. I did begin to look at the Ben Nevis area as well (below), but I decided to focus on the Cairngorms instead this time.

25 years of avalanche locations (red dots) for the Nevis Range

That's all for now. Thanks to the kind of data collected by the SAIS, Ordnance Survey Open Data, and great tools like QGIS and Qgis2threejs, it's becoming much easier to explore, analyse, visualise and understand important datasets. That is kind of what I was attempting here, as a little holiday experiment.

Thanks for reading.

Map data: Contains OS data © Crown copyright and database right 2016

Thursday 22 December 2016

Creating a 3D city model using open data for England

This is a short informational post on how to derive building heights using open data, for areas in England. I am also sharing the data in the visuals you see below. However, if you have access to similar data in your country then you can replicate the method. It's basically the same as others have posted elsewhere, so I don't claim this is a first - I'm just posting it in case others find it useful. And I've also added local authority names to the building outline files, hence the different colours you see. First of all, though, here's an example of what you get at the end of the process.

You can find the raw data in shapefile format here

Of course, this alone would be no fun without having a little interactive version in the web browser, so here you go - knock yourself out! The buildings are coloured by local authority - blue for Trafford, purple for Salford and green for Manchester.

Click for a little interactive extract of the first image

Here's the basic method for adding heights to the building polygons.

  • Download some building footprint data. I used OpenMap Local from OS OpenData and this comes in named tiles (as per the naming conventions of the British National Grid - NH, NN, NS and so on). I used data from the SJ tile, covering Greater Manchester. When you download it you'll get loads of layers but in the data folder you should see one called Building.
  • This building data doesn't come with any height attributes so we need to find a way to add that in. This can be done by using free, open LIDAR data available for England and Wales via the Environment Agency. You can also get aerial photography here, by the way. The data come in a variety of resolutions but I chose the 1m version as I wasn't overly concerned about precision. If you download the 25cm resolution LIDAR data the files will be huge. I downloaded tile SJ89 and both the DSM and DTM products. The DTM is a digital terrain model so it gives us the lay of the land, as it were, and the DSM is a digital surface model so includes the height of anything on the land beneath. The image below probably explains it best.
Source: StackExchange

  • I then add the DSM and DTM files to QGIS (I used version 2.14) as a virtual raster. Otherwise you'll end up with loads of individual raster tiles if you add them one by one because the SJ89 LIDAR data comes in lots of little chunks. You can find out more about this - and how to add a virtual raster - in this extremely useful blog post. The brilliant Owen Boswarva also has some really useful info and links on his mapgubbins site, so check that out too.
  • Because the DTM gives us the height of the land and the DSM the height of any features on it, we need to use the Raster Calculator in QGIS to calculate the height of buildings. But fear not! This is also very simple, and is explained in the blog post linked to in the last bullet point. The only thing I did differently is I just manually selected the area of buildings that covered the SJ89 tile using the Selection tools in QGIS (and then saved it as a new shapefile layer).
  • Before using Zonal Statistics in QGIS to calculate building heights - and bear in mind they are not exact - I just had a virtual raster which was obtained by subtracting the DTM from the DSM layer in QGIS, plus a buildings layer which covered the same area, as you can see below. It was these two layers I used to derive the building heights.
This area is centred on Greater Manchester
  • Then I used the Zonal Statistics tool in QGIS to calculate a mean, min and max height for each building polygon from the LIDAR data. You could just use the mean but I wanted more information so I just left checked these boxes when running Zonal Statistics. The method is also described by Brendan, here, but you should note that in version 2.14 this tool will be in the Raster menu. Note that you may find nothing happens after running Zonal Statistics - don't worry, QGIS has just added the new fields (max, etc) to your shapefile without any fuss so check the attribute table.
Here's an example of my final shapefile, with new height attributes

  • Once you've done that, you should have the ability to analyse building heights and produce a fairly credible 3D model of your area of interest. Just bear in mind that things like pitched roofs, curved roofs and other shapes will not come out when using this Zonal Statistics method. For that, you could of course use the raw DSM data if you wanted.

That's it really. I'll finish with a final big version to which I have added some labels. Right click and you should be able to see it full size in a new window or browser tab.

Hastily patched together, but I hope you get the idea...

Friday 16 December 2016

The 8 English Regions of a Federal UK

The political geography of England can be a contentious topic, particularly when it comes to the craft of drawing boundaries on maps. In fact, people can get quite agitated about it, and probably with good reason. But it’s too important to ignore, particularly at a time when the very nature of the UK as a political entity is being questioned and challenged. In this short essay I outline what I think new regions of a Federal England could look like, based on a combination of cultural, historic, administrative, economic and geographic factors. The boundaries you see below are my interpretation of ideas sent to me by Philip Brown and Nathan Pearce, two former students of the then Department of Town and Regional Planning in Sheffield (since re-named) who now work in professional planning practice. This doesn’t mean they are right, but they do at least have some grounding in the subject, and an unhealthy interest in regional maps and planning. I do too, so I have taken their note and turned it into a series of maps. The final map is shown below, to kick things off. Read the full text to see how I got there. You can see high resolution images of all maps here. You should also be able to right-click maps to open them in larger format in a new browser tab.

Read the rest of the post for the full story

Let’s put the names to one side for a minute because I'll come back to that later. Let’s forget too the big differences in size you can see in the first map. There is some logic there too, I promise. The question of what the English regions should look like is not one with a single correct answer, but I think it is possible to arrive at some kind of best fit compromise where everyone is happy. Okay, that was a lie. But I do think we can draw lines that people can live with and that make sense and work well for governing, planning and economic activity that also reflect historic and cultural factors - and without the need to re-draw local authority boundaries once again. 

Previous regional boundaries have made some sense, but I feel they have never been quite right, for a number of reasons. For example, when I did my PhD it was about the North West region of England, which at the time had recently inherited Cumbria from the old ‘North’ region. Westmorlanders may have been aghast, but it happened anyway so I think we need to revisit a bit of the history of all this before going any further, lest we repeat the mistakes of the past.

I recently co-authored a paper on US megaregions with Garrett Nelson, and after seeing the level of interest it generated, it struck me once again just how much people care about this kind of thing. In England, I think people may care even more, particularly because there has never really been a suitable regional structure - and because pre-existing geographies are layered through the centuries and embedded in our minds. In the context of devolution for other parts of the UK, I can understand why there is unease in parts of England - particularly when there is the perception of ‘London vs the rest’ - though I won’t dwell on that here. What I will say is that the idea of the English regions as ‘the dog that never barked’ by Christopher Harvie (1991), is one that resonates strongly today. Some kind of federal arrangment makes sense, but the question of how to make it work is a vexed one.

I saw in the news recently the leader of the Scottish Labour Party raise the question of a future Federal England within a UK and giving more power to the English Regions. It’s a sentiment echoed elsewhere and in the context of the post-Brexit/Indyref 2 question of what the UK will look like in the future it’s a very important one. On the whole, I’m in favour of this kind of federal arrangment, if only we can just figure out what shape the regions should take, how such a system would function and who would decide how it all works. This structure could provide a useful political and economic counterbalance to London, going beyond the current ‘powerhouse’ or 'engine' style plans for the North and Midlands in England and making permanent a decentralised, fairer political power structure across the country.

As I said above, though, this is a difficult topic to discuss because boundaries drawn the ‘wrong’ way can end up threatening people’s identity, among other things. I was reminded of this recently when Jonn Elledge of the New Statesman and CityMetric published a small English Regions map and Twitter responded somewhat impolitely. So, it’s a difficult topic, but that shouldn’t stop us discussing it. That’s why I waded in to a Twitter discussion between Jonn Elledge, Duncan Weldon and Chris Cook the other day to offer my thoughts, as you can see below.

See some more comments on this on Twitter

Before going any further, though, I think it is useful if we look briefly at some historic, ceremonial and administrative boundaries to see a little bit of what has come before. 

Old boundaries
At this point, I could post any number of different maps that define English ‘regions’, but I’m going to stick to three geographies. You can see lots of others via a quick search, including some interesting telephone call-based ones for the entire country. I use the term ‘region’ somewhat loosely here of course.

First, let’s look at what are termed ‘ceremonial counties’, available from Ordnance Survey. These counties include modern-day Greater London but also a big mix of other, less familiar shapes. These counties have an important history so I wanted to make sure I took them into account and referred to them when making the Brown-Pearce regions of England you see below and above.
These should be pretty familiar

Second, we have what are considered ‘historic counties’, such as Westmorland. These have largely disappeared from our maps, but not from our minds, and this is important. In fact, they often retain a special significance in people’s minds and it’s common to see letters addressed, or goods labelled using these historic names. This kind of emotional attachment is both logical and understandable, so we should pay attention to it in any regional rejig. For example, Middlesex is a name that pops up regularly but it’s much rarer to see it on a modern map. Nonetheless, it is an important mental and historic geography, even if it is no longer an official administrative one. I mention this to highlight the fact that the sense of identity tied to historic place names is powerful and significant. The way modern Greater London intersects with these historic counties is particularly interesting because it has mostly been forgotten, though not by many who live there or who are from there.

Yorkshire is as big as ever

Note the Middlesex/Hertfordshire 'wiggle'

The two examples above may not be ‘regions’ in a formal administrative sense, but they fit my definition here, which is basically bounded spaces that go beyond the local. That sounds a bit waffly, but we must remember that ‘region’ is a term we could apply to Uttar Pradesh, with over 200 million people, or to the old North East Government Office Region of England, with 2.6 million people. Talking of which, the first map below shows English Government Office regions before they were abolished in 2010. As you can see when I add in some of the earlier boundaries in the second map, these align with historic notions of regions in places, but not in others. It’s also worth remembering that ‘region’ itself is often an extremely loaded term. Try arguing that Scotland is just another region, for example, and you’ll see what I mean.

Ye Olde Regions of England

Once again, with meaning

These GOR boundaries weren’t loved; and most administrative geographies aren’t. They were at times a useful governing device but they are not the kind of regions we could get excited about, or angry about; unlike, say, the 1974 reorganisation of local government in England. One issue that strikes me here is how London is too ‘small’ from an economic geography point of view - for more on this, read this excellent piece by Barney Stringer. That is, it’s ‘underbounded’. Greater London contains about 8.6 million people but its economic power and orbit stretches much further beyond the boundary, into and beyond the metropolitan green belt.

I mention this here not to annoy everyone in Berkshire or Surrey, but because it features as one of the guiding principles in the Brown-Pearce regionalisation below. I also agree that if there were to be some kind of Federal England with new regions, then London should be bigger. But how big? 13.42 million people big, that's how big. Read on.

After watching the ‘regions of England’ Twitter exchange from the sidelines, Philip Brown got in touch to say that he and Nathan Pearce (a proud Janner) had something up their sleeves on this subject. Here’s a little extract of what Philip wrote.

"Basically Nathan Pearce and I, when we probably should have been studying for our coursework or something, once had the exact same discussion you, Chris Cook, Duncan Weldon and John Elledge have just had on Twitter."

"Except we were in a University computer room so spent far too long on it as we tried to solve the question of English devolution on cultural ties, geography, and also whilst largely respecting history too, lol! Hopefully these factors combined would make it tolerable to most in modern day England too."

The Brown-Pearce plan splits England into eight ‘Kingdoms’ and promises to ‘solve UK wide devolution forever’… (I like their confidence, albeit tongue-in-cheek). Thankfully, Philip still had the original Word document from when he did this in about 2010, so he sent me it and I republish it below for everyone to marvel at. Please don’t send him hate mail. He’s a very nice chap and a proud Yorkshireman.

"Each Kingdom to have an Assembly of similar powers to those presently in Greater London or Wales, but Parliament shall remain sovereign.

Dumnonia: The counties of Cornwall & Devon.

Wessex: All of the South West England Government Region bar Dumnonia.

Thames & Solent: The counties of Berkshire, Buckinghamshire, Oxfordshire and Hampshire.

Southern: The counties of Surrey, Sussex & Kent.

East Anglia: The East of England Government Region.

Northumbria: Everything from the Scottish border south to the southern borders of Cheshire, Greater Manchester, South Yorkshire & North Lincolnshire with the Dark Peak of the Peak District National Park also included within Northumbria (see Peak District Core Strategy for map of the Park's three regions, namely the Dark Peak, White Peak and South West Peak).

Mercia: Largely the Government Regions of the East Midlands and West Midlands, minus the Dark Peak. Notes on Mercia: the former coalfields of North Derbyshire/North Nottinghamshire are welcome to join Northumbria if they wish. Similarly Lincolnshire is welcome to join East Anglia if they wish - perhaps via a referendum.

County of London: The present county of Greater London plus approximately 10 miles in all directions, pending appropriate boundaries, but with the Mayor of London given strategy over the entire Metropolitan Green Belt. Any modifications outside of the County of London must be made in conjunction with the appropriate local authority and Kingdom. If existing District Councils fall approximately within this 10 mile extension, they shall become new London Boroughs by right, other areas shall either be subsumed within existing or newly created Boroughs, pending appropriate boundaries.

Reflecting on this 5 years on it still largely works so feel free to create it as a map for Messrs Cook, Elledge and Weldon; if you genuinely are so inclined."

Unless you have an encyclopaedic knowledge of English administrative geography, this is quite hard to visualise, so here’s a second map of the Brown-Pearce regions, this time with population figures for each region, which I calculated using 2015 mid-year estimates from the ONS, plus some towns and cities. A brief word or two on methods follows this.

I've added in a few towns and cities this time

This is not a serious policy proposal for the re-establishment and re-organisation of English regions, but I am serious about provoking discussion on the subject. So, for practical reasons, I used existing local authority boundaries as the basis for creating the regions you see above. There are 326 in total across England and in the map legend you can see how many local authorities fall into each region.

For Dumnonia (from the Brythonic kingdom in Sub-Roman Britain), Wessex and Thames & Solent I followed the spirit of the Brown-Pearce plan to the letter. Dumnonia covers the same area as the present counties of Devon and Cornwall, for example. Their Southern region and East Anglia are as described but minus those local authorities which are within 10 miles of the Greater London boundary. This may seem like an arbitrary cut-off, but when you look at where the metropolitan green belt is and the wider commuting patterns of the area, it makes a lot more sense. This can be seen reasonably well in the County of London zoomed-in map below.

The bright lines are commuting flows. The dark green is green belt.

Mercia and Northumbria are more straightforward - I kept these as described but made sure to put High Peak in the Northumbria region, rather than in Mercia - as I think this makes sense on a number of fronts, including the geographic. Note Philip’s generosity when he writes ‘the former coalfields of North Derbyshire/North Nottinghamshire are welcome to join Northumbria if they wish’. I’m not massively keen on some of the names, so I’ve had a go at re-branding them at the end.

Following their instructions and using precise measurements with my GIS toolbox, I ended up with a pleasingly round shape for the County of London, apart from a couple of areas which I thought shouldn't be in the ‘County of London’ (for fear of civil war, but also functional economic reasons). One example was Royal Tunbridge Wells, so I removed this and put it back into the Southern region. Once I’d sorted this little tweak I was left with something I was happy with. I also calculated the land area of each region, as you can see in the third iteration of the map, below.

This time with area metrics, in square kilometres

At first glance, this map might seem somewhat odd, but I think it is actually a pretty good representation of what a Federal England could look like. For example, we have three very large regions in terms of population, with the County of London (13.42m), Mercia (10.34m) and Northumbria (15.28m) the big hitters all with more than 10 million people. Northumbria has the most local authorities, at 73, one more than the County of London which could be thought of as some kind of symbolic gesture, but it’s just a coincidence. The Brown-Pearce plan threrefore increases the number of ‘London Boroughs’ from 33 to 72. I'm sure that won't cause any political problems.

The County of London is quite a nice shape but of course it kind of tramples over some important historic and ceremonial boundaries as you could see above. But I would argue - and have before - that on a functional economic basis London is actually much bigger than Greater London. Things like commuting, housing and economic growth need appropriate economic geographies if they are to be governed and planned properly. Too many important issues are stifled by inappropriate boundaries - a topic not unrelated to the wider political turmoil we find ourselves in here and in the United States. 

The way spaces are divided has very significant implications in the real world. In the same way that political gerrymandering can skew the balance of power in elections, economic underbounding can limit growth and opportunities for development. This is not a new argument, of course, and I’m making it simplistic here, but there is a need to engage with this question more seriously, particularly for London.

Set in contrast to the three big regions are five others with varying characters, histories and identities. The large variations in population I don’t consider to be a problem. So long as the structures are right, this can work well. The examples of US states or German Lander may offer useful comparators here even if they are far from perfect.

But, like I said above, this is really just food for thought for now because I was intrigued and impressed with the regions set out by Philip and Nathan. One thing I wasn’t so keen on was the names, so I’ll end by addressing that.

Summing up
The slightly experimental, playful nature of this piece got lost somewhere above when I began talking about gerrymandering and economic underbounding, so to bring it back down to earth and in the slightly frivolous pre-Christmas spirit that I approached this in, I decided to take the liberty of re-naming and/or re-branding some of the regions. I’ve therefore posted the map below, followed by some further explanation.

Definitely not entirely serious

Northumbria becomes The North of England, because I think it fits better. I know that the historic area of Northumbria covers a much wider region than what we think of today, but it would be a stretch for me or anyone else in South Yorkshire or Merseyside to accept this. I think most people could live with this new name, particularly if they still have a local authority or city region to cling to.

Mercia suffers slightly from the same issues as Northumbria so I have re-named this rather grandly as The Heart of England. There is some historical basis for this and I think it sounds nice, so that’s that.

Dumnonia? Well, I give Philip and Nathan credit for knowing this but it’s just too obscure for me. So, I’m going to go with The Sunshine Coast. Hmmm. What about the inland bits? That is a good point, but I couldn’t think of anything better so I’m happy to receive suggestions but I like the positive vibes generated by the word and the image. And, also, as a native Highlander it's always seemed a very sunny place to me.

Wessex is a difficult one because it’s so fixed in the popular imagination without people really having a precise idea of where it is. Somewhere near Bristol? Something to do wtih Thomas Hardy? That’s why I’ve changed it to Greater Wessex. Including Great or Greater in things always seems to work as people like to be Great. But, slightly more seriously, I think it adds a nice bit of fuzziness that helps soften the blow for non-Wessexian Wessexers.

Thames and Solent works for me and I can’t think of an improvement so I’m leaving it at that. I may just change the ‘and’ to ‘&’. This may actually have been part of the original plan anyway.

For East Anglia, I’m going with Greater again because I think this helps highlight its size and scale. It means that we have Greater Anglia the region and also the train operating company but that’s unavoidable. Greater Anglia it is. This 'Greater' thing really is a winner.

Southern is just bit vague for my liking as it could just refer to the south of England so since it draws from the existing historic counties there I have just added that to the end to make it Southern Counties. I like the way this sounds and we need one that pays homage to historic counties in this way by actually having the word 'counties' in the name.

Finally, we had County of London. If London is going to trample over existing or historic county boundaries that people know and love, I think we at least have to respect that and not use the county designation for the name of a new region. For that reason, and drawing upon international examples, I simply re-named this Metro London. This partly gets to the fact that these areas are definitely not ‘London’ proper but part of its wider economic sphere of influence. They may still be in (e.g.) historic Buckinghamshire but they retain very powerful, important ties to the London metropolitan area. We also have the Metropolitan Green Belt here, which made me think the name was a good fit. Here are those regions one final time, then, with the old Government Office Regions overlaid on top.

New ones are better than the old, I feel

So, that’s that. As I’ve said, this is entirely experimental and is intended to provoke discussion. I’d be very keen to see what other people come up with, either in relation to drawing different boundaries or re-naming the ones presented here. The reason I used the year 2020 in the maps and title of the blog post is that, post-Brexit, we might actually need some ideas on all this. Maybe, if it happens.

Also, in case you were wondering, Wales remains Wales, Scotland remains Scotland and Northern Ireland remains Northern Ireland. This was deliberately just about England. 

If you made it this far, thanks for reading.