Friday, 16 August 2019

Constituency cards

Everything is fine 
This blog post provides some background to my attempt to create a simple 'constituency card' for every UK constituency in the run-up to an election that may or may not happen. This is at a time when, if I may be so bold, it can seem like everything is not as fine as it perhaps once was. I had the idea for this when I saw that there was a new set of official MP portraits that anyone could download and use but then other things got in the way. So, this is part open data experiment and part election prep, though mostly the former. Consider it my attempt to take lots of ugly data and turn it into useful information in an easy-to-digest way. See, I told you everything was fine.

Just want to see the maps? Okay, here you go. Want the underlying data files ? No problem.

An example of one of the constituency cards

Things keep changing

Here's an easy-to-remember short url you can use to go straight to the cards:

What's the point of this?
I'm not a member of a political party and never have been so there is no underlying conspiracy here. Or at least, if there is, nobody told me about it. Aha, but why did I put the '% Swing required' figure on the individual constituency cards? I put it there because I think it's an important thing to know, for everyone with an interest in politics, regardless of whether you're the MP for Knowsley (George Howarth, Labour, majority: 42,214) or the MP for North East Fife (Stephen Gethins, SNP, majority: 2).

The point of this project (which is a spare time thing, not part of my day job) is to provide a single card for all 650 UK constituencies, which tells us who the MP is, what they look like, how they did at the last election or by-election, what it would take for the seat to change hands, plus a bit of other information.

On the last point, I calculated the straight-line distance from the centre of each constituency to the Palace of Westminster. The closest, unsurprisingly, is Cities of London and Westminster at 0.86 miles and the furthest is Orkney and Shetland at 581 miles.

Ideally, people will be able to click the link to the image files on their phone, tablet or computer and then flick between individual cards and make comparisons between places, find out more about individual constituencies and generally learn stuff.

How did I do this?
I compiled a list of official portraits using this blog post on the API, spent ages trying to figure it all out and then once I had a list of official photos, I added to it with other photos in the public domain because there isn't an official photo for every MP. Then I supplemented this data with information on current MPs from mySociety, then I put together a UK-wide geo file and chopped out the loughs of Northern Ireland so it looked right. I then added information on distance to Parliament and the size of each constituency, and mashed it all together. Then, in QGIS, I spent a while sorting the layout, editing and editing and tweaking and tweaking until arriving at the final result. The image below shows you what I ended up with.

The labelling is always tricky

Stuff wot I got wrong or not quite right enough
Sometimes the labelling isn't perfect. That is, sometimes places you might expect to be labelled are not, and some that you think should not be actually are. My labels file has 42,000 or so place names and I use a variety of rules and filters to decide what gets placed on each map but occasionally this doesn't work that well. I could devote tons of hours to it and make further improvements but I think I'm at the point where I'm happy enough with it.

Independent MPs. I'm sorry not to have given the different MPs, who are in different independent groupings, different colours but too many colours would not work well in my opinion. So this is not quite right but to be honest I found it hard to keep track of who is in what independent grouping and who is not.

Occasionally my swing figures are 0.1% out or so compared to some of the figures I've seen elsewhere online but in general they agree with the Target Seats lists on Election Polling.

Colours, maybe. I tried a few different versions without a white dim surrounding the featured constituency on each map but it became a riot of colour at times so I've gone with a 33% opacity white mask layer to dim it a bit. Sometimes this isn't perfect but I like it better than the alternatives, including a dark dim.

Style stuff, etc.
This can be controversial! But I'm quite chilled about it really. I tried my best to make them look good and also to build in some kind of logic and flow to the individual cards but of course they're never going to be perfect but I'm happy enough with them.

If you want to play around with the files you can find them in the files repo. You can style the files using the html colour codes in these - there is a colour for the winning party and one for the second placed party.

Decisions, etc. Well, the name of the constituency goes at the top left and the whole top row of each card is reserved for these names. I have sized everything so the longest constituency names (e.g. Cumbernauld, Kilsyth and Kirkintilloch East) don't run beyond the end of the map image below it. I have also placed the little black-dot-locator inset there as it is close to the name of the constituency and your eyes don't have to move much to locate it, and then you can scan down to the main map image.

I've given the sea a muted blue colour, which also applies to Northern Ireland loughs. I decided not to add lochs and lakes elsewhere as I couldn't decide where I would place the size cut off (e.g. include Loch Morar but not Loch Shin?). Too messy, but I made an exception for Northern Ireland because people are forever making maps without their loughs and they are really big and important.

I included a 1 mile scale bar not because I'm a devotee of the imperial system but because people know how far it is and for most constituencies the bar is big enough. The one mile scale bar is of course tiny on the Ross, Skye and Lochaber card and some of the other huge constituencies.

I included a bit of foreshore for Great Britain (didn't have the foreshore data for Northern Ireland) as I think this makes things look a bit better - usually - in coastal areas although of course it gives some places the appearance of having a lovely sandy beach when they actually don't. But I like to bequeath beaches to people who don't have them - in the spirit of mending the nation's divides.

You'll notice that in lots of maps you can see a nearby city. This is because I wanted people to be able to look at a map and say, for example, "Ah, so Aberavon is near Swansea" or "Oh, right, Meon Valley is kind of between Southampton and Chichester". I don't know about you but some of the constituency names I find quite baffling as they give little clue to the uninitiated where they actually are.

I also included a fairly dull building mask layer. The idea here is you can see the layout and form of the built environment and say things like "this is quite a densely built up area" or "this constituency is quite sparsely populated". You may know this already but I think it adds an extra dimension of knowledge for places I'm unfamiliar with so I like it. Just hard to get the balance right so it doesn't mess too much with the colours.

I decided it would be good to add a party-coloured frame round each photo in lieu of a legend and then the MP's majority below that in large bold text. The MP name and party go in the coloured box below the image. But then I thought it would be useful to show who came second and what kind of swing would be needed for the second place party to win. In some cases they are not likely to win, to say the least, but in a good few things are very close. I make it 171 constituencies (out of 650, so 26% of the total) where the required swing is less than 5%.

The descriptive text below each map has some other information I found useful and I thought I'd add some geographical context with distance from parliament and also area in square miles - these figures make more sense when you compare them with others I suppose. I calculated these in QGIS.

I also added a sources box to the bottom right of the image so it's clear where the data come from.

The font is Montserrat.

Nerd notes
Not too much else to say here but I did use Excel for some vlookup type stuff but basically everything was done in QGIS using the Atlas tool and a range of quite messy looking functions. For example, where the required swing was below 0.05% I had to make sure two decimal places were displayed instead of the default one I used otherwise it would say 0.0% there. For that, I used this:

 format_number( "REQSWING"  ,  if( "REQSWING"< 0.06, 2,1))||'%'

Making sure there were no 0.0% swing figures was fiddly

I used a rule-based symbology for the place name labels - using this rule to only show places in the current Atlas feature:

intersects( @atlas_geometry ,$geometry) 

And then to make sure cities outside the current Atlas feature were always showing I used this on a copy of the same layer, filtered to only show cities:

NOT intersects( @atlas_geometry ,$geometry) 

But how on earth do you get data defined text colours? Well, I couldn't figure out a way to make this happen in QGIS Print Layout as it's not a default option on font colour so I figured out a workaround.

What I did was add the swing figure to the layout using the  "REQSWING" variable in the layer driving the Atlas. I added this as black text. Then I added a white text box over the top of it. Then what I did was set the colour of this text box overlay to match the colour of the party that came second. This was easy because I already created a field called "secondcol" with the html colour code of the party that came second. Then I changed the Rendering blend mode to Screen so that the black text changes to whatever the colour of the second placed party is. Sounds complex but works perfectly - see below for screenshot.

A workaround that works - I couldn't figure out another way

You may also notice that in the five constituencies that have had a by-election since 2017 there is a little '(By-election)' indicator below each Majority figure. Since I created a field in my data table with data on when the last election was (i.e. either 2017, 2018 or 2019) I was able to add a text box with '(By-election)' in it and then set it to be 100% transparent if the value was 2017. Otherwise it is displayed. This is the text I used in the data defined over-ride box on Rendering for that.


WHEN  "last_vote" = 2017 THEN  0

ELSE  100


Again, a bit fiddly but effective

That's about it really. There's nothing super-fancy going on behind the scenes and of course I still haven't really figured out how to make friends with date and time formats. Drives me mad, but I got there in the end. See below for the mess that makes up the data-driven text at the bottom of each card (using the 'Render as HTML' tick box in QGIS).

  • '<b>' ||  replace(  "cname1" ,'Kingston upon H','H')  || '</b> had an electorate of ' ||  format_number("elect17" ,0) || ' and a population of ' ||  format_number("pop17",0)  || ' in 2017. The distance from the centre of the constituency to Parliament is ' ||  format_number( "mi_fr_pow" ,0)  || ' miles, and the constituency covers an area of ' ||     format_number("sq_mi" ,0)  || ' square miles. At the 2017 general election, this constituency was number ' ||  "dec_order"  || ' out of 650 to declare the result, at ' || format_date( "just_time" ,'hh:mm') || ' on ' ||  format_date(  "dec_time" ,'dddd d MMMM') || '.'

Whose shoulders am I standing on here?
Ordnance Survey, mySociety, the amazing team at the House of Commons Library, Philip Brown, Elvis Nyanzu, Alex Parsons, the Parliamentary Digital Service, the very nice voters of the UK, and of course the team who make QGIS.

Without the hard work, expertise, knowledge and experience of a wide variety of people and organisations it is impossible to do stuff like this. I am simply trying to bring data together and make it into useful information. Trying to contribute to the open data ecosystem I suppose.

Who will win the next General Election?

Saturday, 3 August 2019

Playing around with shaded relief maps

Last summer I was playing around with maps of where ships go, but this summer I'm back on land with some shaded relief maps, just for fun - and also a bit of an experiment in styling, labelling and all that. This is all very simple: I took a single tile from NASA's SRTM 1-arc second dataset (basically a map tile with a resolution of about 30 metres, covering a rectangular area about 110km north to south and 60km east to west). This data was captured in 2000 by the Space Shuttle Endeavour from 233km above the earth. The easiest way to download it is using Derek Watkins' 30-meter SRTM Elevation Tile Downloader. Anyway, enough of this for now, here's one of the final maps, which I'll explain below alongside more maps and technical information and stuff about the area I've mapped.

This is the 'single malt' version

The map you see above is of a single SRTM 30 metre tile - NG57004 - and I did it all in QGIS 3.8, no other software involved. There are a few layers here with different blend modes and transparencies plus some basic labelling that I added manually. The area itself is more or less a bit of the Cairngorms plus Speyside in the north of Scotland and is just to the south east of where I grew up so I'm very familiar with it, and if you like Scotch whisky or shortbread you might be as well.

Movie stars, whisky and shortbread

This area is world-famous for whisky production and indeed the Speyside area is one of the five whisky regions of Scotland. It's also where I went for school trips to places like Boat of Garten and Nethy Bridge. If you're into this kind of thing you'll also notice that some of the place names relate to famous Scotch whiskies, including Glenlivet, Knockando and Dallas (as in Dallas Dhu single malt, which actually comes from just outside of Forres, nearby). You may notice in the bottom right corner of the map some places associated with the royal family - including Braemar and Crathie. You might also see Aberlour in the north east portion of the map and that's where Walkers shortbread comes from - one of Scotland's big food exports and which I have seen for sale in airports and shopping malls from LA to Beijing.

You'll find polar bears down in Kincraig and even a bit of surfing at Lossiemouth beach (cold water! I've tried it). But I mainly chose this area because of the lumps and bumps of the Cairngorms - not very high elevation-wise but at 57 degrees north and with five of the highest six mountains in the UK (named on the map) this part of Scotland can be a very harsh environment - though also extremely beautiful. I've shared some more traditionally-shaded versions below but this first one above is deliberately a bit different, since I have gone with something close to a single malt whisky colour scheme. See below for the a slightly more exotic one.

A bit exotic but I like this version

Growing up, I also used to take the bus from Inverness, hire some skis and go skiing at Cairngorm Mountain and remember feeling like I was suffering from severe frostbite on the old 1960s chairlift (in the days when sledges and giant inner tubes were not uncommon and it was all a bit hot tube time machine fashion-wise). Anyway, if you like 50 mile an hour crosswinds, boilerplate ice and zero visibility conditions the Cairngorms are the place to be. If you've visited in summer during midgie season then you'll definitely prefer the former (seriously, see this video).

Right, back to the maps - the next three are as follows: a slightly subtler version of the original above, then a couple more traditional shaded reliefs - one in purple shades then one in orange shades.

I was going for 'single malt' but might be burnt toffee

Some people prefer this kind of thing

This is a lot subtler but not as interesting I reckon

A few other things to say about the maps. I've added a couple of place names to the Tarbat peninsula to the north west of the map - they're not part of the area I wanted to map but I always like to know what's what. Plus Portmahomack is an interesting place and you pronounce it just as it's written (like Drumnadrochit!), unlike loads of places in the Highlands, such as Avoch, Kilravock, Ballachulish and many more. Oh, and Moray Firth is not 'Mo-ray' - it sounds just like 'Murray'.

I've also added a curvy 'Cairngorm Mountains' label and a smaller 'Monadhliath Mountains' label on the west side of the Spey - hopefully it doesn't stray too far north although I'm not 100% sure on what's considered the northern limit of this range.

I did do a version in a more traditional topographical style so I've posted this as well. Below that, you'll see one more version - in greyscale, just to complete the set.

Possibly a bit more realistic
This actually looks quite pleasing

I was just playing around with a bit of relief data here during some down time, and wanted to make a few maps that looked a bit different but were still recognisable as shaded reliefs. I could probably spend hundreds of hours on this if I wanted to get everything right. The underlying data is pretty good resolution (30m, roughly), better than the 50m resolution open data from Ordnance Survey. I do have access to the 5m resolution data but since it's not open I didn't use it this time, but it would give a lot better quality image. We do have 1m resolution Lidar data for small chunks of Scotland, so it's a shame the 5m Ordnance Survey data isn't yet open.

'What does this area look like on the ground', I hear you ask? Well a good chunk of the off-road stuff, including the Lairig Ghru and Ben Macdui is actually on Google streetview, and you can see a screenshot below.

If you're looking for some seriously fantastic NASA-related mapping, check out the amazing work of Joshua Stevens.

That's all for now. The original 300dpi maps are in this Dropbox folder, if you want a closer look.

All six in one

Technical notes: the elevation data you see on the map is released via NASA's Jet Propulsion Laboratory at the California Institute of Technology. The highest resolution data - used here - was released in late 2015. Here's what the SRTM user guide says about the data:

  • "datasets result from a collaborative effort by the National Aeronautics and Space Administration (NASA) and the National Geospatial-Intelligence Agency (NGA – previously known as the National Imagery and Mapping Agency, or NIMA), as well as the participation of the German and Italian space agencies. Together, this international space collaboration generates a near-global digital elevation model (DEM) of the Earth using radar interferometry"

The data were collected from the Space Shuttle Endeavour over an 11 day period in 2000:

  • "SRTM was the primary, and virtually only, payload on the STS-99 mission of the Space Shuttle Endeavour, which launched February 11, 2000 and flew for 11 days. Following several hours for instrument deployment, activation, and checkout, systematic interferometric data were collected within a 222.4-hour period. The instrument operated almost flawlessly and imaged 99.96 percent (%) of the targeted landmass at least one time, 94.59% at least twice, and about 50% at least three or more times. The goal was to image each terrain segment at least twice from different angles (on ascending, or north-going, and descending, or south-going, orbit passes) to fill areas shadowed from the radar signal by terrain"

See this BBC article from 2000 about the SRTM mission - it's a very clear explanation. And here's a picture of the 60 metre long mast that was used to collect the data in the maps above. It's called the Able Deployable Articulated Mast (ADAM). There were radars on the mast and on the Space Shuttle itself.

This is where the SRTM data came from - more info here

* I don't even drink whisky, but I'll lose my Highlander card if anyone finds out so please keep it quiet