Friday, 8 November 2019

Amazing 3D rendering with Aerialod - a tutorial

One of the more exciting developments in recent years for those of us into geospatial things is the arrival at the end of October 2019 of Aerialod by ephtracy. What am I talking about? I'm talking about being able to create the kind of images you see below in only a few minutes using free software and open data. Scroll past the images for some tips on how to do this, and note that this write up was completed on 8 November 2019 and refers to v0.0.1.

The Cuillin ridge in Skye

Part of the City of Belfast in Northern Ireland

This is indeed a little bit of London

A little circular bit of London

This is also London, for Millwall fans

The Cuillins in Skye, featuring Sgùrr Alasdair

This is Sheffield, around the area where I work

Okay, so how do you get up and running? First of all, go to the Aerialod website and download the package you need. It's Windows 64- or 32-bit only for now and you just download and unzip and then run the .exe to launch Aerialod.

You do get some sample raster data in the zipped download (in the 'map' folder) but if you also download the 'Sample Maps' archive next to the software download button you'll get a central Manchester Lidar png and a Mars (yes, the planet) png. This is a nice reminder that Aerialod is able to handle different formats, including .asc, .png. and .tif for example. I haven't tried any other formats though I think you'll be okay with .jpg too.


They don't look like much here, but wait and see!

When you launch Aerialod you'll see something pretty much like the image below - and it will have that blocky sample layer in there. This is useful for playing around with so you can get to grips with navigation etc. Just note that when you zoom or move around Aerilod may briefly look pixelly/fuzzy as it re-renders, so don't worry about that. It sharpens up perfectly once it's done, although with more complex layers it takes a bit longer.

You may be a bit bewildered at first, but it won't last long

Before I forget, be sure to look in the config folder and open the hotkey.txt file, which I've shown below. That's really useful. But I find mouse navigation easier, so read on. Also note that the second section of hotkey actions below combine a left mouse button click with keyboard actions too.


Hit D and be amazedx

If you are finding the interface too small and can't actually see the icons easily (e.g. if you have a 4k monitor or something like that) then you can use CTRL + or - to scale the UI but you could also just edit this bit of the config.txt file (in the config folder) so instead of 1.0 it says 2.0, like below:

view :
{
// 0.5 ~ 3.0
ui_scale : '1.0'
}


Okay, we're all set now so here are the basics of moving things around:


  • Scroll wheel/middle mouse button - you can scroll forwards and backwards to zoom in and out and with the button pressed down you can position the layer wherever you want. 
  • Right mouse button - tilt/pan/rotate etc. Just have a play and you'll see what I mean.
  • And of course with keys, as above, W to zoom in S to zoom out, D to rotate clockwise and A to rotate anti-clockwise.

You loaded the software, figured out navigation with the sample data but now want to render some real world stuff. See below for how to do that.

There are tons of sources for this, including things like NASA's 30 metre SRTM but really it's going to look best with high-resolution DSM or DTM data and for this Lidar is ideal. On the Aerialod page they link to two potential sources of this - the UK's Defra Lidar page where you can download a variety of 25cm, 50cm, 1m and 2m Lidar data for England or get NASA's HiRISE data for elsewhere in the universe.

To get data into Aerialod, the easiest method in my opinion is just to drag and drop a raster file straight in, so that's what I just did with the Manchester.png layer you can see in the screenshot below. To be clear, what you see below is just the result of me dragging and dropping a png file into Aerialod. I haven't done anything else yet.

That was easy!

Before I say more about settings, a word on getting data into Aerialod in other ways. In the top right of the window you'll see a save icon. That will save the raster layer (not the 3D render) to a location of your choice. The next button (the open folder icon) will let you open a new single layer instead of dragging and dropping into the window. The next button (the closed folder icon) is a bit different, but it effectively allows you to stitch together all layers in a single folder, and it's amazing. Just click one item in the folder and it will add them all, as you can see below. The final button (blank page) just starts a new Aerialod blank view.

Stitch multiple files!

Okay, see below for where I've used the open folder button to add a single layer. This is for a single tile (of about 7MB) near the University of Sheffield. In the image below this, I've used the closed folder icon to select the first item and then all layers are added. You don't actually have to select the first item though, just any of the files in the same folder. This is really handy as when you download Lidar data it's more than likely going to be comprised of lots of little chunks as individual files.

Single layer from within a folder

All layers from within a folder

If you get too greedy and try to add a gazillion zigabytes of data, Aerialod may crash. I know, I've tried. Anyway, for the rest of this tutorial I'm mainly going to use the Manchester.png sample file provided on the Aerialod download page - and what you see below is the result of me just dragging and dropping it into the viewer. This covers a good chunk of Manchester city centre and also a bit of Salford and Trafford and to the left of the image below you might just spot Old Trafford - both the football and cricket ground.

Just another sunny day in Manchester


Okay, so the rest of this will cover the main things you need to know but I can't cover everything because a) that would take too long and b) I don't actually know enough to tell you everything and I'm kind of learning as I type here. So the two annotated screenshots are my main contribution for now. Look at the images closely though and you should learn enough to produce great visualisations in not much time at all.

Click on the next two images for a little explainer of what does what and then try it yourself.

This covers the basics, click to expand

Some more tips - including on Focus


Offset - actually quite useful for flood viz
A full screenshot so you can see some of the settings here

That's basically it for now. All that's left to say is don't forget to check the #Arielod hashtag on twitter as well as the @ephtracy Twitter feed. Last of all, don't forget that you can drop all kinds of things into Aerialod, not just terrain models and suchlike. I've experimented with adding in photos of people, which often looks sort of cool but also weird, and I've tried all sorts of other stuff but I'll end with one of the MiniScale raster relief maps of Great Britain which are part of their open data offering. It's not intended to be used this way but I think it looks quite interesting.

This actually turned out alright


Saturday, 2 November 2019

A deprivation by constituency chart

Yesterday I decided to update a little chart I made after the 2017 General Election. It was inspired by a histogram that Owen Boswarva made and the idea was very simple: put England's 533 constituencies into 10 columns, with the most deprived on the left and least deprived on the right, and then colour it by party. The image below is the result. UPDATE: I have now done a full-UK version of this - see below. Also includes an animation and the individual frames which show one party at a time. Read the notes on the UK chart for more information.

Link to slightly higher resolution version

Full size version

Easier to decipher as a gif

Labour

Conservative

SNP

Liberal Democrats

DUP

Sinn Féin

Plaid Cymru

Green

Independent (at 2017 General Election)

OORDEEER!


I did this out of curiosity the last time and then after speaking to my colleague Philip Brown about data, elections and suchlike I decided to update and try to improve the older chart, which was informative but a bit of an assault on the eyeballs. I then saw that the House of Commons Library team re-ran their analysis aggregating the English Indices of Deprivation for the 533 English constituencies, so that was all I needed. And, by the way, the House of Commons Library team are in my opinion doing some of the best data curation, manipulation and analysis out there - really great team. 

Of course, the results are hardly surprising but I didn't expect the sorting to be quite so stark. Naturally I tweeted the graphic and lots of people also found it interesting. So, I've done a slightly different version below which has the constituency names in the boxes - but you'll have to click and zoom to read those.

Full-size version here

Zoomed-in extract of the image above


It's not possible to do a full UK-wide one using a single dataset from the same time point, because the deprivation indices for each country of the UK are slightly different and cover different time period but it would be nice to have been able to - and at least in Scotland the colours would be quite spread across the deprivation spectrum. UPDATE: as you can see above, this last statement is true and I've also done the full-UK chart.

I also did one more version of the graphic, this time in a very long single column vertical one, the original of which is here, with a lower resolution one below. The tends to work great on a mobile but needs a bit of zooming in and out in most browsers. I've added the 'required swing %' figure to this one, showing what percent vote shift it would take for a constituency to change hands.

Click to see full size


Looks like this when full width on screen


There wasn't any great agenda or rhyme or reason behind this, I just wanted to see what it looked like and in particular how much the colours would be grouped and where the obvious anomalies were.


Given what I do for a living I'm duty-bound to point out the following obvious but important things:


  • Correlation is not the same as causation (yawn! but true).
  • I don't believe voting Labour makes you poor, or that voting Conservative makes you rich, as some people online seem to have implied - or even that 'the poor' vote Labour or 'the rich' vote Conservative - and I certainly wasn't trying to demonstrate either of these but understand that's sometimes how people see things.
  • There is a reasonably high amount of variation within most constituencies in relation to levels of deprivation, although at the top and bottom of the scale a lot less than you might imagine - I'm currently working on a project all about this kind of thing.
  • Colours - they are the html values for the party colours from the respective party websites.
  • The aggregation method used to derive the constituency rankings could be done a number of different ways, and therefore can produce slightly different results based on how you do it but the order of places would always be broadly the same.

That's all for now.

Monday, 2 September 2019

Globe projections and insets in QGIS

Proper QGIS boffins may remember Hamish Campbell's excellent post on the topic of 'Azimuth Orthographic Projections with QGIS' from 2014, in which he described a method for creating maps with an azimuthal orthographic projection - or, what non-boffins might call a 'globe view'. The good news is that in QGIS 3.4 you can do this very easily using the built in The_World_From_Space projection and you can easily create new ones with the view of the world you want. See below for an example and then read on for the method. The great thing is that you can also have a globe view and a normal, flat view in the same Print Layout as QGIS now supports multiple projections in the same layout.

The graticules are from Natural Earth

Okay, how to find this 'world from space' projection? The easiest way is to click the little CRS button at the bottom right of the QGIS window, or go via Project > Properties > CRS. Then you just search for it in the CRS search box, as you can see in the screenshot below - where you'll also see a couple of modified CRS versions I created. Just in case you don't know, CRS stands for coordinate reference system and they all have an EPSG code so that's why you'll see 'EPSG' and some numbers. If you're a proper carto boffin you'll know your 4326 from your 3347.

You can see the properties of the projection here

Okay, so that's how you set it. How to create a view centred on where you want? For that, you first need to copy a bit of the text in the above box, which I've pasted below so you can try it yourself.

First, copy this text: 

+proj=ortho +lat_0=42.5333333333 +lon_0=-0.53333333339999 +x_0=0 +y_0=0 +a=6370997 +b=6370997 +units=m +no_defs

Then replace the 42.5333333333 wih the latitude you want and the -0.53333333339999 with the longitude you want. You don't need so many decimal places! So, for example, I did this for New Zealand and this gives me:

+proj=ortho +lat_0=-40.5333333333 +lon_0=-157.3333333339999 +x_0=0 +y_0=0 +a=6370997 +b=6370997 +units=m +no_defs

You can see by looking at the numbers above that I have centred the projection on 40 degrees south and 157 degrees east. In order to create this new projection you just need to go to Settings > Custom Projections > click the green plus symbol and then at the bottom of the window give your project a name and then paste in your new projection information into the box. See below for what this looks like for my NZ projection.

Click OK to activate it

Once you've created a new projection you can just go back to Project > Properties > CRS and search for the new projection and use it. This is what the NZ one looks like, below.

This is not a map without New Zealand

I mentioned above that you can have a globe view like the above in a Print Layout with a different, flat projection in the main map. This is very easy. See below for an example where I have the globe view projection in the top left and a different projection with the same layers in the bottom of the image - in this case using the Equal Earth projection.


One Print Layout, two different projections

This is where you set the CRS for each map

I've put this together quickly so it's not very polished but see below for an example of the kind of thing you could do using this method - I quite like the globe locator inset, and it's also used by a good few news organisations, including the BBC. This used to be quite a lot of hassle in QGIS but now it's very easy and, I think, quite effective.

I've used Layer Effects to add a drop shadow to the globe

The only thing I should add is that in order to get the sea showing in the background here - rather than having the land floating on a blank canvas, I took the 1 degree graticule and converted it to a polygon layer via Vector > Geometry Tools > Lines to Polygons...

One last example below, with lots of cities, just for fun.

Cities are from Natural Earth as well

That's all for now. I hope you find this useful.

Friday, 16 August 2019

Constituency cards

Everything is fine 
This blog post provides some background to my attempt to create a simple 'constituency card' for every UK constituency in the run-up to an election that may or may not happen. This is at a time when, if I may be so bold, it can seem like everything is not as fine as it perhaps once was. I had the idea for this when I saw that there was a new set of official MP portraits that anyone could download and use but then other things got in the way. So, this is part open data experiment and part election prep, though mostly the former. Consider it my attempt to take lots of ugly data and turn it into useful information in an easy-to-digest way. See, I told you everything was fine.

Just want to see the maps? Okay, here you go. Want the underlying data files ? No problem.

An example of one of the constituency cards

Things keep changing

Here's an easy-to-remember short url you can use to go straight to the cards: bit.ly/constituencycards

What's the point of this?
I'm not a member of a political party and never have been so there is no underlying conspiracy here. Or at least, if there is, nobody told me about it. Aha, but why did I put the '% Swing required' figure on the individual constituency cards? I put it there because I think it's an important thing to know, for everyone with an interest in politics, regardless of whether you're the MP for Knowsley (George Howarth, Labour, majority: 42,214) or the MP for North East Fife (Stephen Gethins, SNP, majority: 2).

The point of this project (which is a spare time thing, not part of my day job) is to provide a single card for all 650 UK constituencies, which tells us who the MP is, what they look like, how they did at the last election or by-election, what it would take for the seat to change hands, plus a bit of other information.

On the last point, I calculated the straight-line distance from the centre of each constituency to the Palace of Westminster. The closest, unsurprisingly, is Cities of London and Westminster at 0.86 miles and the furthest is Orkney and Shetland at 581 miles.

Ideally, people will be able to click the link to the image files on their phone, tablet or computer and then flick between individual cards and make comparisons between places, find out more about individual constituencies and generally learn stuff.


How did I do this?
I compiled a list of official portraits using this blog post on the API, spent ages trying to figure it all out and then once I had a list of official photos, I added to it with other photos in the public domain because there isn't an official photo for every MP. Then I supplemented this data with information on current MPs from mySociety, then I put together a UK-wide geo file and chopped out the loughs of Northern Ireland so it looked right. I then added information on distance to Parliament and the size of each constituency, and mashed it all together. Then, in QGIS, I spent a while sorting the layout, editing and editing and tweaking and tweaking until arriving at the final result. The image below shows you what I ended up with.

The labelling is always tricky

Stuff wot I got wrong or not quite right enough
Sometimes the labelling isn't perfect. That is, sometimes places you might expect to be labelled are not, and some that you think should not be actually are. My labels file has 42,000 or so place names and I use a variety of rules and filters to decide what gets placed on each map but occasionally this doesn't work that well. I could devote tons of hours to it and make further improvements but I think I'm at the point where I'm happy enough with it.

Independent MPs. I'm sorry not to have given the different MPs, who are in different independent groupings, different colours but too many colours would not work well in my opinion. So this is not quite right but to be honest I found it hard to keep track of who is in what independent grouping and who is not.

Occasionally my swing figures are 0.1% out or so compared to some of the figures I've seen elsewhere online but in general they agree with the Target Seats lists on Election Polling.

Colours, maybe. I tried a few different versions without a white dim surrounding the featured constituency on each map but it became a riot of colour at times so I've gone with a 33% opacity white mask layer to dim it a bit. Sometimes this isn't perfect but I like it better than the alternatives, including a dark dim.


Style stuff, etc.
This can be controversial! But I'm quite chilled about it really. I tried my best to make them look good and also to build in some kind of logic and flow to the individual cards but of course they're never going to be perfect but I'm happy enough with them.

If you want to play around with the files you can find them in the files repo. You can style the files using the html colour codes in these - there is a colour for the winning party and one for the second placed party.

Decisions, etc. Well, the name of the constituency goes at the top left and the whole top row of each card is reserved for these names. I have sized everything so the longest constituency names (e.g. Cumbernauld, Kilsyth and Kirkintilloch East) don't run beyond the end of the map image below it. I have also placed the little black-dot-locator inset there as it is close to the name of the constituency and your eyes don't have to move much to locate it, and then you can scan down to the main map image.

I've given the sea a muted blue colour, which also applies to Northern Ireland loughs. I decided not to add lochs and lakes elsewhere as I couldn't decide where I would place the size cut off (e.g. include Loch Morar but not Loch Shin?). Too messy, but I made an exception for Northern Ireland because people are forever making maps without their loughs and they are really big and important.

I included a 1 mile scale bar not because I'm a devotee of the imperial system but because people know how far it is and for most constituencies the bar is big enough. The one mile scale bar is of course tiny on the Ross, Skye and Lochaber card and some of the other huge constituencies.

I included a bit of foreshore for Great Britain (didn't have the foreshore data for Northern Ireland) as I think this makes things look a bit better - usually - in coastal areas although of course it gives some places the appearance of having a lovely sandy beach when they actually don't. But I like to bequeath beaches to people who don't have them - in the spirit of mending the nation's divides.

You'll notice that in lots of maps you can see a nearby city. This is because I wanted people to be able to look at a map and say, for example, "Ah, so Aberavon is near Swansea" or "Oh, right, Meon Valley is kind of between Southampton and Chichester". I don't know about you but some of the constituency names I find quite baffling as they give little clue to the uninitiated where they actually are.

I also included a fairly dull building mask layer. The idea here is you can see the layout and form of the built environment and say things like "this is quite a densely built up area" or "this constituency is quite sparsely populated". You may know this already but I think it adds an extra dimension of knowledge for places I'm unfamiliar with so I like it. Just hard to get the balance right so it doesn't mess too much with the colours.

I decided it would be good to add a party-coloured frame round each photo in lieu of a legend and then the MP's majority below that in large bold text. The MP name and party go in the coloured box below the image. But then I thought it would be useful to show who came second and what kind of swing would be needed for the second place party to win. In some cases they are not likely to win, to say the least, but in a good few things are very close. I make it 171 constituencies (out of 650, so 26% of the total) where the required swing is less than 5%.

The descriptive text below each map has some other information I found useful and I thought I'd add some geographical context with distance from parliament and also area in square miles - these figures make more sense when you compare them with others I suppose. I calculated these in QGIS.

I also added a sources box to the bottom right of the image so it's clear where the data come from.

The font is Montserrat.


Nerd notes
Not too much else to say here but I did use Excel for some vlookup type stuff but basically everything was done in QGIS using the Atlas tool and a range of quite messy looking functions. For example, where the required swing was below 0.05% I had to make sure two decimal places were displayed instead of the default one I used otherwise it would say 0.0% there. For that, I used this:

 format_number( "REQSWING"  ,  if( "REQSWING"< 0.06, 2,1))||'%'

Making sure there were no 0.0% swing figures was fiddly

I used a rule-based symbology for the place name labels - using this rule to only show places in the current Atlas feature:

intersects( @atlas_geometry ,$geometry) 

And then to make sure cities outside the current Atlas feature were always showing I used this on a copy of the same layer, filtered to only show cities:

NOT intersects( @atlas_geometry ,$geometry) 


But how on earth do you get data defined text colours? Well, I couldn't figure out a way to make this happen in QGIS Print Layout as it's not a default option on font colour so I figured out a workaround.

What I did was add the swing figure to the layout using the  "REQSWING" variable in the layer driving the Atlas. I added this as black text. Then I added a white text box over the top of it. Then what I did was set the colour of this text box overlay to match the colour of the party that came second. This was easy because I already created a field called "secondcol" with the html colour code of the party that came second. Then I changed the Rendering blend mode to Screen so that the black text changes to whatever the colour of the second placed party is. Sounds complex but works perfectly - see below for screenshot.

A workaround that works - I couldn't figure out another way

You may also notice that in the five constituencies that have had a by-election since 2017 there is a little '(By-election)' indicator below each Majority figure. Since I created a field in my data table with data on when the last election was (i.e. either 2017, 2018 or 2019) I was able to add a text box with '(By-election)' in it and then set it to be 100% transparent if the value was 2017. Otherwise it is displayed. This is the text I used in the data defined over-ride box on Rendering for that.

CASE 

WHEN  "last_vote" = 2017 THEN  0

ELSE  100

END

Again, a bit fiddly but effective

That's about it really. There's nothing super-fancy going on behind the scenes and of course I still haven't really figured out how to make friends with date and time formats. Drives me mad, but I got there in the end. See below for the mess that makes up the data-driven text at the bottom of each card (using the 'Render as HTML' tick box in QGIS).


  • '<b>' ||  replace(  "cname1" ,'Kingston upon H','H')  || '</b> had an electorate of ' ||  format_number("elect17" ,0) || ' and a population of ' ||  format_number("pop17",0)  || ' in 2017. The distance from the centre of the constituency to Parliament is ' ||  format_number( "mi_fr_pow" ,0)  || ' miles, and the constituency covers an area of ' ||     format_number("sq_mi" ,0)  || ' square miles. At the 2017 general election, this constituency was number ' ||  "dec_order"  || ' out of 650 to declare the result, at ' || format_date( "just_time" ,'hh:mm') || ' on ' ||  format_date(  "dec_time" ,'dddd d MMMM') || '.'



Whose shoulders am I standing on here?
Ordnance Survey, mySociety, the amazing team at the House of Commons Library, Philip Brown, Elvis Nyanzu, Alex Parsons, the Parliamentary Digital Service, the very nice voters of the UK, and of course the team who make QGIS.

Without the hard work, expertise, knowledge and experience of a wide variety of people and organisations it is impossible to do stuff like this. I am simply trying to bring data together and make it into useful information. Trying to contribute to the open data ecosystem I suppose.


Who will win the next General Election?
Nobody.