Sunday, 28 January 2018

The Most Densely Populated Square Kilometre in 39 European Countries

A few days ago I published a short piece in The Conversation about population density across Europe, based on EU gridded population data. This was really my attempt to see if I could produce some numbers which better reflect the experience of population density across Europe, rather than just the raw arithmetic average. At the end of the piece I added a table with some stats, including the maximum 1km population density for each of the 39 countries I looked at. There wasn't space in the article to tell you where all these places were, so I'm doing it here instead. So, without further ado, here are the most densely populated 1km squares in each of the 39 countries, from Spain to Liechtenstein. Click to enlarge.

I did this quickly so it's a bit rough and ready as far as the maps go - a few labelling blips here and there but you can get the idea. Scrolling through from most dense to least dense does generally seem to make sense visually though you can't always tell what's what because some places have lots of high rises in them and you don't get a sense of that from the aerial photos.

Notes: the data are from 2011, so a little old now. We should also really consider these estimates, for a variety of reasons, but I think they are likely to be close to truth, based on my analysis of similar datasets (e.g. ONS in the UK and GHSL globally). Note that if you download the EU data you'll have to then join it to some kind of geodata (e.g. shapefile) because it's not already done. This can be tricky with about 2 million 1km cells and rows of data. Why are England, Northern Ireland, Scotland and Northern Ireland separate here? That's just the way they published the data. But I found it useful to get a better look at patterns in different parts of the UK.

Saturday, 13 January 2018

Millennials vs Boomers

This post is a little follow-up to a map I posted on Twitter recently, showing which areas of the UK had more millennials and which had more baby boomers. Here you'll find a couple more maps plus an interactive version. I've also shared the data along with it if you want to explore it yourself - and perhaps create different definitions for each of these categories. There are many different definitions for these groups but the ones I use here both have 19 separate birth years in them. Using this definition, millennials turned 16-34 in 2016 and baby boomers turned 52-70. Here's a more nuanced map graphic, with top and bottom 10 in terms of the millennial-boomer ratio.

You can see an even bigger version here

There's definitely some correlation between in where universities are and the areas with the highest proportion of millennials relative to baby boomers but it's not always as simple as that. It does help explain why West Wales has a blue tinge and a very slight millennial majority (Aberystwyth University - see also Lancaster) but in many more areas there are other factors. The area with the highest ratio of millennials to baby boomers is Tower Hamlets in London, with 3.9 millennials to every one baby boomer. West Somerset is at the opposite end of the scale, with two baby boomers to every one millennial.

Top 10s

You can explore all this yourself in the basic interactive web map I made. Move your mouse over an area to find out more, including total population and total number of millennials and boomers.

Quick, find some avocado on toast

The data come from the Office for National Statistics' mid-year population estimates for 2016 (the most recent data) and are for local authorities. You can get estimates for lower geographies, but I was only interested in local authorities here. Also, note that the data refer to the usual resident population for the UK as at 30 June of the reference year. The data are provided by administrative area, with single year of age and sex. In the spreadsheet I put in the folder, you can see data by area for each individual year, which is quite interesting in itself. There's also another spreadsheet with percentages and a chart that you can edit yourself. Here's a little extract, plus a gif of some area's age profiles.

Kind of like population pyramids, but not split by sex
Just note the Y axis changes to fit the data

For example, over 2% of the population of Christchurch was aged 90 or over in 2016, according to this ONS data. Birmingham had the highest total population (1.1m), including more than 7,500 people aged over 90. In terms of local authorities with the most millennials, the list goes: Birmingham, Leeds, Manchester, Glasgow, Sheffield, Edinburgh, Liverpool, Bristol, Bradford, Tower Hamlets - not exactly in population order but not far off. Greater London had 2.6m millennials and 1.5m baby boomers (1.77 millennials to every one boomer). Finally, compare and contrast the population profile of Tower Hamlets and West Somerset, as shown below.

Quite a few youngsters here, clearly
The distribution is a little top heavy, it would appear

I was motivated to do this because I've been thinking more about differences, societal cleavages and inequalities recently in some of my academic work. I also saw something on this for US counties by the US Census Bureau in 2015 and more recently George Eaton did a piece on it last week. He called this the 'defining schism' of UK politics, though as things stand there is a lot of competition in the schism-sphere. But if we do want to look at things in terms of schisms, my original map is useful, though I do prefer the more nuanced one at the top of this post.

'The defining schism of UK politics'?

As for me, I'm neither a millennial or a boomer, so I guess I'm somewhere in the middle of this particular schism - so I thought I'd make a map and crunch some numbers in order to understand it better. Feel free to take the data and come up with some other figures for millennials vs boomers but I'm pretty sure the results will be similar; if not now, then in 18 months or so.

Notes: as ever, I made the maps in QGIS. The web map was done in Tom Chadwin's fantastic qgis2web. How dare I label you a 'millennial'!? Blame the US Census Bureau - I've stolen their definition. Baby boomer definitions also vary, and by country, but not by that much so as you'll see on the web 1946-1964 seems about right, but of course it's not fact - just my view.

Wednesday, 27 December 2017

How to make a geogif

Back in October I said I'd do a write up of the workflow for creating a geogif, so here it is (and now I've fixed some errors in the original post, see update at end). I've done quite a few of these over the past few years, including ones of the 2017 UK general election, house prices in England and Wales since 1996, commuting in Greater Manchester and US gun homicides. My favourite is one of the coastline of Skye, though that was just for fun. But sometimes a geogif can be quite useful at telling the story of data quickly, particularly online (I think the general election one does this quite well). Here's the end result of the workflow described below - a geogif of the population of US states in 2017. All of the content (maps, QGIS project files, gifs, shapefiles) can be downloaded here.

Click for full size - 1200ms per frame, 1 minute in total

I'm going to assume that if you are following this then you have at least some knowledge of QGIS, but if not then you can probably read my previous posts on the QGIS Atlas tool to help you. But this is definitely aimed at people with existing knowledge of QGIS.

I've packaged up the QGIS project (.qgs file) and the individual shapefile layers I used to create the gif above. You can download them here and then just open the project to see what's going on 'under the hood'. If you open this project (us_state_pop_2017_red_fixed.qgs), you'll see that I've used three layers, as follows:
  • One layer is just a light grey states background;
  • One layer is for the darker grey states that appear sequentially in the gif; and
  • One layer (a duplicate of the darker grey layer) is for the red highlighted states, to give it a more dynamic animated look.
For the dark grey layer I've used a rule to style it, which means that it will show the currently active QGIS atlas feature, plus all previous ones: $id  <= $atlasfeatureid. It was suggested to me on StackExchange by Ian Turton as a simple way of achieving this effect.

The red layer is similar, but the rule on this layer is $id  = $atlasfeatureid, which means that it basically highlights only the current atlas feature as the Atlas iterates through each feature (note that in the new, updated QGIS project files I have used the new Atlas syntax, which is explained below).

Where you set the layer style rules I describe above

When you go to the Print Composer in the project I've shared above, you will not actually see anything much until you activate the Atlas by clicking the Atlas button, as shown below.

Notice the Item Properties in the white box to the right

In the final version of the layout, once the Atlas has been activated, you can see from the Item Properties box on the right what I have used to add values from the layers the project is based on. Basically, I've added in fields and some text, as you can see from the images below.

This one adds a number and state name, plus a dot and a space

This takes the population, divides it by the 2017 US population

This gives the population in 'million' format, to 1 decimal

An important side note here is that I have found it necessary some times to re-order attribute tables in order to get the desired effect in terms of the Atlas running through things in the right order. If you need to do this, install the MMQGIS plugin and use Modify > Sort on a shapefile to achieve this. It works perfectly. Another note is that in later versions of QGIS the syntax for Atlas functions looks a bit different so make sure you are aware of it if you run into problems (e.g.  @atlas_featureid  instead of $atlasfeatureid, though both work for me in 2.18).

Once you have an Atlas project set up you are ready to export the frames. One thing here that is helpful is if you give the frames sensible, descriptive names. You can easily customise output file names in QGIS Atlas (another great feature of the tool). You can see how I've done this below, using the newer QGIS Atlas syntax. The "NAME" in the image below just takes the name of the current Atlas feature from the shapefile's attribute table and adds it to the file name. You can see the result in the folder where I put all the outputs (see screenshot below). Making sure the files have sequential names can save a lot of time later on because for an animation you need to be able to order the files in the order you want them to appear in the final gif.

This creates a file with a number, underscore and state name

These are the outputs from the QGIS Atlas project

Okay, so this is basically part one over with. You can create all kinds of things with the QGIS Atlas and of course you don't even need to use this but for a gif of the kind shown at the top of the page this is my typical workflow. See Lesson 21 and Lesson 22 of Steve Bernard's QGIS YouTube series for a different kind of animation approach.

The next step is to patch the images together into a gif. You can do this lots of different ways, but I tend to use GIMP, which is great - free and open source and despite the slightly worrying name you can normally find it online without running into too much trouble. In fact, you may be relieved to hear that it's the first result on Google. I currently have version 2.8 installed.

I'll assume you've installed GIMP and now want to create a gif like the one at the top of the page. Here are the steps.

File, then Open as Layers...

This first step just gets all the different layers loaded into GIMP. I've done this with a few hundred layers before and it worked fine but if you're getting into the high hundreds then another approach might be best as you could run out of memory.

If you've used QGIS to give the individual layers sequential names then the frames should all load into gimp in order and you'll see the last one first, and the individual layers listed on the right, as shown below.

This is what it will look like in GIMP

If you need to reverse the order of the layers in GIMP, you can do this easily by going to Layer > Stack > Reverse Layer Order. But I want my animation to start at number 1 so I'll keep it like it is.

You could just go ahead and export this as an animated gif, but if you don't optimise it then the file size is likely to be pretty big. Both versions I shared in the online folder are under 1MB (979KB and 530KB) so this step is really useful. Here's how you optimise it: Filters > Animation > Optimize (for GIF).

It won't reduce the quality but it will reduce the file size

Once you've done this, a new GIMP window will open up. The only difference here will be that you'll see the same thing as before but a dashed yellow and black box will appear on top of the images - nothing to worry about, that just tells you which part of the image actually changes from frame to frame. Now would be a good time to save your GIMP project via File > Save As... 

The next step is to Export the image to an animated gif. You do that via File > Export As... and then it's a case of selecting gif as the image format. If you're not used to this it can be a little mind-boggling but basically you just tick the As Animation box and choose your settings. Here you need to be aware of the delay settings in particular because that tells you how long each frame will appear for. As you can see below, I've chosen 1200 milliseconds for this gif because that makes it last one minute overall (50 frames for 1200ms). 

These are the settings used for the image at the top of the page.

In this gif, I used a delay of 1500 so each frame lasts 1.5 seconds

The other thing to think about is how big you actually want your gif to be. I did a version at 1500x1500 pixels but in general I'd recommend going much smaller than this. The two on this page are 1000x1000 and you can choose whatever dimensions you want when exporting from QGIS Atlas.

The tricky thing with this is trying not to cause some kind of cognitive melt-down on the part of the viewer, which is easily done and in a way is a bit of a risk with the US states population gif here. But of course it's not intended just to be watched once and this is just an example use-case to show you the workflow. But there is a serious point here in that a gif is definitely not the way to go a lot of the time, but I do think it's underused and definitely has a place.

Why would you want to do this in the first place?
That's a good question. Take a look at some of the examples I've linked to at the top of the page if you want to see a variety of different use cases. Some are better than others but all of them tell a story. Some stories are better told in other ways of course. Topi Tjukanov has loads of great examples - with more appropriate data - on this Medium post. If you ask me, a geogif can be useful in the following cases:

  • When you want to show movements of things, such as commuters, freight or marine traffic in the Baltic Sea.
  • When you want to show things in sequence, alongside some basic data.
  • When you want to highlight that there are 'a lot' of something, rather than focus on the individual data elements.
  • When you want to quickly compare shapes, as I did previously when looking at gerrymandering in the US. 
  • When you want to show the geographic spread of something over time, such as house price changes.
  • When you want to avoid the work you should be doing and instead feel the irresistible need to create something map-like that moves.
In many cases a geogif will only be an entry point to the data and will not allow users to delve more deeply but in many cases it can be a good way to highlight the big picture, which is still useful thing to do. You can judge for yourself what works, what doesn't and what is and isn't useful. Ultimately, geogiffery can be useful, but it can also just dazzle and confuse, so be careful.

Notes, etc.
Little-known Windows gif fact... If you left-click and hold the mouse button on the [X] at the top right of the Window when an animated gif is in progress, it will pause the gif. Move the pointer off the [X] and unclick to continue. I know of no other way to do this. Sometimes you can pause gifs on twitter, at least on mobile devices.

The shapefiles I use here are from the US Census Bureau Geography pages. The 2017 state population data used in the geogif here is also from the US Census Bureau.

Washington, D.C. is not a state so I didn't include it above but it had an estimated population of 693,972 on 1 July 2017.

I added the QGIS logo to the gif just to highlight the fact that this was done in QGIS.

Alaska and Hawaii are of course not to scale.

"Hey, I don't like red. Can you do a blue version?"

Yes, you can have a blue version

Update: this post was update on 29 December 2017 because the original gifs I posted here contained some errors (order of frames - e.g. Florida and New York were in the wrong order, though the data was correct). I have also updated the Google Drive files and they now contain five different versions of the blue and red gif each, at different speeds (from 1 second per frame up to 2 seconds per frame).

I have also uploaded two QGIS projects in zipped folders - one each for the red gif and blue gif. All the frames are there too, at 1000x1000 pixels.

In the text above I have explained how I re-sorted the state populations shapefile to make sure the Atlas iterates in the right order. You don't have to do this (e.g. I think @atlas_@atlas_featurenumber can be used) but I found it easier just to re-order the shapefile, particularly as I was working with copies of the same layer.