Saturday, 16 March 2019

Alternative Maps

This is a blog post about some maps of places that we don't often see on maps in Britain. I've had these stashed away for a while, since late 2017 when I wrote a piece called 'Alternative Maps and the Future of Brexit Britain', which you can see below. I've been thinking a lot more about this recently as part of my role in the UK2070 Commission, and also because I'm going to Redcar this week. But really this is just some personal musing about maps and in particular maps of the kinds of places that often seem to be left off the map, particularly in conversations at a national level. Scroll down for the maps. There's not too much more to it than that, but I thought looking at some of the less mapped places, with a bit of label re-sizing, was a good way for me to re-think my mental maps of the country, and that's really why I did them. Nothing profound.

You can probably read the text if you click to enlarge

I'm not asking for too much, am I?

I deliberately didn't add inset/locator maps to these

Too often we're blinded by the sun

Only peripheral if you're not paying attention

It's not that far away

A long way from London? 

Part of London?

Limited labels

I've gone crazy with the labels here

The Kingdom of Fife

I always think this part of the world is overlooked

Quite a big garden

So often overlooked

Definitely pretty central

Quite a bumpy landscape

A bit more label inversion here

Technically, this is not Shetland in a box

Greater Trowbridge?

Stafford and Cannock megaregion

There aren't a lot of maps of this area

There are a lot of people here

Like I said, this is just a few maps of places that we don't often see on maps. Or at least we maybe don't think of them as being nationally 'important' in the same we we do about major cities. In some ways that is understandable but I think it can't to any harm to change our mental maps and re-think how we think about the UK, where stuff is and where stuff should be.

Saturday, 9 March 2019

A foto da favela de Paraisópolis

This is a blog post about a famous photograph by Brazilian photographer Tuca Vieira, but also about how emotion and imagery can often be much more powerful than 'data'. I'm just posting it here as a round up of various tweets on the topic I have posted previously so that they have a more permanent place on the web. But first, here's the photo. It was taken from a helicopter above São Paulo in 2004 as part of a newspaper piece on the 450th anniversary of the city. The favela of Paraisópolis is on the left, with the much more affluent area of Morumbi to the right.

Tuca Vieira's famous image

The photo has been used to illustrate many different things, but usually it serves as an exemplar for urban inequality. I have used it to highlight how in spatial analysis near things are not always necessarily more alike (i.e. Tobler's First Law of Geography doesn't always hold) as well as to talk about inequalities. I wanted to use it in a new GIS book so I got in touch with Tuca and he agreed that we could use it (for a very reasonable fee). He also sent some of the other images he took from the helicopter that day, from slightly different angles. Very powerful stuff.

There is a separate story here about how the image took on a life of its own, detached from the photographer, and how hardly anyone credited Vieira or even acknowledged how much effort taking an image like this is. There isn't too much about the image or Vieira's thoughts on it online but see this short interview for more. By the way, Paraisópolis means 'Paradise City'.

Anyway, once I discovered that the city was on street view, I spent quite a bit of time trying to find the exact spot, and I eventually found it. It's taken from a spot roughly above Avenida Giovanni Gronchi, which you can see on the ground in this Google street view image.

My original tweet on this from 2016

The little street to the left separates the areas

You can also read a bit more about it on Tuca Vieira's website, though more recently I have only been able to find this via the wayback machine. The page tells a story about an exhibition in London in 2007 where he was invited, but apparently not so much included. This is how Google translates what for me is the key statement in his piece:

"this photo may make me achieve what should be the great goal of an artist: to provoke a reflection on the world and not on the work and its author".

You can of course now see the scene in 3D in Google Earth, as shown, below. Nowhere near as interesting or as powerful as the original picture but still pretty useful.

Direct link to this 3D view on Google 

Anyway, what really prompted me to look again at this recently was the arrival of Google Earth Studio, a fantastic new tool for creating pretty realistic, smooth animations of 3D scenes around the world. I decided to make a fly-to and orbit type animation of this in Google Earth Studio. The full resolution version is on my web server but I've also embedded a version below (which may not look so crisp).

Notes: as I said at the start, I'm posting this here so that all the information is in one place and not spread between various tweets. This also makes it easier for me to find the information as I'm always forgetting where I put stuff. The image itself also prompts wider questions - e.g. are we only outraged/impacted by this kind of image because the contrast is so stark and so geographically close together? Is it the proximity of wealth and inequality that is so shocking, and if so, would more distance make it more 'acceptable'? Is it only so alarming because we can see it? People will have different answers to these questions, and many more, but it is clear that the image continues to have power and relevance. Type in terms like 'urban inequality' and look at the images and this will probably still be at the top, or very near. Finally, it looks like there is another, newer version of this image on a different Brazilian website. The original seems to be this iStock one by C_Fernandes from 2016.

Sunday, 20 January 2019

A little post about a geogif

This is a short post about map-based animated gifs (i.e. the humble geogif). It's not about general elections in the UK because that would not be a sensible topic. It's a bit about design choices, just in case anyone is interested, and it's also a bit about the method and data. Here's the final gif, below, which I posted on twitter recently. I wanted to make something that told the story of the 2017 General Election in a short time frame - in this case 30 seconds.

This is about 4.7MB

In the original version of this, the large font was too thin so in this version it's bold and easier to read. You could probably tell what it shows and how long it takes to show it, but with this text there is no doubt. This stems from the redundancy principle that you'll hear John Burn-Murdoch talk about in relation to his work with the FT. I've used Raleway as the font, though I think perhaps I should have chosen something slightly cleaner. I'm not that keen on the numbers with Raleway, though it is a lovely font. By the way, this was all done with QGIS Atlas and only one map layer. See bottom of page for the sources.

The constituency names are displayed incredibly quickly below the main text (for 46 milliseconds each). Not because I want them to be read; more for effect so that people can pick out names they know as it runs through. Of course, it's also so that when you pause it you can see which constituency we're up to (more relevant in other formats). Then when the name of a constituency appears, the corresponding hexagon to the right has a black border to it. Not thick enough here probably, but you should be able to see it.

Since the results file from the House of Commons Library is so detailed, I was able to add the time each result was declared, which I did as a digital clock. Again, adding a bit of redundancy in making the times work like a clock, just in an attempt to speed up cognition. Below the clock the number shows the order of the result (from 1 to 650) and whether the result was a hold or a gain from another party. I added a green banner to the top left to say when the General Election was. I did use the numeric format for this in an earlier version, but there are two problems with this: i) I think it's more difficult to process compared to seeing the word 'June' in there; and ii) we can't have our US friends thinking the election was on August 6th. I like to avoid doubt in such situations. That's why I moved house on the 8th of August 😉.

I also wanted to add a chart because you can't tell just by looking who actually wins the most seats. That's why in the final version I added the animated bar chart in the top right (previously it was a boring, non-animated one). This was also done using QGIS Atlas. I think this helps tell the story of the night as it's often the case that in more urban, tightly-packed constituencies (where Labour do well) the counts are quicker and they race off to an early lead. It also gives it a bit of jostling for position/race feel to it, which I like.

At 03:12 Labour are in the lead

According to the data, it wasn't until Daventry, the 498th declared result, that the Conservatives gained the lead from Labour. This happened at 04:09 in the morning.

Celebration or desolation, depending upon your allegiance (possibly)

"Hey, why did you lump my favourite political party into the 'other' category"?

Fair comment. You're right, it's not very nice, but I didn't want to add ten bars so this is what we have. Plus, I thought it wouldn't be too much of a stretch since the lower numbers of the individual groups within the 'Other' category mean they can easily be counted.

I added the numbers to the bars for completeness, and because it wasn't that difficult, and of course the party labels below the bars.

All I wanted to do here was tell the story of the 2017 General Election in a 30 second gif, deliver a lot of information in doing it and have a bit of fun. The idea is that you watch it multiple times to get the story of the evening. Each frame is on for about 0.046 seconds, so the whole thing lasts about 30 seconds. Well, it would have lasted 29.9 seconds but I kept the last frame on for 5 seconds to give viewers a breather and some time to process things before it re-loops.

Now for a few words on the method.

The basic method is to create a set of individual images which are then used as single frames displayed for a short duration in a gif. I do this using QGIS Atlas. You can create gifs from a set of images in a number of different ways, but I use GIMP because it works really well, is open source, not that difficult and you can optimise the animation so the file size is low. The method is described in an earlier blog post I did, from about half way down the page.

When exporting the gif you can choose the duration of each frame, but I like to have the final frame on for longer, at least a few seconds, so that people have time to take in what they are seeing after the rapid fire sequence of frames before it. There's a nice script you can use to set the frame duration of all frames automatically and then you can change the final one manually.

Export options for gifs in GIMP

What I normally do is export individual images from QGIS Atlas at 300dpi and then I batch re-size them in IrfanView, normally to max width/height of 1000 pixels. IrfanView is simple but surprisingly powerful and I've used it for about 15 years now. It can do just about anything, apart from solve Brexit (though I suspect it could do that too). I have found that I get a higher image quality doing it this way, rather than exporting images at 1000 pixels from QGIS.

If I need to convert images to mp4 instead of gif, then I'll use ffmpeg, which is a command line tool. If that scares you, then you'll probably find ezgif very useful - and I believe it uses FFmpeg behind the scenes anyway. The advantage of mp4 is of course that you can more easily pause/rewind and add music and suchlike. But if you upload a gif to twitter it will actually convert it to a slightly different gif format that can be paused.

When I follow this method, gifs on twitter always look nice and crisp.

In a QGIS Atlas of this kind, it's normally the case that only one feature would be showing at a time, but thanks to a tip from GIS guru Ian Turton I've done it cumulatively so that previous features remain on the map and it builds up until the map is fully coloured in.

As above, it's a mix of QGIS, GIMP, IrfanView, FFmpeg, and hardly ever anything else for geogifs. If you've not used FFmpeg for creating/editing/converting video files then you might want to check it out. It can be confusing but it's really, really powerful. I also used it recently to create an mp4 file from a series of Google Earth Studio images, as below (though I cheated and added in the text and audio using Filmora).

Check this out for the method

Data sources
For the hex grid of the 650 UK Parliamentary Constituencies, I used Ben Flanagan's file, which you can download from ArcGIS online. This file is licensed under a Creative Commons Attribution 4.0 International License, which in simple terms means you can use it, adapt it, share it and so on. For the colours, I used those from the BBC website because a) I like them, b) people are familiar with them and c) they know what works. I also used the 'detailed results by constituency' file (see bottom of this page) from the brilliant team at the House of Commons Library.

Wednesday, 2 January 2019

QGIS Atlas by Field

This post explains how to use the Atlas function in QGIS to automate the production of maps from a single layer in order to produce a map output for each column in an attribute table. The Atlas tool is designed to allow users to iterate through the rows in an attribute table, but not through columns so if you want to produce a map for each variable in a single dataset (e.g. annual house prices for small areas) then you may find yourself at a dead end, as I did. Then I saw how Ed Hampson at Savills very elegantly solved the problem so I'm sharing it here. I know of some other workarounds but this solution is my clear favourite as it allows for the production of as many map outputs as you need in a very efficient manner. I've shared the project file in this online folder so you can download it and play around with it yourself. This was done in QGIS 3.4. This is written with the assumption that you already know QGIS and how to use the Atlas tool, so I won't cover that here.

This is the final project, which you can download

The basic principle of this approach is that for the Atlas coverage layer you use a lookup file, which in my case was a table with one row for each of the column headers I wanted to map in my dataset (as shown below). As you can see, the individual rows follow the pattern md_1995, md_1996 and so on. This is because the layer I want to create maps from has one column for each of these fields, from md_1995 to md_2016 - this is the median house price for each area from 1995 to 2016. If you're at all confused and want to see how it works, just download the project folder and take a look yourself by opening the QGIS project I have shared - all the screenshots here come from that.

This is the table I used

Now here's the bit I really like - when setting up the Atlas in Print Layout, the coverage layer here is the table shown above and the Atlas 'Page name' is the 'Field' column in the table above. The normal way would be to set the coverage layer to be some kind of geographic layer and then QGIS would iterate through it spatially, but this is a kind of non-spatial solution that allows us to map by column instead.

The 'Field' here is the Field column in the table shown above

The really important thing you need to do to make this all work is to map the data using eval (  @atlas_pagename )  as the Column to be mapped. Since we have set our Page name in the Atlas as 'Field', and since the data in the Field column in our Atlas coverage layer has the same names as the attributes we want to map (i.e. md_2004, md_2005, md_2006, etc.) when we iterate through the Atlas it will map a different variable each time. Just remember to use a data range in the symbology that allows you to show the highest and lowest value across your entire dataset, otherwise you might find you have some missing values on some maps. For example, if I had set the symbology data range here using the 1995 house price, then by 2016 loads of areas would be above the max value from 1995 and they wouldn't show up on the map.

This is the clever bit

What does 'eval' do, I hear you cry... Well, it's basically this, as described in the QGIS documentation:

  • 'eval' - Evaluates an expression which is passed in a string. Useful to expand dynamic parameters passed as context variables or fields.

Once you've got everything looking the way you want, you can then export the Atlas in the usual way and do all sorts of interesting things with the outputs, like creating a gif for example as I have shown below. I did a tutorial for creating these in an earlier blog post.

How much!!!! Not inflation-adjusted, obviously.

I've added some comments to the expressions in the QGIS project I've shared and these are also shown below to give you a better idea of what's going on here. I know it's a bit tricky to understand without actually seeing it.

This is just how I set the output file names 

This is just what I used to automate the year label

This is what I used to set the highest value each year

This is how I dynamically sized the bars in the chart

This just shows you all the different layers - only 4

You'll see above that this is a normal Atlas, set up in the normal way. The only difference is the way I've set up the coverage layer and how I have set the column to be mapped using the eval expression. Everything else works in the same way. I have also used the aggregate function to pull out the highest value for each year (in the top right of the layout) and I used it to set the height of the individual bars in the chart. I have shared all the individual map outputs from this in the folder, and they look like this.

Final map output for 2016

Okay, I think we're done here but if you get stuck feel free to get in touch. I'm easy to find online or on twitter.

Notes: if you try to load the zipped project folder QGIS may throw up a warning saying it can't find the 'field_transposed_for_atlas' layer but don't worry, it's there. You just need to Browse to the folder where it is located (i.e. the layers folder in the download). I should perhaps really thank Jochen Schwarze here because of his StackExchange tip to Ed on this. The map colours are ones I borrowed from a very nice looking FT map by Steve Bernard. Note that you can add comments inside QGIS expressions using /*comment goes here*/ and after an expression using --comment goes here. In the file names I think it says 'MSOA' but the areas used are actually LSOAs. The house price data is from HM Land Registry but very helpfully compiled as median values by LSOA by the excellent CDRC team. I added a few place names using OS open data. Easy to remember url for the folder is

Wednesday, 26 December 2018

One degree of population

Here I am at the end of the year with another bit of map fun, this time inspired by Bill Rankin's world population histograms from 2008 (check out his website if you haven't already - it's amazing). For no good reason, other than curiosity, I decided to do some mapping of the world's population by single degrees of latitude and longitude. Actually, curiosity is I think quite a good reason, so see the first graphic below, making sure to read the caution on the map (tl:dr not all cells are the same size). Update 28 December 2018 - I added a version doing it by longitude as well. There's a gif at the bottom of the page and the mp4 files are in the repo I link to below. I also did a backwards-forwards looping gif - see below.

I think this backwards and forwards version works well

It makes more sense to compare areas at the same latitude

In the image above, I've picked out the ten most highly populated 1 degree cells but of course they are not all the same size - they get smaller the further away from the equator you are, because the lines of longitude narrow towards the poles. But even so, I think it's probably safe to say the cell where New Delhi is has a lot of people in it, as do the other nine featured above.

Now, we all know that most of the world's population live in the northern hemisphere (I make it about 87%), which is hardly a surprise because that's where most of the land is (68%) but I thought it would be interesting to do an animation of the above map, and that's exactly what I did. Here are some stills from the animation (below) and here is a direct link to the repository where I've put a 15, 30 and a 60 second animation. You'll probably have to download the files before viewing them.

Before that, I've embedded a version from YouTube, with an audio track.

If you live below 30 degrees south you are quite special

People seem to like living where the land is

Above 30 degrees north is where it's at 
I'm one of the 5% - are you?

And by longitude? I did that too and those files are at the GitHub repo link above but here's a fast gif of it for now.

How I did this
The data I used for this is the GHSL world population dataset that I've written about before. I've been using it recently in a paper so thought I'd do something a little different with it. This is the latest data, for 2014. I used the 1km gridded population dataset and re-aggregated it by lat/long so it's not a precise match with lines of latitude and longitude but it's close enough to be a reasonable estimate. I mapped it in QGIS and exported a few thousand frames in QGIS Atlas then patched the video together using FFmpeg.

Saturday, 17 November 2018

How to create an interactive 3D landscape model in QGIS 3

It's been a while since I've posted a QGIS tutorial so with the release of QGIS 3.4 I thought I should do another one. This post is about how to create an interactive 3D model in QGIS 3.4 which you can then export and view in a web browser, like the one you see below. You can see the interactive web version here - it's great fun to play around with, particularly when you press R and watch it rotate. This kind of thing was possible in previous versions of QGIS but I think there are some great improvements with the new version of the Qgis2threejs plugin that you create this with (thanks to Minoru Akagi).

This is Kauai, an island in Hawaii
Make sure you read the instructions - very useful

The basics
All you need to make this work in the way that I have is a) QGIS 3.4, b) a DEM layer (that is, a raster layer with elevation data in it), c) some kind of satellite or aerial imagery, d) the Qgis2threejs plugin which you install via the Plugins menu in QGIS and e) a bit of time.

There are many sources of digital elevation model data but for this I used NASA's 30-meter resolution elevation data from the Shuttle Radar Topography Mission, which is very easy to download thanks to Derek Watkins' tile grabber. You need to register (for free, takes only a few minutes) to get the 30 metre resolution data but if you can't be bothered doing that you can get the coarser 90m SRTM data without registration. As is normally the case with this kind of thing, the area I was interested in was split between two tiles so I patched them together in QGIS (Raster > Miscellaneous > Build Virtual Raster). I chose Kauai in Hawaii because of its size and varied terrain.

Derek Watkins, I salute you

For satellite or aerial imagery, the usual approach would be to add a layer in QGIS using the QuickMapServices plugin (e.g. Google Satellite layer) but in this instance I used a Landsat 15m image from the State of Hawaii's Office of Planning GIS pages. This is a bit better quality than those available via QuickMapServices so I used this.

When you have the DEM data and the Landsat imagery in QGIS 3.4 it will look something like this:

Make sure your image fills the whole map view in QGIS

The preview window is a great addition

The process
You then activate the plugin by clicking the icon (see above) and it's a case of playing around with the options until you are happy with what you see. Since qgis2threejs has excellent documentation, I won't go over all the steps and options here but I would just add the following points - and see point 5 in particular.

  1. Set the level of vertical exaggeration via Scene > Scene Settings... In my case the vertical exaggeration is 3.0.
  2. In Scene Settings under Material you may be baffled by the options - Lambert, Phong, and Toon. I've exported three versions of Kauai, one for each - you can see the difference by opening them in a web browser but basically the lighting will look different depending upon which one you choose.
  3. Right click individual layers to set the Properties. On the DEM settings you'll get a much crisper visual with the resolution set to 400% but the file size will be bigger and it will most likely load slower in the browser.
  4. If your preview looks wrong, with weird, impossibly high extrusion, change the projection in the bottom right of the QGIS window, as below.
  5. In previous versions of qgis2threejs, when you exported the view from QGIS it would automatically open the visualisation in a web browser. When you try this from QGIS 3.4 it will attempt this but if your default browser is Chrome or Internet explorer then you'll get an error message. This is a known issue so you'll have to view in Firefox or Edge (I haven't tried other browsers) to overcome it. This is only a local problem - it's fine viewed over the web in any browser.
  6. If you look at the style panel in the image below, you'll see that I've colorized the raster layer. I did this to give the image a kind of 'morning glow' look to it. See below for a comparison between a colorized and non-colorized image.
  7. Via Scene > Decorations you can now add in a north arrow (very cool, since it is dynamic) and a footer label to add text (which can be html, like I used to style the font and add the QGIS image).
  8. If you want to add labels to a point layer, you can still do this by right clicking the layer in the preview and setting the options. It's no longer Times New Roman font either, so that's an improvement too. You can see an example with labels in a previous map I did of Madeira. I've added another screenshot below to show you how to set this up.
  9. You can also now set the view to be in Perspective (the default) or Orthographic, which is another nice advance.
  10. There are many things you might want to tweak manually that you can't do from within the plugin options but they can be done by editing the html and the js and css files (see below for more).

Don't worry, just change the CRS

Subtle difference, but I like the effect (this is a looping gif)

Setting labels relative to the DEM surface will usually look good

When you export the scene, qgis2threejs will create a number of files and folders which you can then put on a web server so that the world can see how amazing you are. Or, you can just view them locally on your own computer (but not in Chrome or Internet Explorer, like I said above).

Making it look even nicer
You may now want to change the background colour, font size and other bits and bobs. To do this you need to edit different bits of the .html file, the .css file and the .js file created by qgis2threejs. These files will be named as below.

  • index.html (this is the default name, you may have changed it to something else) but it is here you can change the page name and footer text, for example.
  • Qgis2threejs.css - you can edit this to change the north arrow position (I put it in the top right, but it's not there by default) and the colour of the sky.
  • Qgis2threejs.js - you can edit this to change the font size (edit the value of the MinFontSize under label, where you can also edit the leader line colour) and the colour of the base sides and the overall lighting colours. 
This is how you edit the north arrow size and position (css file)

This is how you edit the sky colour (css file)

This is how you edit the footer text (html file)

This is how you edit the side colour and lighting (js file)

This is how you edit the font size for labels (js file)

If you want to edit the font, search for "font-family: arial, sans-serif;" in the css file and change it to something else. You can see some different options for that on this page. In the example below I just changed the font size to 24 and added font-family: "Comic Sans MS", sans-serif; to the css, as below.

You can use lots of different fonts here

This is the bit you want to edit

Last words
The last thing to say is that you should take some time to figure out how to control the visualisation. On mobile, it works really nicely too and upon export there is now a mobile option. It's worth playing around with the different export options to get a sense of what can be done. My favourite thing is to click on a point and you'll see the lat and long plus elevation and then you have two options - Zoom in here or Orbit around here - self explanatory but the orbit option is really cool. You can of course make it rotate just by pressing R.

Comic Sans has its uses, but this isn't one of them

Okay, that's it.