Global terrain maps

A short post today, with some visuals. I used some Blue Marble imagery from NASA - one layer was topography and the other was the colour image of the earth for August - and then I used the prelease v2 of Aerialod to visualise it. I tweaked the Blue Marble colours slightly and the elevation and bathymetry (in the final images) is greatly exaggerated, for effect.

I had a bit of fun with this. And this is the result.

NASA Blue Marble + topography

A few bumps in Europe and North Africa

Some nice colours and interesting bumps here

A view across most of North America

A slightly different angle on South America

I quite like this perspective, very interesting

Gosh, The Himalayas are quite big

So many mountains here!

Another pretty interesting view

Same as the first one, but with a few more light effects

That's all for now :)

With exaggerated bathymetry too

Classic mid-Atlantic wrinkles

Maps without New Zealand should not exist

How to make a 3D population density render for any country in the world

This blog post explains how to make a 3D population density render for any country in the world, using open data and free software. The main tool I'll use (Aerialod) is Windows-only at the moment, but it is amazing. I say 'render' rather than 'map' here because the images you'll generate are rendered by Aerialod, which is, technically, a 'path tracing renderer' rather than a piece of mapping software. But feel free to call them 3D maps - either way, this is an example of what they look like. 

Global population density print

But why do this? Well, partly because it's an interesting and often informative way at looking at population and settlement patterns, but also partly because this particular method produces nice-looking graphics. Perhaps the best answer I could give here is that's it's a combination of aesthetics, insight, and interest. In the image below, for example, we can see a lot of what we might already know, but it's often in the comparisons within and between places that we learn new things and that's also part of the point here - it can give us a new perspective on the world.

Singapore, KL, Medan, Ho Chi Minh City, etc

The ingredients

What do we need to make a population density map like this of a single country? Well, it can be done in different ways, and you may have your own method, but the basic ingredients are outlined below. I'm going to use the example of the United Kingdom as a single country, but the data source I use has data for all countries in the world, so you can pick whatever you like and replicate the method. So, here's what we need:

• Some population data, preferably in raster format
• A tool to render it in 3D

For the data, there are a few different sources, including the European Commission's GHSL data and NASA's GPWv4 but here we're going to use WorldPop population density data. These datasets are all free and open (CC BY 4.0), and all have 1km resolution versions, but WorldPop allows you to download individual countries, for each of the years from 2000 to 2020, so I'll use that here. Just a note of caution now to say that '1km' means '1km at the equator', or 30-arc-seconds - so if you're at 50 degrees north then the cell will still be 1km in height but it will only be about 80% of that width. Also, you should read the WorldPop - Mapping Populations page about the data, so that you understand what it is, and how it was created.

For rendering the world as a 3D population surface, there are again lots of options, the best of which is probably Blender. This is free, open source and extremely powerful but it's not what I'm using here. Instead, I'm using Aerialod by @ephtracy. I've previously written tutorials on how to use this for different kinds of data, but here' I'll focus on population density renders. 

About the vertical dimension of population density datasets

You can just download some data from WorldPop for a single country, in tif or asc format, and then just dump it in Aerialod, but when you do that with population data, something like this will probably be the result (below). This is the 2020 UK population density file from WorldPop, opened in Aerialod. I'm showing you this here as a way of explaining what Aerialod does and what you need to understand before you try to map population density in this way. 

What the tif?

The gbr_pd_2020_1km.tif file from WorldPop (2,372KB) is a simple black and white raster image, measuring 1,250 by 1,321 pixels (you can see this in the image properties when you download it). But the individual pixel values have population counts in them. So, when I open the file and inspect it in QGIS (below) I can see that the highest pixel value is 27,500 - and since Aerialod extrudes the vertical dimension according to individual pixel values, what we see in the image above when we open it in Aerialod is a kind of population density tower 27,500 high and only 1,250 wide by 1,321 high. Not exactly very useful. So, we have a 1km area of London with 27,000 people in it and this then gets extruded 27,500 units on the z dimension in Aerialod, while our x and y units max out at 1,250 and 1,321 respectively. Next step is to fix this.

We need to re-scale this file in order to use it

Scaling population density data in QGIS

This is probably the most important bit. I will say now that there are lots of different ways to do this next bit, but ultimately what we are trying to do is arrive at a situation where we can show population density patterns in 3D that provide a useful overview of the settlement patterns in different countries, or the world as a whole. I used a different approach in the image at the top of this blog post, plus different data (GHSL), but it is based on the same principle. The following is a simple workflow to take a tif file from WorldPop and then rescale it so you can use it in Aerialod. If you were mapping something with much lower actual values (e.g. elevation) then you don't need to do this re-scaling.

• Open the population density tif in QGIS. It should just display as a black-to-white image, with low population counts in darker colours and lighter shades for the highest values. You can see this in the screenshot from QGIS above.
• Right-click the layer > Export > Save As... and then you want to change the Output mode to Rendered image, select a folder to save it to, plus a file name, and then click OK. This will add another raster layer to QGIS. You can see a screenshot of this below.
• You'll see that the new raster layer in QGIS looks exactly the same, but it's not. Instead of the actual population values in each pixel, the values go from 0 to 198. This is what the Rendered image save option does - it scales the image using pixel values in the range from 0 to 255, where 0 is black and 255 is white. 

Saving as a rendered image

• When we add this new rendered image to Aerialod, and then change some of the settings (as in the screenshot below), we do get quite a good visual representation of population density.
• But, there's no land and the old 0 to 27,500 range has essentially been reclassified from 0 to 198, which means that the full range of classes (up to 250) hasn't been used. We'll fix that in a minute but first here's the result, below. As you can see, this does a pretty good job of illustrating the broad sweep of population density across the UK. If you're struggling with Aerialod here, you may need to go back to the links at the top of the page and see my tutorials on it, but all I did here was drag and drop the tif into the main Aerialod window and then change the settings, as per the screenshot.

This works pretty well - but there's no land

How do we get the land to appear? (don't use black for 0)

This next bit is about how to change the raster image before exporting it so that when we add it to Aerialod we can see the land as well. There are loads of ways to do this, but here's one simple way that works well. This step is only really necessary if you want land to appear. I prefer doing this because it then gives me the option of including the land, or not (i.e. you can make the land disappear by changing the Offset value in Aerialod to something like -2.0 in this case). 

• Go to the original raster layer (in our case the one that goes from 0 to 27,500) and then in the Layer Properties, Symbology section, we change the Render type to Singleband pseudocolor. 
• What you need to do here is edit the colours so that the ramp goes from almost black (but NOT black - so don't use RGB 0, 0, 0 or #000000) to white. Aerialod will interpret the darkest colours as low values and white as the highest values - see below for a screenshot on this.

This is one way to do it

• Once you've done this, you can then save the layer as a Rendered image from QGIS, just like we did above. It will then add a new dark to light raster layer to QGIS, just like before. Unlike before, however, when you click on a cell in QGIS with the Identify Features (i) tool, you'll get a value from just above 0 to 255, instead of the population count. 

• Check: are all your highest population spikes the same height, meaning they look truncated or flat-topped? If so, this means you probably need to adjust the light-dark colour gradient again in QGIS before exporting it and loading it in QGIS. This sometimes happens in the highest density areas so it's worth looking closely at it and then if it does happen, re-scale the data in QGIS using a different kind of classification (e.g. instead of Continuous you could try equal Interval or even Quantile). Also, just remember that this approach is quite a simplified visualisation approach to displaying population density data. The vertical extrusion is not super-precise, and of course depends upon the method you use to create the rendered tif. The idea with the approach outlined here is to give a good general overview of population distribution and density in a country.

• N.B. there are loads of different ways you could do this step, including the use of different colour ramp modes (e.g. continuous, equal interval, quantile) or you could re-classify the raster using a scaling factor (e.g. divide all values by 1,000), or do it in an image program, but here I am aiming at simplicity and a method that allows us to quickly arrive at a result that is both interesting and broadly representative of the spatial patterns of density on the ground. 

• So, what does it look like when we add it to Aerialod now? Well, see below for the new image.

Reduce the Offset (on the right) to remove the land

The point of this is to be able to create a 3D render of population density in any country, and this is one way of achieving that. For things like this, where I am striving for broad-brush patterns and workflow efficiency, rather than precise measurement from pixel values, I will often take shortcuts to achieve what I want - so for example, in the image below I have 'reprojected' the data by manually re-sizing it in IrfanView and then saving it as a png file, which ended up being 289KB in size. Try downloading it and dropping it into Aerialod and then replicate the settings I've used below - I think this one has the makings of something quite nice, and you can replicate it for any country. I have also added a similar file for France (as of 21 Nov 2020).

All this from a 289KB png file + Aerialod

I want to map the whole world this way!

Okay, no problem. You can get global population density layers in a few places, but probably the most convenient are WorldPop or GHSL. The WorldPop ones can be found at the bottom of this page - look for the 'Unconstrained global mosaics' link. For the Global Human Settlement Layer (GHSL) data, published by the European Commission, you'll want to head to their excellent Download page. Once again, I'd advise anyone using this data, or any kind of data, to read about what it is and in this instance the FAQ page is particularly useful. 

Most of my population density mapping in the past has used GHSL data. One great thing about the GHSL download options is the ability to pick different projections, so that you don't just have to use WGS84 - this is still good, but the further you go from the equator, the narrower the area represented by each cell. This is not the case with the Mollweide option, since it is an equal-area projection. But this is something you should look into further yourself if you're interested. 

If you do download a global file, you need to be aware that Aerialod can handle images at a maximum of 16,384 pixels in width or height, and the world is more than 16,384km wide. So, if you do download a global population raster, you will need to re-size it in whatever software you use (I always use IrfanView for this) so that it is less than the maximum pixel size allowed in Aerialod - I normally just re-size to 16,000 pixels. 

This means that the 1km cells are no longer 1km but closer to 2km, but at a global scale the pattern is very similar. I've created a small-file-size global raster (2.21MB), with the Pacific in the middle rather than the Greenwich Meridian, and shared it via Dropbox so that you can experiment with it as well. See below for what it looks like when you drop it in to Aerialod with the settings as per the screenshots.

This is the whole world, from a 2.2MB png

Zoomed in, and exaggerated heights

Recap

I could say a lot more about methods and how to tweak this and that, but the aim of this post was just to show you how to create a 3D population density render of any country in the world, and I've done that now, using the United Kingdom as the example country. All I would add is that if you're struggling with Aerialod, then you should go back to my original blog post on it, plus the earlier one linked to in it. It's definitely also worth spending time reading about the data sources I have mentioned.

I hope you have found this useful. If you get a lot of use out of Aerialod, and you are able to, perhaps consider making a donation to @ephtracy - the creator of this great piece of software. I have done so in the past and hope to do so again in the future.

Data sources

I've linked to the data sources throughout this post, but I'll do it again here so they're all in the same place. This is all open data, with each one licenced under the Creative Commons - Attribution 4.0 licence (CC BY 4.0). 

• Worldpop.org - WorldPop really is an amazing project and in fact one of the best data resources currently in existence, in my opinion. It has so much beyond population data, so even if you don't want to map population patterns it's 100% worth checking out. Note that in the data section of the website you'll see both 'Population Density' and 'Population Count' but here I've used the population density data. It's worth spending some time on the website to see what's available and also to understand what the data represent and how it is all collected. You can get this data either as a tif, like I've used above, or as a csv file.
• Also one of the best population data resources currently in existence, the Global Human Settlment Layer has the advantage of going further back in time, with a total of four time points - 1975, 1990, 2000 and 2014. The GHSL project is supported by the Joint Research Centre (JRC) and the DG for Regional and Urban Policy (DG REGIO) of the European Commission, together with the international partnership GEO Human Planet Initiative. If you download this data, there are LOTS of options, but what you need if you want to map population density is the GHS-POP option. I've mostly used the 2015 data, at 1km resolution, in Mollweide or WGS84 projection for the whole earth, but you can also just download single tiles. This data is available in tif format.
• NASA's Gridded Population of the World, version 4 - another excellent source of global population data. The GPW v4 home page explains everything about it, and all the different variants available, as well as previous versions. 

How to create an interactive 3D landscape model in QGIS 3

It's been a while since I've posted a QGIS tutorial so with the release of QGIS 3.4 I thought I should do another one. This post is about how to create an interactive 3D model in QGIS 3.4 which you can then export and view in a web browser, like the one you see below. You can see the interactive web version here - it's great fun to play around with, particularly when you press R and watch it rotate. This kind of thing was possible in previous versions of QGIS but I think there are some great improvements with the new version of the Qgis2threejs plugin that you create this with (thanks to Minoru Akagi).

This is Kauai, an island in Hawaii

Make sure you read the instructions - very useful

The basics

All you need to make this work in the way that I have is a) QGIS 3.4, b) a DEM layer (that is, a raster layer with elevation data in it), c) some kind of satellite or aerial imagery, d) the Qgis2threejs plugin which you install via the Plugins menu in QGIS and e) a bit of time.

There are many sources of digital elevation model data but for this I used NASA's 30-meter resolution elevation data from the Shuttle Radar Topography Mission, which is very easy to download thanks to Derek Watkins' tile grabber. You need to register (for free, takes only a few minutes) to get the 30 metre resolution data but if you can't be bothered doing that you can get the coarser 90m SRTM data without registration. As is normally the case with this kind of thing, the area I was interested in was split between two tiles so I patched them together in QGIS (Raster > Miscellaneous > Build Virtual Raster). I chose Kauai in Hawaii because of its size and varied terrain.

Derek Watkins, I salute you

For satellite or aerial imagery, the usual approach would be to add a layer in QGIS using the QuickMapServices plugin (e.g. Google Satellite layer) but in this instance I used a Landsat 15m image from the State of Hawaii's Office of Planning GIS pages. This is a bit better quality than those available via QuickMapServices so I used this.

When you have the DEM data and the Landsat imagery in QGIS 3.4 it will look something like this:

Make sure your image fills the whole map view in QGIS

The preview window is a great addition

The process

You then activate the plugin by clicking the icon (see above) and it's a case of playing around with the options until you are happy with what you see. Since qgis2threejs has excellent documentation, I won't go over all the steps and options here but I would just add the following points - and see point 5 in particular.

1. Set the level of vertical exaggeration via Scene > Scene Settings... In my case the vertical exaggeration is 3.0.
2. In Scene Settings under Material you may be baffled by the options - Lambert, Phong, and Toon. I've exported three versions of Kauai, one for each - you can see the difference by opening them in a web browser but basically the lighting will look different depending upon which one you choose.
3. Right click individual layers to set the Properties. On the DEM settings you'll get a much crisper visual with the resolution set to 400% but the file size will be bigger and it will most likely load slower in the browser.
4. If your preview looks wrong, with weird, impossibly high extrusion, change the projection in the bottom right of the QGIS window, as below.
5. In previous versions of qgis2threejs, when you exported the view from QGIS it would automatically open the visualisation in a web browser. When you try this from QGIS 3.4 it will attempt this but if your default browser is Chrome or Internet explorer then you'll get an error message. This is a known issue so you'll have to view in Firefox or Edge (I haven't tried other browsers) to overcome it. This is only a local problem - it's fine viewed over the web in any browser.
6. If you look at the style panel in the image below, you'll see that I've colorized the raster layer. I did this to give the image a kind of 'morning glow' look to it. See below for a comparison between a colorized and non-colorized image.
7. Via Scene > Decorations you can now add in a north arrow (very cool, since it is dynamic) and a footer label to add text (which can be html, like I used to style the font and add the QGIS image).
8. If you want to add labels to a point layer, you can still do this by right clicking the layer in the preview and setting the options. It's no longer Times New Roman font either, so that's an improvement too. You can see an example with labels in a previous map I did of Madeira. I've added another screenshot below to show you how to set this up.
9. You can also now set the view to be in Perspective (the default) or Orthographic, which is another nice advance.
10. There are many things you might want to tweak manually that you can't do from within the plugin options but they can be done by editing the html and the js and css files (see below for more).

Don't worry, just change the CRS

Subtle difference, but I like the effect (this is a looping gif)

Setting labels relative to the DEM surface will usually look good

When you export the scene, qgis2threejs will create a number of files and folders which you can then put on a web server so that the world can see how amazing you are. Or, you can just view them locally on your own computer (but not in Chrome or Internet Explorer, like I said above).

Making it look even nicer

You may now want to change the background colour, font size and other bits and bobs. To do this you need to edit different bits of the .html file, the .css file and the .js file created by qgis2threejs. These files will be named as below.

• index.html (this is the default name, you may have changed it to something else) but it is here you can change the page name and footer text, for example.
• Qgis2threejs.css - you can edit this to change the north arrow position (I put it in the top right, but it's not there by default) and the colour of the sky.
• Qgis2threejs.js - you can edit this to change the font size (edit the value of the MinFontSize under label, where you can also edit the leader line colour) and the colour of the base sides and the overall lighting colours. 

This is how you edit the north arrow size and position (css file)

This is how you edit the sky colour (css file)

This is how you edit the footer text (html file)

This is how you edit the side colour and lighting (js file)

This is how you edit the font size for labels (js file)

If you want to edit the font, search for "font-family: arial, sans-serif;" in the css file and change it to something else. You can see some different options for that on this page. In the example below I just changed the font size to 24 and added font-family: "Comic Sans MS", sans-serif; to the css, as below.

You can use lots of different fonts here

This is the bit you want to edit

Last words

The last thing to say is that you should take some time to figure out how to control the visualisation. On mobile, it works really nicely too and upon export there is now a mobile option. It's worth playing around with the different export options to get a sense of what can be done. My favourite thing is to click on a point and you'll see the lat and long plus elevation and then you have two options - Zoom in here or Orbit around here - self explanatory but the orbit option is really cool. You can of course make it rotate just by pressing R.

Comic Sans has its uses, but this isn't one of them

Okay, that's it. 

US 2017 Total Solar Eclipse Animation

I've only just started paying attention to the fact that there's a total solar eclipse in the United States on 21 August this year. What made me really take notice were the imaginative, amusing and bizarre maps people have been posting online - all good fun and often pretty interesting too. What seems to me to be missing - and absolutely essential, of course - is an animated eclipse gif showing the path of totality, towns and cities, a bit of terrain and some roads. So, I grabbed the geodata from NASA and spent a couple of hours playing around with it to make this thing you see below. This is an extract of the full file, in gif format. The full gif is over 100MB, so I've created a video file instead for the final file at the end of this post, below - and also posted on twitter. Click an image to see it in full size.

Note the small part of the path of totality in Montana

Nashville looks like the place to be

I also extracted a series of animations covering the whole US, in separate parts, so that the whole of the path of totality has been giffed, as it were. These are pretty big files and if I add them all here not only will it may crash your browser but it will also probably be a bit too bamboozling with all the animations going on at one time. Here's part 1 and part 2 below. The rest I'll keep to myself for now.

Part 1 - at 50 frames per second

Part 2 - also at 50fps

I just did this out of curiosity and to learn a bit more about the event. I thought it would be interesting to see for myself where the eclipse will go. I remember here in the UK back in 1999 when we had a total eclipse, but I didn't manage to get down to Cornwall to see it properly. Just take a look at this video to see how long ago this now seems. I was in Glasgow at the time and although it didn't get fully dark it was pretty interesting and the birds were very confused. Also, if I didn't do an eclipse map now I might never remember again. The next total eclipse in the UK isn't until 2090 and I just get the feeling I won't be on here making animated gifs of it then.

Notes: mapping software was QGIS 2.14, using the Atlas function to extract the frames for the gif. I used the NASA shapefiles to show where the path of totality and centre line are. I added some place names, roads and US boundary and state lines from Natural Earth and I also added an ESRI shaded relief base layer to give more of sense of the underlying physical geography. I patched the gif together in GIMP and animated the movement of the sun across the land using the vertices of the eclipse centre line. I've added a little glowing corona round the black dot, which is of course supposed to represent the sun.

There were about 1800 points in the original NASA eclipse path file so I used every second one to generate the animation because it still creates a reasonably smooth image. For a proper set of scientific maps, I suggest you check out the NASA Total Eclipse pages, as they are full of great information and maps. They also have an animated gif.

Spinning globes with NASA's Visible Earth

Time for a little fun today, seeing as how it is the 29th of February and therefore a bonus day. I've recently been experimenting with more NASA data and have become somewhat addicted to looking at Visible Earth, an absolutely stunning collection of images and animations of our world. So, naturally, I decided to make a few animated gifs. I did this with NASA's 2012 night globe, the next generation Blue Marble image and the Bathymetry image. You'll see the results below, followed by a little bit of information on how I made the gifs, should you have a burning desire to do so.

Beautiful images of our planet

First of all, here's an animated globe using the classic 'Blue Marble' image of earth. You can find this in lots of different varieties on the Blue Marble pages of Visible Earth. 

This is 75 frames, at 125ms per frame (Credit: NASA)

I also did a version of the spinning globe using NASA's Night Lights 2012 image. This is captured by a satellite 512 miles above the surface in a polar orbit, circling the planet about 14 times a day. And here I am turning it into another animated gif.

This one is 50 frames, at 100ms per frame (Credit: NASA)

Finally, I decided to take a different view and use the Bathymetry image. Bathymetry is the underwater equivalent of land topography. This shifts the focus on to the sea rather than the land, and provides a different take on things.

This is also 50 frames, but this time 150ms per frame (Credit: NASA)

Okay, so that's already three too many gifs for one blog post. But I had fun so it's all good. Read on below if you want more info on how to make these - or similar images. It's very simple.

How to make your own spinning globe (should you have nothing better to do)

1. Find an image from Virtual Earth that you like.

2. Turn it into a square by re-sizing in your chosen image programme. I used IrfanView and/or GIMP for this. I'd aim for no more than 800 x 800 in size. If you don't turn it into a square you'll end up with a spinning rugby ball.

3. Use GIMP (or PhotoShop or similar) and open the image. In order to avoid blocky, pixely edges you need to add an Alpha Channel and then Semi-Flatten it. See below for how to do this in GIMP - the image shows my selecting 'Semi-Flatten' after already adding the Alpha Channel, which is in the same menu. It will still work if you don't do this but instead of a smooth circle you'll end up with a slightly pixellated circle, and it won't look as nice.

Always do this for a smooth globe outline

4. The next step is just to use the Spinning Globe tool in GIMP (as below) and then select some options (also below). As you can see, it's here where you can choose clockwise or anti-clockwise, specify the colours and say how many frames you want - more than 100 will take a little while, but gives a much smoother globe (but bigger file size).

This is a nifty little tool in GIMP

I ticked the first box to make the earth go the right way

5. Once you've done step 4, GIMP will extract individual frames in the layers list on the right and then you're ready to export to gif. This is just done via the File > Export As... menu option. Once there, you just change the file type to gif and select the options you want - e.g. the duration of each frame - I've used 100 to 150 milliseconds above. Then you just click Export. The file will save quickly and then you'll have your spinning globe. 

This is how you get to the gif options

Note that you need to tick the 'As Animation' box

There are other options you can play with and other ways of doing it - including not using GIMP - but I've shown you this here because it's free and easy. You just need to know how.

As for the underlying data, that's another great thing. Nearly all of Virtual Earth is available for re-use, commercially or otherwise. Here's what NASA have to say about it:

"Most images published in Visible Earth are freely available for re-publication or re-use, including commercial purposes, except for where copyright is indicated. In those cases you must obtain the copyright holder’s permission; we usually provide links to the organization that holds the copyright.

We ask that you use the credit statement attached with each image or else credit Visible Earth; the only mandatory credit is NASA."

I'll be back with another frivolous 'how to make a gif' post in four year time.