Saturday, 21 July 2018

A Review of OS Open Zoomstack

This is a long blog post all about something called OS Open Zoomstack. What? Yes, Ordnance Survey have released a product called OS Open Zoomstack, and it is very very good and you should know more about it. So here I am with a review. This is mostly aimed at people who might actually use the raw data to make maps, but also anyone who has an interest in mapping more generally. Put very simply, it's a giant, beautifully styled vector dataset for the whole of Great Britain. It can be used in a webmap (e.g. Mapbox) in a desktop GIS (e.g. QGIS) and other ways (e.g. PostGIS). This is what Ordnance Survey say it is:

"OS Open Zoomstack is a comprehensive vector basemap showing coverage of Great Britain at a national level, right down to street level detail."

But the best way to explain what OS Open Zoomstack is, is to just show you a couple of examples so I've embedded two maps below. The first is a kind of 'night time' view of Glasgow, and the second is a lighter overview map of London. Exactly the same data, just styled a different way and in both I have extruded the buildings by a few metres to give a little 3D effect. But actually both of these cover the whole of Great Britain which you can see if you zoom and pan each of them. Have a play around and then read on below for more information. There are links to full screen versions below.



Full screen - Glasgow


Full screen - London


Besides these examples, why not try a full screen version: as well as London and Glasgow, I've done Manchester, Edinburgh, and Portree (in Skye) but as I said, you can zoom anywhere you like in Great Britain. These places are just where I've set the map to start when you click the url.

If you want, you can edit the url yourself and share a map of any location in Great Britain. You can tilt and pan as well as zoom. To tilt for a different 3D view you may also need to hold down the CTRL/Cmd key or maybe use two fingers on a mobile device or tablet. It's worth trying this on a mobile device just to see how slick it is.

If you're still a little confused at this stage about what exactly OS Open Zoomstack is, take at look at the Ordnance Survey blog and then listen to what Charley Glynn has to say about it in the video. It's worth your time. The documentation is really helpful and even if you're not very experienced you can be up and running with it in no time, particularly if you're using QGIS. Talking of which, the series of maps below were all exported from OS Open Zoomstack in QGIS.

Whereas the examples above involved me uploading the vector tile set to Mapbox and doing a little bit of tweaking, the maps below simply involved downloading the data as a single file GeoPackage, loading it up in QGIS, re-ordering the layers, and then applying style files. You can find the instructions for this in the documentation and if you're an experienced QGIS user it'll be really easy. Even though the whole lot is about 10.6GB, it works very quickly for me on the machines I've tested it on.

This new release from Ordnance Survey is a also a nice demonstration of why the GeoPackage is - shudder - the future for geospatial data. No offence to Professor Shapefile, it's just the way things seem to be moving and this use case, for me, provides a good example that helps demonstrate where the GeoPackage has real advantages over the Shapefile. But I'm not here to talk about that right now.

Take a look at the different images below, and make sure you click to see them in full size. I've taken a range of snapshots from across Great Britain, with a few different zoom levels shown for London.  You'll see on the Knoydart one the impact that the contours have in giving the terrain a nice sense of relief. In my opinion this new product is going to be something of a game changer and so far I can't think of anything I don't like about it. 


Anglesey in North Wales - zoomed out

Central Bath 

Fort William and Ben Nevis

Birmingham city centre

Inverness (although OS use the Gaelic, 'Inbhir Nis')

The terrain of the Knoydart peninsula in Scotland

Leeds

Liverpool

London

Even more London

Zooming closer to the Isle of Dogs

Even closer

About as much detail as you can get

Middlesbrough - lovely detail

Norwich

Isles of Scilly - lovely styling

The 'west side', as they say (Lewis, in Scotland)

Southend

Swansea

This is Unst, Shetland

Wrexham, or Wrecsam

Under the hood
Let's take a closer look at what this looks like behind the scenes. The OS Open Zoomstack trial was announced on 17 July 2018 and runs for three months. You can get the data by following the links on the OS blog page about it. I've played around with two of the download options: the vector map tiles I used in the interactive versions above, plus the GeoPackage. Here's what they look like when you download them (below).


The documentation is also very impressive

If you're not familiar with what a GeoPackage is, and you are a GIS user, I suggest this is the time to get into it. Whereas a shapefile is a set of files that contain attribute and spatial data for a single type of geometry (points, lines, polygons), the GeoPackage is a single file (yes, just one file) which can contain all sorts of different geometries and also raster and vector data too. It's kind of a big container for spatial data. In practice, it will look the same as a shapefile in your GIS of choice but for very large datasets like OS Open Zoomstack I think it's much easier to work with. 

You can see what I'm talking about when you add the GeoPackage to QGIS and you are asked which of the 21 layers within the GeoPackage you want to add. It shows you what type they are, how many features they have and so on, as I've shown below. Once they are added, you just need to follow the instructions to get it all looking nice. This just involves re-ordering the layers and then applying the supplied style file to each layer. Be sure to copy the supplied SVG folder (screenshot below) from the OS Open Zoomstack download into your QGIS svg folder so that the Airports and Railway Stations  layers have the right icons. The folder path should be something like C:\Program Files\QGIS 3.2\apps\qgis\svg.

Here are the Zoomstack layers, all 21 of them 

The folder just needs to be copied to your SVG folder in QGIS

Once you've done all this - and it should only take a few minutes - you'll get a beautifully styled vector base map. As you zoom in and out you'll see that the symbology changes because the team at OS have very cleverly used the scale-dependent rendering options in the supplied style files so that you only see certain features (e.g. Railway Stations) when you are at an appropriate zoom level. This keeps the map looking good no matter what zoom level you are at. Here's what it looks like in QGIS.

Looking good, I'd say

Okay, so this 'review' has really just been a celebration of OS Open Zoomstack. But I think it deserves some love. Anyway, let's take one more look at the data, and how the scale-dependent layer styles work in practice. The two examples below show the Leith area of Edinburgh (I've rotated it 180 degrees so we're looking southwards) at two zoom levels. On the first, you'll see more generalised buildings but on the second you'll see the most detailed building data where you can pick out individual buildings. It's this kind of detail which really makes it work well.


Leith at a scale of 1:19,052

Part of Leith, at a scale of 1:9,526

What else? Here's my list of tips, things I like, and so on about OS Open Zoomstack.
  1. Because it's vector rather than raster data, it's perfect for creating highly detailed background mapping at any scale. Unlike some other products you won't (obviously) get fuzzy pixels when you zoom in, just more detail.
  2. Because it has so many layers in it, you can tailor these maps any way you like. You can also change the styling (colours, fonts and anything else) if you want to create a more unique, bespoke style.
  3. If you just want some data for Great Britain, for a completely different purpose, you can just load up the OS Open Zoomstack layers and use what you like. You could also save individual layers on their own, let's say if you wanted a single shapefile of places.
  4. In QGIS 3.2, I like the fact that I can load up the data, style it within a few minutes and then export high resolution images without even having to open a Print Composer. This is now possible directly from the main map view in QGIS via Project > Import/Export > Export Map to Image... (as in screenshots below)
  5. Because it's Ordnance Survey data, you know it will be comprehensive, up to date and authoritative. For example, the new bridge across the River Ness in the Higlands (below) is of course shown.  

It's a perfect partner for QGIS

You can now set dpi, extent, etc directly from the map view


The latest bridge over the River Ness in Inverness

Finally
As I said above, this is really just a celebration of a new dataset but it is also a review. I think OS Open Zoomstack is exactly the kind of thing the geospatial community in Great Britain needs but I also think, from a desktop GIS point of view, it will be very helpful in making people think again (or for the first time) about the GeoPackage. 

OS Open Zoomstack demonstrates why the GeoPackage is actually far superior to the trusty shapefile in some instances. In others, I'd say it's just another spatial data format that most users can't see the benefit of. I'm not anti-shapefile at all, but this really is an excellent example of a use case where GeoPackage wins hands down.

The only thing I haven't solved yet is how it came to be that some places in the Highlands (where I'm from) now have Gaelic names by default and some don't. I've read the Gaelic place names policy from Ordance Survey (thanks to Liam Mason) but am still none the wiser as to why Inverness is now Inbhir Nis by default and, for example, Shawbost is not Siabost but only a few miles down the road, Tolsta Chaolais is Tolstadh a' Chaolais. But this is nothing to do with Zoomstack so I'll end here.

Happy mapping. I couldn't help myself (see below). Every zoomstack needs a geogif.


What Zoomstack looks like at different scales



Friday, 20 July 2018

Map Layout Design in QGIS Atlas

This is a short follow-up post from the previous one, where I posted an update of my Green Belt Atlas. It's about my on-going quest to try to make clearer, better maps and in particular the choices that went into the final layout. I was quite happy with the style and layout but don't claim it's perfect by any means. Anyway, here goes. Scroll to the bottom of the page for some screenshots of what settings I used in QGIS to create this.

These captions explain things in more detail - click to enlarge

As you can see from the image above, basically everything I've done here has some kind of logic or reasoning behind it. 

The top white space is for the name of each local authority. This means there is a lot of white space when that place is York, as above, but I needed to ensure that when the Atlas shows somewhere with a longer name (e.g. Windsor & Maidenhead) it still fits in.

The font throughout is Raleway, still a favourite of mine, and also free. The only thing I'm not so keen on is the the numbers but I can live with it.

I also added some explanatory/contextual text at the bottom of each map, which pulls in data from the Atlas coverage layer in QGIS. I think this adds some valuable information to help readers interpret things more clearly. I tend not to use black at all these days, and instead go for #222222 (RGB 34, 34, 34) as it's less stark.

The inset map in the top right is also pretty small but works well I feel and for a locator I have just used an off-black dot, again to keep it simple. I do like the kind of faded, cloudy white background effect so that you see enough to give you an idea of what's there but your eyes are still drawn to the main feature.

The really important thing for me was to try to keep things simple but at the same time allowing the reader to get the maximum amount of information out of the underlying dataset. But since this was QGIS Atlas and automated mapping I also had to make sure that when QGIS iterates through each page of the Atlas (all 186 of them), it didn't cause any label or placement conflicts, and this is not always easy. 

If you're thinking about getting into using the QGIS Atlas tool, I've got a tutorial here (with all the data and project files you'll need to play around with it) but it also links to a simpler, previous example.

Anyway, that's all for now. Have a read of all the captions in the image to get more of a sense of my thinking on this little project.


This is the query you use to only show points in the Atlas feature


The is the NOT query used for the background mask layer


And this is the symbology used for the mask layer above

Saturday, 14 July 2018

Green Belt Atlas, version 4

This is a short post about an updated version of my Green Belt Atlas, in which I've produced a map of every local authority in England that contains land designated as green belt. The original idea for this came from GIS guru Bob Barr a couple of years ago and there have been a few versions so far but this one uses the latest data and includes some new features and improvements. Here's a sample map, for York. I'm going to do a follow-up post about some of the technical stuff.

As you can see, York is 82% green belt

I have put all 186 local authority maps in a public folder, so you can check that out if you're interested. I'll post a selection here as well. If you want to find out more about green belts and their history, I'd recommend reading John Grindrod's Outskirts.

See the whole lot here

This version of the Atlas is, I think, an improvement on previous ones I've produced. I'll cover more of the technical details and design choices in the next blog post but for now I'm just posting the maps, and a bit of summary data.

The reason I've done this is because I'm interested in land use, land cover and suchlike, and I know others are too. This version of the Atlas uses the latest data (2016-17 green belt boundaries), which you can find on data.gov.uk, along with data for previous years.

Here are a few green belt maps from the full set, just click an image to enlarge, or go to the full set if you want a  high resolution download. Green areas are land designated as green belt, and I've also shown buildings, roads, railways and I've added some place names. A summary datasheet for all 186 areas can be found here as well as at the bottom of the page.

But wait! Why map this at a local authority level? Well, clearly the green belts were not designated at the individual local authority level, and that's fine. But if we're looking at the kinds of things local planning authorities have responsibility for (e.g. housing) then they do need to be aware of this kind of thing, in my opinion. The Plain English guide to the planning system (paragraph 32) is a good reference point for this:

"This means that the requirement to meet housing needs must be balanced against other important considerations, such as protecting the Green Belt or addressing climate change and flooding."

So, my view is that it's entirely logical and sensible to look at this from a local perspective. 




















Full green belt spreadsheet

Wednesday, 13 June 2018

How many people live in the English green belt?

There was a blog post here about the total population living on land in England that is designated as green belt.

I decided to take it down because I wasn't as confident as I wanted to be about my original estimate and I don't want to spread misinformation. 

The problem is that it's impossible to know for sure with the data we have available, which is a shame. Perhaps in future we'll move to a position where we can easily answer these questions with better open data.

For now, I am just leaving this here as a note to anyone who might have seen the original post or tweet (also deleted).

Normal service shall resume soon.

15 June 2018

Update 26 October 2023: I think the figure is 1.2 million, as I explain in this post


Note
You can find some English green belt shapefiles on this page, though they seem to mostly be historic ones (e.g. 2007-08) unless you click 'Show more' in which case you'll get everything from 2007-08 to 2016-17.

Sunday, 20 May 2018

The Shape of American Democracy, v1.0

The fifth Vice President of the United States, and 9th Governor of Massachusetts - Elbridge Gerry - has not been forgotten. His name lives on in debates about the shapes of political districts and specifically in relation to those very irregularly shaped Congressional Districts we often see in the United States. Thus, gerrymandering remains a hot topic, but it is often difficult to understand without maps. So, in an effort to simplify things, I've created a single graphic that shows the shape of every US Congressional District - or, as I've called it, 'The Shape of American Democracy'. I have put the massive full size version, plus smaller versions, on a dedicated web page, which I may update in due course. Read on for more details about this project.

This is a big version, but you can find a giant version here

What is this?
The graphic above is a single poster that shows all 435 US Congressional Districts of the 115th US Congress, arranged in 15 rows and 29 columns. This is quite convenient, because 15 x 29 is 435, so it allows for a nice, regular layout. The colours (sorry, colors), represent the party representing each district - blue for Democratic, red for Republican. The order of the shapes is also very important. From top left to bottom right, I have arranged the districts in order of their Polsby-Popper score, which is one measure of the compactness of a geographic area, among several. There are others, but I've used this one as I think it provides results that make sense and look right. 

A closer look at the layout
Congressional District table - so you can find them easily

On the right hand side, there is a list, in five columns of 87 rows (5 x 87 = 435, thankfully), which tells you where you can find any Congressional District in the main layout, where I have labelled the rows A to O and the columns 1 to 29. So, if you wanted to find Florida's 18th Congressional District, for example, you would look it up alphabetically on the right, see that it's in N-23 and then look up row N, column 23 to see what it looks like, as shown below. If you look closely, you'll notice that some Congressional Districts are actually whole states. This is because seven states have only one representative: Alaska, Delaware, Montana, North Dakota, South Dakota, Vermont, and Wyoming.


Why do this?
The reason I did this is because I'm interested in the subject and have written about it before. I lived in the US for a short time (OH-3) but I'm not an American and I have no vote. I do have an unhealthy interest in maps and stats, though, as you can see if you look at more of this blog. In previous attempts, I'd done things like make small multiple graphics or animated gifs, but I wasn't satisfied with them. This time I wanted to pack as much useful information into a single graphic as possible, yet I also wanted it to be easy to digest, and The Shape of American Democracy is the result. I'm reasonably happy with it as a first draft, yet almost as soon as I finished, Pennsylvania redrew its districts after the existing ones were struck down by their Supreme Court. I am definitely not doing this to say any single political party is bad or good, or that an irregular shape is necessarily a problem. But by putting them all in one place, where we can also compare and contrast individual districts, I think it gives us a good overview and is a useful conversation starter at the very least.


What does it tell us?
Let's begin with the basics. It tells us that there are 435 Congressional Districts and that some are red and some are blue. By using the Polsby-Popper measure (which I learned about from my colleague Ruth Hamilton), and ordering them like as I have, it also tells us that some areas are much more irregularly-shaped than others. But of course we already knew this information. Where I think a graphic like this adds value is in its ability to pull all this information together into one information set that is, from a cognitive point of view, easier to digest. It provides a kind of at-a-glance summary of US Congressional District irregularity. This is not to say unusual shapes are inherently a bad thing because there may be good reasons for it. Likewise, more regular shapes may not necessarily be a good thing even if, on balance, most people would agree that this is the case. For more on this general topic, I recommend you read Garrett Dash Nelson's piece in Citylab, which asks the question of what a good electoral map would look like.

PA has since changed, but you get the idea

How did I do this?
In brief, I downloaded the latest US Congressional District shapefile from the US Census Bureau website, loaded it up in QGIS and then created an Atlas and then patched it all together in another open source application, called (ahem) GIMP. Creating the Congressional District finder table was a bit more tricky, so for that I used spreadsheets and pivot tables because it was the quickest way I could think of doing it, even if it was a wee bit clunky. I then exported different sizes of the graphic and put it all online in a dedicated single web page.

Click here to download the full size version


Am I saying US Democracy is in bad shape?
I realise the title of the graphic makes it sound that way, but it's not my intention. This is about shapes, and I toyed with the idea of calling it 'The Shapes of American Democracy' but in the end I went with Shape because I thought it sounded better and of course people can make their own minds up about what this might tell them about the state of democracy, if anything. Being from the UK (as a Scot living in England) it would be quite rich for me to criticise anyone else's democracy at present anyway! So, all I'm really saying is that the graphic gives you an impression of the shapes of US Congressional Districts, which form part of the democratic process in the United States, and from that you can make up your own mind about what it all means.


Other things
I originally put this together as a submission for the brilliant Atlas of Design, but it didn't make the cut, so I'm sharing it here instead, and on its own dedicated web page. I made it in January 2018, but then almost straight away the Pennsylvania Congressional Districts were re-drawn, but it still represents the correct information at a moment in time. I could update it if a) I ever find the time or b) someone commissions me to do it. So, that's already two kinds of fails with this project, but I think it is still useful to share it.

That's basically it for now. I haven't been able to obtain the very latest boundaries in the wake of the Pennsylvania re-draw but I might return to this in future if I can to create a version 1.1.


Technical information
When calculating the shapes of Congressional Districts, and their irregularity, there are a number of technical challenges, not least of which is the coastline. What we really want to measure are only those boundaries that can or have been manipulated by people rather than nature. A heavily indented coastline will give a district a very irregular shape, but this is not gerrymandering. So, when I was doing this I used an unclipped shapefile with, essentially, a smoothed coastline. Yet even when you do this you get some anomalies - e.g. Alaska and Hawaii's 2nd District end up on the top row, when really they are just quirks of geography - nobody would claim they were heavily gerrymandered! So this is a limitation but a relatively minor one.

Another thing that you might notice if you look closely is that some Districts might appear to be in the wrong place, based on what they look like. A good example of this is in Florida's 13th Congressional district, which is placed just before Wyoming as the second least irregularly shaped, even though it doesn't look right. But if we look at the portion of the boundary which could actually be modified through gerrymandering, given the coastline and the fact that this is St Petersburg (on a peninsula) it makes more sense. Wyoming does, thankfully, end up in the right place as the least irregular of all the shapes.

FL-13, surely some mistake?

Now it makes more sense - source