Wednesday, 25 July 2018

Visualising Bay Area Commutes with Kepler.gl

A couple of months ago I saw on Twitter that Uber launched a new visualisation tool called kepler.gl. Apart from the fact that it looked seriously amazing, it also looked very useful but I hadn't managed to have a play with it until this week, so here I am with some visuals and a few words.

I've already looked at commuting in the San Francisco Bay Area quite a bit in the past (including this in Wired) so I had the data already and when Bay Area urbanist Alex Schafran got in touch about this very topic a couple of days ago it prompted me to dust off my Bay Area commute data and have a play around. These are the results. 

If you want to have a play yourself, I've put the csv in a shared folder, alongside a ready-to-upload json, some images, and a bonus csv of commuting and image folder for England and Wales as well, which is good fun.

Each line connects two Census Tracts, which have a resident population of just over 4,000 people. 

Commutes to San Francisco (100+) coloured by origin county

Bay Area commutes (colour = destination county)

Commutes flows (50+) within Alameda County 

Commute flows (150+) within Alameda County



Same as above, but with a different base map

Bay Area commutes (100+), coloured by destination county

As above, but showing the Filter tools in Kepler.gl

As above, but showing a tooltip for one flow line

Commute flows (50+) in Contra Costa County

Commutes to Santa Clara County (100+)

Commutes to San Francisco County (50+)

Commutes to San Francisco County (100+)

San Francisco to Santa Clara County commutes (all)

The final four images are just to show you the styles and settings I've used with the data, so that you can understand how it works a little better if you want to have a play around with it yourself.

You can see that lines are based on orig, dest x and y coords

I've applied a single filter here, but you can add more

These are the fields used in the tooltip - very useful

You can choose different base maps and turn layers on or off

That's all for now. Like I said, you can play around with this data if you want or even try the England and Wales dataset I created, which has some useful fields for filtering (e.g. local authority name, distance, flow volume).


Notes: Kepler? Presumably named for Johannes Kepler, the German mathematician. The Bay Area commute data includes commutes within and between the nine Bay Area counties of Alameda, Contra Costa, Marin, Napa, San Mateo, Santa Clara, Solano, Sonoma, and the City and County of San Francisco. The dataset is one I put together for another project, and comes from the American Community Survey. The England and Wales file is from the 2011 Census and includes all commutes of 25 people or more. I've found that you can't always export images from Chrome directly but it worked in Firefox. I haven't tried on Safari but I'm guessing image export works on that. Have a go yourself with the data I used to create the images above.

Saturday, 21 July 2018

A Review of OS Open Zoomstack

This is a long blog post all about something called OS Open Zoomstack. What? Yes, Ordnance Survey have released a product called OS Open Zoomstack, and it is very very good and you should know more about it. So here I am with a review. This is mostly aimed at people who might actually use the raw data to make maps, but also anyone who has an interest in mapping more generally. Put very simply, it's a giant, beautifully styled vector dataset for the whole of Great Britain. It can be used in a webmap (e.g. Mapbox) in a desktop GIS (e.g. QGIS) and other ways (e.g. PostGIS). This is what Ordnance Survey say it is:

"OS Open Zoomstack is a comprehensive vector basemap showing coverage of Great Britain at a national level, right down to street level detail."

But the best way to explain what OS Open Zoomstack is, is to just show you a couple of examples so I've embedded two maps below. The first is a kind of 'night time' view of Glasgow, and the second is a lighter overview map of London. Exactly the same data, just styled a different way and in both I have extruded the buildings by a few metres to give a little 3D effect. But actually both of these cover the whole of Great Britain which you can see if you zoom and pan each of them. Have a play around and then read on below for more information. There are links to full screen versions below.



Full screen - Glasgow


Full screen - London


Besides these examples, why not try a full screen version: as well as London and Glasgow, I've done Manchester, Edinburgh, and Portree (in Skye) but as I said, you can zoom anywhere you like in Great Britain. These places are just where I've set the map to start when you click the url.

If you want, you can edit the url yourself and share a map of any location in Great Britain. You can tilt and pan as well as zoom. To tilt for a different 3D view you may also need to hold down the CTRL/Cmd key or maybe use two fingers on a mobile device or tablet. It's worth trying this on a mobile device just to see how slick it is.

If you're still a little confused at this stage about what exactly OS Open Zoomstack is, take at look at the Ordnance Survey blog and then listen to what Charley Glynn has to say about it in the video. It's worth your time. The documentation is really helpful and even if you're not very experienced you can be up and running with it in no time, particularly if you're using QGIS. Talking of which, the series of maps below were all exported from OS Open Zoomstack in QGIS.

Whereas the examples above involved me uploading the vector tile set to Mapbox and doing a little bit of tweaking, the maps below simply involved downloading the data as a single file GeoPackage, loading it up in QGIS, re-ordering the layers, and then applying style files. You can find the instructions for this in the documentation and if you're an experienced QGIS user it'll be really easy. Even though the whole lot is about 10.6GB, it works very quickly for me on the machines I've tested it on.

This new release from Ordnance Survey is a also a nice demonstration of why the GeoPackage is - shudder - the future for geospatial data. No offence to Professor Shapefile, it's just the way things seem to be moving and this use case, for me, provides a good example that helps demonstrate where the GeoPackage has real advantages over the Shapefile. But I'm not here to talk about that right now.

Take a look at the different images below, and make sure you click to see them in full size. I've taken a range of snapshots from across Great Britain, with a few different zoom levels shown for London.  You'll see on the Knoydart one the impact that the contours have in giving the terrain a nice sense of relief. In my opinion this new product is going to be something of a game changer and so far I can't think of anything I don't like about it. 


Anglesey in North Wales - zoomed out

Central Bath 

Fort William and Ben Nevis

Birmingham city centre

Inverness (although OS use the Gaelic, 'Inbhir Nis')

The terrain of the Knoydart peninsula in Scotland

Leeds

Liverpool

London

Even more London

Zooming closer to the Isle of Dogs

Even closer

About as much detail as you can get

Middlesbrough - lovely detail

Norwich

Isles of Scilly - lovely styling

The 'west side', as they say (Lewis, in Scotland)

Southend

Swansea

This is Unst, Shetland

Wrexham, or Wrecsam

Under the hood
Let's take a closer look at what this looks like behind the scenes. The OS Open Zoomstack trial was announced on 17 July 2018 and runs for three months. You can get the data by following the links on the OS blog page about it. I've played around with two of the download options: the vector map tiles I used in the interactive versions above, plus the GeoPackage. Here's what they look like when you download them (below).


The documentation is also very impressive

If you're not familiar with what a GeoPackage is, and you are a GIS user, I suggest this is the time to get into it. Whereas a shapefile is a set of files that contain attribute and spatial data for a single type of geometry (points, lines, polygons), the GeoPackage is a single file (yes, just one file) which can contain all sorts of different geometries and also raster and vector data too. It's kind of a big container for spatial data. In practice, it will look the same as a shapefile in your GIS of choice but for very large datasets like OS Open Zoomstack I think it's much easier to work with. 

You can see what I'm talking about when you add the GeoPackage to QGIS and you are asked which of the 21 layers within the GeoPackage you want to add. It shows you what type they are, how many features they have and so on, as I've shown below. Once they are added, you just need to follow the instructions to get it all looking nice. This just involves re-ordering the layers and then applying the supplied style file to each layer. Be sure to copy the supplied SVG folder (screenshot below) from the OS Open Zoomstack download into your QGIS svg folder so that the Airports and Railway Stations  layers have the right icons. The folder path should be something like C:\Program Files\QGIS 3.2\apps\qgis\svg.

Here are the Zoomstack layers, all 21 of them 

The folder just needs to be copied to your SVG folder in QGIS

Once you've done all this - and it should only take a few minutes - you'll get a beautifully styled vector base map. As you zoom in and out you'll see that the symbology changes because the team at OS have very cleverly used the scale-dependent rendering options in the supplied style files so that you only see certain features (e.g. Railway Stations) when you are at an appropriate zoom level. This keeps the map looking good no matter what zoom level you are at. Here's what it looks like in QGIS.

Looking good, I'd say

Okay, so this 'review' has really just been a celebration of OS Open Zoomstack. But I think it deserves some love. Anyway, let's take one more look at the data, and how the scale-dependent layer styles work in practice. The two examples below show the Leith area of Edinburgh (I've rotated it 180 degrees so we're looking southwards) at two zoom levels. On the first, you'll see more generalised buildings but on the second you'll see the most detailed building data where you can pick out individual buildings. It's this kind of detail which really makes it work well.


Leith at a scale of 1:19,052

Part of Leith, at a scale of 1:9,526

What else? Here's my list of tips, things I like, and so on about OS Open Zoomstack.
  1. Because it's vector rather than raster data, it's perfect for creating highly detailed background mapping at any scale. Unlike some other products you won't (obviously) get fuzzy pixels when you zoom in, just more detail.
  2. Because it has so many layers in it, you can tailor these maps any way you like. You can also change the styling (colours, fonts and anything else) if you want to create a more unique, bespoke style.
  3. If you just want some data for Great Britain, for a completely different purpose, you can just load up the OS Open Zoomstack layers and use what you like. You could also save individual layers on their own, let's say if you wanted a single shapefile of places.
  4. In QGIS 3.2, I like the fact that I can load up the data, style it within a few minutes and then export high resolution images without even having to open a Print Composer. This is now possible directly from the main map view in QGIS via Project > Import/Export > Export Map to Image... (as in screenshots below)
  5. Because it's Ordnance Survey data, you know it will be comprehensive, up to date and authoritative. For example, the new bridge across the River Ness in the Higlands (below) is of course shown.  

It's a perfect partner for QGIS

You can now set dpi, extent, etc directly from the map view


The latest bridge over the River Ness in Inverness

Finally
As I said above, this is really just a celebration of a new dataset but it is also a review. I think OS Open Zoomstack is exactly the kind of thing the geospatial community in Great Britain needs but I also think, from a desktop GIS point of view, it will be very helpful in making people think again (or for the first time) about the GeoPackage. 

OS Open Zoomstack demonstrates why the GeoPackage is actually far superior to the trusty shapefile in some instances. In others, I'd say it's just another spatial data format that most users can't see the benefit of. I'm not anti-shapefile at all, but this really is an excellent example of a use case where GeoPackage wins hands down.

The only thing I haven't solved yet is how it came to be that some places in the Highlands (where I'm from) now have Gaelic names by default and some don't. I've read the Gaelic place names policy from Ordance Survey (thanks to Liam Mason) but am still none the wiser as to why Inverness is now Inbhir Nis by default and, for example, Shawbost is not Siabost but only a few miles down the road, Tolsta Chaolais is Tolstadh a' Chaolais. But this is nothing to do with Zoomstack so I'll end here.

Happy mapping. I couldn't help myself (see below). Every zoomstack needs a geogif.


What Zoomstack looks like at different scales



Friday, 20 July 2018

Map Layout Design in QGIS Atlas

This is a short follow-up post from the previous one, where I posted an update of my Green Belt Atlas. It's about my on-going quest to try to make clearer, better maps and in particular the choices that went into the final layout. I was quite happy with the style and layout but don't claim it's perfect by any means. Anyway, here goes. Scroll to the bottom of the page for some screenshots of what settings I used in QGIS to create this.

These captions explain things in more detail - click to enlarge

As you can see from the image above, basically everything I've done here has some kind of logic or reasoning behind it. 

The top white space is for the name of each local authority. This means there is a lot of white space when that place is York, as above, but I needed to ensure that when the Atlas shows somewhere with a longer name (e.g. Windsor & Maidenhead) it still fits in.

The font throughout is Raleway, still a favourite of mine, and also free. The only thing I'm not so keen on is the the numbers but I can live with it.

I also added some explanatory/contextual text at the bottom of each map, which pulls in data from the Atlas coverage layer in QGIS. I think this adds some valuable information to help readers interpret things more clearly. I tend not to use black at all these days, and instead go for #222222 (RGB 34, 34, 34) as it's less stark.

The inset map in the top right is also pretty small but works well I feel and for a locator I have just used an off-black dot, again to keep it simple. I do like the kind of faded, cloudy white background effect so that you see enough to give you an idea of what's there but your eyes are still drawn to the main feature.

The really important thing for me was to try to keep things simple but at the same time allowing the reader to get the maximum amount of information out of the underlying dataset. But since this was QGIS Atlas and automated mapping I also had to make sure that when QGIS iterates through each page of the Atlas (all 186 of them), it didn't cause any label or placement conflicts, and this is not always easy. 

If you're thinking about getting into using the QGIS Atlas tool, I've got a tutorial here (with all the data and project files you'll need to play around with it) but it also links to a simpler, previous example.

Anyway, that's all for now. Have a read of all the captions in the image to get more of a sense of my thinking on this little project.


This is the query you use to only show points in the Atlas feature


The is the NOT query used for the background mask layer


And this is the symbology used for the mask layer above