Thursday, 2 August 2018

Watching the ships go by

A bit of summer map fun today, with no real purpose apart from learning about UK shipping density and routes, plus making some shiny maps. Anyone can look at live ship position data, and I've enjoyed it in the past, but it was only when my colleague Ruth Hamilton pointed it out to me that I realised the raw AIS data was available for the UK, via the Marine Management Organisation. So here I am again.

The most recent data I could find was for 2015, so I downloaded it and mapped it on top of a bathymetry layer and then added in a land surface layer, and nothing else. In the maps below I've split the dataset up into shipping types so you can see where they go. This gives us quite a good picture of marine traffic patterns around the UK, and further afield. For technical details, including sources, see the bottom of the page. Click an image to enlarge it, or try opening in a new page/tab to see it in more detail. I've put the high resolution images (400dpi) here if you want to look at them in full size.

Note the location of North Sea oil/gas platforms

Note the loop to the north west of Shetland

Lots of activity off the east coast of England

Quite a bit of traffic to offshore wind farms, perhaps

Lots on the south coast, but also in The Minch

Kind of as you'd expect, but also quite striking

Hmm, quite a lot of cargo

There are also quite a few tankers

I really find the fishing vessels map fasinating

Quite a lot of traffic between England and France

These UK-level overview maps are quite interesting - at least to me - and you can see them pretty clearly at a more local level in the full resolution maps, but I've also done a zoomed-in set for each shipping type, as you can see below. 

Clearly a lot of non-port service craft from Aberdeen

Also a good bit of non-port service traffic around Shetland

Some port service traffic patterns for Scotland and England

Port service traffic is quite busy in the English Channel

Some dredging/underwater activity in the north of Scotland

Note the back and forth patterns off the south west coast

High speed traffic in the Irish Sea area

High speed traffic - to offshore wind farms?

A good bit of military and law enforcement traffic here

Lots of passenger traffic here, as you'd expect

Quite stylish routes I reckon

All hail Caledonian MacBrayne! (and NorthLink, obvs)

England to France: take your pick
There is cargo in UK waters, didn't you know?

There is cargo in the Irish Sea too - it's everywhere!

You want tankers? We got tankers

Fishing in The Minch, and notice the ship tracks down Loch Ness

Note the tracks of the ships compared to the bathymetry

Lots of fish to be caught here

Some nice recreational sailing off the west coast of Scotland

Like I said at the start, this is just a bit of summer map fun with an different kind of dataset. I think it's quite interesting and when you map it in QGIS and add a bit of styling plus terrain and bathymetry data you have some shiny maps. The full resolution images are the ones to download if you really want to see things in detail. There's also a zipped folder with all the images if you want to download them all in one go.

What does AIS stand for? Automatic Identification System.

Where can I read more about all this stuff? The Marine Management Organisation have some great documents explaining the data and what it means.

Can I get data for other parts of the world? I'm not sure, but you can look at live AIS shipping data for the whole world on Marine Traffic. They also have a density maps tool, as you can see below.

This is all marine traffic, on the live web map

How did you make these maps? I used QGIS 3.2. I played around with the styling quite a lot to achieve a look that I was happy with and in particular the feature blending modes (that's what makes the lines look shiny where there are higher densities) and I also applied hillshades to the terrain and bathymetry data, as well as some colorization.

Why did you add the bathymetry layer? I wanted to see if it in any way explained any of the traffic patterns, particularly for fishing, and I think it does. Look closely at some of the maps to see what I mean.

Why did you make these maps? Because I like making maps and find this data interesting.

Can I use these maps? Yes, feel free.

Why did you not include any maps with place name labels? I wanted to keep them super simple. Plus it is of course great fun trying to figure out what's what.

Data sources
The AIS data I used is available under an Open Government Licence (UK) and can be downloaded for several different years, with 2015 being the most recent. The terrain data can be downloaded for free on the EU's Copernicus/Land Monitoring Service website. The bathymetry data can be downloaded from the EMODnet interactive web map via the 'Download products' link. 

The AIS dataset for the full year is pretty big, more than 3 million records, and it's a little unwieldy. But it is also very rich, with data on the start and end time and date for each track, ship length, ship width, ship draught, plus lots of other time and date variables and of course the vessel type. There is an 'unknown' vessel category but I didn't map that here.

Wednesday, 25 July 2018

Visualising Bay Area Commutes with

A couple of months ago I saw on Twitter that Uber launched a new visualisation tool called Apart from the fact that it looked seriously amazing, it also looked very useful but I hadn't managed to have a play with it until this week, so here I am with some visuals and a few words.

I've already looked at commuting in the San Francisco Bay Area quite a bit in the past (including this in Wired) so I had the data already and when Bay Area urbanist Alex Schafran got in touch about this very topic a couple of days ago it prompted me to dust off my Bay Area commute data and have a play around. These are the results. 

If you want to have a play yourself, I've put the csv in a shared folder, alongside a ready-to-upload json, some images, and a bonus csv of commuting and image folder for England and Wales as well, which is good fun.

Each line connects two Census Tracts, which have a resident population of just over 4,000 people. 

Commutes to San Francisco (100+) coloured by origin county

Bay Area commutes (colour = destination county)

Commutes flows (50+) within Alameda County 

Commute flows (150+) within Alameda County

Same as above, but with a different base map

Bay Area commutes (100+), coloured by destination county

As above, but showing the Filter tools in

As above, but showing a tooltip for one flow line

Commute flows (50+) in Contra Costa County

Commutes to Santa Clara County (100+)

Commutes to San Francisco County (50+)

Commutes to San Francisco County (100+)

San Francisco to Santa Clara County commutes (all)

The final four images are just to show you the styles and settings I've used with the data, so that you can understand how it works a little better if you want to have a play around with it yourself.

You can see that lines are based on orig, dest x and y coords

I've applied a single filter here, but you can add more

These are the fields used in the tooltip - very useful

You can choose different base maps and turn layers on or off

That's all for now. Like I said, you can play around with this data if you want or even try the England and Wales dataset I created, which has some useful fields for filtering (e.g. local authority name, distance, flow volume).

Notes: Kepler? Presumably named for Johannes Kepler, the German mathematician. The Bay Area commute data includes commutes within and between the nine Bay Area counties of Alameda, Contra Costa, Marin, Napa, San Mateo, Santa Clara, Solano, Sonoma, and the City and County of San Francisco. The dataset is one I put together for another project, and comes from the American Community Survey. The England and Wales file is from the 2011 Census and includes all commutes of 25 people or more. I've found that you can't always export images from Chrome directly but it worked in Firefox. I haven't tried on Safari but I'm guessing image export works on that. Have a go yourself with the data I used to create the images above.

Saturday, 21 July 2018

A Review of OS Open Zoomstack

This is a long blog post all about something called OS Open Zoomstack. What? Yes, Ordnance Survey have released a product called OS Open Zoomstack, and it is very very good and you should know more about it. So here I am with a review. This is mostly aimed at people who might actually use the raw data to make maps, but also anyone who has an interest in mapping more generally. Put very simply, it's a giant, beautifully styled vector dataset for the whole of Great Britain. It can be used in a webmap (e.g. Mapbox) in a desktop GIS (e.g. QGIS) and other ways (e.g. PostGIS). This is what Ordnance Survey say it is:

"OS Open Zoomstack is a comprehensive vector basemap showing coverage of Great Britain at a national level, right down to street level detail."

But the best way to explain what OS Open Zoomstack is, is to just show you a couple of examples so I've embedded two maps below. The first is a kind of 'night time' view of Glasgow, and the second is a lighter overview map of London. Exactly the same data, just styled a different way and in both I have extruded the buildings by a few metres to give a little 3D effect. But actually both of these cover the whole of Great Britain which you can see if you zoom and pan each of them. Have a play around and then read on below for more information. There are links to full screen versions below.

Full screen - Glasgow

Full screen - London

Besides these examples, why not try a full screen version: as well as London and Glasgow, I've done Manchester, Edinburgh, and Portree (in Skye) but as I said, you can zoom anywhere you like in Great Britain. These places are just where I've set the map to start when you click the url.

If you want, you can edit the url yourself and share a map of any location in Great Britain. You can tilt and pan as well as zoom. To tilt for a different 3D view you may also need to hold down the CTRL/Cmd key or maybe use two fingers on a mobile device or tablet. It's worth trying this on a mobile device just to see how slick it is.

If you're still a little confused at this stage about what exactly OS Open Zoomstack is, take at look at the Ordnance Survey blog and then listen to what Charley Glynn has to say about it in the video. It's worth your time. The documentation is really helpful and even if you're not very experienced you can be up and running with it in no time, particularly if you're using QGIS. Talking of which, the series of maps below were all exported from OS Open Zoomstack in QGIS.

Whereas the examples above involved me uploading the vector tile set to Mapbox and doing a little bit of tweaking, the maps below simply involved downloading the data as a single file GeoPackage, loading it up in QGIS, re-ordering the layers, and then applying style files. You can find the instructions for this in the documentation and if you're an experienced QGIS user it'll be really easy. Even though the whole lot is about 10.6GB, it works very quickly for me on the machines I've tested it on.

This new release from Ordnance Survey is a also a nice demonstration of why the GeoPackage is - shudder - the future for geospatial data. No offence to Professor Shapefile, it's just the way things seem to be moving and this use case, for me, provides a good example that helps demonstrate where the GeoPackage has real advantages over the Shapefile. But I'm not here to talk about that right now.

Take a look at the different images below, and make sure you click to see them in full size. I've taken a range of snapshots from across Great Britain, with a few different zoom levels shown for London.  You'll see on the Knoydart one the impact that the contours have in giving the terrain a nice sense of relief. In my opinion this new product is going to be something of a game changer and so far I can't think of anything I don't like about it. 

Anglesey in North Wales - zoomed out

Central Bath 

Fort William and Ben Nevis

Birmingham city centre

Inverness (although OS use the Gaelic, 'Inbhir Nis')

The terrain of the Knoydart peninsula in Scotland




Even more London

Zooming closer to the Isle of Dogs

Even closer

About as much detail as you can get

Middlesbrough - lovely detail


Isles of Scilly - lovely styling

The 'west side', as they say (Lewis, in Scotland)



This is Unst, Shetland

Wrexham, or Wrecsam

Under the hood
Let's take a closer look at what this looks like behind the scenes. The OS Open Zoomstack trial was announced on 17 July 2018 and runs for three months. You can get the data by following the links on the OS blog page about it. I've played around with two of the download options: the vector map tiles I used in the interactive versions above, plus the GeoPackage. Here's what they look like when you download them (below).

The documentation is also very impressive

If you're not familiar with what a GeoPackage is, and you are a GIS user, I suggest this is the time to get into it. Whereas a shapefile is a set of files that contain attribute and spatial data for a single type of geometry (points, lines, polygons), the GeoPackage is a single file (yes, just one file) which can contain all sorts of different geometries and also raster and vector data too. It's kind of a big container for spatial data. In practice, it will look the same as a shapefile in your GIS of choice but for very large datasets like OS Open Zoomstack I think it's much easier to work with. 

You can see what I'm talking about when you add the GeoPackage to QGIS and you are asked which of the 21 layers within the GeoPackage you want to add. It shows you what type they are, how many features they have and so on, as I've shown below. Once they are added, you just need to follow the instructions to get it all looking nice. This just involves re-ordering the layers and then applying the supplied style file to each layer. Be sure to copy the supplied SVG folder (screenshot below) from the OS Open Zoomstack download into your QGIS svg folder so that the Airports and Railway Stations  layers have the right icons. The folder path should be something like C:\Program Files\QGIS 3.2\apps\qgis\svg.

Here are the Zoomstack layers, all 21 of them 

The folder just needs to be copied to your SVG folder in QGIS

Once you've done all this - and it should only take a few minutes - you'll get a beautifully styled vector base map. As you zoom in and out you'll see that the symbology changes because the team at OS have very cleverly used the scale-dependent rendering options in the supplied style files so that you only see certain features (e.g. Railway Stations) when you are at an appropriate zoom level. This keeps the map looking good no matter what zoom level you are at. Here's what it looks like in QGIS.

Looking good, I'd say

Okay, so this 'review' has really just been a celebration of OS Open Zoomstack. But I think it deserves some love. Anyway, let's take one more look at the data, and how the scale-dependent layer styles work in practice. The two examples below show the Leith area of Edinburgh (I've rotated it 180 degrees so we're looking southwards) at two zoom levels. On the first, you'll see more generalised buildings but on the second you'll see the most detailed building data where you can pick out individual buildings. It's this kind of detail which really makes it work well.

Leith at a scale of 1:19,052

Part of Leith, at a scale of 1:9,526

What else? Here's my list of tips, things I like, and so on about OS Open Zoomstack.
  1. Because it's vector rather than raster data, it's perfect for creating highly detailed background mapping at any scale. Unlike some other products you won't (obviously) get fuzzy pixels when you zoom in, just more detail.
  2. Because it has so many layers in it, you can tailor these maps any way you like. You can also change the styling (colours, fonts and anything else) if you want to create a more unique, bespoke style.
  3. If you just want some data for Great Britain, for a completely different purpose, you can just load up the OS Open Zoomstack layers and use what you like. You could also save individual layers on their own, let's say if you wanted a single shapefile of places.
  4. In QGIS 3.2, I like the fact that I can load up the data, style it within a few minutes and then export high resolution images without even having to open a Print Composer. This is now possible directly from the main map view in QGIS via Project > Import/Export > Export Map to Image... (as in screenshots below)
  5. Because it's Ordnance Survey data, you know it will be comprehensive, up to date and authoritative. For example, the new bridge across the River Ness in the Higlands (below) is of course shown.  

It's a perfect partner for QGIS

You can now set dpi, extent, etc directly from the map view

The latest bridge over the River Ness in Inverness

As I said above, this is really just a celebration of a new dataset but it is also a review. I think OS Open Zoomstack is exactly the kind of thing the geospatial community in Great Britain needs but I also think, from a desktop GIS point of view, it will be very helpful in making people think again (or for the first time) about the GeoPackage. 

OS Open Zoomstack demonstrates why the GeoPackage is actually far superior to the trusty shapefile in some instances. In others, I'd say it's just another spatial data format that most users can't see the benefit of. I'm not anti-shapefile at all, but this really is an excellent example of a use case where GeoPackage wins hands down.

The only thing I haven't solved yet is how it came to be that some places in the Highlands (where I'm from) now have Gaelic names by default and some don't. I've read the Gaelic place names policy from Ordance Survey (thanks to Liam Mason) but am still none the wiser as to why Inverness is now Inbhir Nis by default and, for example, Shawbost is not Siabost but only a few miles down the road, Tolsta Chaolais is Tolstadh a' Chaolais. But this is nothing to do with Zoomstack so I'll end here.

Happy mapping. I couldn't help myself (see below). Every zoomstack needs a geogif.

What Zoomstack looks like at different scales