A few flow maps + data to play with

I've written a fair bit about mapping flows (e.g. migration, commuting) over the past decade or more and here I am again. The point of this post is to a) share some data so that anyone who wants to play with it can have a go; b) talk a little bit about visualising flows; and c) to look at the functional economic geography of England.

First of all, here's the underlying data - it's 2011 MSOA-to-MSOA commute data for England and Wales, which comes from the Census. Yes, it's getting quite old now but it's still a useful dataset. It's a big shapefile, with columns for total commute flow between LSOA and also different modes (train, bus etc) and some other stuff (e.g. distance, area codes and names).

Here are some of the maps I extracted from it. The lines are the commuting connections between places, with an addition blend mode added in QGIS, plus a scaling factor is used to make smaller flows dimmer and larger flows brighter. I've filtered the dataset to show only MSOA-MSOA flows of 10 or greater, otherwise it's a total mess. Here are some of the maps.

Okay, lots of shiny maps to see. With the opacity of the lines set to reflect the volume of the flows you get a slightly better overview of commute patterns. The addition blend mode gives the shiny effect, which is in some ways just a bit fancy but actually it serves a purpose here: making the main economic centres brighter, which fits in well with the underlying economic geography of England and Wales.

Talking of the functional economic geography of England more broadly, there are connections to my previous work with Garrett Nelson on US megaregions (e.g. our interactive map site, with similar shiny maps but driven by an algorithmic partitioning process) and you can get a sense of where the break points are between different areas.

Talking of which, if you're interested in this kind of thing, check out the AMA Garrett and I did on Reddit about the US commute work as part of the PLOS Science Wednesday series. The comments here are pretty interesting, particularly those from commuters about where they draw the line themselves between travelling to city A vs city B.

That's all for today - feel free to download the data and have a play, use the maps as you see fit, or get back to me with any questions or suggestions.

Oh, and if you do download the dataset, you'll also see that I have added origin an destination codes and names for MSOAs and local authorities. This is useful if you want to, for example, only look at all flows into Manchester, or all flows between Leeds and Bradford.


Data notes: MSOAs are small geogaphic areas with between 5,000 and 15,000 people in them, or between 2,000 and 6,000 households. Most have about 8,000 people and 95% of MSOAs had a population of between 5,443 and 11,579 at the time of this dataset is from. Want to know how to do this in QGIS? See my flow map tutorial from a few years ago. Want to know more about the glow effect? See this tutorial on glowing lines in QGIS. And the place labels? I put together a single file of Great Britain place names if you want to use that. I've used a couple of rules on which places to display, plus added some in manually. Thanks to Allan Walker and Richard Mann for making me think a bit more about this again.

Visualising Bay Area Commutes with Kepler.gl

A couple of months ago I saw on Twitter that Uber launched a new visualisation tool called kepler.gl. Apart from the fact that it looked seriously amazing, it also looked very useful but I hadn't managed to have a play with it until this week, so here I am with some visuals and a few words.

I've already looked at commuting in the San Francisco Bay Area quite a bit in the past (including this in Wired) so I had the data already and when Bay Area urbanist Alex Schafran got in touch about this very topic a couple of days ago it prompted me to dust off my Bay Area commute data and have a play around. These are the results. 

If you want to have a play yourself, I've put the csv in a shared folder, alongside a ready-to-upload json, some images, and a bonus csv of commuting and image folder for England and Wales as well, which is good fun.

Each line connects two Census Tracts, which have a resident population of just over 4,000 people. 

Commutes to San Francisco (100+) coloured by origin county

Bay Area commutes (colour = destination county)

Commutes flows (50+) within Alameda County 

Commute flows (150+) within Alameda County

Same as above, but with a different base map

Bay Area commutes (100+), coloured by destination county

As above, but showing the Filter tools in Kepler.gl

As above, but showing a tooltip for one flow line

Commute flows (50+) in Contra Costa County

Commutes to Santa Clara County (100+)

Commutes to San Francisco County (50+)

Commutes to San Francisco County (100+)

San Francisco to Santa Clara County commutes (all)

The final four images are just to show you the styles and settings I've used with the data, so that you can understand how it works a little better if you want to have a play around with it yourself.

You can see that lines are based on orig, dest x and y coords

I've applied a single filter here, but you can add more

These are the fields used in the tooltip - very useful

You can choose different base maps and turn layers on or off

That's all for now. Like I said, you can play around with this data if you want or even try the England and Wales dataset I created, which has some useful fields for filtering (e.g. local authority name, distance, flow volume).

Notes: Kepler? Presumably named for Johannes Kepler, the German mathematician. The Bay Area commute data includes commutes within and between the nine Bay Area counties of Alameda, Contra Costa, Marin, Napa, San Mateo, Santa Clara, Solano, Sonoma, and the City and County of San Francisco. The dataset is one I put together for another project, and comes from the American Community Survey. The England and Wales file is from the 2011 Census and includes all commutes of 25 people or more. I've found that you can't always export images from Chrome directly but it worked in Firefox. I haven't tried on Safari but I'm guessing image export works on that. Have a go yourself with the data I used to create the images above.

WALRUS: the Wirral and Liverpool Regional Urban System

On a recent trip to Liverpool I needed an all-day ticket that would let me use public transport across the city. So I bought a couple of day passes for me and a friend, which meant getting a plastic card that can be topped up, kind of like the Oyster Card in London. But in Liverpool it's called the Walrus card. It's not exactly the same as the Oyster, but it is named after a sea creature, though it took us a few moments to figure out why they chose Walrus as the name. But of course we decided it must really stand for Wirral and Liverpool Regional Urban System (surely, yes?) and I therefore had to make an animated gif of commute flows in the WALRUS, so here I am. Watch it multiple times to see the main travel to work patterns, and scroll to the bottom for a slower version. I've tried to get the colours right so you can see the mix of destinations clearly. Liverpool city centre is near the L of Liverpool and I've also labelled some other locations including the airport (Liverpool John Lennon Airport, as it is officially called).

Behold: commuter flows in the WALRUS

Surely the people who came up with Walrus really meant it as a play on the Beatles song plus a play on this urban and regional acronym. After all, there is a history of this kind of thing in the wider area, with SELNEC as 'South East Lancashire, North East Cheshire' probably the most famous example, from the late 1960s to the early 1970s. But no, if I Google it, I get absolutely nothing, as you can see below or try for yourself. Surely somebody already uses this, it seems too obvious. If not, then I will claim partial credit alongside my good friend for creating this backronym.

"WALRUS, you say?"

From the JR James Archive

There are other examples of urban and regional acronyms in the UK and across the world (feel free to suggest more) but I can't think of one that incorporates a nice local reference like this. Although, if they chose Oyster in London because it means the world is your Oyster then the Walrus thing doesn't work so well here ("The World is Your Walrus"!?). Anyway, it did make travel on public transport much more efficient and it also meant I got to revive my series of animated gifs, for an area I know pretty well.

My Walrus card

The other side of my Walrus card

That's really all there is to it for now. Hope to catch you next time I'm in the WALRUS, which really is a regional urban system. If you get the chance, I'd like to hear about other urban and regional acronyms you may have heard of. I'll end this post with another version of the gif at the top of the pages, this time slowed down a bit more.

The WALRUS in slow motion

From CartoDB to CARTO - the future of interactive mapping?

I've been using CartoDB (now CARTO) for a few years for interactive mapping - and have always loved what it can do - from basic mapping to much more complex analysis. Now, with the re-brand as CARTO and the advanced analytical tools available through the new Builder interface it's on a new level. So, credit where credit's due - I thought I'd do a short piece on this now to give my take on the new interface. But first, here's a little gif of me playing around with some commute data - which you can also download yourself if you want to. The dataset was used as part of a project I've been working on with Garrett Nelson - but hopefully more on that in future.

I'm just playing around turning things on and off here

If you've used the old CartoDB interface, the new Builder one might be a bit confusing at first - though you may not actually be able to get access to it yet. But once you have played around with it for a few minutes it soon becomes pretty intuitive. I uploaded a sub-set of commute flow lines for Minnesota and Wisconsin and then decided to add widgets so that I could filter the data using line distance, FIPS codes and commute volumes - as you should be able to see in the larger image below.

Click to enlarge - change the data view by using tools on right

This is very much just a little data sample, but if you want to play around with the interactive CARTO map you can see it here. It's not very pretty and the origins and destinations don't have place names right now - only FIPS codes - but the principle is the same. The Widget interface takes a little bit of getting used to as well, but is really easy to use once you've figured out what's what. See below for a screenshot.

You can add widgets for different data types

Any negatives to report? For me, not now. I'm just enjoying the enhanced analytical tools at hand. But if I was being greedy... I'm not massively keen on the default legends, there doesn't appear to be an 'addition' blend mode and the snap alignment of shapes in the old map editor has me a little flummoxed, but these are minor grumbles.

I'm not getting paid to promote this and I don't know anyone at CARTO - honest - I just think they have produced something that works brilliantly, is simple yet powerful and allows us to manipulate, analyse and share our data in new ways. There are other tools out there but the new Builder, for me, takes things to the next level for a mass audience. To answer the question in the title of the blog: is CARTO the future of interactive mapping, then? Not the future, but probably a very big part of it.

Notes: really, they didn't pay me. Data used are from the American Community Survey. I wrote a working paper on it already. I also blogged about it on my old blog. Finally, if you're one of the few people in the world not to have seen it, Mark Evans created this beautiful site with the same data.

City Footprints

Earlier in the week I posted a map of London's 'economic hinterland' on Twitter because I've been working with commuting data and wanted to see what the economic footprint of London looks like. But, some people have been telling me that other cities exist - which is a fair point. Since I have the data and I'm just revising a paper on the topic I thought I'd look at a few others, but this time using lower level data - MSOAs instead of districts for the origins. The maps below show the proportion of people from an MSOA who commute to a given area. Only MSOAs with 1% or more going to a particular place are shown and the darker the colour the higher the percentage. These aren't exactly the same as travel to work areas but they do give a reasonable approximation of each city's economic footprint. As you can see, I did quite a few maps - click to enlarge, as ever.

Birmingham has quite a pleasing concentric pattern

Bradford has quite a wide footprint

Bristol - quite a neat footprint

Cambridge is a little wider than I expected

Camden - one of a few London Boroughs I looked at

Cardiff - clearly the major focus in South Wales

Cheshire West - I wanted to see how far into Wales it goes

Derby seems relatively tightly packed

Leeds - the second largest local authority by population

Leicester - another quite tight East Midlands labour market area

Liverpool - extends into Wales and south to Cheshire, as you'd expect 

Greater London - the light areas are only 1% of commuters, but still!

Greater London - same as above, but with some city labels

Manchester - only the 'underbounded' district here, but still dominant

Middlesbrough - an important northern labour market area

Milton Keynes - I think it has quite a wide footprint

Newcastle - clearly dominant in the North East

Norwich - a good example of a large regional labour market area

Nottingham is relatively symmetrical in labour market terms

Oxford - a relatively large footprint here

Plymouth is a major South West economic zone

Reading - I had expected this to be a little bigger

Sheffield is another major northern labour market area

Southampton - somewhat overlaps with London's fringe

Southwark - I wanted to see how this London Borough looked

Swansea - quite a wide footprint here

Tower Hamlets - interesting to see the dominance of eastern origins

Warrington - a strategic hub in the North West

City and Westminster - the ONS group these two together

York - quite a wide Yorkshire footprint

Should I patch all these together in one big animated gif? Of course I should.

Why isn't my city included? Good question. Sincere apologies.

Birmingham doesn't get enough love, and is so often overlooked, so I did a zoomed in version of MSOA flows into Birmingham, with a few place labels.

Click here to see the full size version

Notes: the maps give the impression that they are unclassified choropleths, but that is just for effect and because this is a quick map batch. The colour classification is the same in each (see below) and no areas with less than 1% commuting to a given place are shown. I used the UK Data Service Flow Data website to extract the data and QGIS 2.14 for the maps. Bear in mind that they are a bit rough and ready and only really for comparison. Also, each 'city' here refers to the local authority area, not the wider city-region. But I think it's interesting to compare places. You just need to bear in mind the spatial scale and relative size of the destination places. Birmingham and Leeds ought to have much larger footprints that (e.g.) Nottingham and Sheffield because they contain more jobs. Where are Scotland and Northern Ireland? These datasets come separately and are not part of the English and Welsh MSOA geography so are not mapped here.

I used the same classification scheme for all maps