Crowdsourced City Boundaries

One Friday afternoon a couple of weeks ago, while putting off something more important, I felt the sudden need to see if I could successfully crowdsource some city boundaries. I'd been doing this kind of thing with housing market boundaries for a few years but I thought it would be interesting to see how many shapes I could get people to draw, so here are the results - for fun more than anything else but I think they are also pretty interesting. If you want to download the raw data, you can go back to the original page and click the Download Data button in the bottom right. I've kept it very simple. First of all, the maps are overlaid on a black and white map and the brighter shading shows where more shapes overlap. For places in the UK, I've also overlaid local authority boundaries to give a sense of how the drawn shapes compare to administrative units. The code for this tool was written by Nick Martinelli (thanks a lot) - I just modified it slightly. You should be able to see the detail in the maps when you click on them.

Yes, this is definitely Birmingham

100% Edinburgh - that's it settled

Someone thinks Glasgow is pretty big!

This seems pretty sensible to me

Let's claim a bit of sea while we're at it

This seems big, but is probably not far from the truth

London's functional reach really does go far beyond the boundary

Yes, 'Manchester' is still the classic 'underbounded' city

Now for some US cities - I've just added the ones which had the most shapes drawn for them, starging with Chicago. I see someone has very carefully added in what looks to be an almost exact representation of the actualy city boundary - impressive!

Well done for the detailed city boundary drawing

Los Angeles County (pop c.10m) comes out a bit here

People seem to have mostly drawn the boundary of the 5 boroughs

For New York (and across the US), Jen Nelles helpfully pointed out to me on Twitter that the term 'city' has a different meaning and perhaps I should have used 'urban' or 'metro', which is a good point. When I did this I was only really thinking about the UK but then I got drawn cities coming in from all over the world, so keep that in mind when interpreting these.

Some detailed shapes were drawn for Philadelphia

I find Seattle particularly interesting - nice commutersheds here

Some evidence of people drawing the administrative boundaries here

And now some other cities from different parts of the world - I would have added more but some cities (e.g. Auckland) only had a couple of shapes drawn.

I've left a couple of nearby places in here too

There weren't that many shapes here but it's still quite interesting

Lots of blocky shapes in Tornoto, which I found interesting

And finally, three more images, one of which includes one of the rude shapes drawn by several scoundrels who defaced my map! 

Someone drew a giant knob over Wales - very naughty

A full size GB/IRE version 

I particularly like the way Munich has been drawn here.

And that's all I have to say about this little adventure in crowdsourced procrastination. Thanks to everyone who drew a shape or shared the link online. I've found it very interesting. Feel free to use any of this if you find it of interest. Now time for me to get back to what I should have been doing in the first place.

Notes: I'm did this as a little mapping experiment after seeing Colin Ross do it for Western Sydney (which was inspired by earlier Bostonography work). Earlier work by people such as Kevin Lynch and Patrick Abercrombie is also relevant here. Edinburgh City Council also ran a project like this in 2014 called 'Natural Neighbourhoods' and there are countless other examples across the world. A deliberately prettified/extreme pink example of some earlier stuff I did with this (using housing market data) can be seen on the Rightmove blog. That example involved millions of shapes, whereas we're only just into the thousands for this blog. If you want to see some of my academic work on this, in relation to housing market geographies, take a look at this paper (open access) - an example image is shown below. 

This was generated using data from Rightmove's 'Draw-a-Search' tool

Population Density Comparisons

I grew up in the Highlands of Scotland - well kind of. Perhaps the New York of the Highlands is more accurate (Inverness, obviously). But I am also partly of crofting stock so spent a lot of time in very sparsely populated areas. I lived in Liverpool for 5 years and now live in Sheffield and visit London a lot, and I also like looking at maps and making maps. That's why I was particuarly interested in Steven Kay's population comparison map which compared circular areas centred on London to the populations of Scotland, and also Scotland plus Wales and Northern Ireland.

You can see the original map here

Steven has lots of really interesting, clever maps on his website, many of which do this kind of thing. I also really like his Edinburgh comparison with less populated European nations, as you can see below.

Edinburgh - population about 495,000

Obviously, at one level, such graphics don't really tell us that much beyond the fact that some places are really densely populated, and some aren't - and we all know that. But their power lies in the cognitive trick of the simple spatial overlay. By putting them in the same place and simplifying the shapes to roughly circular ones it makes the comparison more powerful and immediate. And this is why it's both easy to understand and quick to digest. It's also what makes me like it as a simple comparison. There are lots of other examples of this kind of thing online, including one I did in the past looking at the Highland local authority area vs Greater London (which is 32+1 authorities, but also a single political space).

Most definitely not a town planning policy suggestion for Nicola Sturgeon

Talking of the Highlands, the other Steven Kay piece I really liked was his analysis of which parts of Britain and Ireland are closer to Westminster, Dublin, Stavanger, and Oslo respectively. My part of the Highlands is, of course, closer to Stavanger than Westminster. But again, this is just a geographical curiosity and not a reason to suggest we be governed from there! Mind you, their sovereign wealth fund would be nice (current value and nice video - click here).

So, where were we - ah yes... These maps are interesting geographical artifacts but they often have a kind of power that connects with people at different levels - including the emotional. But that's a whole other topic.

For the time being, it's interesting to note the different responses to my tweet of Steven's map. I only noticed it because I went back into the Flickr QGIS feed the other day to change my password after the Yahoo! hack. I liked his map because it made an interesting comparison and even though I know London has 'lots of people' and a good few million more than Scotland, seeing it in this way was quite shocking. For some, that might feel threatening (see notes below) but for others - including me - it's just a comparison which captured my attention as a data/map buff.

Notes: for map nerds, Alan MacEachren's 1995 'How Maps Work' has some good stuff on how maps are imbued with meaning - Part II from p. 213. The section on map connotation (p. 336 onwards) is quite relevant here. For some, the London/Scotland comparison is immediately annoying because of the nation/city comparison (I have a little sympathy here) and for others there is something of an 'incitive connotation' because of the current state of UK politics and particularly the push for Scottish indepdenence vs continued Westminster governance. While I'm on a roll, I'll add that Norway has about £56bn invested in the UK, including ownership of 50% of Meadowhall in Sheffield and 25% of Regent Street in London. So, if you are ever at a loss for words, impress your friends with this knowledge. Tenous links here are that I live in Sheffield and Scotland and Norway have similar populations.

DataFest 2016 - what was all that about?

When I'm not writing obscure blog posts about geo things, I work at the University of Sheffield. Half the time I'm in the Department of Urban Studies and Planning and the other half is spent in the Sheffield Methods Institute. The SMI is an interdisciplinary research centre and we have a strong quantitative methods group here - spanning the breadth of social science. A big part of what we do is running the Sheffield Q-Step programme. Say what? Let me demystify... Q-Step is a UK-wide programme - funded by the Nuffield Foundation, HEFCE and the ESRC, with the aim of significantly improving the quantitative skills of social science undergraduates. There are 15 centres across the UK and I happen to be the director of the Sheffield one. That's why we recently hosted DataFest 2016, a quantitative methods summer school for undergraduate students from the north of England (plus two keen interlopers from Bristol!). Too much text already, time for a pic...

DataFest boffins in the lab, mapping Brexit with QGIS

This blog serves as a little bit of a retrospective round up for everyone involved - and others who might be interested or who plan to put something on in future. DataFest ran over 4 days - a couple of weeks before the start of term - and each day was themed, as follows: Skills, Employability, Analysis, Presentation. We had a mix of practical, interactive and lecture-style content, with some great guest speakers, including:

• Emily Grossman - from the telly, obviously, but also a top scientist and data boffin.
• Joe Twyman - Head of Political and Social Research for Europe, Middle East and Africa at YouGov, and generally all-round good guy (he bought a round of drinks for EVERYONE at DataFest!).
• John Burn-Murdoch - from the Financial Times and one of the world's leading data journalists

In addition, we had input from Jackie Carter from the University of Manchester Q-Step team, Julie Scott-Jones from MMU, Chris Forde from Leeds and our good selves at the University of Sheffield. Even so, getting students to come to something called DataFest, of their own accord, during the summer, is a bit of a hard sell. But somehow, some way, they came and for four days we had 30 students unduly excited about data and methods. Hours-wise, it was like a full module for each student so they learned a lot, including dabbling with coding in R, as you can see below.

Not bad at all for someone completely new to R

We even had nice festival-style wristbands to help create a sense that this was a much-more-fun-than-it-sounds rock festival...

The main stage was absolutely jam-packed

The students who came were mainly second and third year undergraduates and all from social science backgrounds. I'd say we pitched it about right and every single person there really got stuck in and learned a lot - so thanks to all for making it a success. I also did a session on dataviz, as you can see below. At the end of the week, each group presented their findings on the question of 'what caused Brexit?' which was a theme for the week - well done to Group 2, who took away the winnings.

A few technical problems, but we won in the end

Really great presentation by Group 2 - top stuff

Finally, we actually managed to use our rooftop terrace because the weather was amazing all week. This provided a nice venue for eating the piles of pizza delivered during the week and for the final DataFest 'graduation' shot - you can see some of the people who attended below with their certificates.

Thanks guys, good luck for the future

As a bonus, when we were coming out of the computer lab one day, we bumped into some friendly University of Sheffield robots, so I'll leave you with this. They made DataFest 2016 that little bit more interesting.

Acknowledgements: the images here were mostly captured by our marketing and PR guru Sophie Hawley, and the original set-up was done by viral content whiz Hazel Moss. SMI Manager Ruth Bartles decided that 'DataFest' was catchier than 'Quantitative Methods Summer School' and also got all the logistics sorted. Dominos did the pizza... Simon Gallacher and Hannah Broad from our funders (The Nuffield Foundation) also visited on the last day, which was very nice of them.

SIMD 2016: Concentration and Contrasts

The 2016 version of the Scottish Index of Multiple Deprivation was released this week. It's very similar to indices used elsewhere in the UK and other parts of the world. The interactive mapping is great, so I'm not going to do anything on that. Instead, I looked at areas where there are contrasts or concentrations relating to the most and least deprived deciles. With a relative deprivation measure, some places always have to be at the bottom or top, but it's the location, level of spatial clustering, proximity and total number that I was interested in exploring. Here are some maps, starting with Glasgow and then going on a tour round the country. Red areas are in the most deprived 10% of Scotland and blue areas are in the 10% least deprived.

This is probably the bleakest map, in terms of concentrated deprivation

The famous Drumchapel/Bearsden cross-boundary contrast visible here

A bit more mixed towards the south east of Glagsgow's urban area

Alness and Invergordon - some of the more 'hidden' areas on most SIMD maps

Ayr is very notable for its north/south deprivation split

Edinburgh is often contrasted with Glasgow - and you can see why

Again, a somewhat mixed pattern in Falkirk and nearby

Hamilton and Motherwell are also quite mixed in a 'most-least' sense

Inverness - persistent pockets at both ends of the spectrum

A little bit of Fife - which is a bit split in this view

I did another 20 or so maps, in addition to the ones above. They can be seen on a separate Google Drive page if you are interested.

These are all in high-resolution - feel free to use them

I'm not planning to do any more SIMD mapping as there are enough other people looking at it. I just wanted to explore a little bit in relation to how the most and least deprived locations were spread across Scotland. Clearly, very little has changed (and I wouldn't have expected it to) but there have been some recent reports relating to the suburbanisation of poverty in Scotland, including this Conversation piece by Nick Bailey and Jonathan Minton.

Finally, a little SIMD 2016 map insight for you. According to my analysis, there are 31 data zones in the most deprived decile which have a neighbouring data zone in the least deprived decile. Conversely, there are 26 data zones in the least deprived decile with a neighbour in the most deprived decile. In a lot of these, the buildings in neighbouring areas are some way from each other, but in some examples (see below) neighbouring streets are in opposite deciles.

Some neighbouring 'most-least' areas in SIMD 2016

Notes: when you rank places some will always be at the top and some at the bottom, obviously. All other things being equal, however, we might not expect to see such large concentrations of deprivation in single locations. But we know that things are not equal and history and geography have an important role to play in the formation and persistence of these patterns. We can easily blame the politicians of today or those of the past, but I don't think it's that simple. These patterns have persisted for decades and in some cases even longer. For a really good guide to the SIMD2016, read this introduction from the Scottish Government. 

Mapping: I've coloured entire data zones but added in building outlines. Why? Well, the SIMD relates not only to household attributes (such as income) but also to area characteristics (e.g. crime) so the wider environment is important. I wanted to strike a kind of compromise here in relation to what's displayed. Also, the place names are Ordnance Survey open data so the hierarchy in terms of capitalisation and font size is taken directly from the original data. I used QGIS for the mapping. The building outlines are quite generalised so you can't pick out individual houses - this was a deliberate choice as I want to avoid giving the impression that this data says anything about individual people - it's about areas. For reference, there are a total of 6,976 data zones in Scotland. 

Research with QGIS, R and speaking to people

I recently led a piece of research for the Joseph Rowntree Foundation on disconnected neighbourhoods - basically, it looked at the UK's most deprived areas and how connected or disconnected they are to their wider cities in relation to jobs and housing. You can read the brief Findings or Full Report here. This post is just about the methods we used and some of the outputs. We used open source software QGIS and R for the analysis (led by Ruth Hamilton) and we also spoke to policymakers across the country (Rich Crisp and Ryan Powell).

You can see the full report here

We looked at those areas in England, Northern Ireland, Scotland and Wales that fell within the 20% most deprived on the national deprivation indices in each nation and then explored data relating to household moves and commuting (plus lots more). We updated and developed two area typologies to help us make sense of the data - and to see how things changed we produced riverplots (Sankey diagrams) in R (this was done by the briliant Ruth Hamilton).

Created with the Riverplot package in R

I also did a little bit myself with open source software, including updating a 'divided cities' type graphic I produced in the past - looking at the spatial split between most and least deprived parts of 13 cities across the UK, as you can see below. See Sheffield in particular for a very stark divide.

Red = most deprived, blue = least deprived

Our colleagues Rich Crisp and Ryan Powell then spoke with more than 140 policymakers in cities across the UK - another nice 'open source' method. You can read more about this in Chapter 6 of the report but the bottom line is that if we want things to change for the better then we need to take a different approach to urban policy - a more inclusive approach.

As I said, there are two typologies, and we then combined these into a matrix in an attempt to understand area types a bit better and then suggest possible policy responses that might make sense. What each category means is explained in the report but in the figure below you can probably make sense of what 'Gentrifier' areas are and what those labelled 'Disconnected' are.

The shaded areas might be a good policy focus to begin with

The point of this blog post, however, was just to highlight how useful and effective open source software now is, and can be, in real-world research. Advocates will already know this but many more have yet to make the leap so hopefully this will provide just a little bit of inspiration or motivation to do so.

We produced hundreds of maps for the project (too many for the report) so you can probably find one for your area in the online folders. Two examples - one for each typology - are shown below.

Residential typology map of Birmingham

Travel to work typology map for Glasgow

For more on the methods used to develop the typology, see the Annexes in the Full report.

How long is the coastline of Great Britain?

This is a bit of a long read, so if you really want to know the answer to the question in the title of this post, it's very simple: it depends upon how you measure it. Or, you could say that the coastline of the island of Great Britain is infinitely long. But this doesn't really help anyone who wants to walk or kayak or swim round this island, so I'll attempt to answer the question here. Take a look at the image below and you'll see that I've calculated the distance of the coastline round the island of Great Britain as 11,023 miles. 

Quite a lot of coastline for a small island

But hold on a minute, I also calculated it again and got an answer of 3,876 miles, as you can see below. What's going on here? Well, the first image is an extremely detailed digitised representation of the coastline of Great Britain and surrounding islands (bearing in mind 'detailed' is a relative concept). This first map is represented by 2,282,000 individual vertices which create the polygons you see in the image above. 

In the second map, only 0.1% of these vertices are retained, so the geographical features you see below are represnted by 2,282 individual vertices. You can't see much different between the two at the scale you view them at here but if you were trying to navigate your way into a harbour or sea loch on the west coast of Scotland, for example, it would make a big difference. Click the first image to enlarge it and then compare it to the next one and you will see some differences, but nothing too drastic.

The coastline length is a function of how you measure it

At this point, you might be thinking 'hasn't this got something to do with fractals and Benoit Mandelbrot?' - and you'd be right. He wrote a very famous paper in Science in 1967 on exactly this topic, entitled 'How long is the coast of Britain'. The answer is that there really is no definitive answer - it's all about how you measure it. But let's say you want to swim or kayak around the coastline of Great Britain and nearby islands. How far would you have to travel? I tried to calculate this based on a 1km distance from the shoreline and concluded that it could be done by covering fewer than 2,000 miles - even though the coastline seems to be a lot longer. After all, you wouldn't want to go in and out of every little cove and estuary.

Be my guest

I created a little gif based on different ways of measuring the British coastline, starting off with a file that included 100% of the vertices from my original Ordnance Survey map layer (see notes below for more on this). I then created files with fewer and fewer vertices retained, all the way down to a non-sensical shape which retained hardly any of the original points. This is what I got - at 2 seconds per frame (note '% of vertices retained' figure in each image):

Coastline length at different measurement scales

It's a bit difficult to see the difference between some of these images at this scale, so I also zoomed in to the west coast of Scotland to produce another little animation. This time, you can really see more of the difference between the layers I produced. The figures on the graphics indicate what percentage of the original vertices were retained in each case. Below, this, I have also provided a still image with different versions of the coast overlaid on top of each other, just to demonstrate the impact of reducing the number of vertices on the representation of the coastline, and hence its length.

This shows Morar, Mallaig and Loch Nevis 

Each line represents a different level of generalisation

I then decided to take a smaller island and extract the individual vertices (also known as nodes) that make up the shapes you see in the maps above. For this, I chose the Isle of Skye because it's one of the biggest British islands and the coast is highly irregular and indented. Using the version of the original shapefile where I retained 1% of the original vertices, Skye is represented by 772 individual nodes joined together to make a single polygon, as you can see below.

This produces a pretty good approximation of the coastline of Skye for most purposes. At this resolution, the coastline of Skye comes in at 330 miles (530km), compared to 456 miles (733km) at the original resolution. But of course we need to remember that if we had digitised around every single rock around the coastline the length would be nearly infinite. If you measured the coastline with a matchstick, for example, you'll get an extremely high value (and a sore back).

Skye represented with a polygon comprised of 772 vertices

Here's what this looks like when you show them one by one, in an animated gif - just to give you an idea of how it is plotted spatially. This is shown at 15ms per frame, so the dot fairly zooms around the coastline. All of this also gives you a little insight into how a GIS deals with geometry and what goes into the shapes that you see on your screen. It also helps explain why the very detailed, highly accurate spatial data files we can download from Ordnance Survey aren't always the most appropriate ones to use in small scale mapping. Or, maybe I just wanted to make another geogif, but either way I think I learned something.

A dot going round the Isle of Skye at 99,000 mph (forever)

So, how long is the coastline of Great Britain? Well, if you want to swim or kayak around all islands then you should think about training for a distance of around 2,000 miles and if you want to walk the coastline of Great Britain only then it's most likely going to be a bit more, or maybe a bit less - but that depends upon how you plan your route. Despite all the uncertainty, however, I think we can all agree that you'll need to go more than 1,024 miles.

Yes, this is Britain (kind of)

Last of all, I also did a little gif showing the 174 vertices of Great Britain when the file is massively reduced - so I'll end with this.

Another one, just for fun

Notes: I used the OS OpenData Boundary Line product for the coastline. This was a polyline file so I converted it to a polygon and then generalised it several times using the Visvalingam algorithm in mapshaper. Contains OS data © Crown copyright and database right 2015. You'll see if you search online that my measurements are close to those of others - so I'm at least as right or wrong as some people. If you're interested, you might want to look up the coastline paradox as well and, of course, Lewis Fry Richardson. Other big British islands? After the island of Great Britain, it's Lewis and Harris at 741 miles of coastline (1,193km), the mainland of Shetland at 692 miles (1,113km), Skye at 456 miles (733km) and North Uist at 334 miles (537km). Remember that this refers to coastline length and not land area.

Urban road network data for 80 world cities

A recent study published in Nature's Scientific Data caught my eye recently. In the paper, published online on 21 June 2016, the authors describe a method for taking OpenStreetMap (OSM) data and producing usable, toplogically accurate network data from it. It covers 80 of the biggest cities across the world, from Tokyo to Medellin. What caught my eye in particular is that they shared all the data on Figshare, and you can download it in chunks or the whole lot at once (2.15GB). I had a little play with it and made some night time/from space views of the network data, just for fun. Guess the cities below (helpful clues included)... and then tell me which is the odd one out.

Clue: it's not Bognor Regis 

Clue: I have been there (that's not very helpful, sorry) 

Clue: lots of people live here

Clue: rhymes with Few Dork

Clue: lots and lots and lots of people live here

Back on topic now - here's what the study looks like if you haven't already clicked - the authors also provide loads of useful resources in the paper. They also describe how they were able to develop a topologically accurate street network using their GIS-based protocol.

Take a look at the study - it's great

The 80 cities featured

Citation: Karduni,A., Kermanshah, A., and Derrible, S., 2016, "A protocol to convert spatial polyline data to network formats and applications to world urban road networks", Scientific Data, 3:160046, Available at http://www.nature.com/articles/sdata201646

From CartoDB to CARTO - the future of interactive mapping?

I've been using CartoDB (now CARTO) for a few years for interactive mapping - and have always loved what it can do - from basic mapping to much more complex analysis. Now, with the re-brand as CARTO and the advanced analytical tools available through the new Builder interface it's on a new level. So, credit where credit's due - I thought I'd do a short piece on this now to give my take on the new interface. But first, here's a little gif of me playing around with some commute data - which you can also download yourself if you want to. The dataset was used as part of a project I've been working on with Garrett Nelson - but hopefully more on that in future.

I'm just playing around turning things on and off here

If you've used the old CartoDB interface, the new Builder one might be a bit confusing at first - though you may not actually be able to get access to it yet. But once you have played around with it for a few minutes it soon becomes pretty intuitive. I uploaded a sub-set of commute flow lines for Minnesota and Wisconsin and then decided to add widgets so that I could filter the data using line distance, FIPS codes and commute volumes - as you should be able to see in the larger image below.

Click to enlarge - change the data view by using tools on right

This is very much just a little data sample, but if you want to play around with the interactive CARTO map you can see it here. It's not very pretty and the origins and destinations don't have place names right now - only FIPS codes - but the principle is the same. The Widget interface takes a little bit of getting used to as well, but is really easy to use once you've figured out what's what. See below for a screenshot.

You can add widgets for different data types

Any negatives to report? For me, not now. I'm just enjoying the enhanced analytical tools at hand. But if I was being greedy... I'm not massively keen on the default legends, there doesn't appear to be an 'addition' blend mode and the snap alignment of shapes in the old map editor has me a little flummoxed, but these are minor grumbles.

I'm not getting paid to promote this and I don't know anyone at CARTO - honest - I just think they have produced something that works brilliantly, is simple yet powerful and allows us to manipulate, analyse and share our data in new ways. There are other tools out there but the new Builder, for me, takes things to the next level for a mass audience. To answer the question in the title of the blog: is CARTO the future of interactive mapping, then? Not the future, but probably a very big part of it.

Notes: really, they didn't pay me. Data used are from the American Community Survey. I wrote a working paper on it already. I also blogged about it on my old blog. Finally, if you're one of the few people in the world not to have seen it, Mark Evans created this beautiful site with the same data.