Saturday 30 April 2016

A bit about image formats

This may be the most boring blog post you ever read, but I think some people might find it useful. Also, given the title of my blog, it's about time I did something on image formats. Let's imagine you are making some images (maps, maybe, but it could be any kind of graphic) and you come to the end of the project and want to export the final images for use elsewhere for sharing and so on. Do you default to JPEG? Hopefully not. Do you use BMP? I really hope not. Do you default to PNG? (I do most of the time), or are you required to use EPS or TIF? These choices matter and that's what this is about. Take a look at the JPEG image below - click to enlarge it and you'll see that around the edge of the lettering it looks slighly fuzzy or shadowy. That's because the image has been compressed but also because during the compression some of the original image quality has been lost (that's why they say a JPEG is 'lossy').

The jpg format is not very good with big areas of the same colour

Now compare the following maps, all in JPEG format, which I've exported at 75dpi, 150dpi, 300dpi and 600 dpi. To see these and the other images below in their native form, take a look in the Google Drive folder I put them in. 

This is 0.145MB 
This is 0.423MB (2.92 times bigger than the one before)

This is 1.166MB (2.76 times bigger)

This is 3.003MB (2.58 times bigger)

If there were no compression, each doubling in dpi would lead to a file size four times bigger. As you can see, the rate of increase in file size actually declines with each doubling of dpi. But you shouldn't use JPEG for maps like this anyway. So let's look at PNG instead. PNG is also a compressed file format but unlike JPEG it is lossless. It's kind of a non-proprietary version of the GIF file format (which is patented). There are lots of good things about it, including the fact that you can have a transparent background. Here is the PNG equivalent of the above set, with file sizes shown again - as you'll see the file size is bigger than JPEG (without being massive) but if you look at them in their native resolution in the Google Drive folder you'll see that there is no image quality loss around the edge of block colour shapes, particularly the writing in the top left.

This is 0.463MB

This is 1.025MB (2.21 times bigger than the one before)

This is 2.238MB (2.18 times bigger than the one before)

This is 5.353MB (2.39 times bigger than the one before)

So, with PNG you get a bigger file size but the quality is better because it's lossless - and that's why it's normally the most appropriate format for images of this kind on the web. If you need higher quality - for publishing for example - then EPS is often required. Sometimes, it's TIF. TIF is, like JPEG and PNG, a raster graphic format and when you export from QGIS as I have done here there is no compression (that's another story). This means that doubling the dpi leads to a quadrupling in file size - you can see this in the Google Drive where I've taken the image above and exported it again but in TIF formats. 

Uncompressed 75dpi TIF - 2.122MB

150dpi this time - 8.492MB (4 times bigger)

I haven't uploaded the other two TIFs here as they get pretty big quite quickly. The 600dpi file is nearly 136MB. But that's where TIF compression can come in handy. The two images below are 300dpi and 600dpi TIFs, but I've saved them using LZW compression (one of the options in IrfanView - my free tool of choice but there are other ways, like GIMP) and they are much smaller. You can see the full versions of these in the Google Drive folder.

This is only 3.908MB (the original was 33.975MB)

This is only 10.314MB - 13x smaller than the uncompressed TIF

Okay, so where were we? Oh yes, don't make a lovely map and then ruin it by exporting to BMP. Also, don't make loads of maps for a book or other project without thinking about the image format the end user will need. I keep making this mistake, somehow! I haven't even mentioned Abraham Lempel, Jacob Ziv and Terry Welch, but I think I ran out of goodwill quite some time ago so I'll just end by saying that their LZW algorithm is what makes the last two images much smaller than they were originally.

With this post I've reached a new low. I promise something more exciting next time. But if this saves just one person from having to re-produce a map because they got the image format right first time then it's been worth it. Last of all, here is one more extract from a 600dpi JPEG where you can really see the fuzzy effect you get.

The fuzzy effect is more noticeable in blocks of red colour

Other stuff JPG or JPEG? TIF or TIFF? It doesn't matter. If you manually change the .jpg file extension to .jpeg you'll see that it still opens and looks the same. Also with .tif vs .tiff. JPEGs are good for where the colour values change a lot in a short space - like in photographs of people or places. Following tweets from Rob Pattay, I thought I'd add here that you can also reduce PNG size with a bit more post-export compression. I used IrfanView to reduce the 600dpi PNG from 5.23MB to 4.85MB (about 12% less). Also, you might notice that when you click on the TIF images here in blogger they end up displaying as JPEGs.

Sunday 17 April 2016

Where is the Northern Powerhouse?

Depending upon how you look at it or who you've been listening to, the 'Northern Powerhouse' is a 'programme', a 'proposal', an 'idea', an 'area', a 'thing' or - for some - a bit of a dog's breakfast. It's definitely something, but where is it? That's what I set to find out and this is my attempt to understand it. First things first though. The government officially don't define its area, as you can see from the Hansard extract below that says 'The exact extent of the North in the context of the Northern Powerhouse is not prescribed by the Government'.

This seems pretty clear-cut - original source

So, obviously it's just the north of England, right? Wrong. Look at the gif below and see if you spot anywhere that isn't in England. Okay, so North Wales is in and that makes sense. The cross-border economic connections between Cheshire and Flintshire/Wrexham and beyond are very important and long-standing so shouldn't be ignored. Now it's time to look at some maps.

It comes from this lovely video

I attempted to turn the list of areas above into a map, which you can see below. It's a pretty wide definition but of course there is value in not being too specific about what is in or out as you can then be a bit flexible about things round the edges. One example of this would be the parts of Sheffield City Region that are actually in Derbyshire so not in the areas listed in the gif above. But of course this looks pretty big and the Northern Powerhouse is often put into an English, urban context. So the next map adds urban areas into the mix. A third map adds in the parts of Sheffield City Region I just mentioned. As you can see, this then leaves High Peak encircled by the Northern Powerhouse but I don't think that is deliberate.

This is one way of looking at it

This helps highlight the core urban areas (in England)

Now with added Sheffield City Region areas - High Peak left out?

What about connections? An important part of all this agglomeration and city productivity talk is how places connect together to help produce stronger economic growth. That's a big part of why much of the Northern Powerhouse work so far has centred on transport, and why they used my original flow lines map on the front of one of the reports. So I've also attempted to map those here again, in a more colourful way. The final map shows this. I've coloured some of the major urban areas differently and used darker shading for 36 local authorities that seem to me to be at the heart of the Northern Powerhouse idea.

This gives you an idea of how everything connects

One other thing to say is that it's quite interesting if you look at the first mentions of the Northern Powerhouse in parliament. This is what the Chancellor said on 24th June 2014:

  • "Today we have the very welcome news that Abu Dhabi will be investing £1 billion in building new houses in Manchester. That is a step towards it becoming the northern powerhouse I want to see, and it is a £1 billion vote of confidence in our long-term economic plan."

Note the use of 'it' shortly after 'Manchester'. To sum up, then, where is the Northern Powerhouse? My view is that the Northern Powerhouse is:

  • In the North of England
  • A bit in North Wales
  • Has the biggest cities at the heart of it
  • Inherently fuzzy (on purpose)

Not exactly lightning bolt stuff then. But I feel like I now know a bit more, which is always a good thing. Anyway, it seems that we might already have hit Peak Powerhouse.

I wonder what the little bumps were in 2008

Notes: Neil Lee at the LSE has written a good policy paper on the Northern Powerhouse that's well worth a read. Anyone with a decent memory will remember the previous incarnation of this, in Northern Way form, still available in small part online as archived content. I tweeted a version of the last map earlier in the week so I'm wrapping it up here, after a bit of further thinking and to and fro with Buzzfeed star Peter Matthews and expat Westphalian/north west near-native and spatial planning guru Andreas SchulzeBäing. If  you're really keen, go back and read the original Northern Way stuff from 2004. It's still available via the waybackmachine internet archive.

Tuesday 5 April 2016

Deprivation and affluence, cheek-by-jowl

As part of some work I'm doing for a 'fracturing societies' seminar this summer, I've been thinking about the spatial manifestation of inequality in England. Some places will always be rich and some always poor - if we're defining it relatively - but the location of these places, their characteristics and their wider neighbourhoods may all differ significantly. That's one of the reasons I made a 10% most/least deprived interactive map last week and shared it online.

You can explore the map online here

This then raised the question in my mind of how many of the 10% most deprived areas in England neighbour areas among the 10% least deprived. In part, this was inspired by the famous image of São Paulo rich and poor taken by Tuca Vieira in 2007 and widely circulated since then (see below). The photo shows the favela of Paraisópolis on the left, right next to the upscale Morumbi neighbourhood over the wall. You can't really imagine a more stark divide. For more on this, see Teresa Caldeira's article from LSE Cities.

Credit: Tuca Vieira, 2007 - original here

In England, things are not quite as extreme as this example, but there is quite significant socio-spatial inequality nonetheless, as documented over the past few years by many observers, including the SASI research group here in Sheffield. It matters for many reasons, but life chances, health, opportunity and education are just a few of the major advantages/disadvantages experienced by people on either side of the rich/poor divide. 

The answer to my original question of how many of the 10% most deprived areas in England have a neighbouring area in the 10% least deprived is, I found, 75. I set about investigating further and here's what I found. You can see all the maps here but below I've provided a few examples to illustrate my points.

The most deprived area in England with a 10% least deprived neighbour is in Birmingham. On each map I've added in the LSOA name at the top, in addition to the LSOA ranks of the areas in question - as you can see here the one in Birmingham is ranked 38 out of 32,844 so it's right at the very most deprived end of the spectrum. But you can also see that the relationship with the neighouring area isn't like the example from São Paulo above. It's just more a feature of the way the boundaries are drawn. This is the case in quite a lot of the areas, but not all.

Just Google the LSOA name to see an interactive map

One area that's a bit different and where the most and least deprived are almost literally next door is in Gateshead. On one side of the road we have one of the most deprived areas in England right next to one of the least deprived over the other side. Mind you, the location of people within these is still not exactly of the 'cheek-by-jowl' category we see in São Paulo. Most local authorities don't have any areas with 10% most/least neighbours, but Kettering has three, one of which you can see below.

Not many most/least deprived are like this
This is partly an artefact of how the boundaries are drawn

More striking examples of areas where people live next door but in opposite ends of the deprivation spectrum can be found in Leicester, Norwich, Nottingham and Swindon - see below.

You can see this contrast on Google Maps

The Indices of Deprivation 2015 which I used to map the 10% most and least deprived areas do not measure affluence but the areas in the least deprived 10% are significantly wealthier, healthier and have lower crime rates - among many other differences - so it's a pretty reasonable proxy for affluence in many respects. 

All of this raises the vexed 'so what' question once more. Is spatial inequality worse when you can see it? Is it worse, or somehow more grotesque, when rich and poor live side-by-side? Wasn't that what the 'mixed communities' policies of the 2000s were all about? Well, sort of but not really. 

There is a mix of things going on here to create these patterns - sometimes it's housebuilding on brownfield land, sometimes there are physical or artificial features dividing neighbouring areas such as rivers, roads or railway lines, sometimes it's a boundary effect and the people in households aren't really neighbours. Either way, I find it very interesting and troubling at the same time. The plan is to keep working on this. I know it's not exactly São Paulo levels of contrast, but the big gaps between places - and particularly the life chances of young people - are really important.

Click on the image below to see all the maps.

Here are all 75 '10% most' areas with a '10% least' neighbour