After rent and utilities, council tax is often the greatest expense in keeping a roof over my head. When I was last looking for a flat in Edinburgh, I wanted to see a map of council tax rates across the city to help me choose an affordable place to live. Unfortunately, the City of Edinburgh does not provide this resource publicly, and does not make the data open and accessible for anyone to create their own.
The Scottish Assessors Association (SAA) maintains a web portal which aggregates nearly 3 million council rates into a public document called the council tax valuation list. The portal provides record-by-record access, but no bulk access. The SAA apparently wants to profit from public records by charging £10,000 for a copy of the data set! I don’t have £10,000 to spend on data that should be free. Instead I want to recreate the entire list from pieces of the SAA’s website to create my own copy and use it to make my own map and analyze trends in the data.
You might point out that this is a technical solution to a political problem. Instead should I focus on persuading people to open up this data? Sadly, I don’t have time to play politics with local government. But I do want to show what is possible with the data. Perhaps a working demo might be more persuasive than ideas. I am planning to set up a web crawler to periodically download the contents of the SAA’s site to convert the pages of council tax data into a database. Heritrix is the crawler maintained and used by the Internet Archive.
We can enrich the database with geospatial information by looking up the postcode and street address in a geocoding service like Google Maps or OpenStreetMap to get coordinates for the location. I demonstrated the principle by comparing the SAA’s page for Leith Walk council tax rates to a prepared file containing the same information in CSV format. I used a program called html2csv to convert the downloaded page.