Scraping 241 UK council planning portals – 2.6M decisions so far

Re:Scraping 241 UK council planning portals – 2.6M decisions... by mebkorea on Tuesday April 28, 2026 @05:19AM (#47932142)

Cheers, appreciate the feedback. The architect/consultant precedent angle is interesting and a couple of other commenters have already nudged me in similar directions. Tbh... you're likely right that the strongest commercial play isn't B2C £19 reports, it's giving someone fighting a contested case the national pattern across 15 similar pub conversions, the appeal outcomes, what stuck and what didn't. That's a very different product to what I have now but the data supports it. On the T&Cs/legal stuff... I'm not going to pretend I have perfect clarity on it. The position I'd defend is that the data is statutorily public, councils are required by law to publish it, I respect rate limits, and I'm aggregating not republishing in bulk. But there is this grey area between data being public to view and being usable for a commercial product, and I haven't fully nailed it down.

[ Parent | on HN ]

Re:Scraping 241 UK council planning portals – 2.6M decisions... by lifeisstillgood on Tuesday April 28, 2026 @05:32AM (#47932233)

I agree on the “public” data issue - I spent a long time campaigning for better FOSS / data access in government and there are some great people pushing in and outside local and central gov.

But it’s a big mindset chnage (one that will benefit the whole country) but it’s slow.

I think the “push for public policy improvements” angle if genuine will get you a lot more respect and kudos when things get sticky. Good luck

[ Parent | on HN ]

Re:Scraping 241 UK council planning portals – 2.6M decisions... by mebkorea on Tuesday April 28, 2026 @06:07AM (#47932437)

Cheers, and things get sticky isn't lost on me, especially after some of the feedback. Seems all the feedback is converging on the same direction namely open-source the data layer, lead with public-interest, let any commercial product sit on top rather than be the thing. Haven't fully thought it through but maybe this is where it should go. Appreciate the gov-data campaigning context too.

[ Parent | on HN ]

Re:Scraping 241 UK council planning portals – 2.6M decisions... by lifeisstillgood on Tuesday April 28, 2026 @08:13AM (#47933415)

Yes … but…

No-one has figured out how to make money off open source (while sticking to the basic principles. Jeff Bezos makes a fortune off of it)

Most people who open source their code that I have known and still wanted to be paid / recognised for their effort have always been disappointed

Can I suggest you mentally put the work you have done to date into a box marked “the past”, open the data, start yourself as part of the community trying to make government code and data open, and sell your skills - the old “consultancy paying the bills” approach

Trying to make cash off public data will just confuse the message, and start to build resentment. Make a clean and clear statement, Sell your services on top. Expand to other forms of data scraping in government.

It’s a tough road - good luck

[ Parent | on HN ]

Re:Scraping 241 UK council planning portals – 2.6M decisions... by mebkorea on Tuesday April 28, 2026 @09:03AM (#47933996)

Thanks! Yup,... the freemium + paid analysis route is the obvious next step from where I am. You're right that my commercial intent is muddying the message. The sell skills, not data/ consultancy angle isn't one I'd seriously considered. Sounds pretty good, need to sit with it.

A few weeks ago, I'd have said the SaaS product is the play, but now I'm not so sure. Cheers for taking the time.

[ Parent | on HN ]

Re:Scraping 241 UK council planning portals – 2.6M decisions... by PhilipV on Wednesday April 29, 2026 @01:01PM (#47951182)

I would say that you should consider checking whether the councils and/or these planning portals operate under an Open Government Licence : https://en.wikipedia.org/wiki/Open_Government_Licence . These in theory permit commercial use.

I would also see that there is a strong case to say that it is aggregation. Something that would help is redirecting users to the original source content whenever you are referencing it within your search.

[ Parent | on HN ]