robhawkes

The link shows a photo of my grandad with the original R2-D2 he built for the first Star Wars film…

Jack was a master sheet metal worker, and was roped into using his skills on an obscure project to turn the sketches for the droid into something that could be constructed out of aluminium sheets.

I always knew he'd helped make part of R2-D2. However I only found this photo in recent years and have since learned that he was actually instrumental to the construction of the entire droid, particularly with working out how to machine the complex shapes like the dome and legs out of single sheets of aluminium.

He didn't just build one either, he ended up constructing a handful of the droids for various uses in the film. I still have no idea how he did it, especially without modern software and computer-controlled machining.

Unfortunately I never got to know him as he died when I was a baby, though I have a feeling we share a lot in common.

I also have his old Dragon 32 computer that he was using to learn programming in the 1980s, with reams of hand-written code that still works on the computer.

Those are NESO (system operator) grid boundaries. The colours represent the forecast flow of energy over each boundary in relation to the capacity of each boundary. Green means lots of extra capacity, red means over capacity. When a boundary exceed capacity it's likely that this constraint will result in wind farms being turned off to reduce output "behind" the constraint.

The black dots are wind farms and other power assets that don't have any generation data. This is usually because they aren't connected to the transmission system, not that they aren't actually outputting. Or to put it another way, I only have data on power assets connected directly to the main transmission grid.

Thanks! The ultimate fix is to finish upgrading the aging grid. There are other things that can improve the situation however, such as building wind farms away from these constraints, storage (but these can sometimes exacerbate constraints), demand flexibility (eg. place demand above constraints), zonal/regional pricing, and probably more I can't remember off the top of my head.

The demand flexibility aspect is already being explored: https://www.gov.uk/government/news/government-to-make-plug-i...

Yup! Looks like it'll be some form of regional demand flexibility, similar to what suppliers like Octopus Energy (disclaimer: my employer) and others have experimented with over the past few years.

https://www.gov.uk/government/news/government-to-make-plug-i...

Not at all, my stuff is very niche and the other websites and dashboards do a much better job at filling in the higher-level context.

Indeed! That's including available wind generation that was curtailed (not used) due to transmission constraints. So it's the actual output plus the amount of output that was "lost" because we had to switch off some wind farms, even though the wind was there to generate more.

I shed a little more light on the context via my map which you can also find linked on the page: https://renewables-map.robinhawkes.com

Thanks! Always a surprise and a pleasure seeing my stuff on here

Oh cool, that's my website! Let me know if you have any questions about it and I'll do my best to answer them.

Thanks!

That's correct, it's using data from Open Infrastructure Map which is itself based on OpenStreetMap data. That's a good idea to zoom in further to show them off a bit more.

Oh wow that's a blast from the past, feels like a lifetime ago! And thank you.

Ah yeah the FAQ is very out of date and in need of a full rewrite. I'll get to that eventually. And you're right, I'll make sure to add some sort of glossary and perhaps also explain things better in-situ.

Thanks! I would love to make this something that works beyond GB though right now that would require a fundamental change to the way things are set up. It's on my list though, so one day.

I actually have plans to include Ireland and Northern Ireland once I get the GB side nailed down. The data seems to be mostly available though I'd have to really think about how to make it work well on the app as it'd be a pretty fundamental change to how things currently work (assumes a single market).

I'll make sure to get the about page updated soon, it's been a bit neglected with being so focussed on the recent upgrades. Definitely a lot I can explain better, and also ways that I can solve that with better UX in general.

Yup, that's exactly it! When you zoom in you get to see the wind farms and wind turbines using data from the amazing Open Infrastructure Map [0]. I also show the cables for the offshore wind farms.

[0] https://openinframap.org/

Tech stack is a mix of things, but the big-ticket items:

- Dagster for data pipeline

- Parquet files for data format

- Cloudflare R2 for data storage

- DuckDB for data processing

- NodeJS for API

- NextJS + React for the app

- MapLibre for the base map functionality

- Deck.gl for the complex map layer (eg. wind particles)

- WeatherLayers (+ GFS) for the wind particles

- Cloudflare as a caching layer

The main difficulty was learning about the underlying data and industry nuances. I've been working on this map the past few years and I'm still learning new things that force me to change my approach. It's an incredibly complex domain and it's part of the reason I made this, to try and make this complex data a bit more digestible.

The lines are sort of arbitrary in a geographic sense as in reality they are defined based on the intersection with the schematic version of the transmission network [0]. Though yeah, that specific example is an odd one and I'd be interested to understand more about why it was placed in one boundary and not the other.

[0] https://www.neso.energy/data-portal/day-ahead-constraint-flo...

No worries. The Detailed System Prices dataset is lagged by a couple hours so try going back in time.

The simplest answer I can give is that assets place bids and a volume of energy that they are willing to turn down if the system operator needs to. Those bids are either positive or negative in value and this depends a lot on the type of asset, for example wind assets usually bid negative (ie. we pay them to turn down) while gas assets usually bid positive (ie. they pay us to turn down). The reason for that is a lot to do with complexities of the market and also the cost of running that assets, the cost of fuels, etc.

Yup! I have two ways to do it right now, the first is to click the plug-looking button on the map and this'll switch into curtailment mode to filter the assets that are currently being turned down due to transmission constraints, as well as the assets being turned-up elsewhere to balance things out. Also if you click on a curtailed assets you'll see the curtailment as a yellow area on the charts.

The other method is to visit my dedicated page [0] for tracking aggregate curtailment and turn-up, plus the costs for this.

[0] https://renewables-map.robinhawkes.com/curtailment

The most time and energy has been getting my head around the source data [0] and industry-specific nuances.

In terms of stack I have a self-hosted Dagster [1] data pipeline that periodically dumps the data onto Cloudflare R2 as parquet files. I then have a self-hosted NodeJS API that uses DuckDB to crunch the raw data and output everything you see on the map.

[0] Mostly from https://bmrs.elexon.co.uk/ [1] https://dagster.io/

Hey! And thank you

1) Absolutely agree. The current approach for the boundaries is a quick hack until I can implement something more sophisticated. Safe to say an update is already in the works that adds a MW value and more insight into the state of each boundary (and is also more accurate in general)

2) "Please don't tell me that we are paying batteries to _not_ export" – it's actually the opposite, the batteries paid to not export (at least today). You can dig into this yourself via the Detailed System Prices dataset [0] and looking at one of the batteries on the sell stack (eg. KILSB-5)

[0] https://bmrs.elexon.co.uk/detailed-system-prices

Thank you

Some solar power assets are being tracked though unfortunately (for now) I'm only able to display data for assets that report on the balancing system in Great Britain. This means most solar has no data that I can easily access.

I hope to work around this soon.

Semi-related: I also have a page where you can track the wind records as they happen

https://renewables-map.robinhawkes.com/records

Definitely wind roses planned, I posted about them on my LinkedIn page if anyone is curious (linked on the main map).

That's a great point about the controls. Originally the map only showed wind farms and as I've expanded the features the control system (and really the entire UX) hasn't kept up. I need to sit down and have a proper think about how best to design the controls and documentation.

The squiggly ones across the country? Those are the National Energy System Operator (NESO) transmission boundaries for Great Britain. They represent areas of the country that are tracked for the purposes of managing constraints, which is when the transmission network over the boundary can't handle the amount of power flowing over it. When they are overloaded (or rather, before that happens) NESO will step in and "balance" the various zones by turning up or down generators.

Oh wow, I made this! I didn't expect to see my map on here

Let me know if you have any questions or feedback and I'll do my best to respond to them.