dekayed

We already have these in the industry. They're Staff+ Engineers and Architects. It's generally the norm to not be cranking out code at this level, but they make sure everyone is moving in the right direction, assisting managers, and mentoring juniors.

I start with RAISE as well, but follow up with POUTY to get the Y if I blank on the first round.

I believe the US does already have this (went into effect last year, maybe). It applies to corporations which have a revenue over $1B.

Airflow is still a very good option, though there are more competitors on the market. Two options are:

Prefect - https://www.prefect.io/

Flyte - https://flyte.org/

Could you go more into this? How did you go about the evaluation and what metrics were you using? How did you capture the 'developers just hated it' in a way that leadership understood?

I'm not disputing the outcome, just feel like it would be useful for others in a similar situation.

I'd recommend taking a look at Papers We Love

https://github.com/papers-we-love/papers-we-love

I'm assuming that you run Kafka within AWS. Much of the hardware requirements/suggestions I've seen for Kafka are all for non-virtualized environments. If you can get into it, could you share some details...

- What is the size of your Kafka cluster

- What instances types do you use?

- Do you use EBS or use ephemeral storage?

- How much do you over-provision to deal with instance loss?

- Any other gotchas/considerations?

Thanks!