Do you have any advice for setting up a test Kafka environment? I'd love to be able setup a lightweight in-memory Kafka for unit-y tests without going through the whole Docker compose rigamarole.

Depending on complexity and specialisation of your use that's what may work for unit testing:

1) Abstract real library with tech-agnostic interface to remove as much tech details as possible and focus on behaviour.

2) Create simplest in-memory dummy implementation for that interface, basically a mock.

3) Write a test spec for that interface behaviour (does xxx on yyy, returns zzz otherwise) and run it agains both real wrapped interface & mock to be sure they're consistent.

4) Make code to load mock in local testing env

5) Use real kafka when running it on a testing pipeline

Same way you could abstract various things - e.g. once I just abstracted proprietary services and locally flushed everything into local Postgres. You can emulate multiple services there - e.g. database, somewhat-usable queue, etc. And get some extra control nice for tests (e.g. make it to throw errors on your wish or add timeout to simulate net delays). That saved a lot of time for local development for me, without dropping quality.

Check out the Kafka support in Testcontainers: https://www.testcontainers.org/modules/kafka/. It is container-based, but with a very simple-to-use API for spinning up Kafka in tests.

If you truly want something embedded, Debezium's KafkaCluster could be interesting: https://github.com/debezium/debezium/blob/master/debezium-co....

It's used four our own tests of Debezium, but I'm aware of several external users. It spins up AK and ZK embedded. Disclaimer: I work on Debezium

If you're using Kafka Streams it provides a TopologyTestRunner.