radus

It's simple, just tell it not to hallucinate.

Okay, but why?

I'm working on adding features to the snakemake aws batch executor plugin. The existing plugin supports execution on AWS Batch by dynamically creating job definitions based on rule resource configuration, but was missing support for features like using different containers for different rules, consumable resources, secrets, etc. Two approaches:

1) https://github.com/radusuciu/snakemake-executor-plugin-aws-b... (my fork). Just add the features to the batch job building code 2) https://github.com/radusuciu/snakemake-executor-plugin-aws-b.... This is more experimental and not yet fully working. I wanted to try a few things. a) can we rely on existing job definitions (managed through IaC instead). b) can we implement a fire-and-forget model where the main snakemake process runs on Batch as well? c) Can we slim down the snakemake container by stripping off unnecessary features.

Polars has a much more consistent API, give it a shot.

Regarding your plotting question: use seaborn when you can, but you’ll still need to know matplotlib.

> But are you really going to repair it?

Yes

Polars and duckdb interoperate nicely and can enable this flexibility

I like using EFK (ElasticSearch-Fluent-Kibana) for this

I guess then you've got to abstract one level further and formulate your advice as an allegory.

Example: you want to set your local docker context to the production environment, so that when you type `docker system prune --volumes` you delete your production data.

The answer is “it depends”

Quick critique: module contains functions with many parameters, many branches, deep nesting, and multiple return points.

docker swarm is also a decent solution if you do need to distribute some workloads, while still using a docker compose file with a few extra tweaks. I use this to distribute compute intensive jobs across a few servers and it pretty much just works at this scale. The sharp edges I've come across are related to differences between the compose file versions supported by compose and swarm. Swarm continues to use Compose file version 3 which was used by Compose V1 [1].

1: https://docs.docker.com/engine/swarm/stack-deploy/

They were defending rocks. They had not uncovered evidence of native martian life.

See more discussion here: https://scifi.stackexchange.com/questions/160959/is-or-was-t...

You've explained your value proposition vs. celery, but I'm curious if you also see Hatchet as an alternative to Nextflow/Snakemake which are commonly used in bioinformatics.

Example: you use a large plotting library, but it's only needed on some pages. Chunking allows you to selectively load that code only where it's needed.

My approach for this is to use multi-stage dockerfiles. Build tools live and operate in a build stage, and you copy the /venv into your runtime stage. I think this same approach should work with uv so long as it doesn't bundle itself into the venv.

This is great! I’ve been waiting a long time for this and it seems like more debug features are in the pipeline too: https://github.com/docker/buildx/issues/1104

Docs: https://docs.docker.com/engine/reference/commandline/buildx_...

This is my experience as well :/

The term translational is frequently used to describe basic research (eg. with cells or mice) that has applications to human health - findings in lab that make it to the clinic.

I’m not sure that dropping flaming batteries from the sky will be very popular with those on the ground.

Great tool, I use it with this alias:

  alias dive='docker run -ti --rm -v /var/run/docker.sock:/var/run/docker.sock wagoodman/dive'

(as suggested in project the README)