aschleck

Atherton closed their station in 2020 and, through a great twist of irony, it will become a museum for the line that's still alive next to it: https://inmenlo.com/2025/06/16/community-interest-meeting-on...

It's been a while since I thought about this but isn't the reason providers advertise only 3.2tbps because that's the limit of a single node's connection to the IB network? DGX is spec'ed to pair each H100 with a Connect-X 7 NIC and those cap out at 400gbps. 8 gpus * 400gbps / gpu = 3.2tbps.

Quiz 2 is confusingly worded but is, iiuc, referring to intranode GPU connections rather than internode networking.

Bazel makes easy things hard and impossible things possible

I had this problem and complained to AT&T support. I know it doesn't make sense to replace the modem because this is a software, not hardware, issue but they very quickly offered to replace the modem and now my DNS isn't getting hijacked. Would recommend trying it!

This is both a good thing and a bad thing, but Pulumi is way more flexible than Terraform. I wanted to have a cloud-provider-specific submodule that created resources (like EKS and GKE) that then exported output values (think kubeconfig), and then I wanted the parent module to pass those in as inputs to a cloud-provider-independent submodule. Terraform couldn't do it without needing to duplicate a ton of code, or without something heinous like Terragrunt (not sure it even would have worked.) Pulumi makes it trivial and in a language I like writing.

Additionally, our applications consume our cloud configuration (eg something that launches pods on heterogenous GPUs needs to know which clusters support which GPUs, our colo cluster has H100s but our Google cluster has A100s etc.) Writing in the same language in the same monorepo makes it very easy to share that state.

I found that using "helm template" to convert every Helm chart into yaml, and then using Pulumi to track changes and update my clusters (with Python transformation functions to get per-cluster configuration) made my life so much better than using Helm. Watching Pulumi or Terraform watch Helm watch Kubernetes update a deployment felt pointlessly complicated.

1979 16 bit flops on an H100 is with sparsity. See footnote 2 on https://www.nvidia.com/en-us/data-center/h100/. You should be halving it for non-sparse flops.

It's worth noting that just because an H100 has a higher flops number doesn't mean your program is actually hitting that number of flops. Modern TPUs are surprisingly competitive with Nvidia on a perf/$ metric, if you're doing cloud ML they are absolutely worth a look. We have been keeping costs down by racking our own GPUs but TPUs are so cost effective that we need to do some thinking about changing our approach.

I'm not certain but I think part of this is that XLA (for example) is a mountain of chip-specific optimizations between your code and the actual operations. So comparing your throughput between GPU and TPU is not just flops-to-flops.

Cool stuff! Do you have any thoughts about how this compares to https://github.com/fabianlindfors/reshape?

Luiz called large meetings "all-heads" instead of "all-hands" meetings. "Why? Because we're not sailors!"

I never got to interact with him directly but he was a leader worth following. This is a sad day.

Ahh this is a good idea, my friend in fact has been asking me to add it forever. I've been hesitating since the data seems so sparse but I think you're both right. Thank you for the feedback!

A website making trail data from OpenStreetMap easily browsable: https://trailcatalog.org/

Myself and a friend have been working on this for quite a while. It's frustrating because it has come so far (showing trail data worldwide interactively on a tiny budget is challenging), and yet it is so far from being something super useful like AllTrails due to low data quality and a lack of relevant features (photos, reviews, etc etc.)

Because they've limited the dynamism of Python so you can't do as much crazy runtime nonsense, their (vaporware) tech stack is able to (theoretically) compile more of your program into GPU kernels.

I built a cross of React + Wiz (a fantastic frontend framework at Google): https://github.com/aschleck/trailcatalog/tree/main/js/corgi . Totally irresponsible and probably full of bugs, but I was so tired of writing business logic in the same place as my view logic with React and now I'm free of it!

Atomic AI | Software Engineering | onsite South San Francisco, CA | Full-time

jobs: https://atomic.ai/careers#opportunities, company: https://atomic.ai

Atomic AI is fusing cutting-edge machine learning and structural biology to unlock RNA drug discovery.

We're looking for sharp and thoughtful engineers excited to help us design therapeutics to treat untreatable disease. There are two positions currently open.

* Senior Software Engineer, Data - design and build solutions with our in-house experimental biology team to analyze, store, and serve experimental data and enable new analysis workflows (https://boards.greenhouse.io/atomai/jobs/4726839004)

* (junior - mid-level) Software Engineer, Infrastructure - integrate and build broad-spanning systems to create a holistic research platform (https://boards.greenhouse.io/atomai/jobs/4531035004)

I'm the hiring manager for both, please apply via the links above or email me at [email protected] if you have any questions!

I wrote this post to try and give candidates more insight into what it feels like to be a technical coding interviewer, what we're looking for, and in the hope that more candidates might see the interview as a conversation (versus simply a written test.) I'd be very curious to hear what others think.

It's worth noting that Google is also working on JAX these days and that's picking up steam, and one reason is because its API is super clean.

(chiming in here as a founding engineer at Atomic)

So I spent more than 8 years as a SWE at Google, and now work here with both experimental biologists and machine learning scientists. And yes, a lot of the concerns mentioned in this thread are also things I have had anxiety about.

Most obvious to me, being a software engineer at Google felt like being the center of the universe. Coming here, the focus is the scientific research. And yes, the scientists all managed to complete their PhDs so they don't necessarily need me to unblock them every second of their day. But contrary to my expectations, this has been remarkably freeing. I think one particularly important part of our company that makes this work is that, even on the science side, we're multidisciplinary (at a high level, emphasizing both experimental biology and ML.) And so engineering feeling like another arm of that multi-discipline nature is fairly... natural.

The reason I feel it's freeing, and the reason I enjoy working here, is also the greatest challenge. Because the scientists are focused on the science, because they respect me and trust me to figure it out, and because they aren't constantly blocked by me, my job is mostly about dreaming extremely expansively about what I can do to reduce toil and make the scientists more productive. Of course they have feedback and input, but how I use my time and what I build is ultimately my decision because I am the engineer. And I have been able to do some things I am very proud of, like rolling out Bazel and Kubernetes and finding ways to seamlessly bring them into the cloud (we're even multi-cloud now without them even noticing!) On the other hand, it's very challenging because when you work on a product, say Google Photos, as a SWE, you always have some direct tether to the product ("what should we build next? ahhhh, well I guess we could just embed stable difficusion and a million people would immediately play with it".) At Atomic, my tether is very ambiguous. If I do my job successfully, they'll be able to do research more quickly (? effectively?), and eventually we'll be able to produce a therapeutic that hopefully changes the world. Identifying what I can do today to enable that far outcome in the future is very challenging, but it is a far more interesting challenge than gluing some pre-existing software into my UI or running A/B tests to turn a red button blue.

If, like me, you enjoy working on being given ownership over incredibly ambiguous problems, please do reach out!

This role focuses on directly working with the biologists: https://boards.greenhouse.io/atomai/jobs/4726839004

This role is expansive cloud infra: https://boards.greenhouse.io/atomai/jobs/4531035004

And this role is working directly with the ML scientists: https://boards.greenhouse.io/atomai/jobs/4191285004

This is sort of a confusing article because it assumes the premise of "you have a fixed hardware profile" and then argues within that context ("Most scientific computing runs on queues. These queues can be months long for the biggest supercomputers".) Of course if you're getting 100% utilization then you'll find better raw pricing (and this article conveniently leaves out staffing costs), but this model misses one of the most powerful parts of cloud providers: autoscaling. Why would you want to waste scientist time by making them wait in a queue when you can just instead autoscale as high as needed? Giving scientists a tight iteration loop will likely be the biggest cost reduction and also the biggest benefit. And if you're doing that on prem then you need to provision for the peak load, which drives your utilization down and makes on prem far less cost effective.

The permissions management (user/group but with cloud roles) is what I miss most. NFS style approaches assume that I will give every user an unprivileged user on their local machine and deny root access so they can't just create a user with a different ID. S3 style ACLs per-object are just very confusing and no one is going to set those. S3 access policies per role are obviously not a solution because that requires centralizing lists of what the permissions are. I just want to be able to assign a chmod and a chown on a file for a cloud user and group... And don't forget about how fantastic the group tooling was.

I also miss being able to write files with /ttl=72h/ anywhere in the path and knowing they will be cleaned up automatically without me doing any more work. In contrast S3 just rots. Need to find an intern to write an SQS pipeline to schedule file deletions :)

Bazel works well when you're in Google's situation: many teams working on separate but connected projects, in multiple languages, infinite money to fund infrastructure teams and near constant migrations, a mandate that all code (*approximately) must be developed using it, and a culture of vendoring every dependency. Meanwhile the typical OSS project is a group of folks working on a monolith in one or two languages, and the predominant development culture avoids vendoring (for better and worse.)

I'm a strong apologist for Bazel, but I can absolutely see why projects avoid it. The more you deviate from what Google does the more paper cuts you'll get, but the less work you'll have to do. Bazel is great but it's certainly not a clear win in the general case.