bend4r
Hell it was fun riding with the board. Now, in 2021 spare parts are still available and I took the chance to refurbish the board for my child. The aluminum parts are very durable.
Unfortunately, the boards are almost forgotten but still used by a faithful circle of enthusiasts who are enjoying the unique driving feeling and the durable construction details.
For batch workload K8s is not sufficient. You need a workflow engine on top like Kubeflow or Airflow, so run you jobs. Thus. consider K8s more as a cluster operating software, but not as a fully-enabled workload management like slurm.
Furthermore, K8s has so many features you do not need in HPC environments, where high throughput and load occupation is paramount.
Customer is scheduling work load with Airflow on the K8s farm. Coming from a HPC background, I can confirm your comment. For batch-like workload, K8s and Airflow is a bad choice regarding farm utilization and workload placement. Classic HPC schedulers like Slurm are way more efficient on using farm resources for this kind of workload.
Also networking wise is Calico for high speed Infiniband-based networks might be a bad decision. Control plane is too chatty, and slows down spinning up workload. Also do not ever try IP-on-IP encapsulation - it will make your NIC-speed slow, due to CPU-bound encapsulation.
With Mellanox-class NICS, strive for SDN technology or use network transport modes with hardware acceleration on the NIC, like VLAN-tagging.