k2so

Location: Göttingen, Germany Remote: Yes Willing to relocate: Yes, within in Germany Technologies: Python, FastAI, vLLM, PTQ, Inference, Model Training, Transformers, Elastic Search, Vespa, Embeddings, bm25, RLHF, GRPO, trt, prime-rl Résumé/CV: https://bluenotebook.io/about/ Email: [email protected]

I'm Nikhil, an MLE with 8 years of experience across computer vision, search and model inference. I've got a strong product mindset. Looking for product led roles using LLM APIs currently with roadmap of building it in-house.

Location: Göttingen, Germany

Remote: Yes (preferred; hybrid also fine)

Willing to relocate: No (remote, or hybrid within Germany)

Technologies: Python, Go, search & retrieval, ranking, RAG, agentic LLM systems (MCP, tool use), LLM post-training (GRPO/RLHF), evaluation, vLLM, Kubernetes, AWS/GCP, ML Product building

Résumé/CV: https://bluenotebook.io/about (https://linkedin.com/in/nikhil-kasukurthi GitHub https://github.com/nikhil-kasukurthi)

Email: [email protected]

Applied ML engineer with 8 years building ML Products in search & retrieval, and recently agentic LLM systems in production. At Eka.care I built agent-native retrieval (MCP tools, hybrid late-interaction reranking) grounding LLMs in external data, and open-sourced an LLM evaluation library (KARMA, adopted by the Gates Foundation).

Currently doing ML consulting and independent research on agentic search, training models with GRPO to plan, issue, and refine their own search-tool queries. Strong in Python and Go.

I have EU work authorization, no sponsorship needed.

Yeah, for a single GPU inference, considering the higher VRAM and FP4 support on the RTX 6000, it should fit larger models as well than the H100.

Author here. I wanted to train Nanochat d26 to GPT-2 level and had to pick between three H100 variants on Runpod.

SXM was the most expensive per hour but cheapest to finish: SXM: 702ms/step - ~$37 (using vast.ai) PCIe: 1,412ms/step - ~$112 (runpod) NVL: 2,032ms/step - ~$181 (runpod)

My first SXM run hit 1,295ms. Barely faster than PCIe. Nsight OS runtime summary led me to suspect CPU starvation. I found a higher vCPU instance on Vast.ai which hit 700ms. The 128 vCPU SXM instance also hit ~700ms, so it wasn't CPU count.

Looking at the network topology on Runpod and vast.ai, the first instance had GPUs split 4+4 across two NUMA nodes. NCCL's data transfer uses NVSwitch and is unaffected, but the control threads run on CPU. Cross-socket latency on every pthread_cond_signal added up.

NVL was the most confusing result, NCCL kernel times nearly identical to PCIe, but step times 44% worse. Only 4 of 28 GPU pairs share NVLink on NVL, the rest fall back to PCIe. I don't have a full explanation for this yet.

Profiling script: https://github.com/Nikhil-Kasukurthi/nanochat/blob/master/sc... Script with startup checks to ensure instance is healthy: https://github.com/Nikhil-Kasukurthi/nanochat/blob/master/ru...

Happy to discuss, especially if anyone has ideas on the NVL anomaly.

Scroll to visualise steps is such a great idea! Great writeup.

Interesting work, in the examples I can see that quite a few of them have the teracotta/warm-cream colour palette, was that an explicit choice to keep them in the prompts?

From the official frontend-design skill, on multiple occasions, unprompted, even I received the same warm-cream tones for different projects. Wondering if it's a new latent direction the model chooses to go to avoid safe/generic patterns.

I make an argument for Model context protocol, and how it can shape the entire ecosystem

Beyond the developer, the user massively benefits from MCP. Like you said, using any other SDK to build is a very valid approach but then you are tied down to a single client that you have built on that SDK.

If you would like to switch clients, then you have build it yourself. MCP solves this very well since, any MCP supported client can use the same tools/resources that you have built.

A neat trick in Vespa (vectors DB among other things) documentation is to use hex representation of vectors after converting them to binary.

This trick can be used to reduce your payload sizes. In Vespa, they support this format which is particularly useful when the same vectors are referenced multiple times in a document. For ColBERT or ColPaLi like cases (where you have many embedding vectors), this can reduce the size of the vectors stored on disk massively.

https://docs.vespa.ai/en/reference/document-json-format.html...

Not sure why this is not more commonly adopted though

LLMs = Latency? That's how most of us perceive it. When examining the timing breakdown of a request on Claude, you'll notice that the majority of the time is spent in Content Download—essentially, decoding output tokens.

In the blog, I discuss how partial json validation can help in workflow driven LLM products.

Would love feedback on how I can improve, thanks!

In one of my earlier jobs a few years back, we were training deep learning models on VMs with GPUs, back then the tooling was not as extensive (vs-code did not have the remote ssh then) as it is is now.

So, we would use SSH into the VM and do our work. This also involved a lot of debugging of code through vim since it's quicker to make in-place edits and re-run experiments, this taught me a lot on effective debugging and writing code for the VM

Easier to use libraries over highly complicated (supposedly performant) have a significant advantage in driving more adoption.

Recently I was trying to generate text embeddings from a huggingface model. Nvidia triton and text-embedding-inference (built by huggingface) were my two options.

> why large companies are generally incapable of delivering great developer experience. I wanted to curl up and cry while trying to make nvidia-triton spit out embeddings . The error messages are cryptic and you need to have jedi like intuition to get it to work. I finally managed to get it work after like 2 days of wrangling with the extremely verbose and long-winded documentation (thanks in part to claude, helped me understand with better examples)

Triton's documentation starts off with core-principles and throughout the entire documentation, they have hyper links to other badly written documentation to ensure you know the core concepts. The only reason I had endured this was because of the supposed performance gains triton promised but underdelivered (this highly likely being I had missed some config/core-concept and did get all the juice)

On the other hand, text-embedding-inference has a two line front and centre command to pull the docker image and get running. The only delay was due to my internet speed before it started serving the embeddings. Then deploying this on our k8s infra was a breeze, minor modifications to the dockerfile and we are running. And on top, it's more performant than triton!

This is awesome, are you contributing this to candle or is it a standalone package?

I very strongly relate to this, it's been close to 3 months, since I have started working on a blog built on Quarto, and all I have so far is a elaborate design and a half complete blog on an LLM tool I had built.

But like the comments say, I had way too much fun on the journey of a side project, just doing other things like configuring the website and playing around with the design elements like font and how the overall website looks (I'm a data scientist and never usually get to play with design as much as I want to at work). And recently, with claude's help built some cool react elements to push my story further.

Hopefully after this, I ship at least one blog and iterate on the design elements.

This was my first thought too, after reading through their blog. This feels like a no-frills website made by an engineer, who makes things that just work.

The documentation is great, I really appreciate them putting the roadmap front and centre.

Distill for R Markdown is a great alternative too. I really like the layout and the flexibility you get with the plots.

The site can be hosted directly on github pages and you can configure the custom domain on github. Fairly easy to setup and use.

Comes out of the box with GDPR compliance, google analytics and cookie consent

https://rstudio.github.io/distill/website.html

Tomorrow and Tomorrow and Tomorrow - This was best read of 2023 for me. The narration is very immersive and the author makes you very invested in the story. Shows the messy, complicated human side of things. And another plus was that it was about game dev, quite fun to learn about it through fiction.

Hi, great project! I was wondering if you are planning on extending support for dynamodb