art049

CodSpeed | Multiple Positions | On-site (Paris) / Remote (Europe) | Full-time | https://codspeed.io

We’re building the continuous performance optimization platform. We help engineering teams catch performance regressions before they ship, with sub-1% measurement variance. We’re already trusted by teams at Cloudflare, Vercel, Mozilla, Pydantic, and Astral (Ruff, uv).

We’re a small, deeply technical team looking for engineers who want real ownership and the chance to ship end-to-end. Competitive salary, stock options, generous PTO.

Senior Systems Engineer: CPU simulation tooling, language runtime internals (CPython, V8), kernel I/O emulation. C/C++/Rust/Zig.

Senior Full-stack Engineer: React/Next.js, Node.js/TypeScript/GraphQL, GitHub/GitLab integrations, Rust data pipelines.

Senior/Lead AI Engineer: Agent layer, eval methodology, context pipelines, model selection. Python/FastAPI.

Graphic/UI Designer: Visual identity and product experience for a developer tool.

More details at https://careers.codspeed.io. Apply at [email protected]

Yes definitely, this is what we already do with cpu simulation. But we had many people with benchmarks using syscalls for network,fs and other resources. And we found the only solution for this is to actually measure wall time

I hadn't considered this, but it would be really interesting to take it into account. Given that the size of the benchmark suites directly affects the false positive rate, and counterintuitively, the more benchmarks in the suite, the more the chances of false positives, even with super steady benchmarks. (Thanks, it could also be an interesting follow-up article!)

CodSpeed | Founding AI Engineer | On-site (Paris) / Remote (Europe) | Full-time | https://codspeed.io

We're building software performance optimization tools to optimize and measure code performance before it is deployed to production. We avoid regressions that impact UX and help developers solve their performance issues faster. We're already live and trusted by top-tier open-source project teams such as Pydantic, Ruff, and Prisma.

We’re at an exciting early stage and looking for talented engineers who share our passion for helping to enhance the performance of software used by billions, improving the software development lifecycle, and building tools we love to use ourselves.

Apply at https://codspeed.notion.site/Founding-AI-Engineer-cd1bf4fd73...

I never tried xcode instruments. Is the UX good for this kind of tool?

Thanks for reporting the typo. Menu will be there soon.

I thought about this quite a lot and feel like it's super important as well. Often, having the proper infrastructure/tools/resources to automate performance testing is hard. I created codspeed.io to try and fix that.

Is it needed in C? I think it's just a specification difference since most compilers should be able to do the same job, clang and gcc at least.

>This is a neat approach! I'm curious how well it maps to actual perf degradations though. Valgrind models an old CPU with a more naive branch predictor. For low-level branch-y code (say a native JSON parser), I'd be curious how well valgrind's simulated numbers map to real world measurements?

We didn't try it on this specific case but on we found that on branchy code valgrind does aggravate the branching cost. Probably, we could mitigate this issue by collecting more data relative to the branching and incorporate those in our reported numbers to map more accurately to the reality and more recent CPUs.

>My probably naive intuition guesses that some low-level branchy code that valgrind thinks may be slower may run fine on a modern CPU (better branch predictor, deeper cache hierarchy). I'd expect false negatives to be rarer though - if valgrind thinks it's faster it probably is? What's your experience been like here?

Totally! We never encountered a false positive in the reports yet. But as you mentioned since valgrind models an old CPU, it's likely to happen. But even though the cache simulated has a quite old, it still improves the relevance of our measures. When we have some time, we'd really enjoy refreshing the cache simulation of valgrind since it would probably eliminate some edge cases and reflect memory accesses more accurately.

Yes, totally. Most of the time we can even raise a warning with an even lower regression threshold, for example in this benchmark from pydantic-core, there is a less than 1% variance in the results and we can easily detect performance changes of less than 5%: https://codspeed.io/pydantic/pydantic-core/benchmarks/tests/...

Hi HN! We’re Arthur and Adrien from CodSpeed. We’re building a tool measuring software performance before any production deployment, catching performance regressions before they hit production environments and reporting performance changes directly in Pull Request comments. It’s kind of like Codecov but for performance measurement.

Today, the go to solution to measure performance is probably to use an APM(DataDog, Sentry, …), continuously analyzing your production environment. However, since those solutions are operating on real environments they need real users to experience poor performance in order to report issues and unfortunately, performance remains an afterthought appearing only at the end of the development cycle.

Another possibility to measure performance is to create benchmarks while developing and to run them on a regular basis to have an idea of the performance trend of your project. However, with this approach, the variance in the results creates a lot of noise and it’s rarely possible to compare your results with the ones from a co-worker or a production environment.

To make consistent performance measurement as easy as unit testing and fully integrated in CI workflows, we chose a benchmark based solution. And, to eliminate the usual variance associated with running them, we measure the number of instructions and memory/cache accesses through CPU instrumentation performed with Valgrind. This approach gives repeatable and consistent results that couldn’t be obtained with a time based statistical approach, especially in extremely noisy CI and cloud environments.

We have been in closed beta for a few months, already being used by popular open-source projects such as Prisma and Pydantic. Notably, CodSpeed helped Pydantic through their Rust migration, empowering them to make the library 17x faster: https://docs.pydantic.dev/latest/blog/pydantic-v2/#performan...

Today, we’re super excited to finally make the product available to everyone. We currently support Python, Node.js and Rust and are looking forward to integrate with more languages soon.

The product is and will be free forever for open-source projects. Also, we have a per-seat pricing for private repository usage. We have a lot of exciting features planned regarding additional integrations, such as Database and GPU integrations that should come in upcoming months.

Don’t hesitate to try out the product and give your honest feedback. We’re looking forward to your comments!