hassy

This is a great blog post! just taking the opportunity here to comment on this:

> Finally for full scale high fidelity load tests there are relatively few tools out there for browser based load testing.

It exists as of a few months ago and it's fully open source: https://github.com/artilleryio/artillery (I'm the lead dev). You write a Playwright script, then run it in your own AWS account on serverless Fargate and scale it out horizontally as you see fit. Artillery takes care of spinning up and down all of the infra. It will also automatically grab and report Core Web Vitals for you from all those browser sessions, and we just released support for tracing so you can dig into the details of each session if you want to (OpenTelemetry based so works with most vendors- Datadago APM, New Relic etc)

> I'm very biased

Don't write your own load testing tool other than as a fun little exercise. At least not without understanding coordinated omission and thinking about workload modeling (open? closed? hybrid? all of the above?) [1]. Get this wrong and the results produced by your tool will be worthless.

Once you've got that out of the way, don't forget that you'll want a distribution story. It does not matter how efficient your tool might be on a single machine - you'll want to distribute your tests across multiple clients for real-world testing.

"Sure it's easy" you might say, "I know UNIX. Give me pssh and a few VMs on EC2". Well, now you've got 2 problems: aggregating metrics from multiple hosts and merging them accurately (especially those pesky percentiles. Your tool IS reporting percentiles rather than averages already, right?!), and a developer experience problem - no one wants to wrangle infra just to run a load test, how are you going to make it easier?

And, this developer experience problem is much bigger than just sorting out infra... you'll probably want to send the metrics produced by your tool to external observability systems. So now you've got some plugins to write (along with a plugin API). The list goes on.

I'm very biased, but it's 2024. Don't waste your time and just use https://www.artillery.io/

1. https://www.artillery.io/blog/load-testing-workload-models

curl is fantastic. There's also HTTPStat which provides a waterfall visualization on top of curl timings: https://github.com/reorx/httpstat

There's also Skytrace (made by yours truly), which provides timing info as a waterfall visualization inspired by HTTPStat + lots more (syntax highlighting for responses, built-in JMESPath support, command-line assertions and checks etc) - https://github.com/artilleryio/artillery/tree/main/packages/...

The main thing about the advice comes immediately before the bit you’re zooming in on. Want to talk to people who have seen hundreds of companies deal with what you’re dealing? YC is great for that.

Did YC S21 which was an all remote batch. my 2c.

YC is 100% what you make of it. It's not a lean back experience.

I did not meet most of the companies in my batch but I've gotten to know many founders through YC that I would not have otherwise. Founders that have been source of advice and support.

Network of clients - yep don't go into YC expecting to sell to other YC cos. It is easier to get warm intros through the network though.

YC advice, office hours specifically - it's what you make of it too. Expecting a group partner to know your space in great detail is unreasonable but if you recognize that they've seen hundreds of companies with similar problems and make use of that pattern matching, you can get very valuable advice. Some of the advice I did not take and did the opposite and it was the right decision. And some advice that I did not take was exactly right, but I only saw it in retrospect months later.

Fundraising - being a YC company definitely opens doors, and also helps protects you from bad actors who have to think twice before fucking with a YC co. Very valuable for any first-time founder. You have someone to sanity-check everything, terms you're not sure about etc. The bump in valuation is real too.

I'd do YC again in a heartbeat.

Clickable links:

Blog post with demo: https://www.artillery.io/blog/this-load-test-cost-us-how-muc...

Project on GitHub: https://github.com/artilleryio/artillery

This lets you load test with Playwright (very similar to Puppeteer):

https://github.com/artilleryio/artillery-engine-playwright

Yes indeed but wrk2 or Vegeta is still better for this particular use case (unless k6 has support for setting a constant RPS rate, afaik it does not), as otherwise the overhead of establishing a new TCP connection for a single HTTP request will dominate the benchmark.

Yep, k6 suffers from coordinated omission [1] with its default settings.

A tool that can send a request at a constant rate i.e. wrk2 or Vegeta [2] is a much better fit for this type of a performance test.

1. https://www.scylladb.com/2021/04/22/on-coordinated-omission/

2. https://github.com/tsenart/vegeta

Bombardier is cool, but serves a very different use case.

I mentioned elsewhere in the comments that we have a Docker image, and are working on other methods of installing the CLI to alleviate some of these dependency-related concerns.

Getting side tracked here, but there seems to be a common sentiment when it comes to Node.js that it's uniquely insecure. Node.js has indeed had some unfortunate press when it comes to supply-chain security, but every other runtime is susceptible to those attacks (PiPy, Gems, Maven, Rust Crates). Ultimately of course, if you choose to avoid using any software built on top of those stacks, that's your choice.

Artillery specifically is no different to any other Node.js-based project in how large the dependency tree is. VSCode for instance is used by millions of developers has 1.6k dependencies [1].

1. https://github.com/microsoft/vscode/network/dependencies

I haven’t come across Hurl before. It looks cool!

Yes, those dependency trees can be large. Yes, supply chain attacks are a real threat. But Node isn’t that different than Python or Ruby in that regard. How far down the stack do you personally choose to go? I trust you’re familiar with that famous paper published by a certain mr Thompson in the mid-80s?

The world is a big place. There’s a lot of software out there written in Node.js, used happily and productively by millions of developers, many of them in corporate environments.

Given the opinions you expressed elsewhere in the thread here I think it’s clear that this tool is not for you. I hope no one is forcing you to use it.

thanks for trying it out! adding “make timeouts configurable” to the todo list.

you’re right on that YAML-as-JSON thing. If everything is quoted as JSON, those type conversions shouldn’t kick in. Otherwise there’s room for surprises - perhaps we can do something to make those cases more obvious.

“mini” in the sense that it doesn’t do everything that curl does. curl does a whole lot. this tool focuses on more common use cases and makes them friendlier. plus with request waterfalls and assertions it does things curl can’t do.

the number of packages is a bit of a misnomer anyway. Artillery Probe is part of Artillery which does load testing, with multiple protocols, support for multi-step scenarios, publishing to a variety of monitoring systems (Datadog, Prometheus etc) and more. That’s what most of those packages enable.

sure, that’s a valid concern in some environments. fwiw we use Snyk.io for dependency scanning

Yeah so the JSON quoting part is something I’m pretty pleased with. We use a YAML parser (JSON is a subset of YAML) to parse those values, which is what allows for double quotes to be omitted.

Good point on iterating on the query! We already save the body into a temp file, so we can make Probe be able to run queries on a file. Adding it to the todo list. :)

(In my own workflow I use gron a lot for getting an overview of the shape of unfamiliar JSON, super handy tool)

As to an interactive shell… yes, 100%. Kicking ideas around something like that as well!

A few things :)

Syntax highlighting

Request performance waterfalls

Built-in JMESPath queries for JSON (same syntax as in AWS CLI). XML and HTML can be queried too.

Ability to set checks on the response. This one is super handy for quick acceptance tests.

curl is amazing! I love curl. This is not meant to replace it completely. Probe just makes those most common use-cases friendlier, like looking at headers, or inspecting JSON retuned by an API with syntax highlighting, or querying without needing to reach for jq etc.

thank you! we just grab the timings info for the request from the underlying HTTP library, and sprinkle some ASCII art on top. That part was inspired by httpstat [1]

We want to extend those with support for Server-Timing next, and also Core Web Vitals [3] (via Playwright) for web pages.

1. https://github.com/reorx/httpstat

2. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Se...

3. https://web.dev/vitals/

if you’re on a Mac, Artillery has a Homebrew formula too. We have an official Docker image too. Other ways of installing are on the roadmap too (self contained tarballs and binaries).

As to what it does better than curl/wget. Probe is geared towards interactive use with HTTP APIs. So you get syntax highlighting and pretty printing for JSON, built-in querying, request waterfall visualizations, and ability to set expectations on every response.

HTTP/2 is supported.

clickable url: https://www.artillery.io/blog/swiss-army-knife-for-http-test...

Playwright's built-in codegen tool creates scripts that can be run as load tests with this project :)

https://playwright.dev/docs/cli#generate-code

Yes! that's how we ourselves run it. The project includes a ready-to-go Dockerfile: https://github.com/artilleryio/artillery-engine-playwright#u...

clickable link: https://github.com/artilleryio/artillery-engine-playwright