But really Hurl looks really interesting, being editor agnostic is the best solution for your problem, I agree.
A comment I'd have is that it's quite hard to find out who authored this fine piece of software.
I don't necessarily recommend editing the postman config json directly to set up new tests as it's a PITA, but it's generally what I do so I don't need to keep importing and exporting it with Postman.
A tool designed for working with on the shell is likely better than what I'm doing with newman (since the config is not the most accessible), but it also meant I didn't need to rewrite a bunch of existing tests and verify they actually did the same thing.
IMO any testing tool that does not save it's test classes in a human readable format is DoA.
Even for tools that generate the spec from source code, it is usually still possible for user error to define the metadata for an endpoint incorrectly. Dredd catches that.
if you don't have devs there are no bugs, problem solved.
these are different disciplines that deserve to be done well.
Maybe I am biased because I spent the last 10 years in gamedev, or maybe this is another push to make devs do basically everything tech related: but if a developer tells me a feature is done I always look to QA for a nod.
That nod rarely comes, the feature is not done, the developer merely got it to work on their machine.
The difference between devs and dedicated QA people is that devs know the dark corners of the implementation. They know the edge cases and scenarios that they struggled with getting right. That's where most testing focus has to be. QA doesn't know any of that. They can play through some scenarios that are the expected ones from the spec, perhaps have some hunch of what could be the tricky cases, but they don't actually know.
I love seeing all those downvotes for my GP comment. That's all the people working at companies where stuff that devs feel is below them is dumped on QA people, which is the main reason my company is well-known for shipping top quality (without having a QA dept) while our competitors struggle with quality (despite having a QA dept). That's all the evidence I need to back up my claim.
And I believe it's hard to do this kind of thing as an afterthought. If your whole engineering culture is built around having a QA dept then just firing that dept is obviously going to have disastrous consequences.
it's the same with good security folk. sure you can pretend you'll catch 100% of issues, but it's a delusion, good security or quality testing is a totally different mode of thought
If such feature interactions matter then your application has bigger problems than a QA department.
> it's the same with good security folk. sure you can pretend you'll catch 100% of issues, but it's a delusion, good security or quality testing is a totally different mode of thought
Oh I'm not saying that good QA isn't a valuable skill! Of course it is, it doesn't just happen on its own. What I'm claiming is that it's a skill that should be employed as close as possible to the creation of the thing that it's assuring the quality of. So, ideally within the developer themselves.
Same thing with security. You will have a terrible security in your product if you first design and implement it and then put security in there as an afterthought by a dedicated security team. Ideally it's been at the table from day 1. So, a good security team works on educating your devs to do things right from day 1. Just like QA.
QA doesn't exist to check that your crap architecture solves the problem. They're there to be the skeptical person in the chain of custody from developer to production. The old adage "you can't test quality into software" exists for a reason. It's true. And you're right that it's the wrong architecture. But that's dismissing the very real purpose of QA which is to catch these kinds of bad architecture problems by testing for conditions that the dev team possibly didn't consider during development. QA is also there to make sure you're not making any unfounded assumptions about the context the software exists in.
To the extent you actually care about checking all of your assumptions and not just the ones you're cognizant of, QA is very useful.
It allows you to write load/performance tests in JS, commit them to your repo, easily automate them in CI, send metrics to several backends, use protocols besides HTTP, with a modern CLI, and many more features.
There's also a Postman-to-k6 converter[1]. The conversion might not be perfect, but it will give you a head start.
Note that the k6 philosophy is for developers to write these tests, similarly to how you write unit/integration tests, and to break the classic QA-dev cycle.
I don't want to steal Hurl's thunder, it does look great, but it's limited in features compared to existing peformance testing tools, and I'd personally rather write tests in a programming language, than in a bespoke text format.
I feel like I used to be able to run the CLI with just an URL and it would issue GET requests to it -- without needing a .js script to run
Am I crazy, did this never exist? Please bring back/add this feature
I’d hardly call hurl’s config bespoke, it basically amounts to HTTP codes and some header stuff, which basically everyone is familiar with.
From personal experience a couple of weeks ago I needed some http tests. I got a CI-ready Hurl config working on some endpoints for work sorted within like, 5 minutes of downloading it. In comparison I opened the page for K6, saw the “write tests using JS” and basically noped out immediately,
That said...
> I got a CI-ready Hurl config working on some endpoints for work sorted within like, 5 minutes of downloading it.
I'm obviously biased, but I'd argue that k6 is equally easy to get started with. Take a look at the docs[1].
> I’d hardly call hurl’s config bespoke
But it is. It has a unique syntax, and no other tool uses it, AFAIK. I'm not saying it's not simple or easy to understand, but it's not as expressive as a programming language. Which is fine if your tests don't require any logic, but from experience, in order to simulate accurate real-world traffic, which you probably want when writing a load/performance test, you do need some logic, different scheduling options, a way to process and analyze the metrics, etc.
I don't want to turn this into a versus battle, as there is a place for tools such as Hurl, ab, and wrk to coexist with tools like k6 and Locust. I just wanted to mention it as an option for anyone who hasn't heard about it, as any personal biases aside, I do think it's a great tool.
[1]: https://k6.io/docs/
Yeah, I did they start off with JS shenanigans, including importing packages, which immediately makes me concerned I’m going to have to deal with JS packaging dumpster-fire. There’s then one line of CLI args, followed by even more JS boilerplate.
> in order to simulate accurate real-world traffic, which you probably want when writing a load/performance test
I think the mismatch here is that Hurl is closer to being a http testing tool, not a load-testing tool, so most of these features are unnecessary.
> It has a unique syntax, and no other tool uses it, AFAIK.
Taking the example from the front page:
HTTP 200 [Captures] csrf_token: xpath "string(//meta[@name='_csrf_token']/@content)"
Character for character, most of this is bog-standard http verbs, a url, and an html path string. The Jetbrain HTTP tool also sports very similar interface, and I believe there’s a VSCode plugin somewhere that does a similar thing.
I'm tired of having to look at Postman screenshots sent from QA. I'm tired of having to wait for them to press Send once I've implemented a fix. I know they're tired of waiting for me to do that, too. Hurl is something both the devs and QA can speak and write. It can be automated and a part of CI. It makes communicating expectations straightforward. It can be chucked along with PRs as a starting point for QA. I don't see a reason not to use it wherever possible.