I see a single advantage to these frameworks though. They force you to write a test in a specific way that looks like a real-world scenario with clear delineations between setup stages, assertions, and so on. Not that it's impossible to achieve with regular programming language, obviously.
I've seen well maintained and designed cucumber / rspec / robotframework "code" and i've seen some really horrible shit. Keyword re-use is really a joke if you are not familiar with any good practices to actual software development. Granted, in "low code" scenarios, the worst selling point is "you don't need to know how to write code" - which imho is bullshit and from my pov, statements like i quoted echoes that.
Disclaimer; i partake and contribute to RF ecosystem. And i do prefer to write my tests or process automation with it even thought i can write code in multiple languages..
GIVEN("Some setup") {
// code to create the setup
WHEN("some use of the setup") {
// do something
}
WHEN("something else"){
// do something unrelated to the previous when
}
}If you can't code, you shouldn't be writing automated tests. Most of the "benefits" of Cucumber can be gained by using a good reporting framework. I like to use Serenity as I love its reports.
I'd also say "non-coding" rather than "non-technical". Coding is just one aspect of being "technical".
There is benefit in making the tests easy to read but this is better done in actual code, e.g. the screenplay pattern.
I have never seen it work in practice, but that what it is supposed to be about. If your managers are not looking at the gherkin and making suggestions then there is no point in doing it.
Automated tests are then written based on these but not "using" these. By not "using" the Gherkin scenarios for our tests we can be looser with the language and not have to worry about RegEx.
If a manager, or other non-coder, wants to see the tests, they can look at the very pretty report that was generated.
In a previous company I had one use-case that I thought cucumber helped out a lot at.
We were writing software that would be vigourously tested and approved by a customer who didn't have any technical knowledge. Their initial plan was the "Excel sheet of death" test case management system with really fuzzily named tests.
I convinced them to write Cucumber tests, which turned out to be around 100, using the very explit syntax that then got called stupid by the cucumber devs and moved to the "training-wheels" gem. I never expressed what we were doing as cucumber tests. I just told them "Write them like this and it is easy to understand" and they agreed and did so, then they were signed off by our project team and them.
After that I then automated them our side so that we could run them quickly and easily in our pipeline while the customer could still use the tests for their tedious manual user acceptance testing at the end.
It worked out quite well. The customer never knew we automated the tests our side, they were just thrilled that something like 99% tests passed first-time and the 1% that didn't were down to interpretation differences.
The code consists of a series of invocations of functions ("keywords" in RF terms) with parameters. The syntax is tabular.
When I read the intro, it reminds me of Forth.
Although I only gave it brief reading a couple of times. (RF was used by QA at my past job.)
Worst of both worlds, IMO.
I've never seen a manual tester follow the steps that are written down, but I've seen a lot of effort put into writing down the exact steps anyway.
And that's how a lot of testing is done in various companies, especially where manual testing is still common, at least in Poland.
Granted, I've had middling success convincing non technical people to write or even look at tests, but I think it's irresponsible not to try to create this bridge, depending on the audience of the code. Because very few projects are well documented, and this syntax can act as documentation. In my current work, I am using BDD style to create high level flows, where screens are automatically captured, which is useful both for verification and generated documentation, especially when versioned with code.
It can avoid duplication (and its pitfalls), because in unit style tests, there is a text description along with the code, which can drift away from being accurate. If the test is the description, that won't happen.
It creates a more abstract model, because the tests are forced to a natural language, which will tend ot make it less implementation dependant, so the implementation can be swapped out. I've had good success with this approach, seamlessly migrating a complex workflow from one backend to another, and the tests were kinda pricesless.
I also find keyword and flow re-use to be quite high, and it's great how this approach means the more tests you write, the less low level, implementation specific code and scenarios you need to write.
At every client I emphatically insist that if they are not using Gherkin as a “Rosetta stone” for the business folks then they’re wasting their time with cucumber. Don’t go to the trouble of writing gherkin if you’re not using it as living documentation. Save yourself loads of trouble and just use an ordinary test framework.
I've read something along these lines about the choice of OCaml at Jane Street, where they argued that their investor-types are able to proof-read new code, despite not being programmers, because the language "reads like english", similar to SQL or COBOL.
It works really well for us to get a shared understanding between a non-technical business, devs and QA.
That said, we then hand it the Gherkin over to the SDETs who fettle it according to our house style (which is strict and must pass enforced linting) and it does take gardening as the product matures. It's our SDETs who write the test code: we wouldn't dream of letting a non-technical person actually drive the test logic itself.
I'm not impressed at all by Gherkin, BDD, or Cucumber. Never once has any business person gone "oooh, let me read the tests!", And in the rare circumstance they do, what you'll find is a secondary grammar develops where particular step implementations gain loaded significance, far and above the natural language implementation of a step, making things complicated.
At this point, I just tell people to keep notes, make checklists, get familiar with the data model for test data authoring, then go eith God, and always put the User first.
Mileage varies greatly, but there's a culture problrm in a lot of places where "the fiture must go out" always takes precedence over "the spec must be right".
And no one ever wants to rip apart testing tools to figure out how they work.
I presume low code/no code comes from the simplest notion of programming language, which is a medium between human and machine. And the easier it is to read and write means the "better" it is. Therefore pitching it to a non-technical or even semi-technical people is as simple as teaching people to write code at surface level.
However what people don't understand deeply is that any language is a set of symbols and rules requiring a certain amount of cognitive space and a certain time to absorb it. The more ambiguous the symbols are, the closer the growth of the number of the rules to exponential. This is a fundamental point that happens in real-life but is rarely understood, moreover to programmers, surprisingly.
I believe a proper low-code/no-code should have natural language processing capability and understanding context, not just providing syntactical aliases.
I was also a Quality Engineer for a time, and one of our bigger projects was to implement precisely a Natural Language system or something so that PMs or whoever wanted to “live in the future” could write their tests directly in the User Story.
What we did was create a large set of commonly used “phrases” to interact with systems, be it UI or APIs, which we coded the actual test framework for, document them for anyone wanting to write User Stories (in Rally platform, like JIRA), the instructions on what to do if a phrase didn’t exist for their case, etc. It took us 6 focused months, but we delivered, and people were happy. Of course, we are talking about technical PMs, PMs that met with designers and set standard guidelines, PMs that required API documentation and read it, and the like; just not “coding much” PMs.
Albeit this being a company where we already wrote automation tests right next to the developer’s timeframe for their own coding, where a feature got both code/tests right around the same time, 99% of the time, so it was easier for people to take advantage of features instead of shallowly complaining.
So, yes, I concur with you that probably your case would be better if everyone were a developer, I don’t really see the need for those kind of DSL, but if you reeeeeeally need to have other people intervene, and want them to focus more on their end of the bargain (not coding), I think these DSLs have their place: Assuming the whole company has the stomach for it.
When I worked together with a QA team for automated testing, they used Cucumber, which uses Gherkin as an english-but-not-really DSL. Now the idea is obviously, that someone who doesn't know how to code can still describe and read test scenarios. But what ended up happening eventually is that I spent a lot of time writing code to support different Gherkin expressions, but then had to take over writing the Gherkin expressions as well, because even though Gherkin -looks- like english, it still behaves like a programming language. In the end, it would've been much easier for me to just write the tests in a programming language from the start, instead of using an abstraction layer that turned out to serve no purpose.
What I took away from that frustrating experience is that a lot of people think that learning to code is learning a syntax of a programming language, and if you could just do away with that pesky syntax, everybody could code. But, obviously, the syntax ist just the initial barrier, and actually coding is crystallizing and formulating intent - something which programming languages are much better at than natural languages.
But for some reason, this low/no-code idea never seems to die down, especially in some spaces, and I have the feeling I'm just missing something here.