Re:Cucumber: BDD Testing and Collaboration Tools
by hitchstory on Wednesday July 19, 2023 @04:25PM
(#36792689)
BDD is great as a concept but Cucumber/RSpec/Gherkin/Behave/etc. is an absolutely terrible DSL for implementing it:
* It's verbose.
* It's not type safe.
* It desperately needs some kind of inheritance - most user stories branch from other user stories (e.g. put product in shopping cart->purchase product) and the only way people who use it can deal with it is by concealing critical details in code or by writing massively repetitive stories.
* What other language has a parser that routinely asks you to write regexes to get things to parse?
None of these things are fixable and the creators have largely reacted to the complaints with gaslighting: "you're just using it wrong".
It can be used effectively if you align the stars correctly but mostly not.
The one thing it does get right is not being turing complete - there's clear value in not using code to describe scenarios.
Re:Cucumber: BDD Testing and Collaboration Tools
by drewcoo on Wednesday July 19, 2023 @07:08PM
(#36794781)
> Cucumber/RSpec/Gherkin/Behave/etc. is an absolutely terrible DSL
I wouldn't lump RSpec in with the rest of those.
Spec-style frameworks (like granddaddy RSpec) are terser. They don't have layers of indirection. They're intended to help devs think about how code will be used, not to make code look like English.
Re:Cucumber: BDD Testing and Collaboration Tools
by packetlost on Wednesday July 19, 2023 @04:22PM
(#36792655)
BDD is, IME, only really good for testing UI behavior (I've seen some pretty decent JS BDD testing frameworks, but have limitex xp with them). It can be good at that because those behaviors can usually be expressed well enough in natural language. But I've seen it shoehorned into data-centric UI-less applications that are just a massive pain to deal with. Half the point of most testing frameworks is getting more done with less typing/overhead, but Cucumber and BDD is sorta the exact opposite: do less with more typing by trying to describe complex concepts in natural language.
Cucumber mostly exists to upsell naive MBAs and project managers on their "pro" tools.
Re:Cucumber: BDD Testing and Collaboration Tools
by ljm on Wednesday July 19, 2023 @04:18PM
(#36792591)
“If done right” is carrying a lot of weight there.
I don’t mind it if you use cucumber and it still serves the purpose of describing your code’s behaviour, so the output is readable. RSpec’s documentation is a nice example of that.
The problem is if people just use it as a syntax over a webdriver, which means that your test behaviour and logic is arbitrarily split into two different languages, the cucumber output is useless, the tests are needlessly slow, and it’s really hard to debug. Cucumber itself is not able to add value there.
But that’s not unique to cucumber either. I don’t think many people realise that test writing and architecture is its own skill in programming and so many suites turn into write-only cludges that take 12 hours to run on a cluster of 50 CI workers.
Re:Cucumber: BDD Testing and Collaboration Tools
by throwbadubadu on Wednesday July 19, 2023 @04:23PM
(#36792672)
Aren't incapable people the norm and usual problem? ;D
Found it quite neutral, not too bad, but also not really helpful.. some nice ideas especially that cross functional collaboration, but in the end the engineers had overhead and hated to write the wrappers, and the other stakeholders still couldn't or didn't want to read/write docs/test, or however it was intended. Admitted, maybe I belong to that incapable crowd as well ;)
Re:Cucumber: BDD Testing and Collaboration Tools
by hmmdog on Wednesday July 19, 2023 @04:30PM
(#36792770)
Poorly done BDD has made me detest bdd and all of the tools, especially when cucumber is used as a testing tool and not as a collaborative tool.
I would say most companies use cucumber as a testing tool and say they doing bdd. I would actually like to see a company doing true bdd I have not seen one.
Didn’t the founder post here and said something along the lines of hating bdd now?
Re:Cucumber: BDD Testing and Collaboration Tools
by jaggederest on Wednesday July 19, 2023 @04:33PM
(#36792831)
Contrary to many people I've had a bunch of positive experience using Cucumber in very specific situation(s):
- I was an outside consultant working with nontechnical but very process oriented management teams
- There was low trust in existing technical workflows: they had been burned by bugs, failure to deliver, and overpromising so a big part of my task was building rigor and consistency into the engineering workflows
- They were extremely engaged in managing the workflows (the core of the business was doing the thing that we built an app to do)
In that context, they loved having a printed out (!!) copy of all the cucumber tests running on the most recent build so that they could see what was passing and what was failing: Color printer, green line passes, red line fails.
Once I got them into the habit of writing "given when then" statements to define existing features, bugs, and functionality, they were happy as a clam being able to specify things. Eliminated a lot of the usual back and forth over "yes but what exactly do you mean" that non-technical people often create due to not being clear and exact enough for technical purposes.
I would implement the parsing and selenium-based testing that validated whether the cucumber was passing and we would have a shared understanding of where the code was at and whether there were any regressions. In the occasional case where my understanding didn't match what they expected, we had context to work together to figure it out.
All in all it earned me a fair amount of money and lowered the stress of being a consultant for those companies substantially so I have a lingering fondness for it. But like I said it's a very specific situation to deliver that kind of value.
Re:Cucumber: BDD Testing and Collaboration Tools
by hitchstory on Wednesday July 19, 2023 @05:01PM
(#36793319)
I've had, I think, 1 working experience and 6 or so highly negative and it was mostly the nature of the system under of test that made it work or fail.
For simple and very clear Given/When/Then scenarios with clear inputs and results that were easily represented with a small amount of text it worked. Think a trading system with a few codes and a small amount of data on the left and an expected result like a calculation on the right.
Where there was a lot of stories that branched off from one another (e.g. a website with lots of potential user flows) it inevitably led to stories that were too vague or too repetitive or more usually, both.
I'm convinced that it does work for a certain kind of application for which the DSL is capable of representing its scenarios, but it's not the kind most of us develop.
Re:Cucumber: BDD Testing and Collaboration Tools
by jaggederest on Wednesday July 19, 2023 @07:39PM
(#36795060)
The cases I was using it for involved internal applications, so you're right, there was a lot more clarity about "what to do when user click button b" than random people on the internet bouncing around a store and adding things to a cart.
There were also a lot of cucumber scenarios and suites involved. Like think dozens of printed pages in relatively small font. Concise it is not. But the people involved wanted verbose and clear over concise and easy :)
It can be a very reassuring way to develop software for stakeholders - "We added 8 scenarios, and made a further 4 failing scenarios pass this week" - as opposed to "our developers spent 6 months and then quit without delivering any code"
Re:Cucumber: BDD Testing and Collaboration Tools
by drewcoo on Wednesday July 19, 2023 @07:03PM
(#36794731)
All three of those bulleted scenarios are trust failures.
I'm glad Cuke made you money. There are lots of ways to make money from mismanaged companies.
If we were actually professional engineers, we'd have a code of ethics. I'd like to think that the ethical code would involve not doing harm to a business or intentionally obscuring harm being done.
Re:Cucumber: BDD Testing and Collaboration Tools
by kaycebasques on Wednesday July 19, 2023 @04:35PM
(#36792863)
Hi HN, I submitted this because I'm not that familiar with BDD [1] and I'm curious about people's experiences with it. Glad to see that the discussion is progressing as I had hoped!
Re:Cucumber: BDD Testing and Collaboration Tools
by liampulles on Wednesday July 19, 2023 @04:36PM
(#36792892)
Getting cucumber setup is consistently a huge pain in the ass.
But it's worth it - I find acceptance tests are great for testing systems from a data perspective, i.e. what comes in to our app, what we send to other apps, what we respond with, etc. The data coming in and out is ultimately what I care about ensuring.
I haven't yet seen acceptance tests that are readable by "business", but they do help remind me what the usecases in our system do.
I can speak for RSpec and say the output of the test runs can be formatted such that it generates specifications that are as "non-technical" as Cucumber. If you scroll down a bit from https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-trave... you'll see a similar output.
Re:Cucumber: BDD Testing and Collaboration Tools
by bradgessler on Wednesday July 19, 2023 @04:39PM
(#36792930)
With Cucumber you have to write tests and an interpreter. If there's bugs in your interpreter, your entire test suite could be passing when it really should be failing.
Rspec, and other frameworks like it, are good enough these days where you can write expressive tests that can be understood by non-technical people and not have to deal with writing and testing the interpreter.
Re:Cucumber: BDD Testing and Collaboration Tools
by jlkuester7 on Wednesday July 19, 2023 @04:44PM
(#36793040)
My experience with using Cucumber was mostly positive, but had to be done very carefully or it would become a huge mess of rube goldberg code behind the scenes. The number of scenarios and steps can become crazy repetitive unless you leverage Rules, Backgrounds, and Scenario Outlines.
Probably the most important thing was that with a lot of test frameworks, having poorly architected test utility code is annoying and more time-consuming, but has a limited blast radius. Hacky tests can be easily isolated from the other test code. In Cucumber, proper architecture and reusability are paramount. If you screw this up, your tests become unmaintainable really quickly!
Re:Cucumber: BDD Testing and Collaboration Tools
by jlkuester7 on Wednesday July 19, 2023 @04:49PM
(#36793115)
The main value I say in it was that our company has some regulatory obligations to:
- Maintain documented requirements for ALL CODE FUNCTIONALITY
- Maintain tests for ALL REQUIREMENTS
- Be able to produce reports for auditors that prove linkage between the requirements and the tests (to prove that all requirements were tested)
In this context, Cucumber was really nice since it automatically solved all three of these at once! For general testing purposes, though, I think Cucumber has too much overhead for it to be worth using...
Re:Cucumber: BDD Testing and Collaboration Tools
by jaggederest on Wednesday July 19, 2023 @07:46PM
(#36795111)
Yes, those are exactly the scenarios I bring it out for - when someone wants a VERY clear, VERY well defined, completely comprehensive list of every single thing the system can do, and whether they are working today in an automated fashion.
Re:Cucumber: BDD Testing and Collaboration Tools
by _joac_ on Wednesday July 19, 2023 @04:46PM
(#36793063)
I manage a behemot with 13k cucumber tests.
It incentivizes DRY, that for testing is not good.
You end up having step definitions with highly complex implementations, with loops and conditionals.
Admittedly, I never dealt with Cucumber.
* It's verbose.
* It's not type safe.
* It desperately needs some kind of inheritance - most user stories branch from other user stories (e.g. put product in shopping cart->purchase product) and the only way people who use it can deal with it is by concealing critical details in code or by writing massively repetitive stories.
* What other language has a parser that routinely asks you to write regexes to get things to parse?
None of these things are fixable and the creators have largely reacted to the complaints with gaslighting: "you're just using it wrong".
It can be used effectively if you align the stars correctly but mostly not.
The one thing it does get right is not being turing complete - there's clear value in not using code to describe scenarios.
I wouldn't lump RSpec in with the rest of those.
Spec-style frameworks (like granddaddy RSpec) are terser. They don't have layers of indirection. They're intended to help devs think about how code will be used, not to make code look like English.
Cucumber mostly exists to upsell naive MBAs and project managers on their "pro" tools.
i know that 99% is badly written sure.
but bdd cucumber style testing if done right can be good. problem i saw people just incapable.
I don’t mind it if you use cucumber and it still serves the purpose of describing your code’s behaviour, so the output is readable. RSpec’s documentation is a nice example of that.
The problem is if people just use it as a syntax over a webdriver, which means that your test behaviour and logic is arbitrarily split into two different languages, the cucumber output is useless, the tests are needlessly slow, and it’s really hard to debug. Cucumber itself is not able to add value there.
But that’s not unique to cucumber either. I don’t think many people realise that test writing and architecture is its own skill in programming and so many suites turn into write-only cludges that take 12 hours to run on a cluster of 50 CI workers.
Cucumber and WebDriver? Two problematic frameworks does not equal a sane one.
Found it quite neutral, not too bad, but also not really helpful.. some nice ideas especially that cross functional collaboration, but in the end the engineers had overhead and hated to write the wrappers, and the other stakeholders still couldn't or didn't want to read/write docs/test, or however it was intended. Admitted, maybe I belong to that incapable crowd as well ;)
I would say most companies use cucumber as a testing tool and say they doing bdd. I would actually like to see a company doing true bdd I have not seen one.
Didn’t the founder post here and said something along the lines of hating bdd now?
Edit: oh I see cucumber was bought by smartbear and then they fired the last cofounder https://mattwynne.net/new-beginning
- I was an outside consultant working with nontechnical but very process oriented management teams
- There was low trust in existing technical workflows: they had been burned by bugs, failure to deliver, and overpromising so a big part of my task was building rigor and consistency into the engineering workflows
- They were extremely engaged in managing the workflows (the core of the business was doing the thing that we built an app to do)
In that context, they loved having a printed out (!!) copy of all the cucumber tests running on the most recent build so that they could see what was passing and what was failing: Color printer, green line passes, red line fails.
Once I got them into the habit of writing "given when then" statements to define existing features, bugs, and functionality, they were happy as a clam being able to specify things. Eliminated a lot of the usual back and forth over "yes but what exactly do you mean" that non-technical people often create due to not being clear and exact enough for technical purposes.
I would implement the parsing and selenium-based testing that validated whether the cucumber was passing and we would have a shared understanding of where the code was at and whether there were any regressions. In the occasional case where my understanding didn't match what they expected, we had context to work together to figure it out.
All in all it earned me a fair amount of money and lowered the stress of being a consultant for those companies substantially so I have a lingering fondness for it. But like I said it's a very specific situation to deliver that kind of value.
For simple and very clear Given/When/Then scenarios with clear inputs and results that were easily represented with a small amount of text it worked. Think a trading system with a few codes and a small amount of data on the left and an expected result like a calculation on the right.
Where there was a lot of stories that branched off from one another (e.g. a website with lots of potential user flows) it inevitably led to stories that were too vague or too repetitive or more usually, both.
I'm convinced that it does work for a certain kind of application for which the DSL is capable of representing its scenarios, but it's not the kind most of us develop.
There were also a lot of cucumber scenarios and suites involved. Like think dozens of printed pages in relatively small font. Concise it is not. But the people involved wanted verbose and clear over concise and easy :)
It can be a very reassuring way to develop software for stakeholders - "We added 8 scenarios, and made a further 4 failing scenarios pass this week" - as opposed to "our developers spent 6 months and then quit without delivering any code"
I'm glad Cuke made you money. There are lots of ways to make money from mismanaged companies.
If we were actually professional engineers, we'd have a code of ethics. I'd like to think that the ethical code would involve not doing harm to a business or intentionally obscuring harm being done.
[1] https://en.wikipedia.org/wiki/Behavior-driven_development
But it's worth it - I find acceptance tests are great for testing systems from a data perspective, i.e. what comes in to our app, what we send to other apps, what we respond with, etc. The data coming in and out is ultimately what I care about ensuring.
I haven't yet seen acceptance tests that are readable by "business", but they do help remind me what the usecases in our system do.
I can speak for RSpec and say the output of the test runs can be formatted such that it generates specifications that are as "non-technical" as Cucumber. If you scroll down a bit from https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-trave... you'll see a similar output.
Rspec, and other frameworks like it, are good enough these days where you can write expressive tests that can be understood by non-technical people and not have to deal with writing and testing the interpreter.
Probably the most important thing was that with a lot of test frameworks, having poorly architected test utility code is annoying and more time-consuming, but has a limited blast radius. Hacky tests can be easily isolated from the other test code. In Cucumber, proper architecture and reusability are paramount. If you screw this up, your tests become unmaintainable really quickly!
- Maintain documented requirements for ALL CODE FUNCTIONALITY - Maintain tests for ALL REQUIREMENTS - Be able to produce reports for auditors that prove linkage between the requirements and the tests (to prove that all requirements were tested)
In this context, Cucumber was really nice since it automatically solved all three of these at once! For general testing purposes, though, I think Cucumber has too much overhead for it to be worth using...
Just use a programming language.