News for hackers. Stuff that matters.
Re:Sikuli: Automate Anything You See on Screen by jakozaur on Thursday June 09, 2016 @02:29AM (#11867884)
Great project, but it hasn't changed since 2010? Maybe we should add it to the title?

I was hoping that image based testing would eventually overtake Selenium/Appium for web/mobile testing. In principle there are several advantages:

+ It works the way human works. If your button id has change from #submit to #submit-application, your UI test should work just fine. However, if your button became one pixel at top-left corner, test should break. Right now most frameworks do the opposite.

+ If it's based on visual, it would be easier to maintain tests. E.g. Your button change color from light grey to dark grey, would you like to update your tests.

+ Tests would be much easier and faster to read and write. Even less technical folks like current manual QA testing can contribute to automate testing tools.

Re:Sikuli: Automate Anything You See on Screen by charford on Thursday June 09, 2016 @02:43AM (#11867924)
You might be interested in http://www.sikulix.com/, which was forked from this project and is more up to date.
Re:Sikuli: Automate Anything You See on Screen by adamhepner on Thursday June 09, 2016 @03:09AM (#11868007)
Oh, you beat me to it. But the good news is, that SikuliX maintainer seems to be actively developing the next major release version now, and last release is not that old (end of 2015, where the technology don't have to change that rapidly)
Re:Sikuli: Automate Anything You See on Screen by blacksmith_tb on Thursday June 09, 2016 @06:39PM (#11872903)
Re:Sikuli: Automate Anything You See on Screen by hatsunearu on Thursday June 09, 2016 @05:19PM (#11872474)
Link is a 404.
Re:Sikuli: Automate Anything You See on Screen by dack on Thursday June 09, 2016 @03:21AM (#11868031)
I mean, we don't rely on pixels because we usually care more about behavior than we do the actual design. It's too brittle to rely on exact pixel locations in the face of added features and small design tweaks. Maybe it makes sense to have one or two of these types of visual checks, but they should be automated so that regenerating them is easy.
Re:Sikuli: Automate Anything You See on Screen by IanCal on Thursday June 09, 2016 @04:23AM (#11868195)
> I mean, we don't rely on pixels because we usually care more about behavior than we do the actual design.

True, although for some things I do still like a visual test. Your users may be used to clicking a button that looks a certain way in a certain place, and suddenly shifting that might be worthy of requiring a change to a test. You wouldn't want all your behavioural tests done like this though.

> Maybe it makes sense to have one or two of these types of visual checks, but they should be automated so that regenerating them is easy.

I agree. Perhaps there's a nice integration between the two? If you were able to give, say, a css selector and a template image rather than just one or the other to look for you'd be able to report these errors:

* I can't find the button I was looking for, but there is something that matches the css selector [blah] that looks like this: [image] instead of [image]. Do you want to update the image in the tests? Y/N

* I can't find the css selector [blah] but the button appears here [image] with [class] and [id], do you want to update the css selector in the tests? Y/N

If the program is actually still fine then it'd be a quick auto-update, and if it's not then these bits of information are probably the first bit of debug you'd want to do anyway to figure out what broke the tests.

Re:Sikuli: Automate Anything You See on Screen by fizzbatter on Thursday June 09, 2016 @01:26PM (#11870853)
> True, although for some things I do still like a visual test. Your users may be used to clicking a button that looks a certain way in a certain place, and suddenly shifting that might be worthy of requiring a change to a test. You wouldn't want all your behavioural tests done like this though.

A decent example of this would be that i've implemented frontend features before but did not notice an unintended pixel shift. I introduced a bug, albeit just visual, but a bug nonetheless - and i had no idea. Likewise, a minor visual issue like that can get through many stages of review and even deployment.

Pixel interaction may be terrible for early prototyping, but the idea has merit. Or, at the very least, perhaps it could identify by css/dom, and throw warnings for pixel alterations.

I like the idea of letting design people programmatically enforce standards (Though, no idea how to make them do that programmatically lol)

Re:Sikuli: Automate Anything You See on Screen by mmebane on Thursday June 09, 2016 @12:33PM (#11870469)
> Perhaps there's a nice integration between the two?

They both run in Java, so you can combine Sikuli with Selenium pretty easily [0], so long as you're not doing headless testing. I've played around with it a bit, using Selenium to get coordinates of a container element and then having SikuliX operate just on that region of the screen.

[0]: http://www.softwaretestinghelp.com/sikuli-tutorial-part-2/

Re:Sikuli: Automate Anything You See on Screen by tlarkworthy on Thursday June 09, 2016 @03:45AM (#11868100)
The visual matchers are fuzzy matchers, not pixel perfect.
Re:Sikuli: Automate Anything You See on Screen by jakozaur on Thursday June 09, 2016 @03:47AM (#11868108)
It can be some nice fuzzy algorithm with learning capabilities. E.g. if you change background, but button is in same place, you can just update the test.
Re:Sikuli: Automate Anything You See on Screen by mmebane on Thursday June 09, 2016 @12:07PM (#11870297)
From my experience, the fuzzy matching works well enough to handle picking out text or icons on interfaces with semi-transparent backgrounds that change on scrolling (e.g., due to a parallax effect). It's pixel-based and not scale-invariant, though, so it won't automatically handle multiple zoom levels/resolutions.
Re:Sikuli: Automate Anything You See on Screen by marxidad on Thursday June 09, 2016 @11:30AM (#11870047)
you can adjust the fuzziness
Re:Sikuli: Automate Anything You See on Screen by mherrmann on Thursday June 09, 2016 @12:50PM (#11870582)
If you're only worried about your tests breaking because of #submit vs #submit-application you might be interested in http://heliumhq.com. It's a commercial wrapper around Selenium and I'm one of the authors.
Re:Sikuli: Automate Anything You See on Screen by lewich on Friday June 10, 2016 @04:52AM (#11875104)
What is the difference between other wrappers? E.g. Selenide. It looks like you are selling shitty wrapper, not more...
Re:Sikuli: Automate Anything You See on Screen by mherrmann on Friday June 10, 2016 @08:10AM (#11875772)
The only reason you say shitty is because you don't like that it's commercial. ¯\_(ツ)_/
Re:Sikuli: Automate Anything You See on Screen by geoelectric on Thursday June 09, 2016 @09:22PM (#11873667)
Automation lead with a specialty in GUI here,

You do realize we standardize on testing to ID so that you can change everything else (location, parentage, etc.) because that's the stuff that benignly changes during development or for localization, right? The ID is supposed to be semi-permanent and not change in the normal case, so don't make trivial changes like that.

Otherwise, we're fully capable of finding buttons by coordinate, parentage, attributes, whatever. It's just crappy practice to do so. We intentionally lock down some things and not others because we know how software development works and what changes are likely to indicate greater chances of bugs and what aren't. We also code to accommodate different resolutions, adaptive interfaces, visual changes for different languages--your button location and boundaries change when your English 5 character caption becomes a German 25 character caption--etc.

As for the 1 pixel button, it's not just that it might be 1 pixel. If all you care about is static visual correctness, you can do simple bitmap comparison to get that, though that has its own issues. And we can build in basic implicit checks into our frameworks for sane boundaries, z-order occlusion, etc. It's common for control selection to also validate the control could actually be accessed by the user. The bigger problem is that the human interaction details might not work right, It might not visibly depress when you hit it, for example, or might have a hit target that doesn't correspond to its boundaries, or have a janky delay after tapping, or whatever.

Even injecting the most user-like of user events won't always catch those so there's no substitute for actually exercising the interface. And since you're in there anyway, the visual checks become trivial and mostly automatic.

On another subject, there's no way in hell I'll trust a system that takes guesses as to what it's looking at for a regression test suite, which is what nearly all GUI automation suites are. Even if I did use this tool for something less deterministic like model-based GUI fuzzing, I'd need to have another suite built that verified beyond a doubt that the button I expect to be there is there and at least superficially behaves how I expect it to behave. That requires discrete selection and determinism, not fuzzy identification and non-determinism.

The point of a regression suite is NOT to try to make the tests pass, it's to verify that absolutely none of the assumptions I coded into it changed and alert me to manually explore that functionality if they have by failing. Then I can bless the new assumptions with an automation code change or file a bug and xfail the test until the bug resolves. But the assumptions are supposed to be crystalized. You can't trust an alert system unless you know it'd make its decisions to alert the way you would. Unpredictability isn't welcome there.

In this particular case, an ID changing means you removed the control and added another one, conceptually (and possibly practically) speaking, so the automation is supposed to break to trigger a re-examination. Your own code might reference that ID too, you know, so just you having done that means you greatly increased the chances of there being new bugs.