News for hackers. Stuff that matters.
Re:IndexedDB is completely broken in latest Safari by hu3 on Tuesday June 15, 2021 @12:32AM (#27511318)
> Opening an IndexedDB database fails 100% of the time on the first try.

This is something that should have been caught by automated tests.

Re:IndexedDB is completely broken in latest Safari by andrewmcwatters on Tuesday June 15, 2021 @12:37AM (#27511344)
Oh man, if you think browsers are well tested automatically, you’re very far off reality.

There’s basically no automated test suite even for CSS 2.1 that all vendors can rely on that matches behavior specified by the standards.

Luckily standards like that are so mature that it’s not needed as badly, but it definitely keeps incumbents safe from competition.

Re:IndexedDB is completely broken in latest Safari by mh- on Tuesday June 15, 2021 @01:56AM (#27511791)
I assume you're aware of the WPT project (contributed to by all of the major browser vendors these days), and its CSS 2.1 coverage is lacking in a way I'm unaware of.

But for others who don't know:

https://web-platform-tests.org/

https://github.com/web-platform-tests/wpt/tree/master/css/CS...

Re:IndexedDB is completely broken in latest Safari by andrewmcwatters on Tuesday June 15, 2021 @02:08AM (#27511861)
Yes, this is the closest thing we have today, but it is built in such a way that implies you have a fully functioning browser already, which sort of defeats one aspect of the need for such a suite.

They’re also tests that don’t actually point anything out.

Take the CSS2 tests for example, https://wpt.live/css/CSS2/, I can’t do basically anything with these. They don’t tell me whether or not a particular part of the implementation has failed. They’re all basically arbitrary tests that sort of cover the appropriate properties, sometimes, maybe.

The biggest glaring issue is that the test runners run in the browser. Well, if my user agent doesn’t have JavaScript… I can’t “run” the tests.

Further, there are some parts of the standard that can’t be automated simply because it’s up to user agents to decide how the final rasterization turns out. You can’t just compare rasterization output or even reliably look for element heights for verification based on the differences between font rasterization.

This is also why a part of the suite requires manual verification.

What I really need as an implementor is the ability to point my software at some .html files, get some failures, add more implementation, get less failures, and repeat. With, of course, the caveat being that you can only really do this for what is well-defined in the standards.

But you can’t do that easily with the wpts.

Re:IndexedDB is completely broken in latest Safari by jgraham on Tuesday June 15, 2021 @03:47AM (#27512442)
Hi, I work on web-platform-tests.

I'm not really sure I follow your complaint, so I wonder if there's a point of misunderstanding. The tests written for DOM APIs certainly require running javascript to execute the test — that's a given — but most of the rendering tests are reftests. These consist of two files, one using the feature under test and one avoiding that feature [1]. The test passes when the two renderings are identical. That does make these tests difficult to run "by hand", but in terms of implementation it's possible to automate in anything that's capable of rendering to an image. Reftests are preferred over comparing rendering to a fixed image because fixed images tend to be invalidated due to unrelated/permitted changes in the rendering (e.g. a different font or different antialiasing choices). These will typically apply to both test and reference and so are handled automatically.

Reftests obviously aren't suitable to bootstrap getting very basic things working, so there are some manual tests; I suppose one could hope to avoid that by inventing a special kind of output inspection automation just for those basic tests, but it's hard to justify.

IN general, however, the "the ability to point my software at some .html files, get some failures, add more implementation, get less failures, and repeat" is exactly what wpt offers. There is some work required to stand up the necessary infrastructure to run the tests automatically, but it's common for new implementations of the platform to make use of web-platform-tests (e.g. Servo and Flow have both made use of wpt).

In terms of coverage it's very difficult to demonstrate that all the requirements of a spec, and all its interactions with other specs, are completely covered. There have been various proposals for adding test metadata to try to measure coverage, but these have largely proved impractical. Nevertheless some CSS specs have links from the spec to the relevant test cases. And if you find cases that aren't covered by exisitng tests it would be great to get a PR to add new tests [2]

[1] https://web-platform-tests.org/writing-tests/reftests.html [2] https://github.com/web-platform-tests/wpt

Re:IndexedDB is completely broken in latest Safari by andrewmcwatters on Tuesday June 15, 2021 @11:04AM (#27516525)
I’m not sure what you define as “getting very basic things working” but there’s no acceptable threshold for me. Are line boxes very basic? Is overflow compositing very basic? It’s irrelevant whether or not one is bootstrapping.

It is unacceptable to me to use the software I’m testing to test itself if I can’t establish a baseline for the rest of the tests.

Otherwise, all of the tests could be invalid one day based on a regression and I wouldn’t know. But this is explicitly done with reftests.

And I don’t know what you’re talking about with the DOM API tests, there are several layout and compositor tests that require JavaScript and it’s unclear why other than the test author decided to use JavaScript to verify DOM metrics. That’s fine, I just don’t want that in my layout tests.

I’m OK with a subset of the tests just not being automated based on font rasterization and line height calculations. The standards make those expectations clear, but everything else feels like fair game to me.

I can either test for it, or I can’t. If I can’t, it’s not automated. And if it’s automated, I need information about what is failing, and much of the reftests don’t explicitly point out what they’re testing. The assert data is useful when it’s available, but even then sometimes test authors neglect to point out explicitly what they’re testing for.

Plenty of the layout tests in particular are just box model renders that should match with no explanation as to what section or text is being tested against. That’s just not acceptable. It’s not even about interactions between specs, the specs isolated are not well tested.

If you’re testing, let’s say, the width calculation of a box based on children boxes’ widths in normal flow, it needs to say that. And I think I’ve only ever see some of WebKit’s test’s help links reference actual sections.

Otherwise the WPT provide little to no value to me.

My team has been better off writing our own tests because we directly refer to the recommended or living standards texts by section, quote it and link to it, and as a result there’s no ambiguity about what property, calculation, or raster output we’re testing for.

Re:IndexedDB is completely broken in latest Safari by jefftk on Tuesday June 15, 2021 @03:22AM (#27512302)
That seems unrelated to whether WPT could cover this bug?
Re:IndexedDB is completely broken in latest Safari by andrewmcwatters on Tuesday June 15, 2021 @12:30PM (#27517785)
This subthread isn’t discussing IndexedDB.
Re:IndexedDB is completely broken in latest Safari by jefftk on Tuesday June 15, 2021 @02:16PM (#27519357)
It isn't? It looks to me like this subthread started with a question of whether this regression could have been caught by automated tests: https://news.ycombinator.com/item?id=27511318
Re:IndexedDB is completely broken in latest Safari by hu3 on Tuesday June 15, 2021 @12:57AM (#27511461)
Are you speaking from experience?

Because a cursory glance at Chromium src tells me they do have automated tests for IndexedDB:

https://source.chromium.org/chromium/chromium/src/+/main:con...

Re:IndexedDB is completely broken in latest Safari by andrewmcwatters on Tuesday June 15, 2021 @01:12AM (#27511564)
Yes, I am. I'm an implementor. Some of the most fundamental parts of the web don't have comprehensive test suites. IndexedDB is relatively much more modern.

However, if you do stumble on some tests for basic things like the box model, I'd love to know where they are! Tests like those would allow other vendors to know whether or not they're adhering to specification, programmatically.

Earlier standards are very descriptive, rather than telling you how to implement the actual web technologies. That's left as an exercise for the reader. Seriously!

Re:IndexedDB is completely broken in latest Safari by hu3 on Tuesday June 15, 2021 @01:19AM (#27511599)
Interesting. This makes it even more bafling to me that Safari doesn't have basic unit tests for a modern feature such as IndexedDb. It's certainly not for a lack of funds.

I guess applications living in the browser don't align well with Apple's interest in pushing their app store.

Re:IndexedDB is completely broken in latest Safari by andrewmcwatters on Tuesday June 15, 2021 @01:23AM (#27511625)
A part of it is interest, too. Simply because there is a exceedingly small number of vendors, building comprehensive test suites that benefit newcomers is just not in most organizations' best interests.

There actually are some partial tests from the W3C regarding basic web technologies, but they're not really automated ones in the way you'd think.

Re:IndexedDB is completely broken in latest Safari by shawnz on Tuesday June 15, 2021 @01:41AM (#27511715)
I don't think the other poster was talking about automated tests for conformance to the specification, just basic unit tests indicating that the feature works at all by any reasonable definition.
Re:IndexedDB is completely broken in latest Safari by lostmsu on Tuesday June 15, 2021 @02:26AM (#27511964)
That kind of an issue is generally tested against by integration tests. Depending on the root cause unit tests for the entry point code might not have caught it.
Re:IndexedDB is completely broken in latest Safari by shawnz on Tuesday June 15, 2021 @08:53AM (#27514592)
You are right, an integration test would probably be better for demonstrating that this feature works.

I only mean to say that you don't need a whole compliance testing suite to catch a major bug like this.

Re:IndexedDB is completely broken in latest Safari by onion2k on Tuesday June 15, 2021 @01:46AM (#27511743)
However, if you do stumble on some tests for basic things like the box model, I'd love to know where they are!

Its not really a test suite, but back in the early days of CSS we had https://en.m.wikipedia.org/wiki/Acid2 It'd be easy to write a set of tests based on Acid2.

There's also https://en.m.wikipedia.org/wiki/Acid3 but that doesn't quite test things 'properly'.

Re:IndexedDB is completely broken in latest Safari by andrewmcwatters on Tuesday June 15, 2021 @01:59AM (#27511813)
Yes, I fondly remember the Acid tests. Unfortunately, they aren’t very useful for implementors as they don’t cover the entire breadth of the specifications in isolation.

In addition the specs have changed somewhat so now CSS 2.1 and the modules that existed at that point in time have matured beyond what the tests were written for, so a browser matching the spec today would fail.

They were very fun for end users to see whether or not a browser was in compliance, though!

Re:IndexedDB is completely broken in latest Safari by exikyut on Tuesday June 15, 2021 @03:36AM (#27512379)
I wonder if the path of least resistance would be to slowly convince the people that like writing Acid* tests to take things up a notch and rearchitect the next version(s) so that success is a measurement of collective+isolated implementational correctness.

Such an effort would probably involve a higher level of ongoing support from the vendor side of the fence (in terms of motivation and iterative discussion etc), but relative to other options might be, as I said, the path of least resistance.

</armchair_idea>

Re:IndexedDB is completely broken in latest Safari by gok on Tuesday June 15, 2021 @02:06AM (#27511849)
So does WebKit/Safari. Hundreds of automated tests for IndexedDB, in fact.

https://github.com/WebKit/WebKit/tree/main/LayoutTests/stora...

Re:IndexedDB is completely broken in latest Safari by ec109685 on Tuesday June 15, 2021 @01:43AM (#27511726)
Seems like Safari could just adopt these tests…
Re:IndexedDB is completely broken in latest Safari by danjac on Tuesday June 15, 2021 @01:20AM (#27511607)
Forget automated tests; this is something the most cursory QA would have caught. The fact that they evidently have neither is a reflection not of technical ability but plain neglect.
Re:IndexedDB is completely broken in latest Safari by dmitriid on Tuesday June 15, 2021 @03:50AM (#27512461)
Remember when Chrome broke audio for everyone, including their own demo pages? [1]

Testing browsers is a daunting task. A modern browser is 10-15 million lines of code with literally thousands of exposed APIs [2]

[1] https://www.pcgamer.com/a-google-chrome-update-breaks-the-au...

[2] https://web-confluence.appspot.com/#!/confluence

Re:IndexedDB is completely broken in latest Safari by hliyan on Tuesday June 15, 2021 @02:11AM (#27511878)
QA very probably caught it, did a refresh (thinking that it's a problem with their test setup, as it is frequently the case) and then went about testing.
Re:IndexedDB is completely broken in latest Safari by speedgoose on Tuesday June 15, 2021 @01:16AM (#27511578)
Apple never test safari very well, I have seen so many bugs that are easy to spot and reproduce.

My favorite one was websites added to the home screen on an iPad would have the clock displayed above the websites if you rotate the iPad after opening a website. It wasn't fixed for years and it may still be there.

Re:IndexedDB is completely broken in latest Safari by gsnedders on Tuesday June 15, 2021 @03:49PM (#27520568)
It's opening the database _immediately_ after the browser starts that's the problem, which is easily missed given the browser isn't restarted for each test. Provided an IndexedDB test isn't run immediately after any restart, it'll pass.

As with every race condition (cross process, too, so tools like TSAN don't help), testing for the absence of race conditions is hard.