> we tried to reduce flakiness by buffering clicks/interactions with sleep()...

You should have used waits instead of sleep. Wait pools until your DOM is in the state you are waiting it to be, unless given timeout occurs: https://www.selenium.dev/documentation/webdriver/waits/#expl...

It's been a while. I think we did use waits but they were still flakier than you might expect.

It works acceptable for me with a caveat. I do have Retry(3) on UI tests, because yeah, somewhat flaky.

Especially when system has to warm up.

In the end it is Okay

Ouch, that does sound like a painful cycle. It happens, and no one tests as much as they'd like to.

> Portability of the data.

I hear your concern here. The replay data and session data is also saved to disk, so you can save this somewhere. Of course that still leaves the risk of the record & replay tech. I think open sourcing this would solve the portability issue here, and it's something we're actively talking about but haven't reached a conclusion on yet. Anecdotes and examples like this though are incredibly helpful in helping us make that decision.

Thank you for the feedback!

I worked at a couple of dev tool startups that required you to invest some effort to adopt and would require effort to move away from (e.g. if the company went under) - open sourcing ended up being basically a necessity to sell the product (enterprise sales during seed and A). YMMV, but I would definitely encourage open sourcing enough to make potential adopters feel comfortable that they won't have to do a big, urgent migration and/or lose a bunch of engineering investment if you go under or decide to pivot.

I used to work for Testim.is that does this. I’m now a user at Microsoft (which we’re using before I joined).

Overall the idea works very well and Testim had/has a lot of customers using record/playback.