DanyWin

Hi here, Founder of Reduck here. We built a centralized repository of Playwright scripts that are executed over a Chrome Extension and are discoverable / runnable from an MCP. This helps Agents not waste tokens browsing live but can directly do batch actions. This makes Agents 2.5x faster/cheaper on web tasks we tested on. Happy to share insights on how it works / what it unlocks!

Yes, we are working on that! We are preparing to release a feature for people to enable telemetry to contribute to a decentralized and open dataset to train and evaluate models for Selenium code

Exactly! In the future, testers could just write tests in natural language.

Every time we detect, for instance with a vision model, that the interface changed, we ask the Large Action Model to recompute the appropriate code and have it be executed.

Regarding generating tests from bug report totally possible! For now we focus on having a good mapping from low level instructions ("click on X") -> code, but once we solve that, we can have another AI take bug reports -> low level instructions, and use the previously trained LLM!

Really like your use case and would love to chat more about it if you are open. Could you come on our Discord and ping me? https://discord.gg/SDxn9KpqX9

Very interesting indeed!

We are thinking of developing an extension that would connect the browser to LaVague so that actions can be sent to the extension and be executed locally, thus bypassing their barriers

You are exactly right! As I wanted to have a solution that works with many LLMs out of the box, I focused on chain of thoughts and few shot learnings.

Lots of paper show that fine-tuning only helps with steerability and form (https://arxiv.org/abs/2402.05119), therefore I thought it would be sufficient to provide just the right examples and it did work!

We do intend to create a decentralized dataset to further train models and have maybe a 2b or 7b model working well

Thanks! Funny thing, we did not use Vision models but text only with the HTML of the current page. However, we intend to add it to boost performance

Thanks a lot! Love the support <3

This is just the beginning, but it is indeed on the roadmap!

Once we solve browser automation, we intend to support other integrations to further facilitate automation of workflows

It could indeed have an impact on jobs, just like any productivity gains have destroyed jobs.

However, the net gains, in my humble opinion, could be phenomenal. Imagine all the time, mental energy and money spent on navigating through the legacy of today's society? From the legacy legal systems that is super complex, to legacy websites, I believe there is much time to be saved so we can dedicate resources to what truly matters, intellectual pursuits or quality time with friends and family

Here we just provide natural language instructions and the LLMs generate the code appropriate at a given time. If the site changes, we can regenerate the code using the same instruction, so unless the site changes a lot, it is quite robust

There is still a design decision to be made on whether we go for TPMs for integrity only, or go for more recent solutions like Confidential GPUs with H100s, that have both confidentiality and integrity. The trust chain is also different, that is why we are not committing yet.

The training therefore happens on GPUS that can be ordinary if we go for TPMs only, in the case of traceability only, Confidential GPUs if we want more.

We will make the whole code source open source, which will include the base image of software, and the code to create the proofs using the secure hardware keys to sign that the hash of a specific model comes from a specific training procedure.

Of course it is not a silver bullet. But just like signed and audited closed source, we can have parties / software assess the trustworthiness of a piece of code, and if it passes, sign that it answers some security requirements.

We intend to do the same thing. It is not up to us to do this check, but we will let the ecosystem do it.

Here we focus more on providing tools that actually link the weights to a specific training / audit. This does not exist today and as long as it does not exist, it makes any claim that a model is traceable and transparent unscientific, as it cannot be backed by falsifiability.

Exactly! It's not sufficient but it's at least necessary. Today we have no proof whatsoever about what code and data were used, even if everything were open sourced, as there are reproducibility issues.

There are ways with secure hardware to have at least traceability, but not transparency. This would help at least to know what was used to create a model, and can be inspected a priori / a posteriori