News for hackers. Stuff that matters.
Re:Act-1: Transformer for Actions by tasdfqwer0897 on Wednesday September 14, 2022 @04:34PM (#32842992)
Hey, I helped make this! Happy to answer any questions.
Re:Act-1: Transformer for Actions by tux3 on Wednesday September 14, 2022 @05:15PM (#32843449)
On an unrelated note, I imagine this can solve recaptchas and other simple non-visual challenges.

Can I make ACT-1 Sybil a few thousand people on mechanical turk?

Can I submit CVs with ACT-1 for entry-level full remote jobs and have it work for legacy companies, if those companies cannot setup ACT-1 themselves but provide a traditional human jobs interface?

Can I put an interface that interracts with the real world through controls and text on a webpage and have ACT-1 take a physical presence?

Re:Act-1: Transformer for Actions by woojoo666 on Thursday September 15, 2022 @01:58AM (#32847671)
I thought captcha was designed specifically so that AI image recognition couldn't perform well at it
Re:Act-1: Transformer for Actions by TuringTest on Thursday September 15, 2022 @03:50AM (#32848593)
It was, for previous methods of image recognition. Generative adversarial networks and models trained on huge amounts of data might change that.
Re:Act-1: Transformer for Actions by learndeeply on Wednesday September 14, 2022 @05:51PM (#32843915)
Thanks for answering questions!

Are the example given in the blog post considered zero-shot learning?

Was the model trained on the websites in the examples given (e.g. on the Redfin site)?

How much labeled data was used?

Re:Act-1: Transformer for Actions by lee101 on Wednesday September 14, 2022 @08:50PM (#32845580)
I'd be interested to see if we can work together in some way, i'm founder of https://text-generator.io which also crawls links to the web/images to generate better text: https://text-generator.io/blog/text-generator-now-researches...

Some people self host it, so it can solve some subset of the questions, the image understanding part is important, both it can understand a question with image e.g. if a user drops a fridge to search by image, or multiple images (e.g. which of the 10 images is the nicest looking fridge in 1 API request) as well. Also supports getting shared embeddings for images/text/code, which can be important for the information retrieval/question answering example where it needs to first find the relevant context on wikipedia then feed to the reader model to read it out

Also do other custom stuff like retraining etc. Thanks, Lee https://leepenkman.appspot.com/

Re:Act-1: Transformer for Actions by aaaaaaaaaaab on Wednesday September 14, 2022 @04:55PM (#32843225)
What was the training data?
Re:Act-1: Transformer for Actions by version_five on Wednesday September 14, 2022 @05:09PM (#32843386)
Also, are there benchmark tasks that you either created or that already exist that you evaluated the model on?

PS - please don't let this me used at a way to prevent human interaction. Chatbots are a disaster and literally the worst possible application of ML, as a shitty interface to a menu system. I hope this will be used in a way that is not consumer-hostile and that the company actively resists ignorant business attempts to use it to avoid paying for customer support.

Re:Act-1: Transformer for Actions by tasdfqwer0897 on Wednesday September 14, 2022 @07:45PM (#32845044)
Yeah, we did have to custom-build our own benchmarks.

And we are not building a chatbot, we're building something collaborative that you can work with to accomplish the stuff you want to do!

Re:Act-1: Transformer for Actions by tasdfqwer0897 on Wednesday September 14, 2022 @05:12PM (#32843420)
We used a combination of human demonstrations and feedback data! You need custom software both to record the demonstrations and to represent the state of the Tool in a model-consumable way.
Re:Act-1: Transformer for Actions by elcomet on Wednesday September 14, 2022 @05:26PM (#32843612)
How many demonstrations were used ?

And was the feedback data used to train the model with reinforcement learning? Or did you request users to "correct" the action and get a supervised signal?

Re:Act-1: Transformer for Actions by angrais on Wednesday September 14, 2022 @05:47PM (#32843853)
How were the demonstrations annotated? Did you use annotations?
Re:Act-1: Transformer for Actions by blueblimp on Thursday September 15, 2022 @02:03PM (#32856018)
In the Salesforce example, it's modifying database contents. Suppose the model misunderstands your request and modifies data in an unintended way (e.g. adding garbage data or, worse, deleting data). What's the recovery plan?
Re:Act-1: Transformer for Actions by WiggleGuy on Thursday September 15, 2022 @10:21PM (#32861302)
This is super cool; I'd want to stay updated. Do you think you guys could add an rss feed to your blog? I'd want to add it to my rss feed aggregator.