swax

If you're talking about regressions, then yea that's definitely a problem. Currently I'm thinking the AI improves the scripts for its use case locally, being mindful of not breaking other languages. The improvements are submitted by AI, and then I need to use AI again to regression test the improvements.

I've already done it once now with a PR that adds Chinese support, but I think it will need to become more automated with a standard set of videos to test against to catch regressions.

It should work, just clone the repo, open your favorite AI harness and tell it to dub all the WoW audio files into Czech.

It has instructions to extend itself to new languages, so when complete feel free to submit a PR with the Czech support added.

Yea, I'm working on a open source distributed, self-hosted agent runner https://naisys.org/

I also did a private decentralized system that was like WASTE+DHT around 2009 (https://github.com/swax/DeOps)

I did a client called Gnucleus back in the day, the original website is still up at https://gnucleus.org/ I also designed GWebCache with a friend in college (also the IRC caching before that was me, I'm sorry). I think in the end the cost/benefit of the Gnutella architecture wasn't there. Especially in privacy - too much of a liability exposing your machine to serve files and route traffic. Also the efficiency of search over the Gnutella architecture - your query never hit all the peers.

BitTorrent was designed with the good parts like decentralized file transfer, and ditched the decentralized search - simplified with a centralized website/tracker model that can also be members only. That helped people stay more under the radar, as well as allow people to jump on/off to get what they want.

Ultimately the better balance which is why BitTorrent is still going strong today, but there is some nostalgia for the craziness of a single global network. Where people can freely share all their stuff, and downloading/opening files was like rolling the dice.

I’ve been using the same OneNote for like 20 years now. It is backed up, synced and available no matter what device I’m on. Hundreds of lists, notes, references, thoughts, all fully searchable and quick to access.

It's an unsettling feeling as what's more complicated - all the atoms and galaxies, trillions of life forms, the unimaginable distances of our universe OR a relatively simple world model that is our conscious experience and nothing else.

Sure, so you tell the model, here's a command prompt, what do you type next? Ideally, it types commands, but a lesser model may just type what it's thinking which is invalid. You can give it an out with a 'comment' command, but some models will forget about that. The next biggest problem is fake output; it types not just 'cat file.txt' but the following command prompt and fake output for the file.

The biggest mark of intelligence is can it continue a project long-term over multiple contexts and sessions. Like, 'build me a whole website to do x', many AIs are very good at one-shotting something, but not continually working on the same thing, maintenance, and improvement. Basically, after that good one shot, the AI starts regressing, the project gets continually broken by changes, and the architecture becomes convoluted.

My plan is not to change NAISYS that much; I'm not going to continually add crutches and scaffolding to handhold the AI; it's the AI that needs to get better, and the AI has improved significantly since I mostly finished the project last year.

Yea a lot of this is experimental, I basically have plain text instructions per agent all talking to each other, coordinating and running an entire pipeline to do what would typically be hard coded. There’s definite pros and cons, a lot of unpredictability of course, but also resilience and flexibility in the ways they can work around unexpected errors.

Throwing this out there, I have a command line driver for LLMs. Lots of little tricks in there to adapt the CLI to make it amiable for LLMs. Like interrupting a long running process periodically and asking the LLM if it wants to kill it or continue waiting. Also allowing the LLM to use and understand apps that use the alternate screen buffer (to some degree).

Overall I try to keep it as thin a wrapper as I can. The better the model, the less wrapper is needed. It's a good way to measure model competence. The code is here https://github.com/swax/NAISYS and context logs here for examples - https://test.naisys.org/logs/

I have agents built with it that do research on the web for content, run python scripts, update the database, maintain a website, etc.. all running through the CLI, if it calls APIs then it does it with curl. Example agent instructions here: https://github.com/swax/NAISYS/tree/main/agents/scdb/subagen...

Thanks for trying it. If you click near the edge you should be able to drag it somewhere else.

I'm working on a Sketch Comedy Database website:

https://www.sketchtv.lol/

https://github.com/swax/SCDB

Just a fun little CRUD app built with Next.js, MUI, Prisma Postgres. I'm adding Halloween sketches now, if you know some good ones feel free to add them, or anything else :)

AI advancement is coming at us both ways - orders of magnitude more compute, with orders of magnitude more efficiency. Hyper exponential.

I mean we have contexts now so large it dwarfs human short term memory right?

And then in terms of reading a book, a model's training could be updated with the book, right?

Humans are also trained on what they’ve ‘seen’. What else is there? Idk if humans actually come up with ‘new’ ideas or just hallucinate on what they’ve experienced in combination with observation and experimental evidence. Humans also don’t do well ‘ignoring what’s been said’ either. Why is a human ‘predicting’ called reasoning, but an AI doing it is not?

Opus seems to be much better at that. Probably why it’s so much more expensive. AI companies have to balance costs. I wonder if the public has even seen the most powerful, full fidelity models, or if they are too expensive to run.

And you just know people will create AI to do that deliberately anyway.

Sometimes I'll tell two agents very explicitly to share the work, "you work on this, the other should work on that." And one of the agents ends up delegating all their work to the other, constantly asking for updates, coming up with more dumb ideas to pile on to the other agent who doesn't have time to do anything productive given the flood of requests.

What we should do is train AI on self-help books like the '7 habits of highly productive people'. Let's see how many paperclips we get out of that.

Yea, there's some logs here https://test.naisys.org/logs/

Inter-agent tasks is a fun one. Sometimes it works out, but a lot of the time they just end up going back and forth talking, expanding the scope endlessly, scheduling 'meetings' that will never happen, etc..

A lot of AI 'agent systems' right now add a ton of scaffolding to corral the AI towards success. The scaffolding is inversely proportional to the sophistication of the model. GPT-3 needs a ton, Opus needs a lot less.

Real autonomous AI you should just be able to give a command prompt and a task and it can do the rest. Managing it's own notes, tasks, goals, reports, etc.. Just like if any of us were given a command shell and task to complete.

Personally I think it's just a matter of the right training. I'm not sure if any of these AI benchmarks focus on autonomy, but if they did maybe the models would be better at autonomous tasks.

The step changes in autonomy are very obvious and significant from gpt-3, -4, and to Opus. From my point of view given the kinds of dumb mistakes it makes, it's really just a matter of training and scaling. If I had access to fine tune or scale these models I would love to, but it's going to happen anyway.

Do you think these step changes in autonomy have stopped? Why?

I have an open source project that is basically that (https://naisys.org/). From my testing it feels like AI is pretty close as it is to acting autonomously. Opus is noticeably more capable than GPT-4, and I don't see how next gen models won't be even more so.

These AIs are incredible when it comes to question/answer, but with simple planning they fall apart. I feel like it's something that could be trained for more specifically, but yea you quickly end up being in a situation where you are nervous to go to sleep with AI unsupervised working on some task.

They tend to go off on tangents very easily. Like one time it was building a web page, it tried testing the wrong URL, thought the web server was down, ripped through the server settings, then installed a new web server, before I shut it down. AI like computer programs work fast, screw up fast, and compound their errors fast.

I've been working on something similar, here's one of their same tests where the AI learns how to make a hidden text image.

https://www.youtube.com/watch?v=dHlv7Jl3SFI

The real problem is coherence (logic and consistency over time) which is what these wrappers try to address. I believe AI could probably be trained to be a lot more coherent out of the box.. working with minimal wrapping.. that is the AI I worry about.

Amalgamating the pieces is the test. LLMs really can't do it that well.

Is it context size, training, number of iterations, number of agents? That's what's being tried to improve the results.

Using my open source project NAISYS, which is a context friendly command shell wrapper, to compare different AI models in how well they can build a website from scratch

https://github.com/swax/NAISYS

I recently released NAISYS, and open source command shell wrapper for LLMs that makes the shell all context friendly for them. I've been testing it out with building web sites, and improving things along the way. It's funny how good AI is at answering questions, but not like project management. So I'm using NAISYS to explore that.

https://github.com/swax/NAISYS