dagss

    But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.

Sure about that? I thought that what LLMs know about themselves is from humans writing about LLMs?

Certainly the weights that make up the LLMs are not accessible for the LLMs to read. Even if they could interpret them in a meaningful way, which, likely not.

You are both talking about this claim?

    LLMs do fantastic on problems they can copy out of their training set, but they fall apart on actually novel tasks.

I mean, I give Claude what I consider novel problem solving work all the time.

You can make the statement true if you raise the bar for "novel task" to a point where most humans also can't do them. In these threads the goal posts for words like "understanding", "creative", "novel" etc gets moved for LLMs to way past their previous definitions, in fact way past a large % of the worlds human software developers.

To just take one example: Ask for a security review and you get back a list of all the things you would have found yourself after some days of pouring over the code (if you are senior developer well versed in security issues), plus the ones you would have overlooked, but in 15 minutes instead of in several days.

Before LLMs came along this would obviously have been "novel work". A shining example of a human using their intellect! But now that LLMs can do it it is just "copy and paste work"...

When I look at the history of human culture I really see a lot of remixing and "trends" and the leaps are rather small.

What sticks out is really the lack of people doing something totally new. Look back on history and it seems like humans can't do anything except as in response to some input that we re-mix. (Although it is across fields -- a change in economic conditions or scientific outlook may affect music and painting; a small improvement in factory production methods may ripple down to experiments validating general relativity, etc)

Why didn't Mozart create a piece of jazz or rock to mix things up? Why are there no medieval cubistic painters? Why didn't someone understand $scientific_topic before?

Most such questions can be answered by some kind of "the social environment and culture at that time didn't prepare people to do it". I.e., humans couldn't remix it yet from what was already there.

Yes there are leaps but they seem to always be within some small "box" that prior knowledge creates for us. We can jump one inch at the time but no further..

Yes that is what I say so we agree.

"Writing code is probably something LLMs have excelled at the most. But how much of their ability to write code is due to their understanding of it vs. just them recalling code snippets which were present in their training dataset or their context window?"

This makes me think mostly about how humans work.

Programming is to humans like speaking a language. You have ideas for what to achieve and express, and if you are a proficient programmer then outputting those thoughts in a programming language is second nature.

Programming (like most things humans learn) takes practice. It is NOT enough to simply read a specification and then you know how to code (except for transferring skills between similar languages). You repeat, make mistakes, and learn from them.

I would be tempted to say that humans definitely "repeat snippets from their training dataset". We mainly recall and express from our training/experiences. Creativity is mostly recombination of things already seen.

I have used -- for decades.

Yes I have often caught stuff like "not using DB transactions in the right way" or "this is not going to be race safe" for instance from juniors just by glancing on a PR.

I find recent LLMs don't do such mistakes though, in my experience. And also LLMs will find such mistakes if asked to review, if not perfect then more predictable than humans.

I meant it seriously though.

Characterizing the difference between humans and AI is going to be important.

Pointing out that AI is software isn't helpful in answering that question.

What is the defining feature of human minds making them "not software"?

Running agents in MicroVMs and skipping permissions is the most impactful change I have done in my habits for a while.

Have a look into Docker sbx for instance.

...and cars also changed very rapidly during early years, not to mention how often they would break down and how they basically required the user to be a mechanic to repair them on the spot for many years.

So the analogy isn't all that good is it?

You are comparing immature tech with very mature tech.

What technology emerged into the market fully finished?

If your point is "don't use technology until it is mature" then you are of course free to not use it for now.

HN is probably not the best forum for theological debate...

But e.g. modern American evangelical churches are to me quite clearly quite far from the original meaning of the texts with some theology outright invented in 18th and 19th centuries.

It really is the case that what you bring to a text colors what you get out of it, and the more people insist that they are reading a text "as it is clearly written" the blinder they seem to be to this.

Sorry, I didn't mean to make any statements about you personally, at all (and I do feel you are interpreting my comment uncharitably).

It has nothing with knowing you or not to do at all, I was just commenting on the contents of your post, not about you.

I meant that the act that you described (comparing religion with a virus) is in itself using the name-calling-aspect-of "virus". One is borrowing the negative connotation for a purpose, because one has (for other reasons) decided that the religion or ideology is bad.

The descriptive property of virus that you then mention (as in being "viral", packets of information in cell etc) equally applies to liberty, democracy and other things that (at least I) consider good.

The point was that when you do this you participate in name-calling/slurs, not making an argument.

Many things that you personally think are good (I don't know you but say for instance ideas around democracy or liberty) can also be described as viruses in the same way.

Not sure how to read your post. Is your argument that since humans have never been _demonstrated_ to be wiped out by a synthetic virus already, it is not likely to happen in the future?

I'm trying hard to make your post make any sense at all..

I guess you believe everyone gets the same starting point and same opportunities? How did you come to believe such a thing? Have you looked around to see what society is like?

I can agree that not calling it "reasoning" may be correct.

But who knows what human "thinking" is really about. If I find a solution to something it is seldom by painstakingly tracing that A and B leads to C (for that I'd need pen and paper). Rather, thoughts just swirl around and then suddenly a solution, or a hunch about a direction to go in, pops into my mind. Who knows what such thoughts "look like" in humans. It is not all of it I can introspect.

Yes I can sort of follow along some kind of train of thought in my head, but there's a lot going on between each thing I'm consciously aware of that I'm not aware of at all, which probably dominates what you are consciously aware of. (Humans are experts at post-rationalization and so on.)

I see this pattern a lot in AI anthro discussions: (1) Assume humans are some kind of perfect idealistic reasonable beings. (2) Hold LLMs up to the standard of an perfect idealistic reasonable being. (3) Conclude that LLMs fails this test, and are therefore not "intelligent", or in this case "thinking", like humans are.

Problem with the argument is comparing humans in anyway to something that is idealistic, reasonable, intelligent in the sense that is implied in these discussions. Human minds are a mess too and fall short of the same standards, just in very different ways from LLMs.

What do you mean GUI? Of course the observation of the agents behaviour happens by reading the code it emits.

But if you observe that the agent day after day do handle user input safely; and also routinely run an agent that scans for security vulnerabilities and observe it finding cases where input is not handled safely in existing code, you may conclude that the chance of an issue is at the same level at, or probably lower than, if a human wrote it and a human reviewed it.

("Escaping" user input is not good practice though, use parameters, assuming you are talking about SQL.)

Even if the fake is super-fast compared to postgres (1%) in CPU resources: Why spend extra engineering effort to build something that has less fidelity?

CPU for test runs is cheap... and if you use any AI agent at all, tests runs faster than the agent does work. Why does it matter how much faster they run?

The goal of a regression test suite is to catch issues before you deploy to prod. That 95% is a nagging source of doubt. CAN you just release the code straight to prod? Or not?

Many times integration tests including the real postgres and real migration catches bugs for me before they go to prod.

Personally I would just never go with 95% fidelity in tests. Testing with the real postgres is just so good.

And just to get test coverage for the migrations themselves?

(In my case I also use stored procedures, RLS etc that needs those; with your setup that is just not on the table I think or you loose coverage of critical code.)

If you are unit testing something then sure mock out everything you are not concerned with.

But for service layer tests I don't agree with you, including the actual production repository implementation and the DB is very useful.

It is like e2e tests, just leaving the frontend out. I really like having a lot of such tests to be productive with backend development.

To have a 100 such e2e tests complete quickly, spinning up SQL DBs cheaply is essential.

Also for actual e2e tests with frontend I prefer cheap DB clones rather than reusing DB between tests and having to worry about state between tests.

I think perfect layering was more relevant in the 2000-2010 with less powerful machines. Just making pseudo-integration tests and including more than may be strictly needed is fine. It completes quickly enough. When it breaks it is usually obvious what broke without limiting was code is included in the test.

BETA: I have been following https://github.com/superradcompany/microsandbox a bit, very simple to get started with at least.

I feel like the article is leaving the most important part out ; how to allocate that "offset" (sequence number) which allows consumers to keep tabs on how far they have read and query if there are new messages in the fallback. There are a few schemes, some more creative than others, but it is not a trivial problem to solve without complexity or possibility of lock contention (or races with consumers if you do it the wrong way). Most people I guess have writers acquire a lock, for instance on a single row in a state table, for allocating the next sequence number for an event topic.

What is the best way? Perhaps a tool that reads CDC and writes allocated event sequence numbers to another table would work ok, or would that have long latency?

And that CDC processor should then do the NOTIFY too.

Also on the consumer side in some usecases batching can drastically improve throughput. You don't even use LISTEN/NOTIFY then. Just run the consumer in a loop and each iteration process all new unprocessed messages, and store your last sequence number processed between iterations.

It is downvoted because you first talk about "AI psychosis", then acknowledge the very issues people are concerned about.

If our prisons are all random bricks glued together, that doesn't change the practical problem caused by latest AI models more easily exploiting this.

Or to be more specific, issue isn't text/templating as such. It is a fine templating language for text.

But to use a text templating language to generate YAML, which is not only structured, making it a bad match for character based templating with no knowledge of syntax, but indentation-based, making the combination almost seem like a joke.

Helm charts must be the worst engineering solution I have seen in popular use. First time I saw it I didn't believe it, that this was something people did, for real.

Comparing changes in nuclear doctrines with people choosing a different way of life (granted, also for their children) than the majority seems totalitarian, to me.

Minorities are, well, in miniority. Noone is at any point waging "everyone's supper" by trying out alternative ways of farming within their small miniority. (Meanwhile the majority IS risking everyone's supper in some decades).

Nuclear is different from your other examples because the choices of a small minority can drastically affect the vast majority.

Do you feel current western capitalistic culture is perfect and the peak of what humanity can accomplish?

There should be more, not less, experiments in alternative ways of life. I wish there was a lot more examples because we desperately need to change some things and some people need to be first.

As a non-American I don't know much about amish and there could be atrocities I am unaware of, but from what little I know I have always respected Amish for daring to be different, and for living sustainable and not contributing to climate change.

If you trade their belief in God with increased CO2 emissions -- why would that be a rational change to their culture?

So who are really misdirected humans? I would say those who sacrifice the planet on the altar of numbers stored in computer systems in banks...

Reading tip for you is "Sapiens" of Harari. Don't worry, he's an atheist, but he may contribute a more nuanced view on the role of religion in human culture (and he names capitalism as a religion too).

Of course. But I prefer IDEs that group and name the agent sessions by the worktree they are tasked with automatically.

Claude can easily do the merges I need (or myself manually for that matter). I guess all codebases and usecases are very different here and hard to give general advice.

But I do have many years of experience working in a larger team and it's the same problem there (just that people want to merge after some days of working). I'm not sure if AI changes the picture much vs working in a team.

Either way one has to plan ahead a bit and select tasks that are not going to trample on each other. I can typically imagine roughly what the code generated is going to be (at least what files will likely be involved in what way) and when selecting tasks to work on I take into account if it's going to likely cause conflicts.

In my experience if different branches work on different things, Claude have no issue doing "trivial" merges where you just ended up changing different aspects of the same lines. Of course, if two branches rewrite the same pieces of code there's a problem -- so don't do that..

As the sibling comments note, this is kind of off-topic to my post.

But I think git worktrees are a bit more ergonomic, I don't have to think about local vs upstream there's just one place to push.

I like to organize my projects like this:

    myproject/.repo/git  # bare repository .. my own convention..
    myproject/main       # worktrees from ../.repo/git
    myproject/feature1
    myproject/feature2