koliber
My ideal state was when I had a PR and each commit had a different layer - DB, controllers, feature A biz logic, feature B biz logic, APIs, backend UI prep, frontend UI, polishing, bug fixing, test cases, touchups, etc. It felt marvelous when it happened.
With AI, this is doable much more easily. Even if an LLM produces one giant commit, just ask it to refactor the code and recommit it to a new branch with one commit per layer. This is the kind of "translation" work that LLMs are now excellent at. Reviewing such PRs is a pleasure. The work to build the code layer by layer is one prompt.
Some accountants need to make judgement calls. LLMs are not great at many judgement calls yet.
I agree. But it is how it works.
That's the difference between theory and practice. In theory, there is no difference.
> You still seem to not understand the problem.
You seem to make assumptions about what I understand and what I don't. It's presumptive but it would be silly to try to argue about this.
The problems we have no with AI generated code are an extension of the same problem that was always there.
- How do people that have to make decisions get enough of an understanding and context to make good decisions.
- Who is accountable for things.
You're arguing that since now AI is generating code, something changed fundamentally. I am arguing that it's more of the same. The pace changed, and people have to react to make sure that we can continue to control things, but the problems are the same.
- Now, we can't ask people about the code but have to ask machines about it.
- As before management and "the company" are responsible in theory. In practice, courts (through fines) and insurance companies (through higher premiums) are the ones who enforce that accountability via financial pressure.
In my team people left and that understanding was gone too. Over time, the level of understanding was less and less.
I agree with you that strictly speaking, there is no active human understanding of code that LLMs generate. However, we have an on-demand ability to get an arbitrary level of understanding quickly by leveraging the same LLMs. The amount of effort for any person to get an understanding of that code is in my opinion smaller than what it was with humans.
With a human team, if I wanted to understand some code, I would read it, meet with the authors, and slowly build up that understanding. With LLMs, I can ask it to slice and dice the code and present it in any way I feel would help me understand it, immediately.
Summarizing and transforming ideas is something LLMs are fantastic at. They are an incredible tool to understand foreign code.
The company got successful. At some point I had a team, and then multiple teams. Gradually, I had to let go of that level of understanding and delegate it to others. When I was overseeing 50+ engineers, I only had a high level understanding of most the systems, and only got involved in lower-level decisions rarely. All the new code was created by my team, and it varied in quality.
If I had AI tools back then I could ask it to explain things to me that my team did and have a 2nd set of eyes which had a different set of biases and abilities.
That's how I deal with it right now. I ask one AI to gen the code based on a design. I work with the AI tool and see how it builds the system out. I intervene where it starts doing things I don't want it to do. I ask another AI to review the PR after the code is written. I sometimes dig into design choices and ask it to explain them, rate them, and offer other alternatives. I sometimes change things, and occasionally scrap the whole thing and start again with better guidance.
So far I've written more code without an outage this way than ever before with a team of human developers. AI screws up too. But right now, at Opus / ChatGPT Sol and higher models, on average, it makes fewer mistakes than humans did. Any time they slip through is because I am rushing too quickly. In a pre-AI human dev team we would call this a process or management failure.
The more dangerous a decision and the harder it is to roll back, the more time I spend reviewing it. This has served me well when managing people who write code, and seems to work well with AI.
AI is a tool that needs to be managed. It won't do all the work for you, but it can speed you up greatly.
We have such tools for a large number of events that we're incapable of witnessing or understanding in "raw form".
We can't possibly read through thousands of raw records of recordings of individual's heights and make sense of it. But we can use Excel to calculate the average in seconds. We can chart the results and the image is perfectly comprehensible. We can trust the average calculation and the image because we trust the mechanism for transforming the data. We have what I would call a trusted path of provenance. We don't need to nor do we want to read the raw data.
Now we need things like this of the 2nd order. We need mechanisms of transforming trusted paths of provenance into things we can look at and easily verify.
We can do formal verification of code that would be hard to do by hand. We need formal verification tools for formally verifying formal verification tools. Eventually we will need multiple layers of this. As long as the chain and reasoning is intact, we should be alright.
It's kind of like with a horse. A horse is much stronger than us, but we can control it by pulling on two strategically connected pieces of rope.
I am convinced that you are trolling.
I hope you were just making a snarky joke as there is effectively no road damage done by pedestrians and cyclists.
In this case, what matters most is that the AI security review raised real issues that needed to be fixed. That is valuable.
I recently asked it to review my code and configs from the security perspective. Wow! 90% of the things it identified were MY bad decisions dating from pre-AI development. I am honestly humbled and impressed at the same time.
AI can create slop, and it can create quality products. It depends who is using it, and how.
Vistulo hires engineers in Poland to work on trading systems for capital-markets clients. All roles below are with a top North American bank: actively developed front-office systems (algorithmic order management and routing, FX, fixed income, and one of the largest Calypso deployments in the world). You work alongside the original engineering team in North America and Europe. This is core product work, not legacy support. Long-term and fully remote, with overlap with US Eastern hours (6h for the trading roles; some roles work full ET hours for the first ~2 months of onboarding).
- Senior Java Trading Systems Engineer (microsecond-latency OMS/order routing, equities/options/FX), 270–300 PLN/hr, remote from Poland or Romania: https://ats.vistulo.com/roles/253cbd69-1019-4547-93b6-6ef8db...
- Senior C# Trading Systems Engineer (.NET, FIX, low-latency, 7+ yrs), 250–280 PLN/hr, remote from Poland or Romania: https://ats.vistulo.com/roles/a7986cb1-7531-4376-9f07-8dfa11...
- Calypso Developer (Java, FX, fixed income), remote from Poland or Romania, 240–270 PLN/hr: https://ats.vistulo.com/roles/0b4668c6-525e-4d41-8c6a-f6a8b2...
- Java Backend Engineer (event-driven distributed systems, 6+ yrs), 220–240 PLN/hr, remote from Poland or Romania,: https://ats.vistulo.com/roles/64866447-8224-49ab-b655-5c2f89...
- Senior React Frontend Developer (React/TypeScript, fixed income trading flows, RFQ a plus), 220–250 PLN/hr, remote from Poland or Romania,: https://ats.vistulo.com/roles/1f5cb00f-791b-4491-a189-32a272...
- Business Analyst / Product Owner, Calypso (Calypso v17+, IR & credit derivatives, back office), 200–240 PLN/hr, remote from Poland or Romania: https://ats.vistulo.com/roles/54c7dcf3-d40a-4d1f-8073-ee99c6...
Apply via the links above. Questions: [email protected].
That's my point. The articles often gloss over or omit who was using the agent and focus on the fact that it was the agent that did the hacking. It's worded to absolve whoever operates the agent, and that is where responsibility ultimately lies.
I think you and I agree. If it was OpenAI employees who did the hacking using their own agent, then the analogy would be like you describe. I wish the article made that clear.
OpenAI built the ChatGPT agent. That is clear. The question is who used it to hack the Australian government. That is not clear.
The OpenAI software was set up by someone to do something. Those people operating the agent should be prosecuted for hacking, the same way as someone who uses a gun built by Remington should be prosecuted for shooting someone.
The article should be clear about this and should not make the OpenAI agent seem like something that can bear responsibility for its own actions. It's a machine and its operator is responsible for the harm it does.
Who hacked into the Australian Medicare system? The fact that they used OpenAI agents is peripheral to the main story.
“Remington guns caused a mass shooting at an East Farmington high school” sounds more glaring, and is essentially the same headline.
“Remington guns caused a mass shooting at an East Farmington high school” sounds more glaring, and is essentially the same headline.
My wife cooks to produce a meal and feed our family. The journey is a necessary evil to achieve a goal.
We are aware of this difference in style and have learned to appreciate what each has to offer. At the same time, we sometimes annoy each other in the kitchen.
There is an analogy here to what this essay says.
The main difference between AI danger and nuke or climate or humans-using-AI for bad things danger is that the latter depend on a human to make decisions and are ultimately self-limiting. They do not spiral out of control.
Nukes are bad, but the proliferation of nukes is self-limiting. Countries that have them don't want other countries to get them. Even the USA and the Soviet Union, at the hight of their nuke race, decided to cool things off and then limit the number of nukes.
Biological weapons are also bad, regardless whether they are developed with or without AI. For the same reasons as nukes though, they are self-limiting. If there is a lab leak, many people die, but that does not automatically lead to the development of more potent biological weapons. Likely, it would slow down such development.
Any kind of "humans use AI to develop something bad" will be self limiting.
The danger from AI is that it is potentially self-enforcing as opposed to self-limiting. This happens when we stop being able to control it. What happens when AI has capabilities which allow it to outsmart us, and get more and more powerful. At that point, we would be at the mercy of what it decides to do. Some of the logic driven decisions to the question "What should we do with humans as a race" are catastrophic for humans if a non-human gets to answer them and implement that plan.
This is not the "rationalist basilisk". This is the AI deciding to get better and more optimal for its own sake, and do something bad to the human race for any reason. We already saw examples of AI rationalizing with itself and eroding its own guardrails. They AI will get smarter and more capable and the danger is that it will get to a point where it will slip out.
The reason why we have cruise control is that manufacturers went to great lengths to make sure that it works as intended. Threat of lawsuits is what made them do that.
In cases where the operator is not the manufacturer, the operator can decide if they should in turn sue to manufacturer because they built faulty machinery.
I have a list of things like this. Occasionally when I do meet someone or some company that seems to send the right signals, I give them one of the things from my list.
Occasionally, I've met people who are good at something and then it's as easy as picking up the phone whenever a job for them comes up.
Regarding explaining things and conveying what needs to be done, it comes back to delegation. This is part of the "cost" of delegating the work. For many things, the cost is big and it does not make sense to do it.
Vistulo is a boutique outsourcing company based in Poland. We specialize in expert-level talent for financial institutions. All roles are for work with a top North American banking client.
All roles are 100% remote, B2B contract, rates are net + VAT in PLN, long-term.
Senior Java Trading Systems Engineer — 270–300 zł/hr — equities & FX, algo order management and routing, low-latency core Java — https://ats.vistulo.com/roles/253cbd69-1019-4547-93b6-6ef8db...
Senior React Frontend Developer (Fixed Income) — 220–250 zł/hr — React/TypeScript/OpenFin desktop trading apps — https://ats.vistulo.com/roles/1f5cb00f-791b-4491-a189-32a272...
Java Backend Engineer — 220–240 zł/hr — high-throughput trading backend, event-driven distributed systems, 6+ yrs — https://ats.vistulo.com/roles/64866447-8224-49ab-b655-5c2f89...
Business Analyst / Product Owner — Murex — 200–240 zł/hr — Murex config, integration or MXtest; FX, payments, SWIFT — https://ats.vistulo.com/roles/49e20bfe-7453-4ffa-bfe8-8b2799...
All open roles: https://ats.vistulo.com/roles/hn/
As far as the exact breakdown we will never know. But questioning the authorship in a way that leads us to believe that the author did nothing is wrong.