News for hackers. Stuff that matters.
Re:A ChatGPT mistake cost us $10k by wavemode on Sunday June 09, 2024 @06:13PM (#40628059)
No, a lack of monitoring cost you $10K. Your app was throwing a database exception and nobody was alerted that this was not only happening, but happening continuously and in large volumes. Such an alert would have made this a 5-minute investigation rather than 5 days.

If you haven't fixed that alerting deficiency, then you haven't really fixed anything.

Re:A ChatGPT mistake cost us $10k by spencerchubb on Sunday June 09, 2024 @09:49PM (#40629354)
I agree this is more of a monitoring mistake, and little to do with chatgpt
Re:A ChatGPT mistake cost us $10k by rvnx on Monday June 10, 2024 @02:37AM (#40630839)
It could have happened with any programmer writing the code (ChatGPT or not)
Re:A ChatGPT mistake cost us $10k by fragmede on Sunday June 09, 2024 @11:50PM (#40629963)
Right? The log message would have said the id isn't unique, and then it would have taken much less time to debug this problem.

Programming when everything works is easy, it's handling the problems that makes it hard.

Re:A ChatGPT mistake cost us $10k by carbonatom on Tuesday June 11, 2024 @09:24AM (#40645919)
Are you guys able to read this post? When I visit the OP post, I see only this text:

"Under construction "

Looks like the OP removed the post?

EDIT: Found archived copy of post: http://web.archive.org/web/20240609213809/https://asim.bearb...

Re:A ChatGPT mistake cost us $10k by sneak on Sunday June 09, 2024 @11:56PM (#40629992)
TBH, if the backend were written in Go, this probably wouldn’t have happened to the extent it did. Somewhere in a log a descriptive error would have shown up.

One of the reasons I use Go whenever possible is that it removes a lot of the classic Python footguns. If you are going to rewrite your backend from Javascript, why would you rewrite it in another untyped, error-prone language?

Re:A ChatGPT mistake cost us $10k by TheDong on Monday June 10, 2024 @12:29AM (#40630153)
In python it's harder to ignore errors than in Go.

In go, I've definitely seen:

   tx, err := db.Tx()
   defer tx.Commit() // silently ignores the error on committing, which is the important one
That would have masked this error so it didn't get logged by the application.

In python, if you ignore an exception entirely, like I did that error above, you instead get an exception logged by default.

Python's exceptions also include line numbers, where as Go errors by default wouldn't show you _which_ object has a conflict, even if you logged it.

In general, python's logs are way better than Go's, and exceptions make it way harder to ignore errors entirely than Go's strategy.

Re:A ChatGPT mistake cost us $10k by randomdata on Monday June 10, 2024 @01:52AM (#40630619)
> I've definitely seen

What did they say was the thinking behind it? defer tx.Rollback() would make sense, but defer tx.Commit() is nonsensical, regardless of whether or not the error is handled. It seems apparent that the problem there isn't forgetting to check an error, but that someone got their logic all mixed up, confusing rollback with commit.

Re:A ChatGPT mistake cost us $10k by TheDong on Monday June 10, 2024 @02:50AM (#40630910)
"defer tx.Commit() is nonsensical"

It's not pure nonsense, it works in the happy path, and it matches the pattern of how people often handle file IO in go.

    f, err := os.OpenFile(...)
    defer f.Close()
... which is another place most gophers ignore errors incorrectly. Just like the "defer tx.Commit()" example, it's collocating the idea of setup and cleanup together.

Those two patterns are so similar, python handles them in the same way:

    with db.begin() as conn: # implicit transaction, gets automatically committed

    with open(...) as f: # implicit file open + close pair, automatically closed
You're of course right that go requires more boiler-plate to do the right thing, but the wrong code has no compiler errors and works in the happy path, and fits how people think about the problem in other languages with sane RAII constructs, so of course people will write it.
Re:A ChatGPT mistake cost us $10k by bryancoxwell on Monday June 10, 2024 @08:06AM (#40632680)
Unrelated but I’ve never really understood what to do with an error that gets returned by Close() aside from logging it. I’ve also never experienced that error being returned and don’t really understand what could even cause that.
Re:A ChatGPT mistake cost us $10k by randomdata on Monday June 10, 2024 @08:58AM (#40633111)
A write is not guaranteed to be written if close fails. You almost certainly want to do something with the error state to ensure that the data gets safely stored in that case.

If you are only reading, then sure, who cares?

Re:A ChatGPT mistake cost us $10k by randomdata on Monday June 10, 2024 @08:56AM (#40633099)
> it's collocating the idea of setup and cleanup together.

Okay, sure, but Rollback is the cleanup function. You always want to rollback – you only sometimes want to commit. I suppose this confirms that someone got their logic mixed up.

Re:A ChatGPT mistake cost us $10k by owl57 on Tuesday June 11, 2024 @02:43AM (#40643263)
> You always want to rollback

This phrase highlights the confusion. If you learned SQL before Go, you want to rollback only on error.

Every time I write `defer tx.Rollback()`, I cringe and have to remind myself that yes, it's actually ok to call a method called `Rollback` after succesfully writing data.

Re:A ChatGPT mistake cost us $10k by randomdata on Tuesday June 11, 2024 @09:35AM (#40646023)
If you learned SQL before Go, you'd know the pattern works just fine and is arguably a good sanity check.

    BEGIN;
    INSERT INTO ...
    COMMIT;
    ROLLBACK;
It is not some kind of Go-ism. The Go database/sql package actually executes ROLLBACK in the SQL engine. Check out the error returned by it.

Perhaps you mean learned SQL in the context of languages that consider a failed rollback an exception? In that case one needs to be careful to not rollback, else be stricken to handling the exception, which programmers seem to hate doing.

Re:A ChatGPT mistake cost us $10k by owl57 on Tuesday June 11, 2024 @11:53AM (#40647798)
> The Go database/sql package actually executes ROLLBACK in the SQL engine.

No: https://github.com/golang/go/blob/beaf7f3282c2548267d3c89441...

Re:A ChatGPT mistake cost us $10k by randomdata on Tuesday June 11, 2024 @12:14PM (#40648064)
Fair enough, although it wouldn't matter if it did. Nothing would change other than some unnecessary computation would occur. SQL supports it just fine. Anyone who learned SQL first would know that quite well.
Re:A ChatGPT mistake cost us $10k by owl57 on Tuesday June 11, 2024 @01:17PM (#40648932)
Fair as well. It's just sending a command which you know will fail, and even relying on the fact it fails, feels wrong, whether it sends a request over network or not.

BTW I checked and it's not an error in Postgres, only a warning. Still not something I would want in my database logs for the happy path.

Re:A ChatGPT mistake cost us $10k by randomdata on Tuesday June 11, 2024 @01:42PM (#40649252)
You know it will fail if you didn't screw up anything else.

But it's a good sanity check/safety measure to call it anyway incase you made a mistake elsewhere. An errant rollback is more likely to be caught in testing than a dangling transaction.

Re:A ChatGPT mistake cost us $10k by sneak on Monday June 10, 2024 @01:02AM (#40630325)
Yeah, each have their benefits. I run all my go projects through golangci-lint which is required for merge to master/main, so not checking the error value would not have made it to prod.

I suppose there are probably similar checkers for Python that would have caught the passing of a scalar value instead of a function.

Perhaps this is an argument for mandatory linting in CI.

Re:A ChatGPT mistake cost us $10k by asveikau on Monday June 10, 2024 @12:27AM (#40630144)
I think deploying and then going to sleep is the red flag here. They should have deployed the change at 9am or something and had the workday to monitor issues.
Re:A ChatGPT mistake cost us $10k by bruce511 on Monday June 10, 2024 @12:41AM (#40630224)
While I agree, it wouldn't have helped. The first n sign-ups per commit worked. So the problem didn't manifest during the day, only when they stopped committing.

Now granted, if they'd carried on doing stuff (but not redeployed production) it may have shown up in say mid-aftenoon, or not depending on volume.

And of course the general guideline of not deploying anything to production early in the day, or on Friday, is still valid.

Re:A ChatGPT mistake cost us $10k by lolinder on Monday June 10, 2024 @12:59AM (#40630312)
You can deploy and go to sleep if you have monitoring and alerting and someone getting paged. It shouldn't be a human monitoring for issues anyway, so the only reason to choose 9am over bedtime should be that you don't want to risk a late night page, not that someone will actually be checking up actively on the deployment.
Re:A ChatGPT mistake cost us $10k by asveikau on Monday June 10, 2024 @01:08AM (#40630353)
For a big launch, you can have your entire team looking at issues in realtime, rather than a small number of people responding to pages in the middle of the night. I think if it's worth it to you there's a big difference.
Re:A ChatGPT mistake cost us $10k by owl57 on Tuesday June 11, 2024 @02:16AM (#40643060)
That is, if you know what to monitor. When you launch an entirely new feature, you often don't know all the ways it will break.

And it's not trivial to set up a correct alert for this one. Simple HTTP 5xx threshold wouldn't work because it's high enough to not wake you up whenever cloud provider restarts Postgres, it's too high to catch this. You need either per-endpoint failure rate alerts or something else more clever.

Re:A ChatGPT mistake cost us $10k by huygens6363 on Monday June 10, 2024 @01:14AM (#40630393)
IMO way sooner. Some low hanging fruit tests would catch this before it’d even hit git.
Re:A ChatGPT mistake cost us $10k by Edd1CC on Monday June 10, 2024 @11:26AM (#40634610)
Yeah if one instance created one ID, then any integration tests creating more than one user would have failed. There were no testing or logging on a system with live users while doing a refactor between two dynamic languages
Re:A ChatGPT mistake cost us $10k by rnts08 on Monday June 10, 2024 @02:40AM (#40630856)
This gets more and more common, companies and founders does not think about the infrastructure since they believe that their cloud provider of choice is going to do it for them with it's magic.

As soon as you expect paying customers in your system you need to have someone with the knowledge and experience to deal with infrastructure. That means logging, monitoring, alerting, security etc.

DevOps.. amateurs.

Re:A ChatGPT mistake cost us $10k by Kesty on Monday June 10, 2024 @03:09AM (#40630995)
Sure not having tests, is bad. Doing thing with AI without triple checking is dangerous.

But not having error logging/alerts on your db ? That's the crazy part.

This is a new product, is not legacy code from 20 years ago when they thought it was a neat idea to just throw stuff at the db raw, and check for db errors to do data validation, so alerts are hard because there's so many expected errrors.

Re:A ChatGPT mistake cost us $10k by throw156754228 on Monday June 10, 2024 @03:22AM (#40631064)
Should've been picked up at unit, way before monitoring.
Re:A ChatGPT mistake cost us $10k by igammarays on Monday June 10, 2024 @04:20AM (#40631337)
The error didn't show up until multiple entities were created. I can forgive someone for not having unit tests for scaling issues.
Re:A ChatGPT mistake cost us $10k by huygens6363 on Monday June 10, 2024 @07:22AM (#40632360)
Creating more than one entry is not a “scaling issue”.
Re:A ChatGPT mistake cost us $10k by f35caleb on Monday June 10, 2024 @08:50AM (#40633040)
Agreed. I'm chuckling because I was wrestling with this exact same bug on a FastAPI project last night. I caught it because I have a habit of submitting API endpoints multiple times with the same data to see how the database reacts. Got a key collision when I tried to submit the endpoint the second time and figured out that setup didn't create a new UUID each time.

Unit tests are good, yes. Monitoring is also good. But just taking 30 seconds to do some manual testing will catch a LOT of unexpected behavior.

Re:A ChatGPT mistake cost us $10k by withinboredom on Monday June 10, 2024 @10:27AM (#40634010)
I generally follow this process:

1. Get it working: write the code for the desired behavior, not worrying about making it beautiful, testable, whatever.

2. Get it working well: manually testing and finding edge cases, refactoring to get it testable and writing tests to solidify behavior.

3. Get it working fast: optimizing it to be as fast as I need it to be (can sometimes skip this step). No tests should change here, but only new tests.

Re:A ChatGPT mistake cost us $10k by lionkor on Monday June 10, 2024 @01:11PM (#40635716)
Maybe ChatGPT didn't tell them that they needed monitoring?
Re:A ChatGPT mistake cost us $10k by makz on Monday June 10, 2024 @07:07PM (#40640322)
That was hilarious