News for hackers. Stuff that matters.
Re:Idempotence: What is it and why should I care? by ageitgey on Monday August 20, 2018 @09:32PM (#17806032)
This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly.

The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By always passing that key, client can request that the payment be processed a 100 times with no fear of it being processed more than once.

This stripe blog post has as good a description as any: https://stripe.com/blog/idempotency

Re:Idempotence: What is it and why should I care? by rooam-dev on Tuesday August 21, 2018 @02:22AM (#17807455)
Good point, but I'd add that not only when charging money, but any operation that can have side effects if executed more than once (emails, external calls, etc.).

Having idempotent queue consumers/listeners can help with overall systems resilience (someone already mentioned this below). It helps as much as you don't have to worry when restarting a queue broker and/or apps with consumers.

It adds complexity, but I think it pays off in the long run.

Re:Idempotence: What is it and why should I care? by hardwaresofton on Tuesday August 21, 2018 @06:56AM (#17808545)
So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but:

- Express all operations as log messages (ez pz distribution)

- Ensure all operations are idempotent

- Record the operations (this is the log you can distribute if you please)

- Disallow API code modification, only allow accretion/use of new API endpoints.

- All APIs that come up have their own databases, a bit of the CQRS model here (but without events -- just the actions performed)

- When you need to stand up new API servers, start the new ones (with handling code for old operations completely unchanged) next to the old ones, and update the http-server code (like request handlers) to output the new commands. Older servers that don't understand the new commands will ignore (or redirect), and new servers that do understand will process and add to the distributed log. New nodes just stream the replications of the already existing nodes and no one spends any time with an inconsistent view of the database

Of course, writing to a distributed log is slow (pick whichever consensus algo you want, you either have durability with a quorum or best-effort without), but this only is a huge deal if you're doing lots of writes, and for most web applications, that's not what's happening, the vast majority is reads.

CRDTs might even fit in here, because if you want a multi-master setup, you could literally keep the log as a set (I'm not quite sure how truncation of super old records would want) keyed by transaction ID -- Assuming the same request doesn't go to multiple servers, their logs should be easily combinable at the end of the day -- API1 is gonna see events A B and E, API2 might see C and F, and API3 will likely see D G and H.

Honestly everything I've described here is really more like moving the coordination/distributed log problem to the application level (up until now all this action would just happen @ the Postgres/DB level), but I'm not yet convinced it's a terrible idea.

I haven't found the time to actually try to make what I'm describing here a thing but would love to hear thoughts

Re:Idempotence: What is it and why should I care? by Groxx on Wednesday August 29, 2018 @12:56AM (#17865766)
I'm curious about how you anticipate handling new APIs / how this approach helps ensure consistency for people who aren't on the new APIs. Seems like if

    a -> b -> c
becomes

    a    b -> c
     \-> x
then C won't be aware of the new stuff happening in X... unless it's 100% compatible with everything B does, including every log it produces, at the moment it starts receiving traffic, which seems unlikely. In the time until C updates to read from X:

    a    b    c
     \-> x -->^
isn't C (and its consumers) operating on an "inconsistent view of the database", as produced by A?
Re:Idempotence: What is it and why should I care? by hardwaresofton on Wednesday August 29, 2018 @10:26PM (#17873713)
I think I mentioned it earlier, but the idea is that the commands are immutable -- API growth happens through accretion only. New APIs must handle a superset of old ones.

Realistically, this is basically the same as how it's handled in most APIs today -- until you can guarantee (or choose to strictly enforce) that no one use a particular API, it just stays.

In addition to this, new instances use completely different databases, but rely on the replaying the stream of commands that got the old instance there to catch up (and new commands as they come in).

Re:Idempotence: What is it and why should I care? by Groxx on Thursday August 30, 2018 @01:04AM (#17874210)
To be clear, I'm interpreting "APIs" as "a method". So adding a new method is equivalent to adding a new service. If you mean for methods to never increase in number, only flexibility, then yea - I think I follow, this all makes sense. Then new stuff is truly new and disjoint from others, and there's no migration to worry about.

---

Also, since you're mentioning "replaying the stream of commands", I think this means "consistent" is strictly bound to "... at the point in time it has read to, from API X"? Then yea, switching APIs / methods is fine, you just delay the readers. It's event sourcing in a nutshell - there are undeniable benefits between any two "services", it's a compelling design.

I was interpreting it more in a system-wide sense with a large number of services, which is where I don't have a good feel for event sourcing - consumers of C and [others] are not "up to date" with what A has done until they read all data derived from all sources from the same minimum A-timestamp. So without a vector clock (probably) it's generally unsafe to consume from C and Q until they're both up to date, because C is missing stuff from A that Q already handled. Building something that maintains correctness and usefulness in the face of this seems extremely difficult or constrained, unless you accept unbounded delays (in practice: likely weeks of dev time in some cases).

---

And last but not least: CRDTs solve pretty much all of this without synchronization of any kind, yea. Are they still a pain to design? Or have we developed relatively-repeatable strategies nowadays? I haven't kept up much here, sadly.

---

I'll probably have to reread this all a couple times to make sure I'm not totally off somewhere irrational, sorry! Yours was a rather dense comment to comprehend, and I'm not sure I'm following correctly. Event sourcing has been interesting to me for quite a while, but I've never really developed a feel for how to build large, multi-developer(-team) systems out of it and it sounds like you might have an idea.

Re:Idempotence: What is it and why should I care? by pgt on Tuesday August 21, 2018 @08:12AM (#17808857)
You will enjoy the fact-based database Datomic, which is the reason Rich Hickey made Clojure: https://www.youtube.com/watch?v=Cym4TZwTCNU

Datalog is a much, much, much query language better than SQL: http://www.learndatalogtoday.org/

Re:Idempotence: What is it and why should I care? by hardwaresofton on Tuesday August 21, 2018 @09:24AM (#17809291)
I do remember Datomic and I think it's a great tool but I fell out of love with the Clojure ecosystem and JVM-based languages as a whole and don't think I'll be getting back into it/them.

I do remember wanting to check out Datomic (I believe after seeing a talk on how it was being used at a bank in southern america?[0]), but I found it unreasonably hard to find and download/experiment with the community edition -- compare this to something like Postgres which is much more obvious, more F/OSS compliant (I understand that they need to make money) and Datomic doesn't really look that appealing to me these days.

At this point in my learning of software craftmanship I can't do non-statically type-checked/inferenced languages anymore -- I almost never use JS without Typescript for example. Typed clojure was in relatively early stages when I was last actively using clojure, and I'm sure it's not bad (probably way more mature now), but it's a staple in other languages like Common Lisp (the declare form IIRC). The prevailing mood the clojure community seemed to be against static type checking and I just don't think I can jive with that anymore.

Thinking this way right now Datomic wouldn't be a good fit for me pesonally but I believe that it is probably high quality paradigm.

[EDIT] - I found the talk: https://www.youtube.com/watch?v=7lm3K8zVOdY

[0]: https://www.datomic.com/nubanks-story.html

Re:Idempotence: What is it and why should I care? by pgt on Tuesday August 21, 2018 @07:45PM (#17814901)
Yes, Datomic is the killer app for Clojure [^1]. Have a look at Datascript[^2] and Mozilla's Mentat[^3], which is basically an embedded Datomic in Rust.

Hickey's Spec-ulation keynote is probably his most controversial talk, but it finally swayed me toward dynamic typing for growing large systems: https://www.youtube.com/watch?v=oyLBGkS5ICk

The Clojure build ecosystem is tough. Ten years ago, I could not have wrangled Clojure with my skillset - it's a stallion. We early adopters are masochists, but we endure the pain for early advantages, like a stable JavaScript target, immutable filesets and hot-reloading way before anyone else had it.

Is it worth it? Only if it pays off. I think ClojureScript and Datomic are starting to pay off, but it's not obvious for who - certainly not very ever organisation.

React Native? I tore my hair out having to `rm -rf ./node-modules` every 2 hours to deal with breaking dependency issues.

Whenever I try to use something else (like Swift), I crawl back to Clojure for the small, consistent language. I don't think Clojure is the end-game, but a Lisp with truly immutable namespaces and data structures is probably in the future.

[^1]: In 2014 I wrote down "Why Clojure?" - http://petrustheron.com/posts/why-clojure.html [^2]: https://github.com/tonsky/datascript [^3]: https://github.com/mozilla/mentat

Re:Idempotence: What is it and why should I care? by pgt on Thursday August 23, 2018 @04:28AM (#17825536)
Also look at Magic, an experimental typed JVM Lisp with immutable namespaces: https://github.com/mikera/magic
Re:Idempotence: What is it and why should I care? by hood_syntax on Tuesday August 21, 2018 @08:47AM (#17809051)
I take issue with some of Rich Hickey's technical opinions, but Datomic seems pretty cool. I knew Datalog has been around for a while, but had never looked into it further.
Re:Idempotence: What is it and why should I care? by SmirkingRevenge on Tuesday August 21, 2018 @08:01AM (#17808813)
No need to write the distributed log yourself, you just need Kafka.

You basically just independently conceived of, what is becoming a pretty popular architecture these days - event driven systems based on distributed logs.

The source of truth in your system becomes the idempotent distributed log of events (rather than your rdbms or data warehouse) which ought to be "replayable", allowing you to audit or even potentially recreate your application state (databases, etc), at any point in time.

Transitioning to a new version of an api/service means adding another subscriber to the event stream (and perhaps reprocessing some or all of its history), that runs in parallel with the old version - then when everything looks good you can quietly disable the old version.

Re:Idempotence: What is it and why should I care? by hardwaresofton on Tuesday August 21, 2018 @09:08AM (#17809198)
I think I wasn't clear enough about why this solution isn't just a distributed log/event sourcing approach:

- The description above was intended to NOT specify any tech it depends on -- Kafka is the best queueing and long-term-log-storage engine I've ever heard of, but the point is that it shouldn't matter. Kafka doesn't "write" the distributed log, it just stores it.

- I don't want an Event Sourcing solution. It's often combined with CQRS, but the thing is, I don't think you actually need events (I might be wrong) -- you just need the commands, and as long as what they do never changes, you can skip a bunch of problems with event sourcing -- you can more easily compress stuff if you don't allow the semantics of operations to change. Also there's stuff about how you handle things like schema changes, etc -- in event sourcing it's often "recalculate the whole DB", but I think that should not be the solution -- In this system I'm proposing, I'd literally ensure I could reflect the migration as operations in the log, and do it that way.

What you're describing is a Kafka-log based event sourcing system, and it's close to what I mean, but not quite the same -- I want the systems to be able to able to trivially cluster, for example, by hitting an endpoint like `/replicate`, which does nothing but stream log messages out as they hit one particular server. In the system you describe I think the equivalent thing would be a tiny bit harder to achieve since you'd need to be able to listen to another subscriber's messages.

I'm also completely aware that I might go about building this and realize I just took the long way around to the system you're describing but I don't think I am just yet.

Re:Idempotence: What is it and why should I care? by beaconstudios on Tuesday August 21, 2018 @12:56PM (#17811356)
what you've just described is very similar to the data management side of the platform I'm building. I can confirm that given event sourcing, idempotent mutation, and only adding new API endpoints, you're a long way towards what I would consider the ideal online software architecture. So many problems with API maintenance just go away with this approach.
Re:Idempotence: What is it and why should I care? by rectangletangle on Monday August 20, 2018 @10:01PM (#17806233)
This is solid advice. Another common trick is to disable the submit event when the submit button is clicked for the first time, preventing two requests from firing, when the user double clicks. Then re-enable the event, if the request fails. Ideally this is done in addition to server side nonce validation, and not as the only preventative measure, because browser differences, or network issues could cause a double request (more likely if the HTTP GET method is used, instead of a proper POST). Regardless server and client side techniques should be used together to create a better UX, similar to using both client and server side email format validation.
Re:Idempotence: What is it and why should I care? by digitaLandscape on Monday August 20, 2018 @10:37PM (#17806432)
I don't like this because in practice sites usually fail to re-enable the submit button if something goes wrong. Just let the user submit multiple requests if they want to retry, don't take that away from them, just make it harmless.
Re:Idempotence: What is it and why should I care? by jen729w on Tuesday August 21, 2018 @05:31AM (#17808233)
Newbie programmer here, but even I am already implementing finite state machines which handily fix this sort of issue.

Any sort of UI without them now feels archaic.

Re:Idempotence: What is it and why should I care? by hood_syntax on Tuesday August 21, 2018 @08:52AM (#17809082)
Agree 100%. If you don't have explicit state transitions in a UI, you're asking for trouble. I think some people don't realize how much harder they make it on themselves by not setting clear boundaries; they see the up front cost and balk, when it saves a non-trivial amount of headache in the future, not to mention reducing mental overhead when you've got the structure hashed out.
Re:Idempotence: What is it and why should I care? by digitaLandscape on Tuesday August 21, 2018 @11:13AM (#17810301)
Meh. The logic still applies: on a bad network, the first request can hang for half a minute when a retry would get lucky and go through immediately.

Don't handcuff your users for no benefit, just because you can't write a well-behaved backend.

Re:Idempotence: What is it and why should I care? by PinkMilkshake on Monday August 20, 2018 @10:29PM (#17806384)
That's only a partial solution. It may be a user agent or a queuing system (or anything else sitting between a person clicking the button and where the final transaction is recorded) that attempts a retry.
Re:Idempotence: What is it and why should I care? by BoiledCabbage on Tuesday August 21, 2018 @01:42AM (#17807289)
Restricting what the client can do isn't idempotency. Allowing the user to perform the same action (make the same api-call) multiple times, Nd your API coalesce them as if it were 1, is. The point and benefit of idempotency on this scenario is that you silently support "bad" behavior.
Re:Idempotence: What is it and why should I care? by qoi2ijds0 on Tuesday August 21, 2018 @07:48AM (#17808754)
IMO disabling buttons causes more trouble than it's worth - you can never guarantee the user hasn't clicked twice (because don't forget the code that disabling the button in the first place is relying on an event firing that tells you the button was clicked - this doesn't mean 2 events can't be queued before the event handler is called), and then you need a whole chunk of code around re-enabling the button depending on what has happened after the fact which is then a big source of bugs.
Re:Idempotence: What is it and why should I care? by notatoad on Tuesday August 21, 2018 @02:12AM (#17807408)
disabling the submit button is a nice UI feature to indicate idempotency, but it's not the same thing as the function actually being idempotent. both are important.
Re:Idempotence: What is it and why should I care? by genericid on Tuesday August 21, 2018 @08:18AM (#17808880)
This is basically client-side validation.
Re:Idempotence: What is it and why should I care? by enraged_camel on Tuesday August 21, 2018 @05:03AM (#17808118)
I apologize for the noob question, but... is it a good idea to have the client be in charge of generating UUIDs? When dealing with idempotence, they have to be truly random, not just pseudorandom, right? I looked into this recently, and my understanding (from the cursory amount of research I did) was that the UUID generation method depends on the random number generator used by the browser, which can vary a lot.

Why not generate the UUID on the server and send it to the client along with the page request (if using SSR)? I.e. the server generates the UUID, sticks it in a database field, then sends it to the client. When the client responds with the UUID, you can check that against the database to make sure its valid.

Re:Idempotence: What is it and why should I care? by chaboud on Tuesday August 21, 2018 @06:43AM (#17808494)
You can do a mix of the two to reduce server traffic (especially for abandoned sessions) and user latency.

Give the client a client ID that you generate that should be properly random... Have them submit that ID along with their self-generated ID.

Re:Idempotence: What is it and why should I care? by tankerdude on Tuesday August 21, 2018 @12:32AM (#17807003)
Yup. This is actually an interview question that I go through to see if they know mechanisms to prevent the double charge, fulfillment, etc. problem.