felixhuttmann

They could force the invoice sender to verify an email address, and then include this email address in the invoice email. This way, nobody can pose as "coinbase" -- maybe as [email protected], but not as [email protected].

If the explanation is incredibly poor, why is this issue still open: https://github.com/hashicorp/terraform/issues/20718

If terraform crashes during apply, it leaves behind an inconsistent state by design: The lock is still set, and some resources which were created already are not yet in the statefile. Trying to re-run terraform after a crash during apply will generally lead to an error: Even if the lock is removed, resources may still conflict if they already exist.

In contrast, when a kubernetes controller or operator crashes, it can be expected to continue seamlessly where it left off.

It is easier to write kubernetes controllers that are able to continue seamlessly then to write terraform providers that do so, because of the granularity of the persistence of the state machine. Terraform locks the remote state, then applies all resources in the current root module, then unlocks the remote state again. In contrast, kubernetes operators can granularly update individual objects after each API call that is performed.

On gitlab, there has been another way for some time: There is a JWT token CI_JOB_JWT available in an env var which contains the branch name and other info as one of the claims [1]. One can then use this token to obtain production secrets based on whether the branch is trusted.

Github has the same feature upcoming [2], which allows to get also directly AWS or GCP credentials restricted by branch name [3, 4].

[1] https://docs.gitlab.com/ee/ci/examples/authenticating-with-h... [2] https://github.com/github/roadmap/issues/249 [3] https://awsteele.com/blog/2021/09/15/aws-federation-comes-to... [4] https://github.com/sethvargo/oidc-auth-google-cloud

Your STM is missing a proper approach mechanism. The vertical range of the piezo that is used for scanning will only be a few 100 nanometer. A screw is too coarse for that! Stick-slip mechanisms with a ramp (Besocke or Beatle-type STM https://www.researchgate.net/figure/1-Diagram-of-the-Besocke... or page 25 of https://www.bgc-jena.mpg.de/bgc-mdi/uploads/Main/MoffatDiplo...) are one solution. Even with such a mechanism, the 'approach' phase takes many minutes!

I agree. It is also amazing how different the database systems are that are competing against each other today:

Partitioning: 1) DynamoDb: Partitioning is explicit and one of the most important parts of schema design 2) Spanner, Cockroach: Database automatically partitions the key ranges. 3) Postgres: You will probably never reach the scale where you need to partition your dataset!

Transactions: 1) Spanner, firestore - no stored procedures, client-side transactions are important 2) Dynamodb: No stored procedures, no client-side transactions, only transactions where all items involved are known by primary key in advance. 3) Fauna, Supabase: Stored procedures are the way to go! You do not need application code, access your database from the client! 4) Postgres: We have everything, use what fits your particular use-case!

If database internals did not matter, why are they all doing something different and are sometimes quite opinionated about it?

Thank you for the insightful answer.

I see the problem with mutations in foreign systems if those foreign systems do not support idempotency themselves. IMHO, though, stripe should abstract away faults in banks, and figure out how to work around faults in bank's systems using e.g. automated refunds when a duplicate charge is detected, and not just bubble up a 500 to stripe's customers and leave it to them to figure out. If stripe cannot figure it out in an automated way toward the bank whether the request suceeded, stripe's api customers certainly can't either, and stripe should risk double-charging the end customer knowing the the end customer will complain and request a chargeback.

> The most honest answer is that Stripe wasn't built on a data store where transactions are supported

Transactions are not necessary if one can do an insert conditional on the key not yet existing, but then it is required to have the idempotency key from the client enter into the primary key.

Strongly agree that idempotency should be understood as part of the application, and not as part of the transport.

I find it insightful to think about a transport layer that is much less reliable than the internet: traditional/snail mail. In a proper business communication, every invoice has an invoice number such that the recipient can know whether they processed that invoice previously or not.

What I have always wondered about, in the stripe docs it says "Stripe's idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeded or failed. Subsequent requests with the same key return the same result, including 500 errors." which indicates that the idempotency functionality is created in a kind of layer around the main application functionality, and thus the request from the idempotency layer to the main app is not itself idempotent. So the idempotency key protects against faults on the network between the client and stripe's idempotency layer, but not against stripe-internal faults between the idempotency layer and the application. Is that the case? Why is the idempotency not achieved by using an idempotent database transaction or atomic operation? What is the value of responding repeatedly with a 500 if the original 500 was caused by a transient error?

Adyen seems to similarly implement the idempotency in a separate data store (https://docs.adyen.com/development-resources/api-idempotency...).

Both stripe's and adyen's implementation seem to not treat transient faults within their own systems correctly.

Property-based testing is good when there is a property to assert that emerges in a non-trivial manner from the code under test. If you do not have such a problem, property-based testing provides no value over a simple, example-based test.

In the case of the parsing code from the article, the emerging property is that a serialization followed by a deserialization should always yield the original result.

In the case of the 'binary or not' case from the article, the non-trivial, emergent property is that the function never fails with an exception.

Most modern software development is plumbing, stitching together platforms, libraries and frameworks, and there are rarely non-trivial, emergent properties where property-based testing is useful.

It is a software integration problem. Think Oauth. Your car is made by one company, the charging stations by another. The progress becomes very slow.

A few thoughts:

1) AWS dynamodb has a parallel scanning functionality for this exact use case. https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

2) A typical database already internally maintains an approximately balanced b-tree for every index. Therefore, it should in principal be cheap for the database to return a list of keys that approximately divide the keyrange into N similarly large ranges, even if the key distribution is very uneven. Is somebody aware of a way where this information could be obtained in a query in e.g. postgres?

3) The term 'cursor pagination' is sometimes used for different things, either referring to an in-database concept of cursor, or sometimes as an opaque pagination token. Therefore, for the concept described in the article, I have come to prefer the term keyset pagination, as described in https://www.citusdata.com/blog/2016/03/30/five-ways-to-pagin.... The term keyset pagination makes it clear that we are paginating using conditions on a set of columns that form a unique key for the table.

The risk of accidentially deleting the database is real. However, it can be mitigated without introducing another tool by using a second terraform root module (with a corresponding second statefile). So you would have one terraform root module for foundational or stateful things like databases which rarely change and should never be accidentially deleted, and a second terraform root module that holds only the lambda. The former root module is applied only manually, the latter can run automated in a pipeline.

Yes. If we use a null_resource that has the hashes of the source code files as a trigger, then in the `local-exec` provisioner of the null_resource, we can run the build. The build can also be run remotely (we use google cloud build) to be independent of the developer's machine architecture and operating system, which is important for native dependencies. Terraform will not re-run the null resource provisioner so long as the source code does not change, there is no need for a reproducible build.

This works fine only as long as you do not need a step to build or download dependencies, like `npm install` or `pip install`, as part of your run of terraform apply. Otherwise, a more complex solution is necessary, like talideon's above.

Examples:

1) Long-living connections. One part of your application offers large file downloads, so that your users sometimes take significant amount of time to download, and the error rate shall be low. Think people doing `curl -L https://github.com/.../some-commit-hash.zip` in CI. I'd guess this is not served by github's ruby monolith. Or you use websockets. Redeployments typically stop all running tcp connections. Splitting this part off allows you to redeploy the main application frequently without disrupting websockets or downloads.

2) Reduce startup time. For some languages/ecosystems, startup time is significant and scales with the size of the codebase. For example, a typical java enterpise app like keycloak has half a million lines of code and takes about 1 minute only for startup. This is even more relevant when running things in AWS Lambda or google cloud run, and can also be relevant for integration tests. Having several deployables each handling a subset of the functionality may give you better cold start times than having one that contains all code.

3) Resiliency toward resource exhaustion. If two functionalities are served in the same process or VM, and one of them

* has a memory leak

* uses the connection pool to the database inefficiently and thereby clogs it up

* has an endpoint that is overloaded by a misbehaving client

then the other functionality is also affected, which can be avoided by putting them into separate processes/containers/VMs. Splitting services allows to reduce the effort spent on resource hygiene and rate limiting.

4) Binary size. It is often convenient to compile static info or asset-like things into the application, think geoinformation on timezones. This bloats the binary, splitting that part off into an independent deployable can give you a much smaller binary for the main application.

S3 is newly strongly consistent within a single region since last reinvent or so (google cloud storage has been strongly consistent for much longer). However, the cross-region replication for s3 is based on 'copying' [1] so presumably async and not strongly consistent.

GCP datastore and firestore are strongly consistent nosql databases that are available in multi-region configurations [2].

[1] https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.... [2] https://cloud.google.com/datastore/docs/locations

I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 minutes [1], while with AWS IAM, your permissions may be arbitrarily stale. This means that when the GCP IAM database leader fails, all operations will globally fail after 7 minutes, while with AWS IAM, everything continues to work when the leader fails, but as an AWS customer, you can never be sure that some policy change has actually become effective.

In general, AWS more often shifts the harder parts of global distributed systems onto their customers, rather than solving them for their customers, like GCP does. For example, GCP cloud storage (s3 equivalent) and datastore (nosql database) provide strongly consistent operations in multi-region configurations, while dynamodb and s3 have only eventually consistent replication across regions; and google's VPCs, message queues, console VM listings, and loadbalancers are global, while AWS's are regional.

[1] https://cloud.google.com/iam/docs/faq#access_revoke