I just winced in pain thinking of the ways that can bite you. I guess in a cloud/virtualized environment with many short lived instances it isn't even that obscure an issue to run into.
A nice discussion on Stack Overflow:
https://stackoverflow.com/questions/64497035/is-time-from-ti...
Something similar caused my favorite bug so far to track down.
We were seeing odd spikes in our video playback analytics of some devices watching multiple years worth of video in < 1 hour.
System.currenTimeMillis() in Java isn't monotonic either is my short answer for what was causing it. Tracking down _what_ was causing it was even more fun though. Devices (phones) were updating their system time from the network and jumping between timezones.
Bosses actually came to us because our analytics team was trying to figure out who was causing it, because it had been caught by the team doing checks against the data. (a playback period should never have had > 30s of time)
Why would any tenant supplied data affect anything whatsoever?
As a tenant, unless you are clashing with another resource under your own name, I don't see the point of failing.
aws S3 would be an exception, where they make that limitation on globally unique bucket name very clear.
You're inserting a VM with a specific name. If you try to create the same resource twice, the GCE control plane reports that as a conflict.
What they're doing here would be roughly equivalent to supplying the time to the AWS RunInstances API as an idempotency token.
(I work on GCE, and asked an industry friend at AWS about how they guarantee idempotency for RunInstances).
When trying to create the same resource twice, all request should report the same status instead one failing, one succeeding.
In AWS, their APIs allow you to supply a client token if the API is not idempotent by default.
See https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Run_I....
Before I quibble with the idempotency point: I agree with this, entirely, but it is what it is and a lot of software has been written against the current behavior. So I'll cite Hyrum's law here: https://www.hyrumslaw.com/
> GCP control plane is generally not idempotent.
The GCE API occupies an odd space here, imo. The resource being created is, in practice, an operation to cause the named VM to exist. The operation has its own name, but the name of the VM in the insert operation is the name of the ultimate resource.
Net, the API is idempotent at a macro level in terms of the end-to-end creation or deletion of uniquely named resources. Which is a long winded way of saying that you're right, but that from a practical perspective it accomplishes enough of the goals of a truly idempotent API to be _useful_ for avoiding the same things that the AWS mechanism avoids: creation of unexpected duplicate VMs.
The more "modern" way to do this would be to have a truly idempotent description of the target state of the actual resource with a separate resource for the current live state, but we live with the sum of our past choices.
You're right, we did it wrong.
// And paradoxically makes engineers like you.
Disclaimer: I work on GCE.
This is nicer!
More practically, though, the instance name here is literally the name of the instance as it appears in the RESTful URL used for future queries about it. The 409 here is rejecting an attempt to create the same explicitly named resource twice.
Note that whether this creates collisions is entirely under the customer's control. There's no requirement for global uniqueness, just a requirement that you not try to create two VMs with the same name in the same project in the same zone.
You can also batch API calls[1], which also gives you a response for each VM in the batch while allowing for a single HTTP request/response.
That said, if you want to create a set of effectively identical VMs all matching a template (i.e., cattle not pets), though, or you want to issue a single API call, we'd generally point you to managed instance groups[2] (which can be manually or automatically scaled up or down) wherein you supply an instance template and an instance count. The MIG is named (like nearly all GCP resources), as are the instances, with a name derived from the MIG name. After creation you can also have the group abandon the instances and then delete the group if you really wanted a bunch of unmanaged VMs created through a single API call, although I'll admit I can't think of a use-case for this (the abandon API is generally intended for pulling VMs out of a group for debugging purposes or similar).
For cases where for whatever reason you don't want a MIG (e.g., because your VMs don't share a common template). You can still group those together for monitoring purposes[3], although it's an after-creation operation.
The MIG approach sets a _goal_ for the instance count and will attempt to achieve (and maintain) that goal even in the face of limited machine stock, hardware failures, etc. The top-level API will reject (stock-out) in the event that we're out of capacity, or in the batch/bulk case will start rejecting once we run out of capacity. I don't know how AWS's RunInstances behaves if it can only partially fulfill a request in a given zone.
[0]: https://cloud.google.com/compute/docs/instances/multiple/cre...
[1]: https://cloud.google.com/compute/docs/api/how-tos/batch
[2]: https://cloud.google.com/compute/docs/instance-groups
[3]: https://cloud.google.com/compute/docs/instance-groups/creati...
Is that not the conclusion? The tester was clashing with their own names?
The regions used are "us-east-1" for AWS [1] and "us-central1-b" for GCP [2].
1: https://github.com/piercefreeman/cloud-gpu-reliability/blob/...
2: https://github.com/piercefreeman/cloud-gpu-reliability/blob/...
Reminds of me this post on mtime which recently resurfaced on HN: https://apenwarr.ca/log/20181113
Unfortunately, millisecond-precise timestamps proved to be a bit tricky in combination with sqlite.
I suspect the 409 conflicts are probably from the instance name not being unique in the test. It looks like the instance name used was:
which has a 1-second precision. The test harness appears to do a `sleep(1)` between test creations, but this sort of thing can have weird boundary cases, particularly because (1) it does cleanup after creation, which will have variable latency, (2) `int()` will truncate the fractional part of the second from `time()`, and (3) `time.time()` is not monotonic.I would not ask the author to spend money to test it again, but I think the 409s would probably disappear if you replaced `int(time())` with `uuid.uuid4()`.
Disclosure: I work at Google - on Google Compute Engine. :-)