Another comment on this thread pointed out they had a potential collision in their instance name generation which may have caused this. That would mean this was user error, not a reliability issue. AWS doesn’t require instance names to be unique.

Maybe 1 reported. Not saying aws reliability is bad, but the number of various glitches that crop up in various aws services and not reflected on their status page is quite high.

Errors returned from APIs and the status page are completely separate topics in this context.

that was measured from API call return codes, not by looking at overall service status page

Amazon is pretty good about this, if their API says machine is ready, it usually is.

They were almost exclusively user errors (HTTP 4xx). They are supposed to indicate that the API is being used incorrectly.

Although, it seems the author couldn't find out why they occurred, which points to poor error messages and/or lacking documentation.

If not a 4xx, what should they return for instance not available?

503 service unavailable?

It's not the service that's unavailable. The resource isn't available. The service is running just fine.

GCP error messages will indicate if resources were not available, if you reached your quota, or if it was some other error. Tests like OP can differentiate these situations

Yeah, 4xx is client error, 5xx is server error.

Yes, and trying to create duplicate resources is a client error.

Still, 409 seems inappropriate, as it is meant to signal a version conflict, i.e. someone else changed something, and user tried to uplod a stale version.

”10.4.10 409 Conflict

The request could not be completed due to a conflict with the current state of the resource. This code is only allowed in situations where it is expected that the user might be able to resolve the conflict and resubmit the request. The response body SHOULD include enough information for the user to recognize the source of the conflict. Ideally, the response entity would include enough information for the user or user agent to fix the problem; however, that might not be possible and is not required.

Conflicts are most likely to occur in response to a PUT request. For example, if versioning were being used and the entity being PUT included changes to a resource which conflict with those made by an earlier (third-party) request, the server might use the 409 response to indicate that it can't complete the request. In this case, the response entity would likely contain a list of the differences between the two versions in a format defined by the response Content-Type.”

Then again, perhaps it is the service itself making that state change.

That would be confusing. The HTTP response code should not be conflated with the application's state.

There will come a moment in time when you realize exactly what you have stated here and why it is not a good mental palace to live in.

Using HTTP error codes for non-REST things is cringe.

503 would mean the IaaS API calls themselves are unavailable. Very different from the API working perfectly fine but the instances not being available.

What? REST is just some API philosophy, its doesn't even have to be on top of HTTP.

Why would you think HTTP status codes are made for REST? They are made for HTTP to describe the response of the resource you are requesting, and the AWS API uses HTTP so it makes sense to use HTTP status codes.