Many of us now believe that CloudFormation is best described as the assembly language of AWS infrastructure: you can hand-code it, but it's challenging and verbose, and there's a lot of detail you have to get involved in.
CDK is the high-level solution to this challenge. With CDK you describe your infrastructure in terms of high-level constructs such as classes. Your stacks are described as applications instead of YAML templates. You can use many different programming languages including TypeScript and Python, and code is reusable. You can even vendor custom construct libraries to share within your org, or openly via npm.
I've become a convert since CDK came out - it's been ages since I've handwritten CloudFormation templates.
Raw HCL2/Terraform is marginally better IMO with being able to pass around more complex structs and function calls that don't have to be expressed as JSON dicts, but still remaining declarative to be able to create a dependency graph internally.
All of this is why CDK now exists to be able to express these things as TypeScript or Python and spit out the corresponding CFN or TF.
It's also worth noting that AWS added support for additional providers a while back so the CustomResource+Lambda you're describing isn't the only way to interact with non-AWS APIs.
I'm actually curious what you're using - you've shit all over a bunch of things without providing any real alternative.
CF json/yaml is essentially declarative asm for AWS resources.
CDK is how you use popular higher level languages to generate it, and there are some friendlier-than-raw-CF options in between (like SAM).
Basically, CloudFormation is like the shittiest programming language you've ever encountered. People will argue that it's supposed to be simple because it's just YAML, but that's bogus--we clearly need to be able to do complex things in this space or else CF wouldn't provide these hacky functions. We pretty clearly need some sort of expression language if not something more robust, since that's the direction all of these "it's just YAML!" tools are going (CloudFormation, Terraform, etc). CloudFormation might actually make a decent "assembly language" for an infrastructure-as-code backend (although extending CloudFormation for third party services was still far too difficult last I checked--basically you had to run your own lambdas) that some higher-level tools might generate, but it's a mess for anything that isn't a toy.