I'm also going to share your feedback about Stackdriver with our product team as well. Thank you for that!
In AWS for a simple app I'd just whitelist 10/8 and have the DB open to all internal instances (with TLS for added security as required).
I'll follow up by email too, happy to provide detailed feedback on this sort of thing.
Thanks for taking the time to engage with the community on this stuff!
This would be handy with tags in gcp.
(1) Some time ago I needed to get a month of logs (filtered by an expression) in order to compare the logged data against a database. The Console does not have any such option. You can set up "exporting", but that only starts replication from the moment you set it up; no historical data. You can use the APIs or "gcloud beta logging read", but the performance is terrible. I got maybe 200-500KB/sec. After keeping the command running continuously for 3 days (!), I still only had 20% of the log data.
These days, we have exporting set up to pipe all the logs into BigQuery and GCS, just to get around the crappiness. That gives some query capabilities, as well as the ability to quickly grab the original data for processing with other tools.
(2) We're on GKE, and none of the container's labels end up in log entries. Surely this is pretty key stuff. There seems to be no way to customize the payloads that GKE's Fluentd sends to StackDriver.
(3) There's no way to tail logs. The gcloud command doesn't have it, and the Console doesn't either. It feels like such a missed opportunity.
I think Google would be better off ditching StackDriver Logging entirely and instead let the user set up rules to send logs to different services that actually work. The default could be BigQuery. I don't understand the point of having a separate, proprietary query language for logs when you have SQL.
I've looked briefly at the other StackDriver stuff (most of which is still on stackdriver.com, always causing a confusing redirect and often re-authentication), and it's been similarly underwhelming. We use Prometheus with Grafana, and StackDriver seems pointless in comparison.
I think a lot of problems would be solved if there would be a way to setup cloud sql proxy that comes with transaction pooling, especially when using k8s and having a pooling namespace (multiple load balancers for HA) where all other apps connect to.
also cloud sql and ipv6 is not funny. (i.e. a lot of things in gcp is ipv4 only which is sad)
Also agree that Stackdriver is pretty underwhelming, though it does have the perk of providing zero-configuration log capture in GKE.
Lots of problems though -
* UI is super-slow, yet it drops old records when you scroll, so scrolling up and down by more than 100 lines causes a reload of the logs I was just looking at. (Note that e.g. Kibana will do infinite scrolling to load more log entries, but doesn't drop old ones when you scroll, which is much more usable).
* It's really hard to work with large log traces; I'd love a quick way to "show raw logs for this time-period/search" so that I can either copy into a text editor or use browser search to explore. An example use-case here is "select all logs with this request correlator", which could include thousands of logs from a request, then jump around that log stream to look for interesting events. Re-running the search is painfully slow for exploring results like this.
* I've also had problems with alerts; the algorithm used for uptime checks is pretty naive, it just checks if the endpoint was down for the entire threshold period, which doesn't catch a flapping endpoint (e.g. down for 1s, up for 1s). Grafana can be configured to detect that the average latency spiked over the period, even if it's not fully down. This has led to us not being alerted for a production outage, so I've moved our critical alerting off Stackdriver. (Again, the promise here is great; if you could get alerting working as well as Grafana, it would be super-powerful to be able to alert on any synthetic metric generated from any log stream in the cluster. It's just half-baked right now.)
All that said, I've been happy with GCP in general, in particular GKE has made running a k8s cluster a lot easier.