Re:How to use Google Cloud’s free logging service with Go
by ediblenergy on Thursday September 08, 2016 @12:04PM
(#12454253)
oh nice, yeah that is one of the things I read about in the docs but have not tried yet. I haven't really used BigQuery enough yet.
Re:How to use Google Cloud’s free logging service with Go
by skybrian on Thursday September 08, 2016 @12:38PM
(#12454596)
You have to do a bit more yourself in BigQuery. (Define a schema and do a schema update when you log something new.) I implemented logging via App Engine requests which save the data to BigQuery. Not sure if that's better or worse than using the StackDriver service, but it's working.
This setup is not optimal. If you use kubernetes/gke the most optimal choice is relying on GKE logging. You just log to stdout, and GKE takes care of attaching relevant metadata (e.g. instance log came from) and transporting logs reliably and efficiently (relies on fluentd agent provided by the GKE). You don't need to do any setup, except logging to stdout in json format. In my python code I just use python-json-logger.
If you don't want to rely on GKE, instead of sending http calls with logs fluentd is a better choice:
Regarding stack driver - article does not mention that you can export logs from StackDriver to BigQuery. I find it very useful. BigQuery internal version, Dremel, was created for that very purpose - logs analysis.Disclaimer: I'm not affiliated with fluentd or GCP, although I use both and I used to work at Google (primarily search, no ties to GCP).