We use Grafana as UI for CloudWatch. We use dashboard for Kubernetes cluster to discovery relevant pod, set relevant variables etc. We use Prometheus metrics (eg. memory usage) to pin-point interesting timeframe. Then we have linked dashboards to see logs for a specific timeframe of a specific pod. It allows us to limit amounts logs of logs scanned.

In addition, we have links to Grafana in our back-off, where we monitor our batch jobs, so immediately the developer has a timeframe selected and a suitable pod in Grafana to quickly see job metrics and job (pod) logs.