The most important thing to do IMO is to make it part of your acceptance criteria or definition of done for any story/feature. My observation is that observability gets ignored if it isn't baked into the culture.

In terms of implementation, if it isn't on a dashboard or doesn't fire off an alert, it doesn't exist. So put your telemetry and logs on a dashboard, and set up alerts based on your SLA's/SLO's.

Also, make it easy for developers to understand, create and edit dashboards/alerts. If it's hard to understand or hard to do, it won't get done.

I'll add that incentives can really help as well. Observable software is much more easily operated in production environments, and lets tech ops teams diagnose and fix things with much less involvement from Engineers, which will very much appreciate a much lower escalation rate.

Part of our checklist for new features/services is enumerating what needs to be monitored and how, what instrumentation we should care about, etc.

That may well change during development, but new work doesn't get approved without at least a first stab at how we're going to monitor it.