For specifically understanding transformers, this (w/ maybe GPT-4 by your side to unpack jargon/math) might be able to get you from lay-person to understanding enough to be dangerous pretty quickly: https://sebastianraschka.com/blog/2023/llm-reading-list.html

An earlier edition of this book was the textbook for a computational linguistics course I took back in ~2002 (!), it's amazing how much has been added