Here's a sneak peek of something I'm working on
http://fingswotidun.com/avr/AvrAsm/Testbed/
It's pretty much a project to scratch the itch being described in this thread. It gives you an in-broswer 8-bit AVR based fantasy console and Assembler.
It looks quite cool.
The core of the esp-idf is like a wrapper + a tiny bit of logic over the registers, there is a ton of extra stuff packed in there but you don't need to use it if you don't want to and it is pretty clearly separated in the source directory. I really like it because it gets out of my way when I don't want it, and when I do want it there is usually something there to help.
Both the Arduino libs and Espressif SDK are fairly big chunks of code.
Yes, you can program AVRs without libs too, but you did specifically say Arduino. It is also my understanding that ESPs are both bit obscure and fairly complex beasts to be managed without SDK.
While there are still some nice cases of simplicity in the embedded world, the complexity and layers of libraries and frameworks model is encroaching there too.