I think there was some serious experimentation in 2000s, just not in the area of popular systems. Plan9 started to hit the news properly around then. A few OSes in managed languages were developed: MS Singularity, and smaller experiments like JNode and SharpOS.
Well, that's debatable. Linux just crossed the 20 MLOCs bar. Granted, this is mostly driver code which you will only need a small subset of, but still: Linux is a huge complex blob. Linus himself said it's bloated years ago[1] and that it has a high entry barrier for new developers due to its complexity[can't find the link right now].
My guess would be that this is partly due to its classic monolithic architecture and partly due to their developement model[2].
After working with Plan9 for a while I can't help but take my hat off to the beauty and simplicity of the system. Especially as a sysadmin your life would be so much easier. No LDAP, no Kerberos, no NFS, no need to update or deploy software on/to terminals. But as it stands we're living in a land where every computer thinks it is a Mainframe.
Linux has done a great deal for OSS and will without question be around for quite some time but you have to wonder how computing would look like today, if history had taken a few diffenent turns.
FWIW: Is it just me, or is OS R&D gaining a little more traction lately after it almost stagnated in the 2000s ?
[1] http://www.cnet.com/news/linus-torvalds-linux-is-bloated/
[2] https://en.wikipedia.org/wiki/Criticism_of_Linux#Kernel_code...