I'm using the 5x6 layout. For me it seems like the fewest amount of keys (64) I can drop to without having to use layers. I pretty much only use layers for the function keys.

I just ordered a 5x6. I came to a similar conclusion. The 4x6 looks really cool, but I felt I was already pushing it in terms of the amount of change I was talking on, i.e. going from rectangle to a split, going from 75% layout to ~60%, etc. I may try out a smaller model if I really love the Dactyl and want to experiment more with layers.