A buffer per directory is just the right thing for dired to do, for the same kind of reason that it is right that git has a staging area. Your problem is that this behaviour causes friction with how you prefer to manage buffers. Some suggestions:

- With the regular buffer list, if you keep typing letters, the view will restrict to buffers containing that string

- All the usual selection narrowing options, e.g. Helm, work here

- You can rebind 'q' to 'kill-buffer-and-its-windows

- A lot of people like ibuffer, which allows you to group kinds of buffer in its buffer list, e.g. all dired buffers