Yes. The author of Darwin's Blind Spot: Evolution Beyond Natural Selection agrees with you. The view that only natural selection determines evolution is the (neo-)Darwinism view, but there are more modern views as presented by Frank Ryan in aforementioned book.