We did care about performance because 1 MHz was slow even back then and data bus were slow too. However we had to care more about memory size because we couldn't fit much in a few kB or 1 or 2 MB for huge machines. Finally, compilation time for substantial programs were... substantial! You could play games as in that sword fighting xkcd.