Yes. The original PC and PC XT had BASIC in ROM just like all the other home computers of the day. In its most bare bones configuration, you booted into BASIC and used a cassette tape drive for your storage.

I don't know how common that setup was - probably not very. The most typical stripped-down PCs were dual floppy (the hard drive was a premium), and booted either MS-DOS or CP/M.