Originally CPLDs used arrays of PLDs/PLAs which use a wide network of programmable AND-OR gates to implement logic equations with relatively sparse routing resources between logic elements. FPGAs will use a more fine grained element for implementing logic (SRAM LUTs or small clusters of muxes) with denser routing resources.

That distinction is a bit blurred nowadays because Altera has "CPLD" products that are using conventional FPGA architecture equivalent to what was state of the art for FPGAs 25 years ago.