As long as we're being pedantic, it's definitely 'fewer'. 'Less' is only used in cases where you couldn't conceptually talk about a single instance of the thing being described. 'Less distance', but 'fewer miles'; 'less abstraction', but 'fewer layers of abstraction'. Even if in practice it may be difficult to find the edges of a single layer of abstraction, the word 'layers' is still metaphorically talking about a collection of individual items, and whenever this is the case we use 'fewer'.

That's the best explanation of a language quirk I've heard for a long, long time! :)