Sticking to the spec is one way to be portable. Building a portability layer is another one; this is how Common Lisp grows, by taking advantage of the work of different implementations and using standardized tools (FEATURES, reader conditionals) to build a common interface. See e.g. CFFI which has become the (defacto) standard for loading C libraries (https://common-lisp.net/project/cffi/spec/cffi-sys-spec.html...).
Unfortunately the architectural details of the machine will leak through to the program you are writing, and you will need to twist your code in order to get correct behaviour and fast execution.
Case in point is the recent article posted on HN about the platform specific details of getTimeMilliseconds() in Java.
Could you link that article? I searched and couldn't find it on hN.
I believe the article was http://pzemtsov.github.io/2017/07/23/the-slow-currenttimemil... Although I couldn't easily find it in my history or on HN. I had to resort to the Googles.
Writing code in Common Lisp means you're writing something with semantics promised by the Common Lisp specification. Vendors implement the spec (perhaps using nasty #ifdefs), and developers write code against the spec. The spec is a contract, and from the point of view of the programmer, it's indifferent to platform change and so on (up to what the spec says is platform-dependent, like machine integer size or maximum array length).
Just because a particular implementation may not be considered portable, doesn't mean my program (which I may choose to run on that implementation) isn't portable.