> I feel like there's more to aphantasia than just "can you see the apple or not?".

There is. I think the definition is still being worked on. Here is an overview, but I don't see your particular case: https://aphantasia.com/article/science/aphantasia-definition...

Since you can recall scenes you saw, it might not be Aphantasia. Not being able to create visuals of random stuff might be called differently.