I think it’s not a naive question at all. To my naive understanding, categories need to have a composition property (if there is a morphism from a to b, and b to c, then there is a morphism a to c created by composing a to b and b to c), which allows to express “richer” structures than just directed graphs. Now, for the gory details, you better ask a mathematician :]