But you don't have to go that far: Just remove e.g. identity morphisms from categories, and you get something that is more general than categories.
It is of course another matter completely whether these structures are a useful abstraction for what you want to accomplish. I've never come across something reasonable that has arrows and associative composition, but no identities. As for infinity categories, I remain very skeptical that they abstract any problems programmers not concerned with topology (i.e. > 99%) have.
So yeah, categories are a reasonable baseline from where you can build. But you can't show anything relevant and non-trivial that holds in any category. You'll need more structure, hence less generality, for that.
I think people are just butt-hurt that he used the word 'ultimate'.
What you’re linking to is part of a relatively small corpus outside of e.g. algebraic geometry and topology that has successfully been used for nontrivial results. And as another commenter said, while it can be used for e.g. computation, it is extremely poor for complexity theory. There is a reason why basically any analyst (as opposed to algebraic geometer) will use set theory instead of categories for their work.
I’m curious. What makes you think that?