I agree with WalterSear that organizing by feature is best. For things that are shared, I like to have a shared/util directory for the miscellania, grouped by purpose (as needed). So, say, if you end up with a few directives used for form validation, you might end up with a util/validation directory somewhere.

The most important question is "can I quickly guess what directory this directory lives in?"

When you come across directives that clearly feel "utility", you will be tempted to look in a utility folder. When a directive is something tightly coupled to a feature, you might be tempted to first look in the directory related to that feature.