The "AI" capabilities that are making headlines nowadays are mostly based on deep (multi-layer) neural networks with millions, or billions, or parameters. These nets do self-organize into a hierarchy of self-defined feature detectors followed by classifiers, but as you suggest for the most part they are best regarded as black boxes. You can do sensitivity analysis to determine what specific interval values are reacting to, and maybe glean some understanding that way, but by-nature the inner workings are not intended/expected to be meaningful - they are just a byproduct of the brute force output-error minimization process by which these nets are trained.

Neural nets are mostly dominant in perceptual domains such as image or speech recognition, where the raw inputs represent a uniform sampling of data values (pixels, audio samples) over space and/or time. For classical business problems where the inputs are much richer and more varied, and already have individual meaning, then decision tree techniques such as random forests may be more appropriate and do provide explainability.