And it's not just N inputs, M outputs. You also have a processing graph for the intermediate filters. So ideally you want graph nodes represented by some identifier which can then be referenced by other nodes.
Unless I misunderstand you, this is already the case. Simple or complex filtergraphs are declared as a string* by the user as an arg to -vf/-af (per-output) or -filter_complex/-lavfi (global). Filter outputs can have link labels, which allow their consumption by other filters.
*can also be read from a file
*via shell too, if the arg is quoted/escaped.
I think this might make use of ffmpeg less daunting than people attempting to battle a single-line shell input, ordering-sensitive parameters and escaping.
Well, maybe it's redundant, there already is vaporsynth after all.
This is not to say there isn't scope for improvement. A couple of areas for improvement is the consistency of options naming across filters that implement the same feature within those filters, and a clearer syntax to identify the target of an option i.e. protocol, demuxer, decoder..etc