Might not be necessary. Modern statistical machine translation techniques use formal grammars as an auxiliary signal at most.

> find a formal notation to write down arbitrary sentences

Some sign languages might not be representable in SignWriting, but most should be covered. https://en.wikipedia.org/wiki/SignWriting

> collect a large corpus

Many (all?) European (and other) countries have laws requiring news broadcasts, parliamentary debates etc. to be made available in sign language. Similar to the EUROPARL corpus, that could be used as a starting point for signed <-> spoken translation. Also similar to EUROPARL, colloquial language would be underrepresented.

Of course there won't be any SignWriting transcriptions, so the corpus would be essentially unlabeled.

> Build an avatar that accepts that notation as input and translates it into movement

I'd do that before/while attempting to create a large number of transcriptions, both to ensure accuracy and to enable unsupervised pattern mining in the later steps.

> Train some AI to translate video data into your notation

What I mean by unsupervised pattern mining is that the transcription AI would be trained to output a transcription that makes the avatar ouput something like the original video, as well as reconstructing the avatar's input from its movements.

> Train some AI to translate the notation from one sign language to another (and to spoken/written language as well)

It'd probably be easiest to start with translating between the signed <-> spoken pairs in the corpus and then leverage that into signed <-> signed translation.