The only “important” accented letter is “é”. In most cases “è” is completely redundant as it follows the rule that if the next vowel is a silent “e” then the current “e” takes the \ε\ sound. “Ê” makes the same sound but is there for historical reasons. We can determine if we need an open “e” sound by applying like 2 rules (the aforementioned one and e followed by two consonants is always open) and by memorising some of the multigraphs which are part of the phonetics as it is.

Removing “é” makes it harder to determine pronunciation and makes it easier to confuse tenses, eg “il a mange”, if you’re reading you might “a” and misunderstand the tense (he ate vs he eats). There aren’t any consistent rules for the use of “é” apart from the start of a word.

Personally speaking, I rely on “é” a lot more than I do on “è” and I’ve been tripped up by a missing ’ on “e” many times unlike the other one.