It doesn't sound like any reverse engineering happened.
Is this illegal in some way? I mean, it's not a copyright violation if those copies of Stuffit were licensed and not pirated. Is it some kind of shrinkwrap license violation to put it on a VM that accepts jobs from the public online? I dunno. Probably not? Does it seem likely that a company that no longer makes software for this format is going to come after someone for providing a translation service online for a pretty niche hobbyist audience, that they are charging no money for? Doesn't seem very likely to me, I don't know why they'd do that.
I don’t know if the company has any legal right to send a C&D, but that has not stopped some in the past.
It seems pretty unlikely to me they will, for software they don't sell anymore, to a project that is not charging any money, and has a pretty niche audience (there aren't that many people that have use of this service).
No?
2) Even if it were legal to copy or reverse-engineer their file format, the legal team at Smith Micro believes otherwise, and doubtless has amassed a considerable war chest to fight in court for that belief. Do you really want to risk litigation from them as a small-time/OSS developer? It's not enough for something to be legal in order to do it. You have to have reasonable grounds to believe you will spank any challengers to the legality of your actions in court -- or such potential challengers must have been already spanked to the extent that any other challengers will think twice before suing on the same grounds. It's not fair, but it's how the system works.