Remote access has been way faster on vscode for me. I'm not sure what the daemon has to do with speed here, if anything it's one of the reason why remote SSH on vscode is so good.

I think the point is comparing 2 different things. VSCode with a helper daemon running on the server, and Emacs not configured to run as a daemon on the server side, accessing the server via SSH and scp.

I am not sure how the speed of both compares, if one properly sets up Emacs on the server as well, to make it a fair comparison.

Merely for consideration, ControlMaster gets one out of the business of establishing a fresh ssh session per ssh/scp operation: https://man.openbsd.org/ssh_config#ControlMaster (and its friends https://man.openbsd.org/ssh_config#ControlPersist & https://man.openbsd.org/ssh_config#ControlPath )