The system can easily slow to a crawl as memory accesses and allocations are constantly triggering reclaim and causing the system to thrash. The kernel has no way to know that it should prioritize the terminal you are trying to open to kill things rather than try its hardest to keep the offending application running.

The best way to prevent this, if you know an application will use lots of memory, is to run it in a memory cgroup. With a memory cgroup you can limit the applications total allowed memory, reserving some for the rest of the system.

Unfortunately, I don't know of a super nice CLI, but here's an example using cgcreate/cgexec (https://packages.debian.org/stretch/cgroup-tools):

    $ # Create a memory cgroup named 'mygroup', allowing $USER to change the parameters and run tasks in the cgroup.
    $ sudo cgcreate -a $USER -t $USER -g memory:mygroup
    $ # Set the memory limit to 10MB
    $ echo 10000000 > /sys/fs/cgroup/memory/mygroup/memory.limit_in_bytes
    $ # Run a program in the cgroup
    $ cgexec -g memory:mygroup ls