I've experienced this and the 404 related emails on Elastic Beanstalk are indeed annoying, and it would be good if there was any easy way to turn them off just for 404s specifically. However based on my experience this will pass, it won't be like this forever so don't worry. And then it will come back, but then go away again. I haven't had a single email related to this for several months now. After it realizes there's no vulnerability (in my case I don't have php admin) then it seems to stop trying but I don't know how long before there will be new requests trying again.