If you still want to use xargs and ls perhaps avoiding \0's (perhaps so you can easily grep on the data) you can use "ls -Q | xargs rm". The advantage of that is you can still easily use grep on the data as it is still newline delimited but you won't have problems with spaces.

Also ls is never aliased to "ls --color=always" unless you've done that more likely is its aliased to "ls --color=auto" or similar which will work normally.