Does addr2line still work with ASLR? Because that could be a reason not to.
addr2line works depending on the code and compiler/libc. Your libunwind may be able to output relativ code addresses, in which case addr2line will work if you use the stacktrace. GDB should handle this as well. If your c compiler doesn't support it you can just disable compiling PIEs which automatically disables ASLR but that should be reserved for debug builds.
There is no downside to activating ASLR either, so I'm not sure why it isn't.