The datasheet is really skimpy about the power draw, especially compared to what ST Micro tells you. In STM32 parts the Sleep mode is usually pretty bad (say half of normal consumption) and you need to go to Stop mode, where the core and most peripherals are not clocked to get into micro-amp range. However, once there you can typically get by with under 5uA. You can wake up from that in a few microseconds if you clock from internal oscillator with no PLL (so you run at 8-16MHz not 48-64MHz).
In that context, 180uA is pretty bad, but again, the datasheet is not very clear.
What's your source for this? The RP2040 datasheet claims 0.18 mA for Dormant state and 0.39 mA for Sleep. As you've got detailed control over what gets clock gated in sleep state suspect you can bring that 0.39 mA down if you've only got one or two wake events to care about.