I reckon it would be more in the range of minutes (depending on how complete of a linux system we are talking about here). The QSPI peripherial can run at 100MHz if I read the datasheet correctly (so you can read/write data at about 400MBit/s minus overhead). For reference booting linux on an ATmega@24MHz running an ARM emulator and external SDRAM with an memory bandwidth of about 300kbyte/s seems to take about 6 hours [1]. Assuming the SiFive is just as efficient as the ATmega it would take less than a minute (assuming the memory bandwidth is the only bottleneck and not the slow CPU - the SiFive also has 16kB of L1 cache and has a clock rate 10x faster).

[1] http://dmitry.gr/index.php?r=05.Projects&proj=07.%20Linux%20...