News for hackers. Stuff that matters.
Re:Python Pip 20.3 Released with new resolver by cosmic_quanta on Monday November 30, 2020 @09:16AM (#25253899)
It's not super relevant, but I was poking around their CI infrastructure, and I noted the use of a temporary RAM disk to speed-up tests:

https://github.com/pypa/pip/blob/master/.azure-pipelines/scr...

I'm very surprised. Is this common?

Re:Python Pip 20.3 Released with new resolver by diggan on Monday November 30, 2020 @10:25AM (#25254691)
Yeah, I've seen that pattern around and also used it myself. Useful if you're using the disk a lot, for any purpose.

Let's say you have 200 JS libraries that you need to install dependencies, build and test for each commit. Making the JS dependencies install into a common cache directory into /dev/shm will easily speed up the install phase (and make the projects share dependencies cache fairly easily), and you could also make the test data (like fixtures) go through /dev/shm if you need it to.

Re:Python Pip 20.3 Released with new resolver by jackric on Monday November 30, 2020 @09:36AM (#25254112)
Another cool trick - use ephemeral memory SQLite database during unit tests:

conn = sqlite3.connect(":memory:")

But also run against your prod-like database (Postgres/MySQL etc.) later in the CI pipe to catch idiosyncratic database problems.

Re:Python Pip 20.3 Released with new resolver by aitchnyu on Monday November 30, 2020 @10:56AM (#25255060)
For Postgres, there is also disabling fsync delays (20x speed claimed) and even running it all in memory.

https://dev.to/thejessleigh/speed-up-your-postgresql-unit-te...

Re:Python Pip 20.3 Released with new resolver by detaro on Monday November 30, 2020 @09:20AM (#25253943)
It's a fairly straight-forward optimization if disk is a relevant part of your run time and you don't need terribly much of it. Probably not exactly common, because many people don't bother, but also not extraordinary.
Re:Python Pip 20.3 Released with new resolver by sethhochberg on Monday November 30, 2020 @09:59AM (#25254376)
I'd bet its becoming increasingly popular/convenient to do so with the growth of container-based test suites - really trivial to include a "postgres-ram" image for your test stack instead of one that uses disk, and you don't even need to know what tmpfs is to do so.
Re:Python Pip 20.3 Released with new resolver by detaro on Monday November 30, 2020 @10:32AM (#25254766)
If the tooling you use lets you create tmpfs mounts for your container its very easy, yes. AFAIK those are RAM-backed by default. I think from inside a container it could be difficult to have the permissions necessary.
Re:Python Pip 20.3 Released with new resolver by pletnes on Monday November 30, 2020 @10:26AM (#25254707)
Maybe even more so on cloud instances?
Re:Python Pip 20.3 Released with new resolver by aeyes on Monday November 30, 2020 @07:10PM (#25260729)
Pretty common if you run a lot of builds, have a big test suite or pay per minute.

In my environment unpacking a 200MB .tar.gz containing node_modules from previous runs takes half the time on RAM disk vs. SSD local storage. Python doesn't gain as much, packages tend to have fewer small files but there still is a small gain.

How much tests benefit from using a RAM disk depends heavily on what they do, I have seen 10% for React apps.

We also use other simple tricks like running databases with disabled fsync, mv instead of rm, cache whatever can be cached between CI runs, shallow checkouts and so on. Nothing ground breaking but it all adds up.

However, for pipelines which run for five minutes twice a day I don't even bother.

Re:Python Pip 20.3 Released with new resolver by dijit on Monday November 30, 2020 @09:19AM (#25253941)
Yes, it has secondary benefits like not burning out SSDs and wearing out HDDs.

The drawbacks are that memory is kinda expensive in high amounts, and compilers love memory.

Re:Python Pip 20.3 Released with new resolver by acdha on Monday November 30, 2020 @11:48AM (#25255776)
These days, “high amounts” is a lot more than most projects need - even ultra-portables have 8GB or more. Most projects don’t have unit test suites limited on file I/O measured in gigabytes unless they also have a budget.
Re:Python Pip 20.3 Released with new resolver by mixmastamyk on Monday November 30, 2020 @01:33PM (#25257181)
Was very common in the spinning rust days to speed up io bound ops, and mentioned in yesterday’s Amiga thread as built in. Could still be useful if tests use a ton of disk.
Re:Python Pip 20.3 Released with new resolver by davidbrennerjr on Monday November 30, 2020 @12:08PM (#25256053)
Yikes! I'm surprised by the number of people who think temp RAM disk for QA testing came from using containers or the cloud. It was actually in use prior to that technology. And, it was popularized in the use of remote PXE installation which required modifications to the OS after the fact.
Re:Python Pip 20.3 Released with new resolver by detaro on Tuesday December 01, 2020 @12:03AM (#25262511)
Where do you see anybody thinking that "came from using containers or the cloud"?
Re:Python Pip 20.3 Released with new resolver by davidbrennerjr on Friday December 04, 2020 @10:49PM (#25311555)
My contribution is correct. I'm not a parrot. Temp RAM disk started in Linux 2.3 (possibly older than that), it didn't come from builds inside containers or instances in the cloud.