News for hackers. Stuff that matters.
Re:Show HN: I mirrored all the code from PyPI to GitHub and ... by codetrotter on Saturday September 02, 2023 @04:51PM (#37365420)
> https://py-code.org/download

This is a perfect job for task-spooler! :D

To mirror your pypi data, I sshed into my server and did this:

    mkdir -p ~/src/github.com/pypi-data
    cd !$
    wget https://raw.githubusercontent.com/pypi-data/data/main/links/repositories.txt
    xargs -L1 tsp git clone --bare < repositories.txt
And then I closed the ssh connection to my server, knowing that my server will proceed to mirror all of those repositories off yours one by one :D
Re:Show HN: I mirrored all the code from PyPI to GitHub and ... by mrwizrd on Saturday September 02, 2023 @07:37PM (#37366397)
You have very serendipitous timing, I was needing to solve a similar task just the other day and had a hell of a time getting it worked out. Thanks for taking the time to post it! :) Straightforward, but seeing a "worked example" helps a ton.
Re:Show HN: I mirrored all the code from PyPI to GitHub and ... by henrydark on Sunday September 03, 2023 @05:02AM (#37368876)
I didn't know about task spooler. Is it better than using xargs with a parallel pool?

    xargs -L1 -P20 git clone --bare < repositories.txt
Re:Show HN: I mirrored all the code from PyPI to GitHub and ... by codetrotter on Sunday September 03, 2023 @09:03AM (#37370086)
Yes, for me it is better because if you do it your way you have to keep your ssh connection open until all of the git clones have been done, which in this case takes several hours.

(Or you could also run your way in tmux or screen.)

With task-spooler, it puts all of the commands (in this case, the individual git clone commands for each of the repos) in a queue and it runs the commands independently of my ssh session, so I can quickly add a bunch of jobs like this to the queue and immediately disconnect my ssh session.

Re:Show HN: I mirrored all the code from PyPI to GitHub and ... by allarm on Sunday September 03, 2023 @06:05AM (#37369139)
I’ve recently worked on something similar that involved cloning multiple repositories and found gnu parallel to be ideal for that task. Parallel gives more control than xargs.