Playing with Go: Embarrassingly Parallel Scripts

Re:Playing with Go: Embarrassingly Parallel Scripts by hannibalhorn on Monday December 03, 2012 @10:42AM (#4865865)

Go is great for stuff like this, especially when part of an actual system.

That said, if this was just an adhoc job (to figure out which domains point to a specific IP address) you can just use "xargs -P" or GNU parallel and it becomes a pretty basic shell script, along the lines of: cat domains.txt | xargs -P 1000 -n 1 host

[ Parent | on HN ]

Re:Playing with Go: Embarrassingly Parallel Scripts by skeltoac on Monday December 03, 2012 @11:13AM (#4866043)

Until last week I didn't know that xargs could invoke commands in parallel.

  xargs -n1 -P8 dig A <hosts.txt | grep -v ';' | grep $TARGET_IP | sed 's/\.\s.*//' >hosts-matched.txt

[ Parent | on HN ]

Re:Playing with Go: Embarrassingly Parallel Scripts by scurry on Monday December 03, 2012 @02:17PM (#4867257)

So what's the difference between xargs and parallel? I thought the point of parallel was that it was xargs with the addition of running things in parallel. But if xargs can do that already, is there any reason to use one over the other?

[ Parent | on HN ]

Re:Playing with Go: Embarrassingly Parallel Scripts by eggoa on Monday December 03, 2012 @03:40PM (#4867655)

It's mostly about how they handle special characters.

http://www.gnu.org/software/parallel/man.html#differences_be...

[ Parent | on HN ]

Re:Playing with Go: Embarrassingly Parallel Scripts by hannibalhorn on Monday December 03, 2012 @03:42PM (#4867674)

In addition to the above, it's worth nothing that parallel also supports running the jobs on multiple remote systems via ssh, giving you an easy way to take advantage of a whole cluster.

[ Parent | on HN ]

Re:Playing with Go: Embarrassingly Parallel Scripts by prodigal_erik on Monday December 03, 2012 @02:52PM (#4867423)

For what it's worth, GNU Parallel seems to be fairly new; at work I have an ubuntu distro from this year and it's not in the package repo yet. xargs is POSIX so you can expect it everywhere, though no parallel option is specified (merely encouraged).

[ Parent | on HN ]

Re:Playing with Go: Embarrassingly Parallel Scripts by skeltoac on Wednesday December 05, 2012 @11:19AM (#4876935)

xargs is already in your path.

[ Parent | on HN ]

Re:Playing with Go: Embarrassingly Parallel Scripts by mistermcgruff on Monday December 03, 2012 @10:52AM (#4865921)

xargs -P is a godsend for one-off jobs needing parallelism

[ Parent | on HN ]