https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Great story. I won't claim to be a wizard, but here's mine:
grep -Eo "mailto:[^\"']*" webpage.html | sed 's/^mailto://' | sort -u
There is a whole heap of tools in the package, and they're definitely worth a look if you like to poke at the web from a term. Plus they handle a lot of the nasty corner cases that will pop up if you try to treat HTML as simple ASCII text.
document.documentElement.innerHTML.split('\n').forEach( (line) => line.includes('mailto:') ? console.log(line) : 0 )
But please push this further (is there a shorter way to say document.documentElement.innerHTML ?)
a = document.createElement('a');
a.download = 'emails.txt';
a.href = `data:text/plain,${Array.from(document.querySelectorAll('a')).map(a => a.href).filter(s => ~s.indexOf('mailto')).join(',')}`;
document.body.appendChild(a);
a.click();
`document.getElementsByTagName('a').map(elem => elem.href)`
[...($$('a'))].map(e => e.href)
can you share the smart line of javascript? You have 60 seconds, go.
More to the point, though: the junior data scientist might not have been entirely on the wrong foot there. There's a tradeoff between "I need to do this thing quickly" v. "I need to do this thing repeatably and robustly"; whipping up a quick shell one-liner to do it falls into the former category, and also can drive some ideas for implementing the latter category (where you'd almost certainly want to be properly parsing the HMTL, properly accounting for commas in the email addresses (and yes, email addresses can have commas per RFC 5321), etc.).
This is great! I can't believe I've never run into "cut" before. In similar spots, my workflow would have been filtering the lines with grep and then opening the file in VI to finish the task.
Cut has a mirror image program, "paste", but I've not yet hit a use case for it.
Here's an example:
I was working at a consultancy one day as a data engineer, and across the desk was a junior data scientist. We were preparing for a meetup and the manager said "this website has a bunch of email addresses on it I'd love to have them in a CSV" (they were in a format where it wasn't easy to copy + paste them)
The junior data scientist said "I can do that" and the manager said "only if it takes less than 2 minutes", the junior, thinking he'd have to: setup a python environment, find a parsing lib, write code to pull the page, parse the dom, search for the email addresses, write them to a file,
said it could be done in an hour or two.
I viewed the code on the page, and saw the html wasn't minified, and the "a" tags were 1 per line, so I :
use wget to pull the webpage cat webpage.html | grep 'mailto:' | cut -d '"' -f 5 | cut -d '"' -f 1 > emails.txt
^ unix wizards will tell me a better way of doing this, but it took me literally 60 seconds to go from "I wonder if" by the manager to "here's your CSV"
THATS WHY WE USE COMMAND LINE TOOLS, they are a swiss army knife