Knowing the right incantations is useful in that context and keeps me from installing better tools until they become part of the distro we deploy with.
I wrote some of the tools on coldtea's list, and I'm pretty much the same way in that I stick to distro tooling in vanilla setups. But I spend enough time on my local workstation that having "better" (read: creature comforts) is worth it there! But yeah, if you spend most of your time in vanilla setups, then tools not in the standard distro repos are a hard sell.
I feel like people significantly underestimate the difficulty of being 100% compatible with another tool. Not even GNU grep is strictly POSIX compatible, for example. To get POSIX compatibility, you need to set the POSIXLY_CORRECT environment variable.
And then what do I gain from this? Do you think distros are going to start throwing away GNU grep for ripgrep? No, I don't think so. So what then? A few people will have the pleasure of using ripgrep with grep's flags, even though they are already significantly similar? Not. Worth. It.
Now... If someone does think it's worth it, then they are more than welcome to start that journey. Most of ripgrep is factored out into libraries, so in theory there is a lot of reuse possible. But if you want to support compatibility all the way down into the regex engine, then you might be hosed.
That seems to miss a huge lesson: backward compatibility eases transition, new features retain users.
Therefore, your assertion makes only sense if there would be no point in attracting existing users to use a tool whose added value is entirely irrelevant.
ripgrep is an evolution on ag, which in turn is an evolution on ack. All three tools have similar defaults, so in fact, ripgrep preserves some amount of backward compatibility with previously established tools in terms of the default mode of operation. Just not GNU grep.
This goes further in that the intersection between ripgrep's features/flags and GNU grep's features is quite large---certainly much larger than the differences between them, which is just another form of preserving backward compatibility. This was done on purpose for exactly the lesson you're claiming I missed: backward compatibility eases transition.
(The context of this conversation was a 100% backward compatible version of ripgrep with GNU grep. See my other comments on ROI. Just because I can argue against 100% backward compatibility doesn't mean I've missed the importance of backward compatibility.)
You'd also loose the ability to improve upon arcane/ad-hoc and obsolete flags and syntax choices though too.
And for the "same flags" to make any sense, you'd also be constrained to produce the same output and even recreate the same quirks.
At which point, might as well just use the originals.
That's why I wrote about it being nice to have them all in distros -- perhaps in a single package like coreutils.
Then wherever you are, they'd be just a "package-manager install rustutils" away.
Generally, if one is not an admin of random hodgepodge of systems (e.g. in big enterprises), then you are:
(1) doing work on your own workstation
(2) administering machines you control the provision (e.g. a person administering a startup's Cloud servers)
(3) doing development on some kind of vm
In those cases you can quite easily install a bundle package and make sure those tools are there.
I, for one, don't go out in unknown systems day after day -- even if we have 100s of servers we manage, we DO manage them.
I'm glad these tools work for you and I hope the community keeps making them, I just threw in my perspective.
Why?
1. Because you end up with a more complex Dockerfile. I've seen curls, clones etc during build - having a remote call for random tools in our build _feels_ like a bad practice.
2. There is also the issue that a fleet of application servers will have different tooling available, which can be frustrating in a pinch. I've done some gnarly things across many hosts, expecting a common set of tools. At AWS, we were responsible for non-nuclear team's services during oncall and would ssh to their boxes. Logging into a box with a different shell would be annoying.
[2] > for i in `cat hosts.txt`; do ssh $i '...'; done
For companies whose main activity is processing data, various build pipelines (not necessarily compiles) might be what production servers do all day.
To answer you, I meant on a Docker image build step- which is where you’d want to install the tool so it’s accessible to people debugging the application during the container run context.
It strikes me as an odd choice to accept the least common denominator of tools. As a technologist, that sounds frustrating. And as someone in leadership, I can’t imagine hamstringing my team by making them work within an environment that won’t let them use the best tools for the task at hand.
The popular package management tools rely heavily on FHS and like to install stuff into directories that require root permission.
Imagine if there was a tool that could download binaries of modern tools from a certain repo and install it to our ~/bin. Imagine if we could use new command line tools as easily as we can download a fully functional complex applications securely into a browser tab just by typing its URL!
You can also add ~/lib (for example) to your LD_LIBRARY_PATH path if you wanted to install custom libraries and even have your own man page path too.
All of this is already possibly on Linux and Unix however I'm not sure it's something you actively want to encourage as allowing users to install whatever they want on servers lowers the security to the level of the worst user on that server. If it was really deemed necessary that users should be able to install whatever they want then I'd sooner roll out a dedicated VM per user so at least their damage is self-contained (barring any visible networked infrastructure)
Of course you can, technically. The parent laments why it's not easier to achieve. Already you're talking about manually building for example. Where's a package manager that will allow for that too, not just the central repo? (not just asking if such manager exists in some form, asking where it is in modern popular distros).
>All of this is already possibly on Linux and Unix however I'm not sure it's something you actively want to encourage as allowing users to install whatever they want on servers lowers the security to the level of the worst user on that server.
Which also touches the parent's question. Why is it not easier AND safer? It's not like we don't have security models that allow for such things...
Anything short of that would be sacrificing security for the sake of convenience.
I think it is fair to hope that after 25 years of Internet, it should be easy to bring in new tools from the Internet into our local environment without requiring root access in a safe and trivial manner.
I don't know if this is possible with plain Nix (not the OS).
It even used to be easier to make stuff run from your homedir. We are moving from it, not towards it. It is really a shame for the few multi-user setups out there.
That way I don't even have to learn the new command (or risk forgetting about ls / be frustrated when I can't use it).
jq: https://stedolan.github.io/jq/ (lightweight JSON client)
On the other hand, you may want to keep some smart dotfiles around that installs them in ~/bin
(Joey is a boss, did not know of this project of his -- thanks for pointing out)
I use "x" in oh-my-zsh plugins instead of una. https://github.com/robbyrussell/oh-my-zsh/tree/master/plugin...
I use "ag" instead of ripgrep. https://github.com/ggreer/the_silver_searcher
I use "fpp" instead of vgrep. https://github.com/facebook/PathPicker
Anybody has tried them and has an opinion on which is better?
One could argue that tool like this makes no sense since make is installed almost everywhere but I'm already using fish shell so one more non standard tool makes no difference. I appreciate simplicity of "just".
Consider also the "fuzzy finder" FZF.
Httpie is my goto http cli for all systems. Pretty much PostMan (that a lot of ppl use for no reason) but without annoying UI and on a single line. Formats JSON and colors it right in the terminal as well. Has a super nice query, header and post body syntax. Works well with sessions and cookies.
I can't see anywhere in the httpie docs about capturing variables from and/or running tests on the HTTP responses - it's an extremely powerful feature of Postman when you're debugging non-trivial HTTP flows.
$ userId=$(http get https://jsonplaceholder.typicode.com/posts/1 | jq '.userId')
$ http get "https://jsonplaceholder.typicode.com/users/$userId" | jq '{name,email}'
{
"name": "Leanne Graham",
"email": "[email protected]"
}Was this a thing back when the BSD and GNU greps were being developed? Was it common for people to have huge directory trees (e.g., `node_modules` or `.git`) lying around that were causing significant search slow downs? Not sure, but it seems to have taken a while for it to become a popular default!
There are of course other tricks, and most of those are inspired by changes in hardware or instruction sets. For example, back in the day, a correctly implemented Boyer Moore was critical to avoid quadratic behavior by skipping over parts of the input. But that's lessish important today because SIMD routines are so fast that it makes sense to optimize your search loop to spend as much time in SIMD routines as possible. This might change how you approach your substring search algorithm!
... so what's my point? My point is that while sometimes optimizations are classic in the sense that you just need to change your heuristics as hardware updates, other times the optimization is in the way the tool is used. Maybe you can squeeze the former into existing tools, but you're probably not going to make much progress with the latter.
I remember when I first put out ripgrep. Someone asked me why I didn't "just" contribute the changes back into GNU grep. There are a lot of reasons why I didn't, but numero uno is that the question itself is a complete non-starter because it would imply breaking the way grep works.
>Was this a thing back when the BSD and GNU greps were being developed? Was it common for people to have huge directory trees (e.g., `node_modules` or `.git`) lying around that were causing significant search slow downs? Not sure,
Sure, actually. It was called “build” and it was moved out of source code hierarchy to not interfere with tools. Pretty clever, and you don’t have to patch every tool each time new build-path appears. This should lead us to some conclusion, but I cannot figure out which. Do you?
also that approach seems dependent on all projects you might want to grep (in this example) building cleanly into an external directory, which is naturally never going to be 100%: some people don't know why other software supports those options, some people don't care, some people think that's the wrong solution to the problem, etc. Ultimately someone comes along and builds up a big enough body of experience that they can account for and fix some fraction of the brain dead behavior out in the wild, and the rest of us get a useful tool.
gz-sort: http://kmkeen.com/gz-sort/
The only reason for using ack these days is if you’re already invested in a Perl workflow and can use some of the Perl magic powers, and don’t care about comparatively poor performance. That’s not many people.
Other reasons that you may want to use ack:
* You want to define your own filetypes
* You want to define your own output using Perl regexes
* You need the portability of an uncompiled tool in a single source file that you can drop into your ~/bin when you can't install or compile anything on the box.
* You don't want to have to deal with a Rust runtime.
You can define your own file types with ripgrep on the command line (albeit not in a config file, so you end up needing a wrapper script or an alias).
ripgrep has pre-compiled binaries that work out of the box on Windows, Mac and Linux.
"Rust runtime" isn't really a thing. It's like you saying you don't want to deal with a C runtime.
I'm glad to see that ripgrep lets you define your own types. That ripgrep does it with globbing vs. the ways that ack allows is just another of the differences between the tools that people have to decide about.
I believe that defining the output using Perl regexes was what I was referring to with “Perl magic powers”. I work at FastMail, the backend of which is mostly Perl, and have been unable to convince most people to use ripgrep because they like Perl and ack and occasionally use some of that fancy superpower. I, on the other hand, lack that experience (I’m most used to Python, Rust and JavaScript, and have never seriously worked in Perl) and thus don’t really get anything out of ack over ripgrep, without putting in a fair bit of effort to figure out what I need to do in a particular case. I’m more likely to feed the output through sed or vim if I want to rewrite it, actually!
I’m very glad that ack existed; it led the way with better tools, and I used it heavily for some years. Thanks for making it!
One of the things I'm working on right now is a cookbook that will ship with ack 3, currently about thisclose from going to beta. I want to show examples of the magic powers and see how powerful it can be. Your comment reinforces in my mind that it's an important thing to have.
> I’m very glad that ack existed; it led the way with better tools, and I used it heavily for some years. Thanks for making it!
You're very welcome. I'm continually pleased to see what's come out after it. A search tool arms race is a good thing.
It's interesting how git grep makes the ack feature of "ignore your VCS dirs" not so important any more. I wonder how the grep-alike tool world would look if git weren't so dominant, and/or git didn't include its own grep.
So I can have regex like 'abc\K(def)(?=ghi)'. That will highlight ONLY 'def' in the text but only if that string is preceeded by 'abc' and 'ghi' follows.
- ripgrep cannot do that as it does not support PCRE
ag support --passthrough, but I haven't yet felt the need to try that as ack --passthru has been baked into my workflow and aliases for ages.
I use ripgrep (rg) otherwise by default.
I'm pretty sure the PCRE lookaround support you're referring to is exactly what the GP meant by "Perl magic powers."
# Do "ack2 --help" to learn more (or "ack2 --man" to learn even more).
alias ack_passthru '\ack2 -h --flush --passthru --color \!:*'
# USAGE: SIM_COMMAND | hllog
alias hllog "ack_passthru --color-match='white on_red' '(\*[FE],\w+):' \\
| ack_passthru --color-match='white on_red' '^(Error-\[.*?\].*)' \\
| ack_passthru --color-match='white on_red' '(\*[FE],\w+)\b.*?([a-zA-Z_.0-9]+),(\d+)|\d+' \\
| ack_passthru --color-match='white on_red' '^FOO Info\s+::\s+\K([1-9][0-9]* Runs (Failed|Missing))' \\
| ack_passthru --color-match='red' '^UVM_[EF][ROATL]+\s+:\s+[1-9][0-9]*' \\
| ack_passthru --color-match='black on_white' '(\*W,\w+):' \\
| ack_passthru --color-match='black on_white' '(\*W,\w+)\b.*?([a-zA-Z_.0-9]+),(\d+)|\d+' \\
| ack_passthru --color-match='black on_white' '^(Warning-\[.*?\].*)' \\
| ack_passthru --color-match='yellow' '^UVM_WARNING\s+:\s+[1-9][0-9]*' \\
| ack_passthru --color-match='yellow' '^FOO Info\s+::\s+\K([1-9][0-9]* Runs with Warnings)' \\
| ack_passthru --color-match='yellow' '^(FOO Warning\s+::.*)' \\
| ack_passthru --color-match='white on_red' '^(FOO Error)\s+::' \\
| ack_passthru --color-match='white on_red' '^(FOO Fatal)\s+::' \\
| ack_passthru --color-match='yellow' '^(\*WARNING\*[, ].*)' \\
| ack_passthru --color-match='white on_red' '^(\*ERROR\*)' \\
| ack_passthru --color-match='white on_red' '^\s*(ERROR:)' \\
| ack_passthru --color-match='white on_red' '^:\s*(ERROR:.*)' \\
| ack_passthru --color-match='red' '^:\s*(FAILURE:.*)' \\
| ack_passthru --color-match='yellow' '^:\s*((WARNING|LIMITATION):.*)' \\
"Example in the linux repo:
rg --color always PM_RESUME | rg '^|PM_SUSPEND' --colors 'match:fg:green'
Thinking of stuff like fd and:
ripgrep: https://github.com/BurntSushi/ripgrep (grep/ag alt)
xsv: https://github.com/BurntSushi/xsv (csv tool)
exa: https://the.exa.website/ (ls)
una: https://github.com/jwiegley/una (multi-compression utils wrapper)
tokei: https://github.com/Aaronepower/tokei (loc stats)
And of course this: https://github.com/uutils/coreutils