News for hackers. Stuff that matters.
Re:A common mistake involving wildcards and the find command by svnpenn on Saturday February 08, 2020 @05:51PM (#22279155)
As another user mentioned, many POSIX and/or GNU utilities havent aged well. I respect trying to stay portable, but peoples needs change over time and these tools simply havent kept up. Like the other user, I use Fd now instead:

https://github.com/sharkdp/fd

As well as Silver Searcher:

https://github.com/ggreer/the_silver_searcher

while Grep performance is pretty good, its also gotten pretty stale with regard to its defaults and options.

Re:A common mistake involving wildcards and the find command by anonsivalley652 on Saturday February 08, 2020 @08:46PM (#22279873)
For grepping, rg is faster and saner.

https://github.com/BurntSushi/ripgrep

On end-user systems with fast I/O, it is usually a better use of resources to have a battery-/suspend-/reboot-aware indexing system that has path/extension white/blacklists and prioritizes file monitored changes. This way, searching can happen against an optimized text search DBMS much faster than waiting for zillions of IOPS that are basically wasted by instead spending them ahead-of-time in the background/idle indexing text files/metadata/structured data once when it changed/s and already know exactly where all occurrences of kWhateverCondition_PP3V42_G3H or \A[AB]{1,3}c+d\z live under ~/Projects without reading any actual files.

Re:A common mistake involving wildcards and the find command by eikenberry on Saturday February 08, 2020 @09:11PM (#22279949)
Do either of these tools have a "grep" mode, so that I can alias grep to it and get similar, but improved, behavior? 25 years of habit is a lot of reinforcement to overcome and I'd love to just be able to take advantage of these tools with an alias.

As is I just have a grep wrapping shell script to give it some sane defaults which works fairly well.

Re:A common mistake involving wildcards and the find command by burntsushi on Sunday February 09, 2020 @08:25AM (#22281977)
Author of ripgrep here.

> As is I just have a grep wrapping shell script to give it some sane defaults which works fairly well.

Yes, that's what I did for about ten years before I wrote ripgrep. :-)

> Do either of these tools have a "grep" mode, so that I can alias grep to it and get similar, but improved, behavior?

Another commenter already mentioned `rg -uuu`, and that's pretty much the right answer. In a large number of cases, if you `alias grep=rg`, then most things will continue to work as you expect. If you really do not want any kind of smart filtering, then yes, you'll want `alias grep="rg -uuu"`. But otherwise, things like `command ... | rg pattern` and `rg pattern file ...` will continue to work just like grep.

ripgrep also tries to use the same names for flags as grep, wherever possible. So you'll find a lot of overlap there too.

Please note that ripgrep is not and is not intended to be a drop-in replacement. But I didn't go out of my way to be incompatible with grep either. So a lot of it should be quite familiar!

Re:A common mistake involving wildcards and the find command by ryannevius on Sunday February 09, 2020 @01:05AM (#22280649)
The closest I know of is `rg -uuu`. From the manual:

> -u, --unrestricted ...

> Reduce the level of "smart" searching. A single -u won't respect .gitignore (etc.) files. Two -u flags will additionally search hidden files and directories. Three -u flags will additionally search binary files.

> rg -uuu is roughly equivalent to grep -r.

Re:A common mistake involving wildcards and the find command by anonsivalley652 on Saturday February 08, 2020 @09:21PM (#22279982)
No, you're asking oranges to be bananas. Read the man page or user guide. It's not grep, it's better and already has saner defaults. https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md
Re:A common mistake involving wildcards and the find command by svnpenn on Saturday February 08, 2020 @10:23PM (#22280195)
Yeah and its also 10 times larger

https://github.com/BurntSushi/ripgrep/issues/1481

Re:A common mistake involving wildcards and the find command by reificator on Sunday February 09, 2020 @02:54AM (#22280961)
If you go on a system with grep aliased and run your single grep command, are you going to be able to tell which binary was larger?

If you look at the size of two disk images for different distros, and one has grep while the other has ripgrep, are you going to be able to tell which is which?

10x larger than grep is noise in 2020. Making that tradeoff in exchange for significantly improved performance seems reasonable.

Not to mention a large part of that difference is likely due to grep relying on libc, while ripgrep has to bundle its own runtime. And that's not even bringing up memory safety.

I'll take ripgrep at 10x the size of grep any day.

Re:A common mistake involving wildcards and the find command by burntsushi on Sunday February 09, 2020 @08:18AM (#22281941)
Yes. If that disk space is more important to you than correctness, performance and better Unicode support, then yeah, you should definitely keep using the silver searcher.

I gave more details in a response on that issue: https://github.com/BurntSushi/ripgrep/issues/1481#issuecomme...

Re:A common mistake involving wildcards and the find command by dharmab on Saturday February 08, 2020 @10:46PM (#22280274)
Which is absolutely a good trade-off for a modern desktop system.
Re:A common mistake involving wildcards and the find command by detaro on Saturday February 08, 2020 @10:46PM (#22280270)
the previous issue on the topic has some pointers
Re:A common mistake involving wildcards and the find command by kragen on Saturday February 08, 2020 @05:58PM (#22279191)
Wouldn't you have the same problem with fd if you invoke it on unquoted globs? The problem is the semantics of the shell, not find.
Re:A common mistake involving wildcards and the find command by wffurr on Saturday February 08, 2020 @06:24PM (#22279331)
Yes, but fd has implicit pattern matching without requiring the * character:

    Features:
        Convenient syntax: fd PATTERN instead of find -iname '*PATTERN*'
Re:A common mistake involving wildcards and the find command by pwdisswordfish2 on Sunday February 09, 2020 @05:12AM (#22281357)
Seems a misfeature. Much of the time, I use find to look for files with a specific extension, which makes the trailing asterisk harmful.
Re:A common mistake involving wildcards and the find command by johnp_ on Sunday February 09, 2020 @07:18AM (#22281747)

    fd -e py
Also neatly bypasses the glob issue.
Re:A common mistake involving wildcards and the find command by userbinator on Saturday February 08, 2020 @10:14PM (#22280171)
As another user mentioned, many POSIX and/or GNU utilities havent aged well

What do you mean by that? Users have gotten more stupid and uneducated over time? I've seen "when in doubt, escape it" in several books about UNIX and shell, and escapes and wildcard expansion are one of the most beginner lessons in a lot of other materials. However, the Internet has let everyone have a voice, for better or worse, and as a result anyone who barely knows something can now write a misguided "tutorial" about it.

In other words: don't blame the tools, nor advocate writing dumbed-down replacements which lack the composability and generality the originals had; blame the proliferation of barely-correct educational material that has spread cancerously over the Internet.

Re:A common mistake involving wildcards and the find command by svnpenn on Saturday February 08, 2020 @10:22PM (#22280191)
I have written hundreds of shell scripts, several which are hundreds of lines long. When I say they havent aged well, Im speaking from experience. When I use something like:

    find -name *.jpg
I know what it actually does. But what it actually doesnt isnt what it should do. The current behavior returns matched files... unless no files match. Then it returns the literal string "*.jpg". That behavior is usually going to be desired by no one. Same with:

    grep $SOMETHING
it going to fail if the variable has any spaces. Again something most people arent going to want. These both can be worked around with Shell options and "IFS" respectively, but people shouldnt have to do that. The default behavior should be sane, but in many cases the POSIX standard isnt that. For goodness sake, the "venerable" find tool doesnt even have a "maxdepth" option:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/f...

Re:A common mistake involving wildcards and the find command by userbinator on Sunday February 09, 2020 @12:33AM (#22280564)
You completely ignored my point, which is that none of this behaviour should be surprising or unexpected to anyone who has taken the time to think about how it all works together, and that this flexibility is the inherent way in which the UNIX command line is so powerful.

it going to fail if the variable has any spaces

Quoting and escaping. I mentioned that already, it's a basic part of the understanding and is not surprising at all.

Again something most people arent going to want.

You're assuming a lot. The fact that the variable substitution could expand to multiple arguments (because it's really just simple textual substitution) is in fact very powerful and useful (you can pass multiple arguments in one variable, for example), and you can always quote if you really want it to expand to one argument, the same way you would quote strings containing spaces to make them one argument. This is very consistent; observe that if $SOMETHING contains "foo bar" (without the quotes), then

    grep $SOMETHING
is exactly the same as

    grep foo bar
but what you seem to be suggesting is that it becone

    grep "foo bar"
...what? Where did those quotes come from? More importantly, how do you propose to remove them? My point is, the current behaviour is simple and consistent as well as powerful and flexible, and I suspect people are just going "it doesn't work the way I thought it would" and downvoting my comment when they really haven't thought about the perfectly logical explanation for why things are the way they are. The original developers of the shell language were most certainly not stupid.
Re:A common mistake involving wildcards and the find command by DavidSJ on Sunday February 09, 2020 @02:41AM (#22280925)
The current behavior is not simple and consistent. If it were, then *.jpg would “expand” to the empty list of arguments, rather than remain unchanged, in the degenerate case of no match.
Re:A common mistake involving wildcards and the find command by olejorgenb on Sunday February 09, 2020 @04:22AM (#22281223)
I almost never want the split-on-ifs-on-expansion behavior. That's one of the reasons I use zsh. When I want I can either use an array or $=SOMETHING.

The default split behavior basically forces you to always quote expansions which is tiresome. It also is confusing because almost no other programming language works like that.

And when you assigned SOMETHING you did wrote SOMETHING="foo bar" (with quotes) so it sorta makes sense visually also.

Re:A common mistake involving wildcards and the find command by UncleMeat on Sunday February 09, 2020 @03:04PM (#22284011)
Our tools should be dead simple whenever possible rather than "if you make mistakes then git gud".
Re:A common mistake involving wildcards and the find command by pwdisswordfish2 on Sunday February 09, 2020 @02:55AM (#22280962)
So, do you prefer people have this misconception instead? https://unix.stackexchange.com/q/544189/348263

Also, for the case of passing multiple arguments to a command, just use an array:

    SOMETHING=(foo bar)
    
    grep "${SOMETHING[@]}"
It will handle arguments with spaces correctly, unlike using a string variable and splitting it on spaces. There is no reason to ever use space splitting, unless you're maintaining some legacy interface where you can only pass a string instead of an array.
Re:A common mistake involving wildcards and the find command by porker on Sunday February 09, 2020 @04:36AM (#22281258)
I don't understand escaping properly.

Why, on some machines (all varying distros of Ubuntu) does

    find -name *.xml
return nothing and I have to use

    find -name '*.xml'
whilst on other machines the first command returns results? The shell is always bash, it's a stock install - so why the variation?

I get that it should always be quoted, but not why it sometimes works when unquoted and other times doesn't.

Re:A common mistake involving wildcards and the find command by scolby33 on Sunday February 09, 2020 @10:25AM (#22282517)
Perhaps `shopt -s nullglob` has been set?

“If set, Bash allows filename patterns which match no files to expand to a null string, rather than themselves.” [1]

[1] https://www.gnu.org/software/bash/manual/html_node/The-Shopt...

Re:A common mistake involving wildcards and the find command by bmn__ on Sunday February 09, 2020 @10:52AM (#22282644)
> so why the variation?

Difference between dash and bash?

Shell options? `man shopt` and grep for glob

Re:A common mistake involving wildcards and the find command by BeeOnRope on Sunday February 09, 2020 @01:06AM (#22280652)
That's not going to directly solve this problem since the globs are expanded before the invoked process is even called with its arguments.
Re:A common mistake involving wildcards and the find command by mikenew on Saturday February 08, 2020 @10:09PM (#22280145)
> Smart case: the search is case-insensitive by default. It switches to case-sensitive if the pattern contains an uppercase character*.

This is what I want as a default for pretty much any text search.

Re:A common mistake involving wildcards and the find command by josteink on Sunday February 09, 2020 @01:59AM (#22280811)
Emacs have had this since forever. It’s pretty neat.
Re:A common mistake involving wildcards and the find command by Lio on Sunday February 09, 2020 @04:09AM (#22281182)
Vim also provides it if you use:

  :set smartcase
Re:A common mistake involving wildcards and the find command by Aardwolf on Sunday February 09, 2020 @09:01AM (#22282123)
Imho also anything that outputs or uses multiples of 512 bytes (blocks) by default

Sure, disks may have sectors and there may be some use cases, but we use bytes, kilobytes, kibibytes, etc now for data sizes.

Re:A common mistake involving wildcards and the find command by weberc2 on Saturday February 08, 2020 @09:00PM (#22279913)
In this case the shell hasn’t aged well. Besides the globing feature, the bash language is arcane and few can actually reliably write correct bash despite it being ubiquitous. Personally I would like to see a completely reimagined take on the shell.
Re:A common mistake involving wildcards and the find command by eru on Saturday February 08, 2020 @09:41PM (#22280039)
Have you looked at powershell, scsh or fish?
Re:A common mistake involving wildcards and the find command by AlexCoventry on Saturday February 08, 2020 @11:26PM (#22280397)
shellcheck helps a ton, though.
Re:A common mistake involving wildcards and the find command by choeger on Sunday February 09, 2020 @06:27AM (#22281591)
I would argue that this case shows that the shell(s) have a problem with a feature being enabled by default. In hindsight it might have been smarter to put globbing expressions in special quotes than the other way round.
Re:A common mistake involving wildcards and the find command by fit2rule on Sunday February 09, 2020 @06:11AM (#22281544)
For me, Silver Searcher often fails to find matches in files that 'grep' will, easily. I've not been able to determine why this is, but it happens often enough that I consider SS to be highly unreliable, even if its fast.

So, it seems that there are still some issues to be sorted here.

Re:A common mistake involving wildcards and the find command by bmn__ on Sunday February 09, 2020 @10:55AM (#22282660)
This claim needs adequate proof, please provide an example. Otherwise the HN reader has to assume you made a user error.
Re:A common mistake involving wildcards and the find command by fit2rule on Monday February 10, 2020 @03:56AM (#22287457)
Maybe it has something to do with my not understanding its case sensitivity rules .. which I've learned about in this thread as being different.

As soon as it happens again I'll bring it here as an example, but like I said I stopped using silversearcher because of it getting in my way like that.