- Use a regex to find a string in some text: use regexes, get a &str

- Escape all the regex reserved characters in that string: Okay this requires mutation but we can't mutate a &str, let's loop over the &str's characters (using the Chars iterator) and push them into a new String with the proper escapes

- The new String we own, so we can easily turn it into a regex and use it to search the original string.

Problems with well-constrained allocation paradigms do really well in rust. Problems with reading/parsing/storing messy data structures from external data just really, really hurt.

I think a big part of the problem is that the ownership analysis layer has the dual crises of being (1) really complicated and too hard to keep in your head as a single model and (2) specified in an ad-hoc way that resists formal rules. So eventually you end up in of those puzzles like you mention where... there's just no answer! No one's been where you have, and the lack of rigor means you end up trying to reverse engineer the compiler front end trying to find the trick that works. And then you give up and write it in Go or Python or whatever.

It is an error to access a place, when an access conflicts with a loan, and the loan is live.

In fact there are an infinite number of correct Rust programs which will never access memory incorrectly but which will still be rejected by the compiler, for the simple reason that Rust's authors, talented though they may be, have made no progress at all at solving the Halting Problem.

What Rust actually accepts is a subset of correct programs. And this subset is somewhat informally defined as "whatever the compiler could manage to prove". Real code hits against this limit occasionally, and when it does there's really no option other than "join the Rust team" or "try voodoo".

The unstated problem is that there are patterns from other languages that are actually invalid and Rust is correctly denying, but not properly communicating it, which causes frustration. You can always make the argument that the failure modes are benign or rare, and for those cases noone is stopping you from actually using unsafe.

[1]: https://manishearth.github.io/blog/2015/05/27/wrapper-types-...

And this is why this argument persists. This is, to people outside the community, a semantic evasion:

1. It relies on a Rust-internal definition for "actually valid" (conforming to Rust's specific set of provability requirements) that doesn't correspond to what the rest of the world views as "correct" (not behaving incorrectly). Think about stuff like allocate-through-a-session-and-free-in-a-block paradigms (Apache was famous for this), or run-once-and-exit, or garbage collection, etc... Those things aren't "invalid" in any reasonable sense, they're just not what Rust programs do.

2. The definition for "valid" is (and this is my point above) entirely Rust-internal and ad hoc. It's not that we refuse to conform to your rules, really, it's that we don't know what they are!

> Fun fact: while at Mozilla I heard multiple anecdotes of [very intelligent] Firefox developers thinking they had found a bug in Rust's borrow checker because they thought it was impossible for a flagged error to occur." However, after sufficient investigation the result was always (maybe with an exception or two because Mozilla adopted Rust very early) that the Rust compiler was correct and the developer's assertions about how code could behave was incorrect. In these cases, the Rust compiler likely prevented hard-to-debug bugs or even exploitable security vulnerabilities. I remember one developer exclaiming that if the bug had shipped, it would have taken weeks to debug and would likely have gone unfixed for years unless its severity warranted staffing.

Sometimes people think that they're right and that the compiler is wrong, but often, (not not necessarily always!) it's that they forgot some important piece of context.

(Oh, and Rust absolutely can do "allocate in a block and free at once" stuff, or "run once and exit" stuff...)

To recap, here are the rules:

It is an error to access a place, when an access conflicts with a loan, and the loan is live. Access means source code construct to read or write, whether the construct is actually executed in runtime is irrelevant. Place is static approximation of a set of bytes in the memory. Place is either local (x), field (x.f), or upvar (local captured in closure). Indexing (x[i]) and method calls (x.m()) are approximated as local (x). Loan is either shared (&x) or mutable (&mut x). Read access conflicts with mutable loan, write access conflicts with all loans. Expression is live from the definition to the end of the containing statement. Declaration is live in connected region of control flow graph from the definition to potentially multiple last uses.

Like all type checking technology, the rule is about static behavior, not dynamic behavior. So halting problem is irrelevant.

Rust accepts all programs that follow the rule in static behavior. It's not a subset, and it's not a best effort to prove. It's the complete statement.

"Access" means source code construct to read or write. Rust doesn't care whether the construct is actually executed in runtime.

    let (left, right) = (&mut foo[..split], &mut foo[split..]);

I see these are limitations but not show-stoppers in any way.

Yes, sure, the rule in question may be simple in the abstract (it's a compiler, after all -- they're just software doing straightforward things). But the ability for the poor programmer to detect which rule is being violated where is a lot more limited than the compiler is designed for. Thus the user with the upthread complaint, which is hardly unique.

I mean, at the end of the day if Rust wants to be an everyday language for this kind of everyday problem, the analysis paradigm needs to be communicated much better to everyday hackers (via docs, error messages, whatever). When Rust was new this seemed like just a technical problem to be solved with software maturity. I guess at this point after several years of regularly returning to play with Rust and being frustrated every time, I've mostly given up.

Not to belabor this growing thread, but here's the disconnect. I complained the rules were ad hoc and complicated, you replied that they were simple, and when challenged on an edge case your treatment is to add another clause to re-define a term you used in isolation earlier.

That's what "ad hoc and complicated" means.

Actually, my elaboration of "place", local/field/index, is nearly complete. The only thing missing is upvar, which is local captured in closure. Pre-NLL "live" is simple: expression is live from the definition to the end of the containing statement, and declaration is live from the definition to the end of the containing block. Post-NLL "live" is connected region of control flow graph from the definition to potentially multiple last uses. Really, that is the whole story.

There are two changes currently in development. One changes definition of "place" to include field captured in closure, in addition to local captured in closure. The other is more drastic, changing definition of "live": a loan is live where the loan is origin of a variable and the variable is live. This is great, because current approximation is equivalent to union of all origins.

I hope to evolve my understanding so that I can do the right thing at design time to minimize unnecessary clone()s.

Whereas in Rust you know that it's probably possible to use a &str there instead of a String. If you just bang on it a little longer, you can make it just a bit more optimal. The clone()s and the Vec::new()s are all explicit, making them feel heavier than those other languages, when in reality they're still quite a bit lighter.

You usually don't have to optimize them out! Your code will probably be faster than Java even with a bunch of clones, etc! But there's this temptation that's really hard for programmers to resist, to sink hours and hours into making things just slightly more optimal given the opportunity. You have to consciously talk yourself out of that if you want to be productive.

That's… not exactly true. `Vec::new` is completely free so that's a different debate, but much as in C++ or C allocations in Rust are much more expensive than in managed languages:

1. system allocators genuinely suck, all of them, though some more than others (iirc macos' is especially bad)

2. managed languages can much more easily specialised allocation strategies (freelists, bump allocators, type-custom allocation strategies), this is either difficult / impossible (no custom allocators) or way more painful (manually pass custom allocators in) in Rust

As a result, allocations in Rust are really tremendously slow by default, it's not too rare to see questions about Rust programs which are slower than managed equivalents, in release mode. Because the rust program is allocation heavy (relatively), it might have 10% the allocations of the non-rust program but the allocations are 100 times more expensive.

Edit: I do know that languages where strings are immutable use that fact to do lots of optimization (automatically sharing "copied" strings and just cloning reference-counters, for example), but you can accomplish some of this in Rust too with Rc<> or even persistent data structures if you really want to. And of course there are cases where it's more efficient to actually mutate a string, which these languages can't do. But it sounds like you're talking about something else?

That's 100% correct but there's one problem: how do you write down your future improvement opportunities?

Skip 2 optimizations here, 3 optimizations there and after 30 PRs you can easily have notes spanning 100 bullet points on what to optimize one day in the future Soon™.

I am still not at what I'd call an expert level in Rust -- and the skill ceiling is quite high -- so maybe with time I'd start to automatically spot these optimization opportunities?

But before I am at that level I'd prefer to either keep hugely long notes or, ideally, have a tool recommend optimizations to me.

The better tool is to profile the code, then optimize hotspots. There's a Rust Performance Book with a list of profilers known to work with Rust programs[1]. Rustc does support Profile Guided Optimization[2] which is often pretty good at speeding up the output even without any code changes.

[1] https://nnethercote.github.io/perf-book/profiling.html [2] https://doc.rust-lang.org/rustc/profile-guided-optimization....

> The simplest tool is to just grep for various constructs that copy or allocate.

Do you happen to have a curated list somewhere?

Not sure of a list for things that copy their data. The Clone trait is the obvious one, and it requires calling clone() to make the copy. EG `val.clone()`. So searching for `.clone()` will get you those. But other things like to_string are expensive, and From/Into are sometimes expensive.

[1] https://docs.google.com/presentation/d/1q-c7UAyrUlM-eZyTo1pd...

This took 2 min: https://play.rust-lang.org/?version=stable&mode=debug&editio...

and I don't recall what's the last time I used the regex crate.

I'm certainly doing all the regex stuff incorrectly, no idea how to match regex-reserved characters, no idea if I'm using the iterator API correctly (its the second example in the regex docs..), etc.

But the idea is simple, search the text for the regex, escape all the reserved characters, use that regex to search for occurrences of the same regex.

I guess that if you want a fully-vectorized, zero-allocation, zero-copy, multi-threaded, gpu... version of this that maxes out the theoretical peak of the hardware available, then you'll probably end up putting a considerable amount of work here.

But otherwise, for someone with literally 30 seconds of experience with the regex crate, it was as straightforward to implement as you mentioned. Just c&p their second example, and just added some glue. This is how I'd implement this in Python, or Java...

EDIT: other responses suggest cloning, but that seems to suggest that one has to explicitly and deliberately think about cloning. I instead just played a game of type golf using functions like `.to_string()` or `.collect` to "clone" stuff behind the scenes. My code is probably horrible, but is the first thing that came to mind.

If you don't really know Rust and in particular understand the borrow checker and how it works (and why it exists), you are going to have a bad time.

Rust in general has a learning curve, but once you do learn it there is a payoff. If Rust code compiles (and if you didn't use 'unsafe'), you can be pretty damn sure it is free of a long list of potential bugs including concurrency bugs, buffer overflows, null pointer dereferences, alignment problems, double-free, use-after-free, stack smashing, most memory leaks, and so on. The runtime cost of that assurance is less in Rust than in any other language I know, and sometimes the safety of Rust actually lets you do things in higher performing ways that would be dangerous to code in C. In C/C++ you have to knit your own straightjacket, and yours may not be as well thought out or efficient as the one the Rust compiler issues you.

I've really started to appreciate Rust's safety. If the code compiles there can still be bugs, but they're going to be higher up at the logic/algorithm level instead of the annoying sorts of bugs that constantly crop up in C/C++.

As for a "borrow cow", you probably want to read https://doc.rust-lang.org/std/borrow/enum.Cow.html and https://deterministic.space/secret-life-of-cows.html. For what you are doing, it is an optimization to avoid unnecessary clones that is absolutely not required: make your code work with String and worry about Cow<'_, str> later.

And here's my branch (you can see the latest commits to see the file I'm modifying): https://github.com/ahelwer/tree-sitter/tree/testfile-separat...

I haven't had a chance to go through and add clones everywhere, and will be away at a PT appointment for the next hour or so, but would appreciate any pointers you can give.

  diff --git a/cli/src/test.rs b/cli/src/test.rs
  index ac4807bf..8b9882ab 100644
  --- a/cli/src/test.rs
  +++ b/cli/src/test.rs
  @@ -410,16 +410,17 @@ fn parse_test_content(name: String, content: String, file_path: Option<PathBuf>)
           .map(|b| String::from_utf8_lossy(b).to_string())
           .map(|s| escape_reserved_regex_chars(&s));
       
  -    let suffixHeaderPattern : Option<String> = suffix
  +    let suffixHeaderPattern : Option<String> = suffix.as_ref()
           .map(|s| String::from(r"^===+") + &s + r"\r?\n([^=]*)\r?\n===+" + &s + r"\r?\n");
       
  -    let suffixDividerPattern: Option<String> = suffix
  +    let suffixDividerPattern: Option<String> = suffix.as_ref()
           .map(|s| String::from(r"^---+") + &s + r"\r?\n");
   
  -    let headerRegex = suffixHeaderPattern
  -        .and_then(|s| ByteRegexBuilder::new(&s[..]).multi_line(true).build().ok())
  -        .as_ref()
  -        .unwrap_or(&HEADER_REGEX);
  +    let headerRegexFromSuffixHeaderPattern = suffixHeaderPattern.as_ref()
  +        .and_then(|s| ByteRegexBuilder::new(&s[..]).multi_line(true).build().ok());
  +
  +    let headerRegex = headerRegexFromSuffixHeaderPattern
  +        .as_ref().unwrap_or(&HEADER_REGEX);
   
       // Identify all of the test descriptions using the `======` headers.
       for (header_start, header_end) in headerRegex

    error[E0382]: use of moved value: `suffix`
       --> cli/src/test.rs:416:48
        |
    406 |     let suffix = FIRST_HEADER_REGEX
        |         ------ move occurs because `suffix` has type `std::option::Option<std::string::String>`, which does not implement the `Copy` trait
    ...
    414 |         .map(|s| String::from(r"^===+") + &s + r"\r?\n([^=]*)\r?\n===+" + &s + r"\r?\n");
        |          ------------------------------------------------------------------------------- `suffix` moved due to this method call
    415 |
    416 |     let suffixDividerPattern: Option<String> = suffix
        |                                                ^^^^^^ value used here after move
        |
    note: this function consumes the receiver `self` by taking ownership of it, which moves `suffix`
       --> /Users/ekuber/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:451:38
        |
    451 |     pub fn map<U, F: FnOnce(T) -> U>(self, f: F) -> Option<U> {
        |                                      ^^^^
    
    error[E0716]: temporary value dropped while borrowed
       --> cli/src/test.rs:419:23
        |
    419 |       let headerRegex = suffixHeaderPattern
        |  _______________________^
    420 | |         .and_then(|s| ByteRegexBuilder::new(&s[..]).multi_line(true).build().ok())
        | |__________________________________________________________________________________^ creates a temporary which is freed while still in use
    421 |           .as_ref()
    422 |           .unwrap_or(&HEADER_REGEX);
        |                                    - temporary value is freed at the end of this statement
    ...
    425 |       for (header_start, header_end) in headerRegex
        |                                         ----------- borrow later used here
        |
        = note: consider using a `let` binding to create a longer lived value

    let suffixHeaderPattern: Option<String> = suffix
        .as_ref()
        .map(...);

    let headerRegex = suffixHeaderPattern
        .as_ref()
        .and_then(|s| ByteRegexBuilder::new(s).multi_line(true).build().ok())
        .unwrap_or(HEADER_REGEX.clone());

Edit 2: to further drive the point home that this should be a bug, this is the current output for a similar case while I was trying to minimize this:

    error[E0308]: mismatched types
      --> src/main.rs:10:22
       |
    10 |         .map(|s| bar(s));
       |          ---         ^ expected `&Struct`, found struct `Struct`
       |          |
       |          help: consider using `as_ref` instead: `as_ref().map`

`Option::deref` is a good choice here.

If only it was spelled CoW and ARC I would very quickly grasp it (probably).

But it's an extremely minor inconvenience and arguably makes the whole thing very memorable

2. Given you mention "maps of option", probably you are doing something like opt.map(|x| &x.field) or equivalent which is obviously invalid (since you are returning a reference to a subobject of a function argument). Instead, you need to do opt.as_ref().map(|x| &x.field) (or as_mut() if it's mutable).

Honestly the friendliness of the community is a big part of why I'm learning Rust. I found it so much harder to get free help from volunteers with c code issues. Not that I deserve it or anything, but it's really nice and I try to contribute back by writing PRs to improve the docs once I understand something.

Long suggestion: Working with strings in Rust requires that you have a C++ mental model of what strings are and what properties (in the abstract sense of the word) they have. Rust will make sure you can't misuse them, and it will make them as ergonomic as they can be within that C++-like model, but it's a fundamentally different concept from what you have in languages like C# and Java, and if you're trying to look at things from that other perspective, you're going to keep running into walls over and over.

More broadly, when you're using Rust you need to be in the basic mindset of a C++ programmer. From there Rust will make things significantly easier in many ways, but if you don't start with the right mental model, you're going to have a bad time. Strings are probably the most striking case of this because the way they get treated in virtually all higher-level languages is so wildly different from the way they get treated at the low-level.

Just do the first regex search, then escape into a String with regex::escape, then build a regex from the String and search with it.

Of course what you are doing is probably wrong since you should use a string matching crate rather than escaping characters and using a regex matching crate.

Just lol at the idea that you won't run into borrow checker issues when dealing with strings though.

I think what's being called "trivial" is doing a bit of regex searching. It's probably accurate to call that trivial for an experienced Rust programmer, but if you're just beginning, I don't think it's helpful to call anything trivial. I still remember my first exposure to Rust. It was different. It took a bit to grok. But once it clicked, things were much better.

As the maintainer of the regex crate, I invite you or anyone to ask for help using regexes. The regex repo has Discussions opened up, so it's appropriate to ask for help, even if they are beginner questions: https://github.com/rust-lang/regex/discussions

As usual though, try to provide as many details as you can. Giving the source code you have but can't get to work is a great start, for example.

https://doc.rust-lang.org/rust-by-example/ and https://github.com/rust-lang/rustlings might be good places to visit to gain that experience, as well as improving the specific diagnostics to be more actionable.

Yes, I figured that to be the case, but still wanted to say, "Discussions are open for questions, even if they are beginner related, as long as it's related to regex'ing somehow." :-)

https://play.rust-lang.org/?version=stable&mode=debug&editio...

https://github.com/danielzfranklin/drm-fourcc-rs/blob/main/b...

https://play.rust-lang.org/?version=stable&edition=2018&gist...

But what you really want is to get your head around ownership and borrowing, especially when it comes to the scoped RAII that rust has (e.g. freeing a String while a reference to it still exists and is being used would give you the "temporary value dropped while borrowed" error).