News for hackers. Stuff that matters.
Re:Astral by akprasad on Tuesday April 18, 2023 @01:57PM (#35617435)
I love Ruff and I'm glad that Charlie and the rest of the team are able to work on such tools full-time. I'm also happy to see that the author of Maturin (Rust+Python interop) is involved, as Maturin is a fantastic project with great ease-of-use.

For those who aren't familiar, Ruff is a very fast Python linter that supersedes a variety of tools like isort, flake8, and perhaps eventually Black [1]. My understanding is that since Ruff is an all-in-one tool, it can parse the underlying Python files just once -- and since this is the most expensive phase of these kinds of tools, adding any extra functionality on top (such as linting, formatting, ...) is very cheap. And that's to say nothing of the fact that Ruff is built on Rust, with the usual performance wins that Rust projects tend to have.

I'm a little apprehensive of how development will look given a VC raise with the expectation of VC returns. But for now, I'm delighted and can't wait to see what the team comes up with.

[1]: https://github.com/charliermarsh/ruff/issues/1904

Re:Astral by Night_Thastus on Tuesday April 18, 2023 @02:33PM (#35617978)
I thought Black was an auto-formatter, similar to clang-format, but for Python. Ruff seems to be only doing linting, not formatting. Am I missing something?
Re:Astral by senko on Tuesday April 18, 2023 @02:37PM (#35618038)
Ruff will be getting more autoformat capabilities: see the link in parent poster and also read the first comment there, from one of Black maintainers.
Re:Astral by Night_Thastus on Tuesday April 18, 2023 @02:41PM (#35618113)
Pretty bold to try to replace Flake8 and Black with one tool. I hope they succeed. Would be great to have a tool that does both and is substantially faster.
Re:Astral by senko on Tuesday April 18, 2023 @02:43PM (#35618152)
Indeed.

I do hope they'll expose Ruff as a Python module / API in the future. I'm currently using Black to format Python code that's in-memory (never gets written out to disk).

With Black (as an imported Py module), it's just a matter of passing in a string and getting one back. With Ruff as it is now, I'd have to write that out to disk, spawn Ruff process, then read the formatted file back in. Do that for many files and the speed advantages disappear, and actually it's slower.

Re:Astral by BiteCode_dev on Tuesday April 18, 2023 @02:54PM (#35618309)
You can mount the file system in memory for that, but I see your point.
Re:Astral by zamnos on Tuesday April 18, 2023 @09:03PM (#35622839)
Why doesn't ruff reading/writing from stdin/out, combined with passing the data in with python subprocess.run and os.pipe work here?

    cat bad.py | ruff check --fix --stdin-filename stdin.py - > good.py
Substituting bad.py and good.py for in-memory os.pipe objects for your data. This still involves spawning a subprocess, which can add up, but you're not having to read/write to disk at least.
Re:Astral by senko on Wednesday April 19, 2023 @02:54AM (#35625099)
It does work. The tradeoff is:

* run ruff command many times (with different input file each time) * write all the files to "disk" (tempfs actually so it doesn't involve disk access) and run ruff once

Currently with black I avoid that by importing it as a module once so I get the benefits of both (and is simple to boot).

In other comments here I've read that a person heavily involved in Python-Rust interop is also part of the team, so there's hope they'll expose ruff as a module too and magically whisk my dilemma away :-)

Re:Astral by NegativeLatency on Tuesday April 18, 2023 @03:13PM (#35618612)

    ruff check . —-fix
Re:Astral by swyx on Tuesday April 18, 2023 @02:16PM (#35617715)
if it's any consolation, Accel is in my mind one of the "good ones". they have a record of backing "good" devtool companies that manage open source and commercial concerns well - Vercel, Sentry, etc. (i'm sure theres more, those are just the two that i'm closest to)
Re:Astral by echelon on Tuesday April 18, 2023 @03:00PM (#35618390)
What are the good VC companies? Not just for dev tools, but in general. Where do you find this information?
Re:Astral by swyx on Tuesday April 18, 2023 @03:16PM (#35618658)
just being "in the scene" for a few years :)

tbh i care more about individual investors than the fund brand, but its a fallback whenever meeting someone new

Re:Astral by pid-1 on Tuesday April 18, 2023 @03:01PM (#35618402)
Even if no VC was involved, how do you make money developing an open source linter?
Re:Astral by darkstar999 on Tuesday April 18, 2023 @03:10PM (#35618560)
They are probably looking for an ecosystem around the tool, like how github built on git.
Re:Astral by qbasic_forever on Tuesday April 18, 2023 @04:40PM (#35619750)
How did Hashi corp make money (and an eventual billion dollar plus IPO!) developing open source tools like vagrant, consul and terraform...

Clearly the creator of ruff is going to expand from an open source tool into a company focusing on Python tooling and developer experience, services, etc. There is a market for stuff like that just based on pycharm and jet brains' success in the space alone.

Re:Astral by nwellnhof on Tuesday April 18, 2023 @04:54PM (#35619967)
HashiCorp made a $270 million loss last year, more than 50% of their revenue.
Re:Astral by spullara on Tuesday April 18, 2023 @04:56PM (#35620003)
A lot of the money comes from Vault which is open source but only used by larger organizations that want support.
Re:Astral by andygeorge on Tuesday April 18, 2023 @07:59PM (#35622272)
Vault namespacing also requires an enterprise license, so yeah if you're doing anything useful with Vault at scale, you're paying for it.
Re:Astral by quadrature on Tuesday April 18, 2023 @05:53PM (#35620848)
im sure they have a ton of ideas. my bet is that they might launch something in the CI/CD space. like building python packages/images fast, speeding up python testing or tackling something in ML inference.
Re:Astral by mixmastamyk on Tuesday April 18, 2023 @06:25PM (#35621270)
Telemetry probably. Remember the telemetry in your terminal folks, on here? :cry:

But you might be able to do support contracts, like ansible. Tough road though.

Re:Astral by nine_k on Tuesday April 18, 2023 @06:34PM (#35621370)
Support contracts. Custom features built to order. Other commercial products based on the open-source libraries for parsing and analysis. Running a hosted service.

All this while remaining a team of 2-3, hopefully, with all company-running stuff minimized and outsourced.

Re:Astral by panchicore3 on Tuesday April 18, 2023 @09:54PM (#35623303)
Look at sentry
Re:Astral by sph on Sunday April 23, 2023 @11:44AM (#35677207)
Sentry requires a platform to work, it's much easier to monetise. I'm not even sure it's possible to self-host?
Re:Astral by klibertp on Thursday April 20, 2023 @09:35AM (#35639682)
I think we're going to see a shift in developers expectations towards the cost of their tools, as a side-effect of Copilot & co. If you already pay $100/mo in subscriptions for IDE, its plugins, and services it integrates with, then forking additional $5 for a linter doesn't look as absurd as it used to...
Re:Astral by cauthon on Tuesday April 18, 2023 @03:31PM (#35618876)
just starting to dabble in Rust, and I've seen PyO3 mentioned a number of times for writing rust bindings. What's the difference between Maturin and PyO3?
Re:Astral by shadaj on Tuesday April 18, 2023 @03:34PM (#35618910)
PyO3 is the library that enables Python <-> Rust bindings, Maturin is a build tool for packaging PyO3 Rust libraries (which export Python APIs) as Python packages!
Re:Astral by cauthon on Tuesday April 18, 2023 @05:37PM (#35620623)
thanks!
Re:Astral by lozenge on Tuesday April 18, 2023 @03:39PM (#35618979)
The main reason pylint is slow is it tries to infer the types of the variables, and it predates type annotations so it can go through multiple functions/control paths across multiple files to figure these out. This approach is obviated by type annotations and type checkers.

Ruff is built for speed from the start and doesn't look at type annotations because you should run a type checker alongside.

Re:Astral by zx8080 on Tuesday April 18, 2023 @08:28PM (#35622551)
Just to be sure. Is Ruff faster by lacking type checking?

Also, does Ruff work well on codebase without type hints?

Re:Astral by npage97 on Tuesday April 18, 2023 @08:53PM (#35622749)
Ruff is definitely faster by lacking type-checking. That requires way more analysis.

Ruff will work fine without type hints.

I believe it rightfully leaves it to mypy for those who want those features.

Mypy transpiles itself to c using mypyc and that can still take a while to complete when caches get invalidated.

Re:Astral by rattray on Tuesday April 18, 2023 @09:37PM (#35623208)
Ruff is faster than many tools which don't do type checking.

Ruff works well regardless of whether you use type hints.

Re:Astral by fsckboy on Wednesday April 19, 2023 @01:17AM (#35624514)
that's dodging the question a little bit? let's say Ruff is faster because it's so good, how much faster than type checking alternatives would this great tool be if it did type checking?
Re:Astral by viraptor on Wednesday April 19, 2023 @02:26AM (#35624900)
Without building the tool to compare, it's really hard to tell. In my experience of rewriting some Python and Ruby code into compiled versions, you can expect 10-20x improvements. But it's both really dependant on the code itself and on how you're going to reimplement it - most of the time it doesn't make sense to do perfect 1:1 rewrites.
Re:Astral by lozenge on Wednesday April 19, 2023 @06:29AM (#35626473)
That depends on how you do type checking and more specifically type inference. Mypy infers types of local variables but not function boundaries as it expects you to annotate function arguments and return values.

In mypy a variable taking values of two types is an error (unless explicitly annotated) which means you can assume the type from the first assignment statement and then use that assumption every other time the variable is used/assigned. In pylint it allows variables to take multiple types and only emits errors when an operation is invalid against one of the types. E.g. `x[0]` is valid for lists and tuples so you could assign it a list or a tuple depending on some condition.

The number of possibilities it considers quickly multiplies. Eg if your function is of the form `if blue_condition: x = blue_action(x)` that's a doubling of the possibilities and ten flags -> 1024 possibilities. (pylint has some heuristics on when to give up.)

The advantage of pylint is that it doesn't require your code to fit a certain style. But nowadays people have type annotations and they prefer to use them even if it means changing their code style a bit here and there.

Other type inferers with different approaches are pyright, Jedi, Pyre, pytype etc with different tradeoffs.

Re:Astral by toyg on Tuesday April 18, 2023 @04:18PM (#35619430)
I actually cursed Maturin a few years ago, as it refused to compile on OpenBSD and hence broke some lib I planned to use.

Python is becoming a bit too reliant on Rust. Rust is good but, in term of portability, is just not as mature as C. If your lib relies on Rust, please please please test it on something beyond Linux and Mac.

Re:Astral by criddell on Tuesday April 18, 2023 @05:50PM (#35620813)
> please test it on something beyond Linux and Mac

That’s asking a lot. If your platform is poorly supported by Rust, then maybe that’s where your efforts should be directed. If you fix that, lots of interesting stuff beyond one library is unlocked.

Re:Astral by senko on Tuesday April 18, 2023 @06:10PM (#35621084)
Theo de Raadt engaging the Rust community would be a sight to behold :)
Re:Astral by steveklabnik on Tuesday April 18, 2023 @06:29PM (#35621322)
I am not aware of him saying anything about Rust beyond this: https://marc.info/?l=openbsd-misc&m=151233345723889&w=2

Which doesn't mean he hasn't, of course!

Re:Astral by robocat on Tuesday April 18, 2023 @07:43PM (#35622101)
That email (from Dec 2017) ends with “Such ecosystems come with incredible costs. For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space.”. I presume Theo is actually complaining about using more than 3 GB (?) of memory, but still it really shows the different cost-versus-benefit decisions that we all make.
Re:Astral by steveklabnik on Tuesday April 18, 2023 @07:52PM (#35622202)
Yes. My recollection is that around that time, llvm and/or rustc itself would sometimes go over that. I don't know off the top of my head what memory usage looks like right now.
Re:Astral by jeffparsons on Wednesday April 19, 2023 @01:10AM (#35624472)
Is it actually reasonable to expect compiler toolchains to work on old or underpowered hardware? Or is the real problem that cross-compilation is still a special case, rather than being the only way compilers work? If it wasn't still normal to depend on your target environment being the same as your build host, would anyone really want to do serious development work directly on their RPi/Gameboy/watch/whatever, just because that is the target runtime environment?
Re:Astral by robocat on Wednesday April 19, 2023 @02:52AM (#35625082)
I am wildly guessing that Theo’s beef is more that rust uses a lot of memory (paraphrase: 640kb should be enough for anybody). OpenBSD does integration builds on a variety of different systems, and maybe Theo noticed the OpenBSD/386 build failing due to lack of necessary memory?
Re:Astral by jeffparsons on Wednesday April 19, 2023 @05:11AM (#35626026)
That seems a reasonable guess, but it brings me straight back to wondering why such a diversity of build environments is necessary or useful. And my wild guess is that it's mostly because cross compilation is still a second class citizen in most languages today. Though I guess it could be a kind of cultural expectation that you should be able to compile the whole OS on the hardware you're running it on.
Re:Astral by steveklabnik on Wednesday April 19, 2023 @01:27PM (#35630863)
My understanding is that the policy of the project is that the base system must not be cross compiled. My understanding of why is that they want to be able to fully bootstrap from base itself.
Re:Astral by thayne on Wednesday April 19, 2023 @07:26AM (#35626863)
The rust compiler runs on i86, the complaint is that it can't compile the rust compiler, because it uses more memory than is addressable on i86.
Re:Astral by criddell on Wednesday April 19, 2023 @12:55PM (#35630482)
That’s what the person was getting at. Does it matter if it can be cross-compiled elsewhere?
Re:Astral by steveklabnik on Wednesday April 19, 2023 @01:26PM (#35630839)
It matters to Theo, due to policies created for specific objectives. It does not matter to many other people.
Re:Astral by jeffparsons on Wednesday April 19, 2023 @10:56PM (#35635835)
I guess at the end of the day this is all that really matters. If there are specific goals that Theo has around this, then there's no point in me second-guessing whether those are "reasonable"; it's a matter of values and preferences and whatever.

Whereas if this conversation was about something with broader stewardship, like _Linux_, I'd be saying this is silly, you shouldn't be compromising other things just so you can build your RPi kernel on an RPi.

Re:Astral by senko on Thursday April 20, 2023 @08:28AM (#35638936)
I think latchin onto this example misses the crux of his raitonale:

> In OpenBSD there is a strict requirement that base builds base. So we cannot replace any base utility, unless the toolchain to build it is in the base.

> Such ecosystems come with incredible costs.

Basically, the cost of adding rust to the OpenBSD base system currently far outweights the proposed benefits (reimplement fileutils), especially considering people will probably not want to pour in the effort needed to rewrite those with strict POSIX compliance (which is another requirement).

He's not saying rust isn't useful but that it wouldn't be a net benefit to have a hard dependency on it in the base system. In the BSD world folks take a very strict and conservative view of what can go in base (and for good reason IMHO).

This is different from the GP comments about just making it possible to use rust programs/libraries in OpenBSD, so we're definitely on a tangent here.

Re:Astral by Aeolun on Tuesday April 18, 2023 @06:38PM (#35621415)
> please please please test it on something beyond Linux and Mac

Why? Simply because you use something that even 95% of the unix folk do not?

I can totally see how they wouldn’t be concerned with that. Just as I’m not concerned with making my JS lib work for those weirdos that still run IE 5.5.

Re:Astral by ohgodplsno on Wednesday April 19, 2023 @05:21AM (#35626099)
>If your lib relies on Rust, please please please test it on something beyond Linux and Mac.

No, thank you, I'll just add Windows. Your choice to use platforms that nobody else does save for a few specialists does not constitute a need on my part to support your favorite. In the same way that if I package for Debian, Ubuntu and RHEL, it's your problem that it's not on Arch or unsupported by your custom Hannah Montana Linux. Feel free to submit PRs though.

Re:Astral by Cort3z on Wednesday April 19, 2023 @02:48AM (#35625059)
Ruff is great, but it lacks a lot of the rules[1]. As far as I understand it reads every file in parallel, if that is still the case there are categories of issues it cannot detect. In my case ruff cannot fully replace my existing tools, but makes for a great companion.

1: https://github.com/charliermarsh/ruff/issues/970