awused

>It's the problem with classical stage-oriented compilers. A compiler designed for diagnostics from the beginning

I'm not sure any compiler, even one "designed for diagnostics," can gather the "necessary metadata" from inside my brain based on my original intentions. If the compiler could unambiguously interpret what I meant to write, it wouldn't have needed to fail with a compilation error in the first place. Whenever I see something like "perhaps" or "maybe" in a compilation error, the information just didn't exist anywhere the compiler could possibly get to, and it's just a suggestion based on common mistakes.

Honestly I found myself coding very much the same way in Rust as I did in Python and Go, which were my go-to hobby languages before. But instead of "this lock guards these fields" comments, the type system handles it. Ownership as a concept is something you need to follow in any language, otherwise you get problems like iterator invalidation, so it really shouldn't require an up-front architectural planning phase. Even for cyclic graphs, the biggest choice is whether you allow yourself to use a bit of unsafe for ergonomics or not.

Having a robust type system actually makes refactors a lot easier, so I have less up-front planning with Rust. My personal projects tend to creep up in scope over time, especially since I'm almost always doing something new in a domain I've not worked in. Whenever I've decided to change or redo a core design decision in Python or Go, it has always been a massive pain and usually "ends" with me finding edge cases in runtime crashes days or weeks later. When I've changed my mind in Rust it has, generally, ended once I get it building, and a few times I've had simple crashes from not dropping RefCell Refs.

Lifetimes and borrowing are very much a correctness thing and aren't just for tracking when memory is freed. While you won't have use-after-free issues in a GCed language, you will still have all the other problems of concurrent modification (data races) that they prevent. This is true even in single-threaded code, with problems like iterator invalidation.

Honestly I assumed you had read the article and were just confused about how tokio was pretending to have preemption. Now you reveal you hadn't read the article so now I'm confused about you in general, it seems like a waste of time. But I'm glad you're at least on the same page now, about how checking if something is ready and yielding to the runtime are separate things.

Actually, no, I misread it trying to make sense of what you were posting. This is just more mundane non-blocking sockets. Yeah, if the socket never would block, it won't block.

Are you sure about that?

>Rust embraces abstractions because Rust abstractions are zero-cost. So you can liberally create them and use them without paying a runtime cost.

>you never need to do a cost-benefit analysis in your head, abstractions are just always a good idea in Rust

Again though, and ignoring that, "zero-cost abstraction" can be very narrow and context specific, so you really don't need to go out of your way to find "costly" abstractions in Rust. As an example, if you have any uses of Rc that don't use weak references, then Rc is not zero-cost for those uses. This is rarely something to bother about, but rarely is not never, and it's going to be more common the more abstractions you roll yourself.

You're confusing IO not happening because it's not needed with IO never happening. Just because a method can perform IO doesn't mean it actually does every time you call it. If I call async_read(N) for the next N bytes, that isn't necessarily going to touch the IO driver. If your task can make progress without polling, it doesn't need to poll.

>I'm not familiar with tokio so I did not know that it maintained buffers in userspace

Most async runtimes are going to do buffering on some level, for efficiency if nothing else. It's not strictly required but you've had an unusual experience if you've never seen buffering.

>filled them before the user called read()

Where did you get this idea? Since you seem to be quick to accuse others of it, this does seem like trolling. At the very least it's completely out of nowhere.

>it could still have read() yield and return the contents of the buffer.

If I call a read_one_byte, read_line, or read(N) method and it returns past the end of the requested content that would be a problem.

>I assumed that users would issue reads of like megabytes at a time and usually receive less.

Reading from a channel is the other easy example, if files were hard to follow. The channel read might implemented as a quick atomic check to see if something is available and consume it, only yielding to the runtime if it needs to wait. If a producer on the other end is producing things faster than the consumer can consume them, the consuming task will never yield. You can implement a channel read method that always yields, but again, that'd be slow.

>The proposal is obviously not to yield 1 million times before returning a 1 meg buffer, is this trolling

No, giving a illustrative example is not trolling, even if I kept the numbers simple to make it easy to follow. But your flailing about with the idea of requiring gigabyte sized buffers probably is.

>This sort of design solves the problem for any case of "My task that is performing I/O through my runtime is starving my other tasks."

Yeah, there's your misunderstanding, you've got it backwards. The problem being described occurs when I/O isn't happening because it isn't needed, there isn't a problem when I/O does need to happen.

Think of buffered reading of a file, maybe a small one that fully fits into the buffer, and reading it one byte at a time. Reading the first byte will block and go through epoll/io_uring/kqueue to fill the buffer and other tasks can run, but subsequent calls won't and they can return immediately without ever needing to touch the poller. Or maybe it's waiting on a channel in a loop, but the producer of that channel pushed more content onto it before the consumer was done so no blocking is needed.

You can solve this by never writing tasks that can take "a lot" of time, or "continue", whatever that means, but that's pretty inefficient in its own right. If my theoretical file reading task is explicitly yielding to the runtime on every byte by calling yield(), it is going to be very slow. You're not going to go through io_uring for every single byte of a file individually when running "while next_byte = async_read_next_byte(file) {}" code in any language if you have heap memory available to buffer it.

>Trying to solve the problem by frequently invoking signal handlers will also show in your latency distribution!

So just like any other kind of scheduling? "Frequently" is also very subjective, and there are tradeoffs between throughput, latency, and especially tail latency. You can improve throughput and minimum latency by never preempting tasks, but it's bad for average, median, and tail latency when longer tasks starve others, otherwise SCHED_FIFO would be the default for Linux.

>I read the blog about this situation at https://tokio.rs/blog/2020-04-preemption which is equally baffling

You've misunderstood the problem somehow. There is definitely nothing about tokio (which uses epoll on Linux and can use io_uring) not responding in there. io_uring and epoll have nothing to do with it and can't avoid the problem: the problem is with code that can make progress and doesn't need to poll for anything. The problem isn't unique to Rust either, and it's going to exist in any cooperative multitasking system: if you rely on tasks to yield by themselves, some won't.

The way you used it in your parent comment didn't make it clear that you were using it properly, hence my clarification. I'm honestly still not sure you've got it right, because Rust abstractions, in general, are not zero-cost. Rust has some zero-cost abstractions in the standard library and Rust has made choices, like monomorphization for generics, that make writing zero-cost abstractions easier and more common in the ecosystem. But there's nothing in the language or compiler that forces all abstractions written in Rust to be free of extra runtime costs.

"Zero-cost abstractions" can be a confusing term and it is often misunderstood, but it has a precise meaning. Zero-cost abstractions doesn't mean that using them has no runtime cost, just that the abstraction itself causes no additional runtime cost.

These can also be quite narrow: Rc is a zero-cost abstraction for refcounting with both strong and weak references allocated with the object on the heap. You cannot implement something the same more efficiently, but you can implement something different but similar that is both faster and lighter than Rc. You can make a CheapRc that only has strong counts, and that will be both lighter and faster by a tiny amount, or a SeparateRc that stores the counts separately on the heap, which offers cheaper conversions to/from Rc.

>but Tokio forces all of your async functions to be multi thread safe

While there are other runtimes that are always single-threaded, you can do it with tokio too. You can use a single threaded tokio runtimes and !Send tasks with LocalSet and spawn_local. There are a few rough edges, and the runtime internally uses atomics where a from-the-ground-up single threaded runtime wouldn't need them, but it works perfectly fine and I use single threaded tokio event loops in my programs because the tokio ecosystem is broader.

You don't even need other runtimes for this. Tokio includes a single-threaded runtime and tools for dealing with tasks that aren't thread safe, like LocalSet and spawn_local, that doesn't require the future to be Send.

There's nothing buggy about a future that never yields because it can always make progress, but people prefer that a runtime doesn't let all other execution get starved by one operation. That makes it a problem that runtimes and schedulers work to solve, but not a bug that needs to be prevented at a language level. A runtime that doesn't solve it isn't buggy, but probably isn't friendly to use, like how Go used to have problems with tight loops and they put in changes to make them cause less starvation.

>You are right. I concede nil is not useful.

You say this sarcastically, but it is actually true. A nil pointer is not useful. Once you have determined that a pointer is nil, you have confirmed that the function returning it at all was a waste of both space and time. Though it's actually only half true: nil pointers are worse than useless and they provide negative utility, because they allow invalid code to compile. A better design - which is also more efficient, even considering the overhead of tagged unions - is to not return the pointer/value at all if it would be useless. Other languages allow for this, even C does allow for it with manually tagged unions. Go is rather unique, especially among modern languages, in how it doesn't provide any mechanism for it, so people use what is available to emulate that.

For the rest of it, well, you've contorted yourself into some really interesting positions.

You are now claiming that they are useful when you must discard them as useless when the error is non-nil. At least with you being caught in such a plain and clear lie I can be done with this.

>I love nothing more than being wrong.

That's odd, because you've contorted yourself into defending absurd positions based on personal definitions no one else shares, including the Go authors, just to avoid admitting you were wrong. The same Go authors responsible for both the proverbs, which you quote as gospel, and the style guide, which you claim doesn't apply when it's inconvenient because it directly contradicts your claims in plain english.

>No, it is very much written from a consumer perspective. We're talking about from the producer perspective.

This is just a lie. It's written about Go and doesn't split its perspectives, the producer should only ever return meaningful non-error values with an error if explicitly documented, and the consumer should only ever expect those when documented.

>you point to a good example of where you can find un-meaningful return values when there is an error.

Okay, skimming the release notes for 1.21. The slog package, brand new in 1.21, contains a MarshalText method (and others, but this is the first I noticed). It returns ([]bytes, err) and it is not documented to be idiomatic as per your personal definition. Therefore, by your interpretation of the style guide and your own personal standards, it is not idiomatic. What's more, a nil/empty slice is potentially valid output from marshaling something, so the slice is entirely useless and must be ignored if err is non-nil. It is not valid to try to "derive meaning" (your terminology) from the slice being nil or not, so it is well and truly useless if err isn't nil.

Moreover, I'll assert that if Either existed in the stdlib, it would return Either[bytes[], err] instead of ([]bytes, err), and it is a limitation of Go that they're using (T, error) which does not offer the appropriate semantics.

>Clearly the vast majority of the standard library, especially in the newer stuff, does return meaningful values upon error

This, too, is simply a lie.

>Yes, it says you cannot trust functions, unless documented, to be idiomatic.

That is not what it says, in fact, it almost says the opposite. It is idiomatic Go to never return useful values if error is non-nil unless explicitly documented. It never affirms your particular, personal, definition of what "Idiomatic Go" is and directly contradicts it.

>If you believe that the stdlib is idiomatic

I believe most of the stdlib is idiomatic Go, especially the newer stuff. Your excuse earlier was that the older stuff in the stdlib isn't idiomatic but the newer stuff is. But the newer stuff doesn't match your personal standards for being idiomatic either. I would challenge you to point to a large body of code that actually matches your definition of idiomatic, I do not think it exists.

>Based on what? ... In fact, Go Proverbs even says so.

And the Go style guide says otherwise: if err is non-nil, you shouldn't even check the other return values. The Go proverbs are just words, it doesn't make them true or even good ideas. When Go proverbs don't agree with how Go code, including idiomatic Go code is written, reality wins over a bad idea. The Go style guide happens to better match real code written by real people, even the people directly responsible for the proverbs.

>I'd love to see some real-world code you think is idiomatic

To be fair, I have been talking about the Go programming language and idiomatic code written in that language. You seem to have a different idea of "Idiomatic Go" from everyone else's. Most of the stdlib, including the newest additions, is idiomatic as judged by other people but evidently not by you.

Since you seem unwilling or unable to follow an argument you started this seems more like an attempt to bait a response that you can report to the mods instead of an honest conversation. If you want to know why the alleged tangent was relevant, well, the posts are still there.

>Go Proverb #5: Make the zero value useful.

Yeah, it's a nice quip, but that's all it is. It sounds nice on first read to someone who doesn't program much. But it is inaccurate and not followed by Go, and is explicitly against the Google style guide.

The sophistry trying to paint a nil pointer as "useful" is just trying to defend a position you've dug yourself into in the process of this argument, so it doesn't really need to be addressed again.

>An empty set upon error is more meaningful than nil, usually.

This in particular is just a mistake in Go. Nil maps, unlike nil slices, cause panics, so people try to avoid ever returning them.

>But it does not match the current semantics of idiomatic Go code using the (T, error) pattern.

But it does match how (T, error) is actually used the majority of the time. The impedance mismatch is that code that currently has the semantics of Either, which is the vast majority of idiomatic Go, needs to use (T, error).

So you've defined idiomatic Go code in your own way, that no one else matches, such that no idiomatic Go code actually exists. If no idiomatic Go code exists, then it's definitionally true that all idiomatic Go code is without flaws, but I don't agree with your personalized definition of idiomatic Go in the first place.

>That may be true, but of no relevance.

So you agree with this, and it rebuts the entire contents of the first post I responded to. I wonder why you were posting irrelevant, off-topic content yourself. But I'm glad you finally agree that Go isn't flawless.

>whatever haphazardly written Go code you happened to find

Well, if the Go stdlib is haphazardly written and unidiomatic then I think the burden is on you to demonstrate that idiomatic Go, as per your definition, exists.

>more baseless ad-hominem and ignoring the points

I'll boil it down to one sentence: Go forces programmers to use (T, error) when it is not appropriate, and Go would be better if it did not.

I recommend you read your own posts sometimes. Your claim of "in Go values must always be useful" was rebutted multiple times. That you're trying to move the goalposts to something neither of us was arguing speaks loudly. You also dance dishonestly around the actual point I made in that last post. Again, I'd recommend following your advice about reading a thread before responding to it.

>As you explain yourself, they cannot be equivalent representations.

That's the point. They are not equivalent, they represent different things, and Either is a better fit for the actual code even in Go most of the time. Go shouldn't force people to use (T, error) when Either[T, error] is the correct choice.

But that's twice now you've resorted to ad-hominem attacks instead of responding to the content, so I'll take your implicit admission that you have no rebuttals.

>Most notably, you can derive meaning from its nil-ness.

This is sophistry. If I try to "use" a nil pointer I get a crash. I have to carefully check that it's non-null even if error is null. You can "derive" the same "meaning" from Result[T, error] being an error instead of a T. You can "derive" the same "meaning" from Option[T] being empty. There is no special meaning that a null file gives me that I can't take from a Result containing an error.

There isn't some big philosophical difference that Go is taking a principled stance on, just a practical one: with those you get type safety, and if you do it wrong you get a compilation error. In the Go way if you do it wrong you get a runtime panic.

>that would not be idiomatic. That would just be a terrible API design and unkind to the users of your API.

That is the vast majority of the stdlib and the vast majority of all popular Go libraries. If idiomatic Go code is code where (T, error) means T is always a useful value even if error is non-nil, then there is vanishingly little idiomatic Go code in existence.

>as its intent is to create a dependence between two variables.

This is nonsense. To use your personal specific terminology, Either encodes a dependence between variables that already exists, it doesn't create it. That dependence exists in Go too, Go isn't a language where the fundamentals of programming change, it's the same in C where people write methods that take both a result and an error pointer.

Either is an option to use when there is a dependence. If there isn't a dependence, and both are always present, you can and should return (T, error) and not Either[T, error]. No one is trying to force Go to always use Either when (T,error) would be appropriate, just like you are not forced to in other languages. You just have choices in those languages you do not have in Go, and overwhelmingly people choose more appropriate types than (T, error) when given the choice.

responding to your edit: >Either is not a suitable representation of (T, error). They have very different semantics. There are data structures which can serve as a suitable representation of (T, error), but Either is not it.

It's odd that you acknowledge this, but then claim that I somehow claimed the opposite. Perhaps you should follow your own advice about reading. Either represents a subset of the four cases that (T, error) covers, and even in Go the two cases that Either covers are the only ones in the vast majority of usage. In Go, most, but not all (and no one is claiming all), uses of (T, error) would be better expressed as Either[T, error].

In fact, the different semantics is the entire point. The point is not to keep the semantics the same but change up the syntax. In Go (T, error) is used commonly, in idiomatic Go unless the stdlib is unidiomatic, to emulate the semantics of Either[T, error]. If (T, error) doesn't have the right semantics for your program - and rarely are all four cases considered - then a more appropriate type with matching semantics should be used instead.

I think most of the stuff in this repo is too much and trying to beat a square peg into a round hole, but the little things like Option and Either patch a hole in Go. I don't think I'd use them without them being in the stdlib, though, which is where such fundamental types belong. Those aren't even really functional, unless you count null pointers as necessarily imperative.

>... read some blog post about how functional programming and type theory will save the world, instead of actually being productive

I see the opposite side in Go a lot, where without testing or trying anything they dismiss everything they aren't already using right now as useless ivory tower academia, which is its own set of popular blog posts. Seems both sides have a lot of time to argue on the internet though, oddly the people who actually write code tend to be the productive ones regardless of philosophy.

Yeah, there are counterexamples, but the only way to know is to read the comments or source code of the function you're calling. (T, err) doesn't convey any useful information and, in the overwhelming majority of cases, err != nil means T is a meaningless default value that should be ignored or a null pointer.

By and large I think the stuff in this repo is too much and doesn't fit Go. I don't particularly want Go to pretend to be functional, but Either and Option at least would be nice to have in the stdlib and help prevent this exact issue where there are rare exceptions to normal practices. I don't see them getting widespread use without being part of the stdlib though. If Either/Option were common in Go but io.Reader was one of the few APIs returning (T, error), that would convey a lot more information.

This is nonsense. This isn't about idiomatic Go or not, there is only one way to do things in Go, so a function doing things in that one way doesn't communicate anything to the caller. If you try to open a file, and the file doesn't exist, you have to return a useless nil pointer alongside the error and there is no way to magic up a "useful" T. Usually err != nil means T == nil, so trying to blindly use T assuming it's "useful" will panic and crash your program.

The idiomatic Go way to work around this is to write comments saying "sometimes T is non-nil even if err is non-nil, you need to handle this" and hoping your callers read your comments.

Funnily enough, your philosophy is far more true in a language with proper sum types. In Haskell/Ocaml/Rust, returning a tuple of (T, error) does mean that both T and error should both be "useful", because if they weren't the function would have chosen to return one or the other but not both. You're reading meaning into Go code where meaning can't be present, because there's no choice to be made, and ignoring languages where you actually can have the semantics you want Go to have.

It's not just cross-platform programs that suck. Applications moving to gtk4 from gtk3 add an extra 100-200ms of startup latency every time because of OpenGL initialization and shader compilation (there is currently no cache), varying a lot depending on your CPU. Applications that used to open instantly now have have a noticeable pause, even if it's still short compared to the worst Windows applications. It's been reported multiple times for well over a year now and no improvements have been made.