News for hackers. Stuff that matters.
Re:Go 1.20 Released by valyala on Wednesday February 01, 2023 @08:25PM (#34620789)
Could you provide a few links to popular open source projects, which switched to generics and got better code because of this?
Re:Go 1.20 Released by cpuguy83 on Wednesday February 01, 2023 @09:10PM (#34621150)
Can you point to one that had the opposite effect?
Re:Go 1.20 Released by valyala on Thursday February 02, 2023 @12:11AM (#34622342)
No. I suppose, maintainers of popular open source projects couldn't justify introducing generics into their code. This is exactly my story - I was actively seeking places in VictoriaMetrics codebase, which could benefit from switching to generics. Unfortunately, not a single place has been found yet. There was a promising place for generics [1], but it has been appeared that it is better from readability and maintainability PoV to use plain old interfaces instead.

[1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast...

Re:Go 1.20 Released by cpuguy83 on Thursday February 02, 2023 @03:28PM (#34632497)
Tacking on generics on a working code base is difficult.

I effectively used it in this little experimental CLI library: https://github.com/cpuguy83/go-cli/blob/main/command.go

It's pretty simple, but the nice thing is it can use any flag library you want (stdlib flag package, pflag, whatever).

Re:Go 1.20 Released by valyala on Friday February 03, 2023 @02:03AM (#34638045)
Try substituting F generic type with the FlagSet interface - and the code will become more clear without the need to use generics.
Re:Go 1.20 Released by cpuguy83 on Friday February 03, 2023 @01:09PM (#34645305)
That totally changes it as it requires type assertions to use. With generics `Flags()` returns a concrete type and not an interface.
Re:Go 1.20 Released by valyala on Friday February 03, 2023 @03:40PM (#34647437)
The question is whether the Flags() should return the concrete type instead of FlagSet interface? Another question is whether the additional complexity related to generics worth the ability to get the concrete type from Flags()?

In fact, the generic-free implementation based on the FlagSet interface is more flexible, since it allows storing multiple different FlagSet implementations in the same Cmd.

Re:Go 1.20 Released by cpuguy83 on Saturday February 04, 2023 @11:51AM (#34655838)
I mean the generic type def there could be changed such that different Cmd's can have different flag implementations.

The main point is that generics come in handy when building libraries so that you aren't forcing callers of your library into specific types or loosing some type safety.

Re:Go 1.20 Released by jppittma on Wednesday February 01, 2023 @09:38PM (#34621339)
Suppose you're the maintainer of a popular open source project, and you have a million users running a working, stable version using code generation.

What priority would you assign a bug that migrates the existing, working code that other people are doing god knows what with downstream? What would the theoretical benefits "better code" be to your downstream, and how do they weigh against the cost of "I ran go mod tidy, and our build broke?"

Re:Go 1.20 Released by jen20 on Wednesday February 01, 2023 @11:35PM (#34622125)
Re:Go 1.20 Released by valyala on Thursday February 02, 2023 @12:17AM (#34622393)
I doubt this library is useful. Some of the provided functions such as Clone(), Clear(), Copy() and Equal() are substituted by simple and readable three-line loops. Others are plain harmful. For example, Keys() and Values() may lead to huge memory allocations when working over big maps, while DeleteFunc() worsens code readability and maintainability comparing to a simple 3-line loop. Additionally, it may slow down the resulting code.
Re:Go 1.20 Released by jen20 on Thursday February 02, 2023 @01:12AM (#34622784)
> I doubt this library is useful

It is.

> Additionally, it may slow down the resulting code.

FUD.

Re:Go 1.20 Released by valyala on Friday February 03, 2023 @02:23AM (#34638161)
Agreed, it's my fault that I didn't explain why maps.DeleteFunc() can be slower than a simple loop with "if" statement inside. The reasons for the slowdown are:

- Go compiler may fail to inline the callback passed to the maps.DeleteFunc(). This will result into an additional overhead for callback calls per each item in the map.

- Go compiler may move some variables inside the callback from stack to a heap. This will result in an additional memory allocations comparing to a simple loop, leading to an additional load on garbage collector.

Re:Go 1.20 Released by oauea on Thursday February 02, 2023 @01:13AM (#34622793)
Useful enough to be included in the standard library. https://github.com/golang/go/issues/57436#issuecomment-14125...
Re:Go 1.20 Released by valyala on Friday February 03, 2023 @02:17AM (#34638128)
That would be a mistake :(
Re:Go 1.20 Released by oauea on Friday February 03, 2023 @01:28PM (#34645663)
FUD
Re:Go 1.20 Released by 0x62c1b43e on Thursday February 02, 2023 @02:51PM (#34632071)
By that logic we should remove bytes.Clone, strings.Split, bytes.Equal, strings.TrimLeftFunc, etc. from the standard library.
Re:Go 1.20 Released by valyala on Friday February 03, 2023 @02:15AM (#34638116)
The bytes.Clone() uses non-trivial trick under the hood - `append([]byte{}, b...)`. Compare it to a trivial loop behind maps.Clone().

The strings.Split() implementation is non-trivial because of performance optimizations.

The bytes.Equal() is actually written in highly tuned and optimized assembly in order to achieve high performance for inputs of various lengths.

Now compare this to trivial implementations behind generic-based functions for maps. And do not forget that these implementations may hurt performance because of excess memory allocations in Keys() and Values() functions or because the compiler may fail inlining the callback passed to DeleteFunc().

Re:Go 1.20 Released by 0x62c1b43e on Friday February 03, 2023 @01:48PM (#34645997)
Those are good points about some of them being nontrivial.

Though part of why they’re optimized and in the stdlib in the first place is because they’re such common patterns. So without them people would end up writing trivial, unperformant, custom versions. So now that more routines can be moved into the stdlib, they can benefit from optimization later.

(I’m not sure how much the maps routines specifically can be optimized, but stdlib routines routines can generally be more aggressive with unsafe or asm or being coupled to the runtime and its quirks, like bytes.Clone, strings.Builder, etc.)

And there are still plenty of ubiquitous patterns that have been worth including in the stdlib even if they’re usually just simple loops that aren’t very optimizable. Like strings.Index is an easy loop to write, but it comes up so often. Or strings.Cut is basically just an if-statement. But it makes code clearer about its intentions; and optimizations to these down the road benefit everyone.

It’s also true that maps.Keys and maps.Values allocate slices, and that you could avoid this with a loop, but strings.Split, bytes.Split, regexp.FindAll, os.ReadDir return slices and are still worthwhile as opposed to specialized iterators for each one. As with any code, you’re conscious of memory allocations where it counts, and optimize as needed.

In fact, now that generics make it possible, the Go team has discussed using iterators (https://github.com/golang/go/discussions/54245), which would benefit strings.Split even further in addition to all the other slice-returning functions.

So generally you have three options for those slice-returning functions:

- Custom inline loop for some of them. More verbose, will probably be naive and not benefit from stdlib optimizations. - Return a slice and iterate over it with a for loop. Creates allocations that could probably be avoided. - Create a customized iterator for that type. Unfortunately, you can’t really use an ordinary for loop, and extra custom iterators for each type. - Use generic iterators to benefit from the optimized functions and also avoid allocation overhead.

So part of the motivation is that now with generics there’s a variety of further optimizations available even to old functions like strings.Split and regexp.FindAll, in addition to opening up common patterns and optimizations for maps/slices/etc. to be included in the stdlib.

Re:Go 1.20 Released by valyala on Friday February 03, 2023 @08:56PM (#34650555)
Agreed with most arguments.

A few remarks:

> Like strings.Index is an easy loop to write, but it comes up so often

Actually, strings.Index() is very non-trivial function partially written in assembly in order to achieve high performance [1]. This function is used in Go projects *much more frequently* than functions from the golang.org/x/exp/maps package.

> strings.Cut is basically just an if-statement

No, strings.Cut() has non-trivial code when comparing to a trivial loop for map copy or for map delete [2].

> It’s also true that maps.Keys and maps.Values allocate slices, and that you could avoid this with a loop, but strings.Split, bytes.Split, regexp.FindAll, os.ReadDir return slices and are still worthwhile as opposed to specialized iterators for each one.

The *key* difference between maps.{Key,Value} and the mentioned functions from the standard library is that it is trivial to write the `for k, v := range m` instead of maps.{Key,Value} and avoid memory allocations, while it isn't trivial to write the corresponding code without memory allocations, which substitutes strings.Split() or other mentioned functions from the standard library.

[1] https://github.com/golang/go/blob/86c4b0a6ec70b07ab49d3813a5...

[2] https://github.com/golang/go/blob/86c4b0a6ec70b07ab49d3813a5...

Re:Go 1.20 Released by bboreham on Thursday February 02, 2023 @04:41PM (#34633541)
Re:Go 1.20 Released by valyala on Friday February 03, 2023 @01:53AM (#34637973)
That's a good example of using generics-based sorting! I think it would be more beneficial to work on compiler optimizations, which could optimize the standard interface-based sort.Slice and sort.Sort to the same level. This shouldn't be too hard to do for the most widespread case when the compiler knows the actual type hidden behind the sort.Interface, so it could inline methods of this interface.

This optimization then could speed up other similar interface-based algorithms.