> Michael Knysze significantly increased throughput of memory allocation for large blocks by redesigning the memory allocator's "mcentral" data structure to reduce lock contention. The new allocation code is more than twice as fast for blocks of 12KB or larger.

Excellent! Nice to see these sorts of infrastructure enhancements.

The fun thing about the Go compatibility promise is that it doesn't work as well as the go developers probably imagine; just last week one of my older Go project was discovered to be broken; wouldn't compile anymore. golang.org/x/crypto was responsible, they used some code that wouldn't work in more modern go compilers anymore. Granted, I had not maintained the project in 3 years but I also had everything vendored just in case go decided to break anything upstream, and the broken parts didn't even touch on anything that the compatibility promise excludes.

I think the Go devs should improve in that section a bit.

I had the same issue. The issue was a security one https://nvd.nist.gov/vuln/detail/CVE-2019-11840 which required a breaking change to fix so their options where leave everyone vulnerable but could compile or break anyone that uses this code and fix it. I think they made the right choice.

Also golang.org/x/crypto is not a core library so doesn't actually fall under the compatibility guarantee but they seem to try their best not to break it.

Neither of those things applied though; I had x/crypto vendored, so it didn't change it's code either and the CVE is unrelated to the actual code that broke (I used blake2b) and again; it was vendored, so my code wouldn't break but be vulnerable.

How did vendored code that didn't change break your build?

The easiest way to accomplish that is definitely with //go:linkname

You can use a comment of the format '//go:linkname [localname] [method]' to link some function as another one. Notably, this includes unexported private methods in the go runtime and stdlib.

I've seen code in the wild that uses this to grab go's map hash algorithm, get monotonic time, and other things.

The go authors have used this hack themselves in the stdlib often enough because the runtime doesn't expose some knob, and rather than thinking that perhaps other developers may want that knob too, they instead use such hacks, but they also don't support said hacks (because how could you, obviously keeping all private functions stable forever is silly and impossible)

I updated the compiler.

That was golang.org/x/crypto's fault for using internals not covered by the compatibility promise [0], sorry about that.

However, we try to hold x/crypto itself to the same compatibility standard as the standard library, so I think simply updating x/crypto at the same time should have made it a smooth update.

[0]: https://news.ycombinator.com/item?id=23164795

I just compiled a fairly complex (CGo etc...) project a few weeks ago which I hadn't compiled since 2014, and it just worked.

So I guess YMMV...

Of course, not every project will fail, but in my case it failed and I think that is the issue.

Do you remember the specifics? I am not aware of syntax constructs that were removed in Go, I guess ever.

EDIT: I'm sad folks are downvoting for asking a question...

Older versions of the /x/crypto library no longer compile on newer go versions, there wasn't any indication this would happen and I vendored the library to avoid the API breakage in that library. From my standpoint, that completely ruins using that library as after 3 years either I vendored it and it breaks because of something the go compiler does or I didn't vendor it but the API changed. Either way, my program will break eventually and there is no way to avoid it.

I'm sorry you had this experience. I'm genuinely curious as to what the breakage was so hopefully we can prevent it for some other folks, hence why I asked if you remembered the specifics.

Basically, x/crypto/blake2b received a patch to switch how it detects the presence of AVX and SSE at runtime, the symbols required for that to work with earlier versions where removed. While I understand this is technically covered in the compatibility promise, it's very frustrating when it happens with a library developed by google, especially when, at the time, the general recommendation was to vendor them so as to not suffer from breakage when they update.

I believe this commit has background: https://github.com/golang/crypto/commit/d6449816ce06963d9d13...

To be fair though, that package was depending on a private API to begin with.

Well, that's for the better still.

If you use vulnerable crypto (willing to go with old unpatched vendored code) then you don't really need crypto at all..

@zaarn explained in another comment that the vulnerability did not impact him, he did not exercise that code path.

I understand that, but it's not really relevant.

The code path that failed to build was still related to the included crypto (even if it was due to compiler change).

If that was their only way to fix the crypto issue, then it's totally fine, because that way they fixed it for people who did need and exercize the crypto code path -- and those take priority over people who "bundled some crypto code but didn't exercize that code path and their build still broke".

I exercized a code path that broke, it's not like unrelated code broke, it was code I was actually using in a commit in the upstream library that changed not much for the end user and then was broken by the compiler fairly soon after.

The issue is related to AVX detection in the CPU, which has the reasonable failure path of "don't build it with AVX".

Isn't /x/ experimental by definition? They introduced a breaking changed to an experimental library. This does not fall under the scope of the compatibility promise. The compatibility promise works great.

One of those point under "Expectations" can be explain in this page

https://golang.org/doc/go1compat

Nope, none of these points apply to the specific change, atleast as far as I am able to tell. The code that broke did not use unsafe when I last updated it and then it simply didn't work anymore in the current go version.

I believe one of those probably applies.

> Use of package unsafe. Packages that import unsafe may depend on internal properties of the Go implementation. We reserve the right to make changes to the implementation that may break such programs.

I suspect the crypto code you vendored used unsafe somewhere within it (whether to use `//go:linkname` or for other reasons), which would except it from the go1 compatibility promise.

I agreed with jeffsmith82's comment.

That comment isn't applicable either; I had x/crypto vendored so any fix upstream wouldn't break my app.

I'm afraid blake2 used assembly and runtime.support_avx2, which doesn't seem to be covered by the compatibility promise: assembly is part of the toolchain, and the symbol it used was an internal one.

See:

https://go-review.googlesource.com/c/crypto/+/110355/

https://go-review.googlesource.com/c/crypto/+/110016/

So, my code broke because a library that was provided by the go developers used some unstable feature I had no idea about and had no real way of knowing short of checking a dependency three levels below my project? And to top that, that library was certainly subject to API breakage, the entire reason I vendored it in the first place.

It doesn't seem reasonable to me. When I developed my project, vendoring /x/ projects was recommended exactly due to the breakage, so now it breaks my projects because of recommendations of the golang team.

I'm pretty sure x packages are not part of go core.

x packages should still compile on newer go compilers in my opinion. Their compatibility isn't guaranteed if you don't vendor it, which was a recommendation when I started the project and as I pointed out in the comment you're responding to, it was vendored, so it didn't break itself.

>On Windows, Go 1.15 now generates executables that use address-space layout randomization (ASLR) by default. ASLR uses position-independent code to randomize the addresses of various data areas on startup, making it harder for attackers to predict target addresses and create memory-corruption exploits.

Impressive this is only landing for Windows now after PPP made a spectacle of the lack of ASLR back in 2013. http://pwning.net/2013/04/21/exploiting-a-go-binary/

Ehh I agree with you but I also haven't seen an "actual" exploit for a go memory corruption vulnerability outside of POCs.

Like I know it's possible with effort and maybe it was just lack of interest or adoption (like lets be fair its not a browser writing or pdf rendering engine type of language...) so while that situation was not great it never struck me as a burning problem with practical impact in the ecosystem either.

Interested in any pointers if possible.

What are some examples of medium-large Go projects where the code is well-written? By that, I mean code that would make a good example for someone trying to learn the "right" way to do things.

I was briefly learning Go a while back. I'm curious how well gorotuines and channels have held up as building blocks, and how extensively they are used in real projects.

These days when I see a fast, reliable CLI tool I think "this has got to be written in Go" and a lot of the time they are. Some examples: the new GitHub CLI, Docker, Hashicorp tools, ngrok (though v2 is not open source), Caddy, the list is quite long. I can't vouch for the quality of that code, but I suspect some of those projects are good starting points. And I've found the Go standard library source to be really helpful (and generally fairly easy to understand).

Speaking from some experience, Go is probably the best option these days at "do one thing well" command line tools - the mix of library support, native binaries, and the simple, stable language make it a good fit for a weekend project that opens files, munges strings and pushes them over standard protocols.

When Go has to support a large application that defines a platform in itself, the downsides of the language rise to the forefront, but that level of expression usually isn't entailed in a Unix-style CLI tool.

(Rust also has more potential for performance scaling in this space, but with the "if you have the time for that" kind of caveat.)

One thing I hate is that the go flags package sorts the options by default. Ordering options in a thoughtful way is definitely a plus in an api. I know there are (not very intuitive) ways to override this, but I just can't understand why they would make such a bizarre choice even given their penchant for being opinionated.

edit: Now that I read this over, this complaint seems a bit misplaced in this thread, but I guess I'll leave it due to its sort of relation to cli programs in go.

> Rust also has more potential for performance scaling in this space, but with the "if you have the time for that" kind of caveat.

When you have a CLI for some network-based service, the network latency is going to be magnitudes larger than the performance differences between Go and Rust anyway. (Or at least in 99% of cases.)

some suggestions are that in the future, the I/O latency is going to get very low, network speeds very fast, etc.

[0] https://www.youtube.com/watch?v=SM3PhGGllL0&feature=youtu.be [1] https://www.ece.rice.edu/~willmann/teng_nics_overview.html

Unless that 10-gig NIC messes with the speed of light, I don't see it reducing the RTT between me and some server in the US by any significant amount.

not sure about docker/kubernetes. I've read somewhere that after investigation of source code there were conclusions that source code is not using good practices, a lot of duplication of code and so on thus making conclusion that project has some risks in technical debt or something like that. Would not be good to learn from project which is not well written :)

It depends on what you want - big side of things, check out docker/Moby though it's a little hard to follow due to their approach on using Docker to build, maybe Terraform?

On the other, more modular side, try something like the Gorilla toolkit e.g. https://github.com/gorilla/mux/

I'll be honest, I rarely use channels other than for timeouts/deadlines.

Prometheus is a very readable and understandable code base which does use those building blocks.

> By that, I mean code that would make a good example for someone trying to learn the "right" way to do things.

You’re in luck! Go is unlike a lot of other languages, and that there’s usually only one way to do something, and it’s usually the right way, or at least an acceptable way.

I would suggest you just pick a part of the standard library, such as net/http and just read away since you probably have an understanding of the fundamentals.

You can probably checkout his code

https://github.com/tj/

Its remarkable how many go projects are very simple and small.

I don't think its worth going through other go projects to learn best practices because at that level, there's not enough complexity to learn anything that isn't painfully obvious.

There aren't really a lot of design patterns or clever tricks to learn.

Kubernetes (although there are some shenanigans with codegen), Cockroach DB and CoreDNS come to mind. There’re lots really - the language makes it hard not to do it “right” way.

Go-routines are quite ubiquitous especially in server-style projects.

Go-ethereum

A good presentation did the rounds a while ago[0] that does a deeper dive but the main things for us are the performance, smaller binaries and the improvements in tooling.

The one thing I wish they added though? `go new` and have it initialise a new project structure that stopped tiny package syndrome and bizarre Makefiles in repos! [1]

[0] https://docs.google.com/presentation/d/1veyF0y6Ynr6AFzd9gXi4...

[1] https://rakyll.org/style-packages/

Re "go new": I think part of the problem with that is that people have quite different ideas of what a good project structure is. It also varies a lot between a simple CLI tool (for which main.go and maybe go.mod is enough) and a large multi-package server project.

Can you explain what you mean by "bizarre Makefiles"? I've never used Makefiles for my (admittedly fairly small) Go projects, because "go build" just works.

I've seen a number of projects where the root directory is a load of other folders and then there's a Makefile and it's a kafkaesque nightmare.

I agree on the toplevel `main.go` vs the `./cmd/myapp/main.go` style needing catering for that only comes out as the project evolves... maybe it should be a go vet addition instead :D

Go modules are removing the need for much of the Makefile magic. You still have use of a Makefile if you need vendoring or are dealing with non-public branches, but things can be much simpler.

I'm the author of that presentation, and I'm surprised to see the very similar content and structure, to say the least. I left a comment on the Reddit thread[0] to see if I'm missing something.

[0] https://www.reddit.com/r/golang/comments/ginp22/whats_coming...

Thanks -- I replied in full there. I reused your title, which I can see was confusing. However, this is all my own writing and research, primarily from the official (draft) release notes, new linker design doc, golang-dev, and GitHub issues. But your slides were a minor source, so I still should have attributed them. That would have helped you and not hurt anyone -- sorry!

> A new time/tzdata package was added to allow embedding a static copy of the time zone database in executables.

This is a BIG improvement. One of the big benefits of Go is static binaries. Having the TZ data includes will make deploying even easier.

The heck? This seems silly, and like an active nuisance. Timezone data changes multiple times each year. You can see this by the tz-announce email dates [0]

Why would I want to have to update recompile and deploy a go binary 2-10 times a year just because Morocco decided to remove DST from one of its regions?

I'd much rather the go authors not encourage me to do something dumb like encode tzdata in a nonstandard un-updateable location in my binary rather than having it in a standardized location that the OS will update for me as appropriate.

[0]: https://mm.icann.org/pipermail/tz-announce/

First point: making something possible doesn't necessarily mean encouraging it (unless you meant it in the "slippery slope" sense).

Second point:

The way many people use containers (especially when using minimal images meant to "just run a static binary") effectively boils down to the same thing: having a "un-updateable" blob that you need to rebuild in order to get an updated version thereof.

If you have an OS with a package manager, that runs without being "reimaged" for weeks, months, years, then having a process that keeps the "OS" up to date (TZ data, security updates of shared libraries, ...) is good idea.

If you have a static binary in a minimal image, running in an environment where any software update entails "re-imaging" your machine (VM, container, ...), then your process already has to consider that running the same binary for months and years without any updates is very bad idea.

Statically linking critical components like your crypto libraries already begs for a timely and automated rebuild process. The TZ data fits the same pattern as shared libraries.

There are points to be made in favour and against shared libraries. Whatever your position is on that topic, I think it can be transferred directly to TZ embedding.

Go will use the system timezone data, this is just a fallback the programmer must opt-in.

Tz data is not always available (see alpine base image, or scratch). When it is available, it may be out of date. It may never get updated in fact.

What do you mean by a benefit? Most compiled/native languages can make static binaries.

To paraphrase rob pike incorrectly from an only mildly misleading quote, "The key point here is our programmers are not researchers. They learned java, python, javascript. They're not capable of understanding a brilliant language, so we built go".

The comparison isn't other compiled languages that can easily produce static binaries (like ghc/haskell, rust, c++ (with about 2000 lines of meson or cmake or makefile, ...), etc).

The comparison is other languages Go programmers might use, like python and javascript. If someone cared about having a good type system, they wouldn't be using Go, so we have to compare Go's capabilities to other languages that are targeted at people who don't want any brilliance or type-theory, but just want a "simple language" where it's accepted as part of the "programmer condition" that you write bad code and get things done.

However, in reality, it's worth pointing out that there are very few other languages that have as seamless a static linking experience as Go. Sure, it's quite possible to statically link C(++), rust, etc, but it's more involved than it is for go. Furthermore, for those languages, there's a larger library ecosystem that dynamically depends on C libraries, so i.e. in rust you may have trouble linking any crate that depends on openssl (which quite a few do transitively). Go is the only language I know of that's created an entire culture of eschewing the C ABI as much as possible, that uses linux's syscall interface even though that means there are various things that are just plain broken (like nsswitch).

So yeah, while it's true that go isn't the only language that can be statically linked, it's one of the few that focuses on it to such an (arguably unhealthy) extreme.

> c++ (with about 2000 lines of meson or cmake or makefile, ...)

Making an static binary in C/C++ is a single command-line parameter in most compilers.

What’s the actual quote ?

"The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt. – Rob Pike 1"

"It must be familiar, roughly C-like. Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical. – Rob Pike 2"

Sources:

https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2014/Pa...

https://talks.golang.org/2012/splash.article

The irony here is that Go type system is weaker than any of those three languages on their latest versions.

So while they learned Java, Python, JavaScript, according to Pike they are still not good enough at their skills, so they must downgrade to Go.

The word "can" is very different than "does by default".

Even just doing a static binary for windows and osx will require some VERY complex build systems for most languages. Not with Go. It's simple.

I hope other languages put as much effort as Go has into static linked binaries.

I'm also pretty happy about what landed in the cryptography packages.

Session ticket keys, which are a major weak link in Forward Secrecy in TLS 1.0-1.2 [0], are now rotated automatically every day, and dropped after 7 days [1]. Session keys are also dropped once they get too old [2], instead of becoming an increasingly powerful Forward Secrecy liability.

The tls.Config.VerifyConnection callback is a much more powerful and usable way to customize peer verification [3].

X.509 root CAs are now extracted on macOS by linking directly against Security.framework with some assembly glue, instead of using cgo [4]. (Warning: awakens Cthulhu.)

crypto/x509 now ignores the 20-year-deprecated Common Name field by default [5] and has a consistent rule about invalid hostnames [6].

crypto/elliptic now provides MarshalCompressed and UnmarshalCompressed for compressed point encodings [7].

The clever TLS 1.3 downgrade canary in the server random is now enforced client side [8].

PrivateKey and PublicKey types implement Equal, giving a consistent method to build type-safe interfaces for them [9].

math/big has a new Int.FillBytes method for fixed size, zero allocation byte serialization [9], which we used all over the crypto packages.

RevocationList and CreateRevocationList were added to crypto/x509 to generate spec compliant CRLs [11].

crypto/ecdsa has two new functions, SignASN1 and VerifyASN1 that operate on encoded signatures instead of big.Ints that everyone was encoding/decoding anyway [12].

Plus some performance, cosmetic, correctness, and documentation improvements! Not a bad cycle overall, even if it made sense to focus on smaller scope tasks for a while.

[0]: https://blog.filippo.io/we-need-to-talk-about-session-ticket...

[1]: https://go.googlesource.com/go/+/43f2f5024b

[2]: https://go.googlesource.com/go/+/6ea19bb668

[3]: https://go.googlesource.com/go/+/62a3f2e27c

[4]: https://go.googlesource.com/go/+/6f52790a20

[5]: https://go.googlesource.com/go/+/d65e1b2e41

[6]: https://go.googlesource.com/go/+/9d1e120c42

[7]: https://go.googlesource.com/go/+/5c13cab36b

[8]: https://go.googlesource.com/go/+/a6c6e59655

[9]: https://go.googlesource.com/go/+/b5f2c0f502

[10]: https://go.googlesource.com/go/+/c9d5f60eaa

[11]: https://go.googlesource.com/go/+/5d47f870a6

[12]: https://go.googlesource.com/go/+/8c09e8af36

> Session ticket keys, which are a major weak link in Forward Secrecy in TLS 1.0-1.2, are now rotated automatically every day, and dropped after 7 days.

Wooooah, that's awesome! Up to now I think only Caddy has been doing this automatically & by default, I'm happy to see the std lib able to offer this too. Maybe we can simplify some of our code base. (Although, on second thought, Caddy can rotate the keys distributed across a cluster so as to improve TLS performance behind load balancers, for example; so we might still need to use our own home-brewed rotation. Which is fine!)

> The tls.Config.VerifyConnection callback is a much more powerful and usable way to customize peer verification [3].

Also looking forward to using this in Caddy! Right now we use VerifyPeerCertificate which is... fine, and definitely better than without it, but I am sure plenty of users will be happy to have the extra flexibility and added robustness of VerifyConnection.

Everything else looks stellar too -- keep up the good work!

Thanks! As a service/app developer, I'm not familiar with a lot of that terminology, but I'm very thankful that someone on the Go team is.

(Int).FillBytes is great, thanks! I had resorted to using Bits and reimplementing the Bytes logic on top of that...not pretty.

This is a well-written article for lwn subscribers, by HN's very own @benhoyt. If you read lwn, I definitely recommend subscribing; it consistently has high-quality technical content.

I was surprised to see ASLR was not enabled by default on Windows. Is enabling ASLR by default a common thing on Windows?

It is commonly enabled for C++ compilers, but ASLR defends against attacks that Go code is not generally vulnerable to.

I'm not sure if that is accurate, ASLR defends against quite a range of attacks that are the result of code execution, which AFAIK can still occur in Go, Rowhammer comes to mind here as a universal crack against languages.

There is no downside to activating ASLR either, so I'm not sure why it isn't.

Does addr2line still work with ASLR? Because that could be a reason not to.

addr2line works depending on the code and compiler/libc. Your libunwind may be able to output relativ code addresses, in which case addr2line will work if you use the stacktrace. GDB should handle this as well. If your c compiler doesn't support it you can just disable compiling PIEs which automatically disables ASLR but that should be reserved for debug builds.

Speaking of the future, does anyone here have an idea as to the the time frame when generics could land in a release?

An year or two.