News for hackers. Stuff that matters.
Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by dchest on Thursday January 25, 2018 @06:47PM (#16235851)
Nothing wrong with it. In fact, I wish spawning processes was more common. It's beneficial for security.
Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by sreque on Thursday January 25, 2018 @08:38PM (#16236391)
Spawning a process, even if its just vfork() followed by exec(), is very very expensive compared to spawning a thread, and even then most apps created threads on startup and then just cache them for the lifetime of the application. Spawning a process 20 times a second honestly isn't terrible, but even hundreds per second can be noticeably slow IIRC.
Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by azinman2 on Thursday January 25, 2018 @09:10PM (#16236522)
I thought Linux treated forks and processes the same, where as the former is just a special case?
Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by saghm on Thursday January 25, 2018 @09:23PM (#16236578)
`fork` is how you create a new process on Linux. Threads, which GP was talking about, are different, especially Go threads, which are not the same as OS threads.
Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by ereyes01 on Thursday January 25, 2018 @10:59PM (#16236944)
I believe GP was referring to the `clone` system call, which both fork and pthread_create use under the covers. The difference is what happens to the memory-resident data- the forked/child program shares the address space of the parent until either writes to a page, in which case the page is copied such that each process gets its own unique copy. This is known as copy-on-write (COW). Pthreads, on the other hand, outright share the process's address space and must implement their own synchronization via locks or whatever.

Obviously, the performance of fork vs. pthread_create could differ dramatically depending on what the program does.

Goroutines are a layer of abstraction above this. They might run on different threads concurrently- the Go runtime controls what happens here and may differ on various architectures / OSes. If you break into a running Go program on Linux with gdb, there's definitely a bunch of pthreads running, maybe for goroutines, and probably for garbage collection, and other stuff. (If you want to actually debug go code, you should of course use something like delve).

http://man7.org/linux/man-pages/man2/clone.2.html

Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by kjksf on Thursday January 25, 2018 @07:29PM (#16236095)
Process isolation is good for security.

Parsing text data in ad-hoc, non-standardized, not documented, not defined format is really bad for security.

Just spawning a process creates as many security problems as it solves.

If it was done right, it would look like Chrome architecture, where untrusted, isolated processes can do dangerous work but communicate with trusted process via well defined IPC protocol.

Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by pjmlp on Friday January 26, 2018 @01:39AM (#16237430)
Yes, but there is no need to spawn them all the time.

A parser service daemon, or a pool of them can be used instead, getting requests from the main application process.

Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by mschuster91 on Thursday January 25, 2018 @06:50PM (#16235868)
> It's beneficial for security

... and for RAM usage. Java applications all have a tendency to bloat the longer you keep them running.

Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by paulfurtado on Thursday January 25, 2018 @10:20PM (#16236792)
It's not bloat or memory leaks per se, the JVM just does not return memory to the OS after it is freed. To limit its memory usage, tune the heap size. To fully allocate the heap on startup for consistent usage, use -XX:+AlwaysPreTouch
Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by chinhodado on Thursday January 25, 2018 @11:41PM (#16237068)
Last time I checked, I still couldn't control the max heap free ratio, because apparently that option/flag just doesn't work with Java 8's default GC.
Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by axaxs on Thursday January 25, 2018 @11:00PM (#16236946)
Java has always been 'use memory to increase speed...sometimes'. You can tune it some, sure, but that's what it's known to do.
Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by nurettin on Friday January 26, 2018 @12:27AM (#16237216)
Back when I tried running a jruby application on 800mb ram, it bloated, then started throwing "OutOfMemoryError"s and "Insufficient Class Space" or something similar. Apparently jruby was generating too many new types at runtime to accommodate rails framework. Garbage collector was pretty garbage at it's job back in 2011.
Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by readittwice on Friday January 26, 2018 @01:36AM (#16237423)
AFAIR JRuby creates a Java class for each Ruby method. Classes were part of the Permanent Generation, so sometimes this generation was too small escpecially when using Rails. It was enough to just increase the size of this generation with -XX:MaxPermSize (-Xmx doesn't increase the permanent generation's size).

The Permanent Generation was named that way since objects in there were never collected. For most applications this isn't really a problem. Running JRuby+Rails just allocated a lot of classes in this generation, so the default size was too small. But still, the permanent generation was quite small compared to the heap size.

I wouldn't really call the GC bad because of this, IMHO they were already quite good back then. And in Java 8 the permanent generation was replaced with the Metaspace, objects in there can be free'd and the Metaspace can be expanded at runtime so it's less likely to get these OOM errors for the permanent generation.

Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by nurettin on Friday January 26, 2018 @03:45AM (#16237802)
That was my experience as well. Increase maxpermsize and reboot every night.
Re:How a fix in Go 1.9 sped up our Gitaly service by 30x by arghwhat on Friday January 26, 2018 @12:48PM (#16241178)
That's mostly just a Java thing, though. Java has a weird relationship with memory.