News for hackers. Stuff that matters.
Re:What it's like to use Haskell by implicit on Wednesday March 26, 2014 @02:12PM (#7475200)
We deal with Haskell resource leaks the same way you would in C++ or Java.

We have production monitors on every host that show basic metrics like memory, disk, and CPU utilization. Atop that, we added a tracker for the number of suspended Haskell threads. (that is, threads which are not blocked on I/O, but are also not running)

We found that the machines are usually able to handle requests as soon as they come in, so if the number of Haskell threads goes above 0 for any length of time, the machine is about an hour away from melting down.

We can restart the process without losing any connections, so this leaves us a very comfortable margin of error.

Once we know we have a problem, it's usually pretty simple to run the heap profiler on the process and look at recent commits. We continuously deploy, so there's only about a 10 minute delay before a particular commit is running in front of customers. This makes tracking regressions down really fast.

Even in cases where we can't figure out why a bit of code is leaking, we can almost always identify it and revert it until we understand what's going on.

Re:What it's like to use Haskell by pmahoney on Wednesday March 26, 2014 @07:33PM (#7477467)
> We can restart the process without losing any connections

Would you mind expanding on this a bit? I'm not too familiar with Haskell, but I am familiar with various was of blocking new connections while allowing existing connections to complete, either at the load-balancer level or built-in each individual process.

What Haskell stack are you using, and how are graceful restarts accomplished?

Thanks.

Re:What it's like to use Haskell by implicit on Wednesday March 26, 2014 @10:15PM (#7478152)
One of my coworkers wrote a really cool bit of software to do this. I want him to open source it.

Basically, you can share a single socket amongst many servers. The OS ensures that just one process accepts each connection.

You can therefore have a manager process that owns the socket and passes it on to application processes.

To update, start new processes, then politely tell the old ones to go away.

Re:What it's like to use Haskell by dllthomas on Wednesday March 26, 2014 @11:06PM (#7478330)
One really cool thing in Linux is that you can actually pass file descriptors between processes over unix domain sockets.
Re:What it's like to use Haskell by enigmo on Thursday March 27, 2014 @01:13AM (#7478739)
Windows has supported this for ~14 years too.
Re:What it's like to use Haskell by dllthomas on Thursday March 27, 2014 @01:16AM (#7478750)
Good to know. Does it work for everything that's an fd in Linux? I know you've got to treat sockets and files differently in some cases (or at least did once)...
Re:What it's like to use Haskell by enigmo on Thursday March 27, 2014 @02:12AM (#7478895)
It works for most kernel handles, sockets might be a little more normal starting with Win7 but I stopped doing Windows development around then.

Here are the official docs: http://msdn.microsoft.com/en-us/library/windows/desktop/ms72...

Re:What it's like to use Haskell by dllthomas on Thursday March 27, 2014 @02:59AM (#7478998)
Looks like there's a separate function for sockets.

Still, cool stuff there too.

Re:What it's like to use Haskell by samstokes on Wednesday March 26, 2014 @10:44PM (#7478258)
einhorn [1] implements this model and is pretty effective. Used in production at Stripe and other places. (It's written in Ruby, but can run application processes in any language.)

[1] https://github.com/stripe/einhorn

Re:What it's like to use Haskell by DonPellegrino on Wednesday March 26, 2014 @11:13PM (#7478364)
Basically, catch SIGINT, then stop listening to a socket/port. Finish all current requests and exit. The "watcher" parent process will restart the process with the new executable. Repeat for all other processes listening to the socket/port.
Re:What it's like to use Haskell by shadytrees on Wednesday March 26, 2014 @07:50PM (#7477558)
I can't answer for grandparent, but you should check out https://github.com/notogawa/graceful
Re:What it's like to use Haskell by joehillen on Wednesday March 26, 2014 @06:51PM (#7477231)
Except in Haskell you can build ekg right into your server. http://ocharles.org.uk/blog/posts/2012-12-11-24-day-of-hacka...
Re:What it's like to use Haskell by benmos on Wednesday March 26, 2014 @04:00PM (#7476157)
"we added a tracker for the number of suspended Haskell threads" - would you mind sharing how you did that? I couldn't see any obvious GHC APIs for it.
Re:What it's like to use Haskell by implicit on Wednesday March 26, 2014 @04:17PM (#7476285)
It looks like you're right. I misspoke.

We track total threads, working or not. It works great as an indicator because it tends to stay below the number of CPU cores on the server.

Re:What it's like to use Haskell by benmos on Wednesday March 26, 2014 @04:26PM (#7476364)
I take it you mean OS-level threads then?
Re:What it's like to use Haskell by implicit on Wednesday March 26, 2014 @05:59PM (#7476946)
We track Haskell threads.

edit: Found the code. :)

We rolled our own implementation. Our WAI application action increments a counter and decrements it again whenever an HTTP request is received and completed.

It doesn't track threads created as a part of HTTP request handling, but we don't allow those actions to forkIO anyway. There hasn't been any demand for it.

Re:What it's like to use Haskell by benmos on Thursday March 27, 2014 @03:36AM (#7479080)
Ah, right, that makes sense - thanks.
Re:What it's like to use Haskell by thinkpad20 on Wednesday March 26, 2014 @06:06PM (#7476984)
... so, how? Isn't that what he was asking?