News for hackers. Stuff that matters.
Re:Twitter open-sources a high-performance replicated log se... by ryanobjc on Tuesday May 10, 2016 @01:43PM (#11669047)
I was interested until I saw the Zookeeper dependency.

I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

Re:Twitter open-sources a high-performance replicated log se... by nicobn on Tuesday May 10, 2016 @02:00PM (#11669189)
For better or for worse, Java runs a big chunk of this world. Not running a Java application because "ewww Java is gross" will prevent you from enjoying some of the most exciting database technologies out there.
Re:Twitter open-sources a high-performance replicated log se... by gshx on Tuesday May 10, 2016 @02:08PM (#11669248)
Would also be interested to hear more about the "deployment nightmares with Zookeeper". For us, it has to be one of the most stable pieces of 3rd party server software we run.
Re:Twitter open-sources a high-performance replicated log se... by slantedview on Tuesday May 10, 2016 @02:35PM (#11669462)
General things: when your session fails to migrate, when log compaction can't keep up with writes, when logs fill up your disk (ZK used to require external cron jobs to prune logs), when watcher notifications arrive late, when the leader's GC exceeds its heartbeat timeout, and so on.

Lots of things can and do go wrong with Zookeeper. I suspect it depends on the use case, but building a Zookeeper dependency into any system is potentially asking a lot of users/operators.

Re:Twitter open-sources a high-performance replicated log se... by growse on Tuesday May 10, 2016 @03:07PM (#11669697)
Well, like many things, if you deploy it on sensible hardware and put some thought into how its operated, zookeeper isn't particularly problematic. It's certainly not a 'nightmare'.
Re:Twitter open-sources a high-performance replicated log se... by sciurus on Tuesday May 10, 2016 @09:48PM (#11672098)
As an operator I do feel that pain but the alternative of every piece of software that needs distributed coordination reimplementing it, likely poorly, it isn't great either.
Re:Twitter open-sources a high-performance replicated log se... by curun1r on Wednesday May 11, 2016 @12:37PM (#11676823)
There are (IMHO, superior) alternatives like Etcd and Consul that could be used.

Ideally, these kinds systems should be built such that the distributed coordination piece is pluggable and the implementation can be chosen based on deployment concerns.

Re:Twitter open-sources a high-performance replicated log se... by vruiz on Tuesday May 10, 2016 @03:23PM (#11669845)
I have similar worries with kafka, that’s why I’ve been working on and off in my own log based message queue for while now [0]. So far is nothing but a humble beginning to something targeting very simple use cases, but AFAIK there aren't any lightweight solutions in this space.

[0] https://github.com/ninibe/netlog

Re:Twitter open-sources a high-performance replicated log se... by kinkdr on Tuesday May 10, 2016 @03:16PM (#11669784)
Exactly this.

When I saw the header I thought to myself, "Yay! Now I can run something like Kafka, without depending on Zookeeper or having to install JVM!! Give me stable drivers for Python and Go and I am sold!"

If I have to install Zookeeper and JVM, why not use Kafka?

Re:Twitter open-sources a high-performance replicated log se... by jlisam13 on Tuesday May 10, 2016 @02:22PM (#11669349)
'plus systems software in Java', mind sharing an explanation?
Re:Twitter open-sources a high-performance replicated log se... by ryanobjc on Tuesday May 10, 2016 @03:37PM (#11669958)
Java at first is great. Once you get in to large and complex loads, the GC tuning becomes load specific. While tuning GC can be fun (for the first dozen times), the bigger problem happens when your load shifts and your GC tuning becomes subtly wrong.

Also even IF you are tuned properly, there is a world of 99-percentile you'll never get to.

Additionally, the difference between openJDK and Oracle JDK becomes something admins learn to hate you for. Complex shell scripts to invoke java, and lots of standard unix tools just don't work super well. eg: pgrep and pkill. you can tweak it, but it takes a little while to learn the many tips and tricks.

It's not all bad, most Java programs are deployed in a static-all-batteries-included fashion, so you rarely worry about system-installed library versions. So that makes deployment a little less hassle. You never have to recompile for cross platform. The profiling tooling and other stuff is pretty good, and the more you're willing to pay the better the tools get.

Re:Twitter open-sources a high-performance replicated log se... by sulam on Wednesday May 11, 2016 @12:42AM (#11672693)
You are wrong about never getting to the 99th. I did not work on this service at Twitter, but many services at Twitter are built on the JVM, and if this service is like most there, then it probably has SLA targets that are sub-millisecond at the 99th and sub 10ms at four 9's. The only caveat to this is that it has to write to disk, which may keep it from achieving truly high performance, but this is not the fault of the JVM. I worked on one service at Twitter that had to use microseconds on its dashboards just to see any variation at all throughout the day.
Re:Twitter open-sources a high-performance replicated log se... by coredog64 on Tuesday May 10, 2016 @07:48PM (#11671608)
I find that 'jps' works as a good replacement for pgrep.
Re:Twitter open-sources a high-performance replicated log se... by ryanobjc on Tuesday May 10, 2016 @10:35PM (#11672268)
sure, but jps isnt pkill.

The point is you cant use binary name to make your way around anymore. The binary name is 'java'. Standard unix tools just don't work. Yes there are work arounds, but over time things end up being just a little more complex than they should be.

Which means if someone is comparing zookeeper and etcd, well etcd wins major points for being unixy and easy to deploy. Copy 1 binary, done. ZK loses major points here. Gotta make sure the JVM is installed, but do you need the OpenJDK or the Oracle one? if the latter, well apt-get and yum are less helpful.

It's all just little globs of annoying details that add up to be a small pain. Nothing horrible, but if you could make a choice to avoid that, why not?

Basically I guess what I'm saying is there is probably a market for replacing all the Javay distsys stuff with Rust/Go versions. I mean look at etcd!

Re:Twitter open-sources a high-performance replicated log se... by lmm on Wednesday May 11, 2016 @04:50AM (#11673613)
> Basically I guess what I'm saying is there is probably a market for replacing all the Javay distsys stuff with Rust/Go versions. I mean look at etcd!

Or in the other direction. If your main system runs on the JVM and your sysadmins are used to the JVM tools then having a piece of infrastructure that's just another .jar is wonderful, and C/Rust/Go/Ruby/etc. infrastructure elicits groans. Mixing platforms will always be harder than a common platform. So the infrastructure market depends on where you think the future of applications is.

Re:Twitter open-sources a high-performance replicated log se... by glibgil on Tuesday May 10, 2016 @02:26PM (#11669381)
Java has garbage collection and is about two times slower than C
Re:Twitter open-sources a high-performance replicated log se... by sgt101 on Tuesday May 10, 2016 @02:56PM (#11669622)
For good C, which is hard to write - for me and other mortal humans.

On the other hand, as a mortal, I can write a-grade-above-code-that-an-idiot-would-write-just code in Java at about 10 times the speed I can write dire-useless-risible C code.

The comparison is pointless though, good modern languages like Rust and Julia are developing and LLVM is enabling further development.

Re:Twitter open-sources a high-performance replicated log se... by glibgil on Saturday May 14, 2016 @05:15PM (#11698068)
You can call the comparison pointless, but it is the answer to the question of why Java is not a systems language
Re:Twitter open-sources a high-performance replicated log se... by Chyzwar on Tuesday May 10, 2016 @05:31PM (#11670831)
C dont have garbage collection. In some cases Java can outperform C.
Re:Twitter open-sources a high-performance replicated log se... by glibgil on Saturday May 14, 2016 @05:15PM (#11698070)
Yeah, that's what I said. C doesn't have gc so it is a systems language. Java has gc so it is not a systems language
Re:Twitter open-sources a high-performance replicated log se... by slantedview on Tuesday May 10, 2016 @02:26PM (#11669379)
Agree re: Zookeeper. It looks like DistributedLog uses BookKeeper, which I'm guessing is where the Zookeeper usage comes from.

In terms of simplifying the deployment, they could have went embedded with Atomix[1] instead. Perhaps next time.

[1]: http://atomix.io/

Re:Twitter open-sources a high-performance replicated log se... by asimuvPR on Tuesday May 10, 2016 @02:00PM (#11669190)
Mind sharing a bit more?
Re:Twitter open-sources a high-performance replicated log se... by ryanobjc on Tuesday May 10, 2016 @03:51PM (#11670088)
The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_.

So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good.

There are other operational issues people have noted above as well.

Re:Twitter open-sources a high-performance replicated log se... by rad_gruchalski on Tuesday May 10, 2016 @04:54PM (#11670574)
Then configure it using IP addresses. I've been using Zookeeper like this for 4 years in public EC2, swapping machines in and out and it always worked. What's the problem.
Re:Twitter open-sources a high-performance replicated log se... by sciurus on Tuesday May 10, 2016 @09:35PM (#11672046)
How is this an improvement over using DNS? When you swap in a new EC2 instance it has different IP addresses from the old instance. So now not only do you have to restart your other Zookeeper servers in the cluster, you have to update their configuration then restart them.
Re:Twitter open-sources a high-performance replicated log se... by curun1r on Wednesday May 11, 2016 @12:46PM (#11676908)
He's probably using ENIs to decouple the IP address from the instance.
Re:Twitter open-sources a high-performance replicated log se... by ryanobjc on Tuesday May 10, 2016 @05:52PM (#11670971)
i think you do know the problem, which is not everyone has that set up. Yes, if you are cloud, and can roll with static IPs then great for you. Some people still have to be on-prem and reusing IPs isn't always an option.
Re:Twitter open-sources a high-performance replicated log se... by ttwwmm on Sunday May 15, 2016 @03:01PM (#11702230)
We run ZooKeeper (and Kafka) within Docker containers and were able to solve this problem with a patch to the client: https://issues.apache.org/jira/browse/ZOOKEEPER-2184

You also need to disable the JVM's insane DNS-caching behavior (config change only).

Re:Twitter open-sources a high-performance replicated log se... by neeleshs on Tuesday May 10, 2016 @10:01PM (#11672150)
Re:Twitter open-sources a high-performance replicated log se... by eropple on Tuesday May 10, 2016 @04:51PM (#11670552)
You're right in that Zookeeper can be a hassle in a dynamic environment, but in my experience Exhibitor largely removes that concern.
Re:Twitter open-sources a high-performance replicated log se... by neilc on Wednesday May 11, 2016 @04:46AM (#11673600)
This behavior should be fixed in Zk 3.4.7+ and 3.5.0+