News for hackers. Stuff that matters.
Re:Dragonfly: Alibaba P2P file distribution system by ksec on Wednesday June 13, 2018 @11:30AM (#17304019)
Ok, this is huge.

>At Alibaba, the system transfers 2 billion times and distributes 3.4PB of data every month, it has become one of the most important piece of infrastructure at Alibaba. The reliability is up to 99.9999%.

I took at look at their repo and it turns out there are surprisingly lots of good stuffs in it which never gets much spotlight or attentions.

Re:Dragonfly: Alibaba P2P file distribution system by justinsaccount on Wednesday June 13, 2018 @11:45AM (#17304147)
It's a cool project, but I'm not sure how huge that is.

If you have 5000 nodes running this, that could be as small as distributing 23.8 gigs/day to each node over the course of the month.

Re:Dragonfly: Alibaba P2P file distribution system by oh_sigh on Wednesday June 13, 2018 @01:24PM (#17305184)
3.4PB/month is only 111,000 GB/day. That's not particularly huge in this day and age, except to consumers.
Re:Dragonfly: Alibaba P2P file distribution system by Xorlev on Thursday June 14, 2018 @12:13AM (#17309229)
I imagine this traffic is fairly bursty as well.

Also keep in mind this is a data distribution system. In the case of large data pushes, new builds, etc. it's important that all peers get the new data on a timely and reliable manner.

I suspect you're underestimating the problem this project solves by focusing on a mostly irrelevant data rate stat.

Re:Dragonfly: Alibaba P2P file distribution system by justicezyx on Wednesday June 13, 2018 @12:37PM (#17304664)
> The reliability is up to 99.9999%

"up to"... Certainly looks like a wrong wording.

Re:Dragonfly: Alibaba P2P file distribution system by zild3d on Wednesday June 13, 2018 @01:09PM (#17305020)
I read it in the "we have now gotten it up to X" sense, such as "I'm now up to my 5th beer"
Re:Dragonfly: Alibaba P2P file distribution system by gcbw2 on Wednesday June 13, 2018 @11:23PM (#17309014)
except reliability reduces a total, not sum integers like beer count.

if I write a shell script with a scp line right now I can tell you it has 100% reliability. up to X% reliability means that "in the very nice and controlled environment, with the own devs attacking every production problem, trying to get 100% got us up to X%"

Re:Dragonfly: Alibaba P2P file distribution system by quixoticelixer- on Thursday June 14, 2018 @07:32AM (#17310849)
Except that is clearly not what they where trying to communicate.
Re:Dragonfly: Alibaba P2P file distribution system by unicornporn on Wednesday June 13, 2018 @02:58PM (#17305995)
That's the way to read it.