I wonder if that is happening in this specific case? Although if it were the case the vendor should still be honest about it. Just saying they limit uploads to 2 Mbps is better than giving the run-around.
Its to reduce their maximum bandwidth capacity required. I don't see it as a problem, considering their price points. They're selling you storage, not "slam 1TB of your data into our storage system in a day". If you're looking for that, ship a hard drive to Iron Mountain.
EDIT: Even AWS limits how fast you can upload to S3, and built an appliance for you to rent and ship back and forth if you need to move data faster. That station wagon full of tape is still alive and well.
I'm on gigabit fiber and use S3 to backup hundreds of gigs per month to S3. I've never seen them limit upload speeds, it is clearly saturating the connection for the entire duration of my upload. I would expect that because I am paying for the storage, they would be happy to let me write data to their machines as fast as I like. Is there a citation you can provide from their docs that supports your statement? Genuinely curious, because my experience has been different.
To the point that some of these sync or backup providers limit bandwidth, I have definitely experienced that. Tested SpiderOak and Dropbox and upload speed was horrid. Dropbox in particular was disappointing because they can't even claim to have the extra encryption overhead SpiderOak does, it was just shit speed every day. I'm paying a premium for gigabit fiber to the home and you really can tell who over-promises and under-delivers quickly. Fortunately my 'roll your own' backup + sync works well and is price competitive so I'll stick with that.
I don't understand why you'd think this. You're paying for storage, not an SLA as to how fast you can fill it.
> I'm paying a premium for gigabit fiber to the home and you really can tell who over-promises and under-delivers quickly. Fortunately my 'roll your own' backup + sync works well and is price competitive so I'll stick with that.
This is the preferred solution if a) commercial services are too slow for you and b) you're willing to spend the time to implement and manage it. It appears, based on commercial services out there, that there is no competition based on upload speeds.
They should be looking to partner with someone who has bandwidth problems in the other direction. By combining a backup service's upload bandwidth and a streaming video service's download bandwidth into one AS, you can get a more balanced stream, and qualify for free peering.
A great model would be to parter with CDNs; they pour content out to eyeball networks, but you could run a distributed network of your storage system across all of their POPs.
That ZOMG WHAT A DEAL! of a plan is kinda worthless...
"Slam" is a bit of a loaded word, since... if they are selling 1TB of storage, shouldn't we get 1TB of storage?
That's the same crap that ISP's tried to pull with UNLIMITED INTERNET!!! (as long as you stay under 30gb per month)
You do, they're just not allowing you to store it in 24 hours. Some services (Backblaze, if I recall) allow you to ship a drive to get around this limitation.
Notice that all services do this? If you can do better, build one! Prepare to go broke from the peak bandwidth requirements you'll need to build your networking architecture to support such transfer rates, but I always encourage experimentation and learning lessons over complaints.
The limitation is actually the pipe that connects you to Amazon, not an inherent limitation within S3 or other services within Amazon on connection speed. If you have a good enough connection, or peering with Amazon things go amazingly fast.
When I worked at an ISP, we slammed about 20 Gbit/sec into S3 without issues, but even then data we were backing up -- about 300 TB of data a day -- at that rate took 1.4 days to upload to the cloud, so we ended up backing it up in-house instead. (we needed to store the data for 7 days, after that it went bye bye).
Seems like the perfect usecase for S3; inbound transfer is free, and you're only paying for a rolling 7 day window of storage with lifecycle rules :/
I think it would be a good selling point for a service like this to allow higher upload speeds.
Hold on, this is hacker news. VCs, this is a great idea!
No, no of course it's not. Initial seeding is a competitive moat for the first mover. Moving a few hundred gigs to a new backup company just to save a few bucks? I don't think I could be bothered, because I KNOW how long it will take.
Backup services especially have low operational requirements for their hardware and network connection, since once the files are uploaded they only need to be verified periodically.
SpiderOak is definitely overselling the 1TB as well as another one that pops up once in a while called as the "unlimited" plan for $149 a year. This is clear from the disproportional pricing structure - $79 a year for 30GB that jumps to $129 a year for 1TB and then to $279 a year for 5TB - which entices users to go for the higher amounts because they appear to be great deals. What people with residential broadband connections may not realize is that a) uploading even 1TB of data will take a long time and b) SpiderOak cannot, and does not, provide any minimum guarantees on the upload or download speeds (assuming everything else in between SpiderOak and the user looks fine).
BTW, why store photos and videos on encrypted storage? For that I use Office 365's OneDrive: everyone in my family gets a terabyte for $99/year and I really like the web versions of Office 365 because when I am on Linux and someone sends me an EXCEL or WORD file, no problem, and I don't use up local disk space (with SSD drives, something to consider).
As for OneDrive, I tried it for a while but it didn't work out. Their clients and web interface were terrible and their API was severely lacking. I expect more functionality when I'm sacrificing my privacy.
I ended up going with Google Drive in the end, as you can get 1TB for $9/month with an Apps for Work Unlimited account (I actually seem to have Unlimited under that plan, which isn't supposed to happen until 4 users). That of course means sacrificing encryption but I trust Google enough to make the privacy tradeoff in exchange for extra features (OCR, Google Photos etc.).
A little off topic, but Google really seems to be upping their consumer game lately with Google Music, Youtube Red, Google Movies + TV, etc. I am now less a user of other services like GMail and Search, but Google gets those monthly consumer app payments from me. I have the same kind of praise for Microsoft with Office 365.
but its an ad hominem argument, thats for sure.
You are assuming that you are the only one using that uplink and that server
> easily capable of hashing and encrypting the data much faster than SpiderOak was handling it
I can believe that there was upstream congestion somewhere outside my network (speeds to Google, Amazon indicated that there were no issues inside) or that their server was overloaded but the engineer who investigated seemed to attribute it to the client:
> Additional changes to storage network configuration will not improve the situation much. There is an overhead limitation when the client encrypts, deduplicates, and compresses the files you are uploading"
In February SpiderOak dropped its pricing to $12/month for 1TB of data. Having several hundred gigabytes of photos to backup I took advantage and bought a year long subscription ($129). I had access to a symmetric gigabit fibre connection so I connected, set up the SpiderOak client and started uploading.
However I noticed something odd. According to my Mac's activity monitor, SpiderOak was only uploading in short bursts [0] of ~2MB/s. I did some test uploads to other services (Google Drive, Amazon) to verify that things were fine with my connection (they were) and then contacted support (Feb 10).
What followed was nearly __6 months__ of "support", first claiming that it might be a server side issue and moving me "to a new host" (Feb 17) then when that didn't resolve my issue, they ignored me for a couple of months then handed me over to an engineer (Apr 28) who told me:
"we may have your uploads running at the maximum speed we can offer you at the moment. Additional changes to storage network configuration will not improve the situation much. There is an overhead limitation when the client encrypts, deduplicates, and compresses the files you are uploading"
At this point I ran a basic test (cat /dev/urandom | gzip -c | openssl enc -aes-256-cbc -pass pass:spideroak | pv | shasum -a 256 > /dev/zero) that showed my laptop was easily capable of hashing and encrypting the data much faster than SpiderOak was handling it (Apr 30) after which I was simply ignored for a full month until I opened another ticket asking for a refund (Jul 9).
I really love the idea of secure, private storage but SpiderOak's client is barely functional and their customer support is rather bad.
[0]: http://i.imgur.com/XEvhIop.png