Re:25 Gbit/s at home, part 1
by virtuallynathan on Thursday September 07, 2023 @02:32PM
(#37423509)
I just upgraded to 10GbE service from Ziply Fiber in the Seattle area:
Speedtest.net - ~9.3Gbps symmetric
iperf to Denver: ~9.3Gbps symmetric (~8 threads)
iperf to Minneapolis: ~9.3Gbps symmetric
AWS S3 Download with small files: 1.3Gbps
Usenet download: 7-9.5Gbps
Mounting my ZFS pool to an AWS instance via SMBv3 in us-west-2: 3Gbps (not clear what the limit was here)
Will try to test Steam and such.
Well, now you're just making the rest of us jealous.
AWS S3 Download with small files: 1.3Gbps
That's $53/h in data transfer costs from us-west-2...
>Mounting my ZFS pool to an AWS instance via SMBv3 in us-west-2: 3Gbps (not clear what the limit was here)
this one sounds interesting and definitely a writeup I would read about.
Re:25 Gbit/s at home, part 1
by virtuallynathan on Saturday September 09, 2023 @09:20AM
(#37445039)
I had some media files I wanted to verify the integrity of -- I was using tdarr.io in a podman container on my local server. I then spun up an m7a.12xl spot instance, mounted my local storage to it via SMBv3. I ran tdarr on that instance, and connected it as a worker node to my home server instance. tdarr ran ffmpeg to decode every video frame to /dev/null, and reported any errors. It was able to process a few thousand FPS or about 2-3Gbps. I found some corrupted files, and was able to restore them from another backup.
There was no practical reason to do this, it cost about $100 to double my processing speed.
The bigger benefit of all of that bandwidth is going to be multiple users. If you work from home and have a big family, or just live with a few roommates you're going to be seeing huge quality of service improvements from one person not being able to tank the whole connection.
Re:25 Gbit/s at home, part 1
by doublerabbit on Thursday September 07, 2023 @02:40PM
(#37423605)
Not with my 800k DSL you won't.
Yes, this is why I went with the medium 10gbps package from Init7, with only a 1gbps internal network. Each port is 100% independent bandwidth-wise, and maybe I'll eventually add a 2.5gbps adapter on a suitable host.
Yeah, that is also what I found. 'Just' 1gbs fiber, but downloads are often bottlenecked by the server, especially things like GitHub.
You sure it is the server and not routing?
Indeed. My cheap router, when bufferbloat mitigation is turned on (codel), maxes out its CPU at 650Mbps.
Re:25 Gbit/s at home, part 1
by grepfru_it on Thursday September 07, 2023 @02:47PM
(#37423716)
Its not just your router, also intermediate hops. I saw this same thing back when 1gbps was first being rolled out 20 years ago. My downloads would hard lock at 100mbps from a lot of sites while other locations would pull at the full gbps speed. Turns out some router in the middle would max out at a much lower limit. OPs test is showing the same thing. FWIW, Steam and Battle.net use P2P to distribute data so you get multiple streams using different routes. But GitHub, as example, has a single route between their datacenter and you and goes through a bunch of routing shenanigans once you hit their POP.
Get ready for a whirlwind of delayed network improvements from your favorite transit provider for the next 3-5 years just as we saw in the early 2000s!
Nope. There are plenty of servers that do push 1 Gbit, my provider has very good peering, and really close to several CDNs.
the one nice thing is copying big files to and from my work computer at 100 MB/s, but my work network is mostly 1 Gbps anyway so it wouldn't get any faster if I had faster internet...
Not sure what you see, but I always multiply transfers times 10 to estimate bandwidth, since there is some overhead in the TCP/IP protocol. So 280MB/s I would estimate being about 2.8gbps...
Re:25 Gbit/s at home, part 1
by masterteague3 on Thursday September 07, 2023 @02:21PM
(#37423330)
To convert MB/s to gb/s (MegaBytes per second to Gigabits per second) simply multiply by 8 (8 bits per byte). So 280MB/s is 2.24gbps (moving the decimal place).
> but I always multiply transfers times 10 to estimate bandwidth, since there is some overhead in the TCP/IP protocol
20%ish seems high though, especially for file transfers which are probably almost all full packets near 1500 MTU cross WAN.
There is another variable as well. Most network equipment I've worked with reported throughput with incorrectly labeled kibi/mibi/gibi, and the ISP uses kilo/mega/giga.
The former system is multiples of 1024, the latter the mathematical and legal definition of x1000
The X10 rule of thumb is probably pretty spot on at the end
What godforsaken gear reports in binary prefixes? That's just... horrendous.
I think ethernet (1500mtu standard) has about 30 bytes of packet routing information per (max) 1500 bytes of data. That works out to 2% minimum. On smaller data packets (less than 1.5kb) it would be a much higher percentage. TLS on top of that, once you're out of the handshake, is about 40 bytes. those 40 bytes would be inside that 1500 window. Assuming 1460 bytes of data (best case) with 40 bytes of TLS and 30 bytes of ethernet overhead - that's up to ~5%.
There maybe further overhead when that packet gets to your router/modem and out to your ISP, and their switches, consider that there's likely another wrapped header on there accounting for 30 more bytes and possibly causing a packet to get split into two smaller ones. I think 10% would be the highest "data rate" vs "port rate" overhead you'd see, without involving a VPN or complex network.
I am not sure if the original poster is talking about downloading over https and, whether he is mentioning the data rate or the file size. For instance you download a 1gbyte file in 4 seconds - your effective data rate in terms of what you want to see on your disk is 250Mbytes/second ; but if you measure the actual data rate it would be more due to both TCP/IP and, application layer overhead.
There is a more concrete example - in the article I've showed iperf3 output, which gave 23.5 Gbit/s - that is actual data. Raw speed was ~24.9-25. Everything else is overhead in that case.
Re:25 Gbit/s at home, part 1
by candiddevmike on Thursday September 07, 2023 @01:49PM
(#37422799)
And here I am with 300Mbps (Charter/Spectrum) that tanks to 50Mbps after 5:00 PM for the "low" price of $99.99/month...
I get the same impression using PS+ and trying to stream some AAA games. Even though I have a 1gbps connection, I'll get timed out from my VM running Deus Ex: Mankind Divided just trying to load a new map sector.
On 5/5 I've found Usenet obliterates torrenting in the throughput department.
Yeah but you pay for UseNet access separately. Some usenet servers charge extra for access over 100mbps or 1gbps.
You do need to pay but I would stay away from any provider still charging more based on how much bandwidth or transfer you're using, they are likely just extremely outdated or plain overpriced not actually better. Frugal Usenet for $50/y is where I'd point anyone but there are tons of good alternatives too so that's not the only place people should check out by any means. I've pulled about 50 TB in the last 2 months (NAS upgrade) and most files come in around 4 gbps out of 5.
Is there a modern guide to UseNet that you'd recommend?
I followed the guide here https://www.reddit.com/r/usenet/wiki/index/. I ended up doing Frugal for the provider, nzbgeek for the indexer, and nzbget for the client. Mapping to torrent terms the indexer is the torrent search site, the client is the client, and the provider is (effectively, in this mapping) who you pay to leech from. Depending how deep you get into it it may be worthwhile to look at a second provider on a different backbone (for greater availability) and/or another indexer but I'd start with just that and see how it works for you then add if needed.
Config wise signing up a provider is like buying anything else online. Setting up the client isn't much more difficult than setting up a torrent client (select default folders, set limits, etc) beyond you need to manually enter the info to connect to the providers.
Re:25 Gbit/s at home, part 1
by UI_at_80x24 on Thursday September 07, 2023 @02:11PM
(#37423177)
Which is why you break the task into multiple threads.
Any sane Usenet client will do this by default but limitations from providers are per account not per connection.
Re:25 Gbit/s at home, part 1
by Phelinofist on Thursday September 07, 2023 @01:48PM
(#37422786)
280MB/s steam? On a good day I get 25MB/s (Vodafone 400/25) on any German mirror
Your max theoretical would be 50MB/s on that connection, so 25MB is not terrible. I think there is some notable overhead on steam (check summing or encryption) as with my 1gbps connection previously I would never get past 100MB/s (800mbps).
Re:25 Gbit/s at home, part 1
by Phelinofist on Thursday September 07, 2023 @03:15PM
(#37424140)
I regularly max out at 70MB/s for other services, so the connection is actually pretty good IMHO
Speeds seem to vary based on the game and based on releases, interestingly. Some larger new-releases (Baldur's gate) were only hitting <100MB/s. I imagine it comes down to how loaded up the local steam POP is and maybe even the specific server hosting that game's files.
Aside from that release, almost everything that is a good size (>10GB) has gotten north of 100MB/s pretty regularly.
Re:25 Gbit/s at home, part 1
by bravetraveler on Thursday September 07, 2023 @02:56PM
(#37423860)
I get about the same; full link speed at 2Gbit. Steam will absolutely chew CPU to support this
I see a notable difference on a 7950X3D simply by messing around with the power/performance governors.
cable is shared media
Re:25 Gbit/s at home, part 1
by Phelinofist on Thursday September 07, 2023 @03:14PM
(#37424124)
Indeed, but no other service I use suffers from this to this extend.
Unless you are say say pro competitive gamer (lag) or doing sone fringe science, anything above 1gbs real speed is useless these days.
I have it, or slightly below, with pings around 7ms, and... as a regular user with some modest 10tb movies and games collection, 99% of the cases 100mbit is more than fine, and the rest covers rest.
No place for anxious ocd types, but low ping and overall reliability of the connection are way more important to me, even for fetching 70gb+ games. Not only due to wfh, but also for that.
Re:25 Gbit/s at home, part 1
by 10000truths on Thursday September 07, 2023 @02:10PM
(#37423162)
Multiplayer games usually care more about latency than throughput. Modern FPSs can use as little as 250-500 kbps of bandwidth, but they'll play like crap if your RTT to the game server is >100ms.
Or if you want to download a 4k torrent on the fly. A 40gb file will take only 2 to 5 minutes to download. Almost instantly.
Re:25 Gbit/s at home, part 1
by formerly_proven on Thursday September 07, 2023 @01:57PM
(#37422938)
Steam downloads are always very CPU intensive for me, it seems to need about 1 core per 100 megabit/s. I'm guessing they're using somewhat slow compression, perhaps LZMA. Likewise, game file validation is CPU limited (it only uses a single thread) at around 500 MB/s for me. Probably an outdated algorithm choice as well.
Steam - ~280MB/s (2.2gbps)
Battle.net launcher - ~140MB/s (1.1gbps)
25GB Torrent with 1000+ seeders - ~70MB/s (~560mbps)
2GB iso from github - ~80MB/s (~650mbps)
fast.com - 1.4gbps
speedtest.net - 2.7gbps (using ISPs endpoint 2ms away)
Using a download manager like IDM or jDownloader will help for http downloads, but most hosts will limit your speed even with 16 connections open. I've managed to see 2gbps moving data to/from servers (scp) with softether configured to use 16 connections. The reality is with a single connection (majority of ssl transfers) you'll be limited by the sending side in almost all cases.
Overall it seems that while you can get connected and run an iperf to your ISP or multi-connection speedtest to a server hosted by your ISP or peered with your ISP, you'll be pretty much limited to <1gbps speeds regardless of your home network throughput.
Knowing this I would have simply went multi-gig (2.5g) for all in-home networking and saved a good chunk of change on networking equipment.