News for hackers. Stuff that matters.
Re:A man launched a new ISP from his garage (2018) [video] by state_less on Saturday March 07, 2020 @11:22AM (#22512340)
I like the P2P nature of the internet, but how do you get connected to a peer on good terms? Comcast, Verizon and AT&T limit not only who but what devices you can route packets for (e.g. hotspot data).

Any ideas on finding good peers to work with? I'm hoping for few usage restrictions first and then quality links characteristics next (e.g. bandwidth, latency, cost, etc...).

Re:A man launched a new ISP from his garage (2018) [video] by jcrawfordor on Saturday March 07, 2020 @11:47AM (#22512478)
As a new ISP, you're not going to get any peering arrangements. Really, as an existing consumer ISP not in the top 10 or 20, you're not going to get any peering arrangements. Consumer ISPs generate very asymmetric traffic (more down than up) which tampers with the core "equality" assumption of traditional peering arrangements.

You'll need to purchase transit from an existing provider, such as CenturyLink's commercial arm or Verizon Business, there are plenty more. This won't be free but it also probably won't be as expensive as you think. In a city you can get a Gigabit dedicated to you for $2k a month, maybe less. Shop around. Most of these providers will put you on the phone with a network engineer as part of the quoting process and they can be helpful in recommending what service level and equipment you get.

IPv4 allocations are hard to come by unless you want to bid on them at auction which can be expensive. You probably want to get a block from your provider, but they'll be hard to talk into giving you more than say a /29 and they might charge you more for it. Often they don't, though, they just put you in a queue sorted by customer size and you'll never get to the top of it. As a small ISP, CGNAT is probably going to be your only choice for IPv4. This also has the downside of being non-portable since the addresses will always belong to your provider. Maybe save up money to bid on your very own. Once you get one you'll want to start participating in BGP which is complicated anyway, so get to that later, once you have customers.

Re:A man launched a new ISP from his garage (2018) [video] by rahimnathwani on Saturday March 07, 2020 @01:49PM (#22513250)
'Consumer ISPs generate very asymmetric traffic (more down than up) which tampers with the core "equality" assumption of traditional peering arrangements.'

Shouldn't this be in favour of the ISP? If the traffic isn't balanced, then doesn't the peer that's sending more (e.g. YouTube, Netflix) usually pay?

I have no experience of setting up an ISP, so this is just based on what I've read. I'd love to understand it better.

"IPv4 allocations are hard to come by unless you want to bid on them at auction which can be expensive."

Looking online, it looks like the prices are $15 to $30 per IP address, depending on the size of the block. So I guess this cost swallows the first month of revenue from each new customer.

Re:A man launched a new ISP from his garage (2018) [video] by jcrawfordor on Saturday March 07, 2020 @09:09PM (#22515827)
> Shouldn't this be in favour of the ISP? If the traffic isn't balanced, then doesn't the peer that's sending more (e.g. YouTube, Netflix) usually pay?

No, these providers end up peering or placing CDN edge nodes on the provider network and essentially don't pay for their transit. They get this arrangement because it reduces the distance that the high-tier ISPs have to move their traffic.

The basic problem is this: ISPs enter peering arrangements with each other to avoid the bureaucracy of both paying each other to access the other's customers. As a consumer ISP, you only have end users, not service providers (or at least you have many more end users than service providers). So, your customers want to access to other networks. No other networks want to access your customers. This means that you don't have the negotiating position to set up peering agreements (because only your own customers would be affected by not having one).

In general, peering agreements are becoming very strained over the last five plus years because the underlying assumption of traffic symmetry is virtually never true for consumers now. In practice most of the high-tier ISPs business is providing consumers with access to Netflix. Netflix doesn't care about getting access to anyone else. So no one likes to peer with consumer ISPs, even the big ones, because there's nothing in it for them. This has produced some very high profile disputes with major consumer ISPs dropping peering with tier-1 providers.

> Looking online, it looks like the prices are $15 to $30 per IP address, depending on the size of the block. So I guess this cost swallows the first month of revenue from each new customer.

Yes, but you don't buy addresses onesy-twosy. If you're going to buy your own block you're probably going to want at least a /24 (255 addresses). That runs upwards of $5k. Most small ISPs start up with very little capital and that's out of reach. Plus you'll need to participate in BGP which is out of the technical expertise of most of the people I know running WISPs, although sometimes you can pay your transit provider to handle BGP for you.

Re:A man launched a new ISP from his garage (2018) [video] by Scoundreller on Saturday March 07, 2020 @02:52PM (#22513628)
> 'Consumer ISPs generate very asymmetric traffic (more down than up) which tampers with the core "equality" assumption of traditional peering arrangements.'

It's a good situation for the ISP to start a DC business with unmetered ports to help balance things!

Re:A man launched a new ISP from his garage (2018) [video] by jcrawfordor on Saturday March 07, 2020 @09:12PM (#22515845)
This is a lot of how the major ISPs smooth out the peering situation, they also offer colocation and carrier hotel services to large customers. Unfortunately, the popularity of Netflix and other streaming services has largely upset this balance and it's very hard for any ISP to keep traffic equality.
Re:A man launched a new ISP from his garage (2018) [video] by Scoundreller on Sunday March 08, 2020 @01:36AM (#22516658)
At least with Netflix, you can get their Appliances and cut your inbound Netflix bandwidth by 80%+ for the cost of power and 4U.
Re:A man launched a new ISP from his garage (2018) [video] by iptrans on Sunday March 08, 2020 @03:42AM (#22516841)
Yeah, if you have at least 5 Gbps of Netflix traffic on your network...

No small or startup ISP is going to qualify for a Netflix appliance. Also to cut your Netflix bandwidth by 80+%, you need more than 6 Gbps of Netflix traffic, as Netflix requires 1.2 Gbps of inbound bandwidth just to fill the cache.

Netflix share of Internet traffic is about 15-20%, so in practice Netflix appliances are used by ISPs with 25-35 Gbps of bandwidth and up. This means ISPs with tens of thousands of subscribers or more.

Re:A man launched a new ISP from his garage (2018) [video] by Scoundreller on Saturday March 07, 2020 @02:46PM (#22513585)
It's kinda a strange thing in Canada.

The incumbent telecoms generally refuse to peer domestically with independent ISPs and smaller transit providers that cheaper DCs use.

One ISP reported that Bell would only connect to them in Equinix in Chicago, even though Equinix has a PoP in Toronto. https://www.peeringdb.com/net/1550

This means if you subscribe to internet with the local telephone monopoly (Bell) in Toronto, your traffic to toronto[1-8].voip.ms goes through NYC or Chicago. Have seen some routes that add Washington DC to the mix too. A 5km trip made into several thousand km.

The incumbents even don't peer much at our big IX: http://torix.ca (they buy ports, but don't peer freely).

In a lot of ways, because the independents will peer with everyone and love IXs, your traffic often stays a lot more local in Canada.

Re:A man launched a new ISP from his garage (2018) [video] by dylz on Saturday March 07, 2020 @04:03PM (#22514033)
The US is the same. All major ISPs refuse to peer with any independents at all; refuse to be on routeservers; and demand some minimum of like 50-100 Gbps for a peering session, with insane requirements.

I've seen numerous ones also demand things like RPKI/ROA and good filtering - while themselves having everything ROA and IRR invalid, announcing customer space with no IRR records whatsoever, etc.

Re:A man launched a new ISP from his garage (2018) [video] by ksec on Saturday March 07, 2020 @05:25PM (#22514650)
What are the major iSP's incentives for not peering?

I am starting to think may be Government should run a minimal Ix every x km2 and force all ISP to connect with it.

Re:A man launched a new ISP from his garage (2018) [video] by redis_mlc on Saturday March 07, 2020 @10:40PM (#22516204)
> What are the major iSP's incentives for not peering?

In the US, each peering agreement is a private agreement (ie. a contract) between 2 parties, usually with roughly equal bandwidth traffic.

If one party doesn't like you, perceives you as a competitor, or you don't have similar bandwidth, then no deal.

Examples of typical rejections are Cogent, who compete on price, and Netflix, who need last-mile acess to consumers.

https://en.wikipedia.org/wiki/Peering

Re:A man launched a new ISP from his garage (2018) [video] by isclever on Sunday March 08, 2020 @09:15AM (#22517855)
For Canada, specifically Bell, they both an eye ball network and a transit provider. Bell has no incentive to peer at local IXs and "give away" part of their service (the eyes, as that is the lucrative part) when they can sell on the fact they are a large national network AND you can get access to their end customers as well.

Edit: clarification on the part that is more valuable.

Re:A man launched a new ISP from his garage (2018) [video] by Scoundreller on Saturday March 07, 2020 @05:01PM (#22514476)
So how do independents interconnect their routes?

I think the Canadian issue is that incumbents aren't even peering with transit providers.

Re:A man launched a new ISP from his garage (2018) [video] by dylz on Saturday March 07, 2020 @06:08PM (#22514907)
For this context, when I say independents, I mean smaller providers - content providers, independent ISPs that aren't large enough yet. They will obviously peer with Google, Akamai, etc.

Independents more or less just buy transit/blend and deal with it.

Here's an example of a large incumbent's peering policies: https://www.spectrum.com/content/dam/spectrum/residential/en...

The first requirement completely removes the possibility of ever getting a peering session with a small ISP or independent provider. So does the fourth.

Another one: https://www.xfinity.com/peering

> minimum of four mutually agreeable geographically diverse points in the US. Interconnection points must include at least one city on the US east coast, one in the central region, and one on the US west coast

This makes "launch an ISP from garage" bordering on "never" because they do not want regional providers getting settlement free peering - they just want them to buy IP transit from them.

If you look at virtually every major US/CA ISP on peeringdb, they will be Restrictive / Selective or Private peering only, most likely.

Re:A man launched a new ISP from his garage (2018) [video] by 2J0 on Saturday March 07, 2020 @10:31PM (#22516162)
Does enough independent isp demand exist to aggregate the traffic to provide breakouts and satisfy Spectrum's 8 point term?

This reminds me of writing a bit to catch the BT SIN Supplier Information Notes ephemeral notices: BT and later OpenReach were required by law to advertise every sweetheart deal to all comers on this domain. This might take place for only long enough for the page to come and go from the google cache (which BT had some way of automating content exclusion from, the cache page link in search results leading to a 404 was a dirty little trick to claim the advertising and universal offering components of statutes, which didn't last long for a ruse because the entire uk industry seemed to automatically file in court to obtain even the simplest of provisions, I know we weren't alone) anyway that was how we discovered BT would lease twisted copper pairs from CPE to exchange and you could hop onto commercial backhaul from there, so we had 4 wire SHDSL modems either end and the tent was a enormous £495pq British pounds or about $3kpa for a symmetric 10Mb/s last mile to clec

Re:A man launched a new ISP from his garage (2018) [video] by dylz on Saturday March 07, 2020 @11:12PM (#22516304)
> Does enough independent isp demand exist to aggregate the traffic to provide breakouts and satisfy Spectrum's 8 point term?

You're very close to describing standard IP transit. The regional-ness of small ISPs and indies make it such that interconnections throughout the entire US make it infeasible, especially considering the cost of such numerous transport and XCs.

I can get a blend from the DC - they aggregate enough capacity to peer with Telus for example, so the ASPath would go me -> DC -> Telus over IX or me -> DC -> Transit -> Telus.

If I got on the exchange directly, Telus peers selectively and does not do routeservers, so my traffic to them would not go over the exchange anyway, so what is the point?

Another annoyance is that in many facilities cross connects cost more than bandwidth itself - I get spam from Cogent daily offering cheap bandwidth, but the XC monthly cost costs more than several gigabit of Cogent bandwidth, so why bother unless I need a 100G port at once?

Re:A man launched a new ISP from his garage (2018) [video] by johngalt on Saturday March 07, 2020 @11:43AM (#22512459)
It's somewhat ad hoc. Generally you'll want to find who owns the actual infrastructure, and get as close to an agreement with them as possible.

I.E. There will be some huge Zayo fiber that is resold by a bunch of different carriers. They are B2B only and don't sell directly to end users and have fairly high minimum commitments. Find the closest point that you can connect to it, and do wireless from that point outwards.

Or you can do an interconnect/backhaul type agreement (L3 used to be great at these before they were acquired by Clink). This is where you are only buying a link back to a colocation. From there you can do direct peering or pick from several transit providers at pricing you can only get in that sort of environment.

Fundamentally the problem is:

If the fiber is there, then likely you'll have 50 carriers all reselling it and you are just number 51.

If the fiber isn't there, then you'll have a difficult time providing suitable bandwidth.

Re:A man launched a new ISP from his garage (2018) [video] by lapinot on Sunday March 08, 2020 @01:11PM (#22519159)
As others have said, as a small isp you're gonna need to start off by buying transit. But the core philosophy is to peer with your peers, might sound stupid, but you should look towards other isp that look like you. There are already a lot of diy internet communities, most doing one kind or another of wireless mesh to connect to end-users. Some have as numbers and can peer at regional colo centers.

* guifi.net (spain, mostly in catalunya, as49835)

* ffdn.org (france) (see also: gitoyen.net, as20766)

* freifunk.net (germany) (see also: community-ix.de)

* ninux.org (italy, as197835)

* riseup.net (as16652)

Most networks can be found on peeringdb.com. You can also start poking at local NOG mailing lists (https://en.wikipedia.org/wiki/Internet_network_operators'_gr...)