News for hackers. Stuff that matters.
Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by jimktrains2 on Wednesday May 02, 2018 @04:54PM (#16981078)
> Actual ISP network engineer her

So then you must intimately understand the depth of the lies and anti-consumer business practices that spur such networks.

> Real five nines network infrastructure takes significant capital investment.

5-9s is 5 minutes of downtime a year. I've never had a commercial, let alone residential ISP hit that mark.

> Actual, reliable internet costs real money, both in the salaries of people to engineer and architect it, salaries for 24x7x365 clued-in NOC staff, equipment, salaries and expenses for field technicians to build it. ...

Going back to my first point, almost no one is unwilling to pay for their internet connection.

They're unwilling to pay less to get TV service they have no use for and internet than they are internet alone.

They're unwilling to look at the cost of network access in other countries and be happy when comparing that to the cost of internet in a major, densely populated area in the US.

They're unwilling to accept transfer caps and accept the lies they they help the network by making people share when they know that transfer is irrelevant and bandwidth is the limited commodity.

They're unwilling to accept lies of bandwidth speed that isn't attainable even under the best of conditions.

They're unwilling to accept shady and underhanded business practices such as random, "accidental" charges.

> It just needs to be channeled the right way so that they can figure out what it actually costs to run a reliable ISP. I'm all in favor of new startup ISPs.

However, as an "actual ISP network engineer", you must understand the (very arguably unfair) regulatory issues that the big players have put in place to make this impossible to do. Google failed at starting an ISP, and it's not like they are tight on either cash or lawyers.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by Spooky23 on Wednesday May 02, 2018 @05:24PM (#16981278)
It’s pretty hilarious to hear a cable network engineer get on a soapbox about reliability.

As an enterprise customer, I only get 99% commitment, and frequently get credits. They’ve improved it in the last 5 years, but our whole region had an outage when the single fiber connecting to their core routers was cut by a house fire.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by adrianN on Thursday May 03, 2018 @12:47AM (#16983416)
I'm no network engineer, but when you have a large network with lots of equipment, you want the individual components to have a couple more nines than the resulting network. Maybe that's what they meant.
Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by walrus01 on Wednesday May 02, 2018 @05:00PM (#16981118)
Google did not fail at starting an ISP. They decided that the ROI for doing individual residential home FTTH was not worth it, considering the cost per house passed. Then they went and bought Webpass, which has a totally different condo/apartment based business model, which has been wildly successful.

Starting a small ISP based out of extreme dissatisfaction with huge, evil incumbents is something that I encourage any plucky upstart to do. If you don't like Comcast, Charter, RCN, etc, if you have sufficient clue, go do it yourself. Just don't try to do it with $20,000 and some good wishes, because something with much deeper pockets like Webpass will come along and obliterate you.

The point that I am rather bluntly trying to get across to these folks is that if they're going to invest significant amounts of their personal time/engineering resources into it (unpaid labor?), that they should treat it like a serious business and not a nonprofit. Raise sufficient funds, somewhere in the $500,000+ range to start, to build a network that actually does meet five nines through sufficient diversity in topology and routing.

Compared to random YC companies that seem to raise $1m to $5m to do some vaguely defined web2.0 SAAS thing, the amount of money that a group of people need to raise to build the core of a serious ISP is not huge. Elsewhere in this thread I threw out a figure like $750,000 in beginning CapEx if they're serious. For some good reasons, however, it's easier to get $5m of series A funding from a VC for a pure software development business than a wholly-owned-facilities-based startup ISP.

Even if you completely ignore things like colocation costs, IP transit costs, IX/peering costs, office rental, office utilities and overhead, running a serious ISP with live, clued in humans watching the network takes a lot of money per year for salaries.

3 to 5 NOC persons for 24x7x365 NOC phone coverage. At a certain point if you peer with the big boy ASNs they will expect a live human to answer the phone at 3am when something unusual happens in traffic between your mutual networks. That live human needs to be somebody who has the equivalent of 'enable' on your peering router and sufficient clue to wield it.

2 network engineering/architecture positions, multiply by expected fully loaded yearly salary and benefits package for each person.

at least 2 field technicians

vehicles, vehicle insurance, tools for technicians

At NYC salaries? In a greenfield scenario, I could easily spend $900,000 a year in just payroll and benefits, just to begin to attempt to duplicate what some other serious ISPs already have in place for manpower.

And again that is before you spend one dollar on equipment.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by danShumway on Wednesday May 02, 2018 @05:43PM (#16981412)
> At NYC salaries? In a greenfield scenario, I could easily spend $900,000 a year in just payroll and benefits

I kind of don't understand what you're getting at here. They're a nonprofit. The majority of their work is coming from volunteers. They aren't paying New York salaries.

This is like going to Linus and saying "You can't possibly expect Linux to be sustainable - do you have any idea how much money Microsoft pays its developers?"

Nonprofit/open models aren't less work than commercial ventures; the difference is that people and companies donate their time/money because they want the project to succeed. If they end up with a competitive enough product, then businesses will start sponsoring more development, because it'll be a cheap alternative to dealing with Comcast.

At that point, large businesses operating in the city will pay/sponsor New York salaries to maintain the parts of the project they find useful. Exactly like every other Open Source or community run project on the planet.

I have plenty of doubts about volunteer mesh networks, but I'm not so skeptical to say they literally can't work. I think you're looking at one particular model of the market and assuming that anything deviating from that model is broken, rather than just targeting different segments or trying a different strategy or even just accomplishing different goals.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by smallnamespace on Wednesday May 02, 2018 @06:32PM (#16981734)
> The majority of their work is coming from volunteers.

> the difference is that people and companies donate their time/money because they want the project to succeed.

And that volunteering time and energy are probably valuable then, especially if they can do the work of a real ISP.

I think one real question to ask though is what is actually more efficient—donating volunteer time and energy to a nonprofit, or donating money and having that nonprofit hire professionals instead?

GP doesn't sound like he is saying a volunteer mesh network literally cannot work, just pointing out that a network is inherently expensive to run, whether it is 'fully accounted for' on a payroll, or merely hidden in the cost of 'free' volunteering effort. It's an open question of whether asking for people's time or their money is better in the long run. Capitalism tends to suppose the latter.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by danShumway on Wednesday May 02, 2018 @07:25PM (#16982040)
> I think one real question to ask though is what is actually more efficient—donating volunteer time and energy to a nonprofit, or donating money and having that nonprofit hire professionals instead?

That's a good question. Judging from software development (which may or may not map perfectly to hardware infrastructure) both seem to work pretty well. Depending on who you ask you'll get different answers about whether closed (purely paid) or open (either volunteer or mixed) is "winning".

But certainly they seem to both work, on the whole. Neither strategy is so radically inefficient that it's safe to dismiss a company that relies on one over the other.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by pasbesoin on Wednesday May 02, 2018 @08:43PM (#16982410)
The incumbents are really good at suppressing competition.

One response/workaround: Guerrilla networking.

Why, in part? Because there's nothing much to target, financially. In some ways, legally. Instead, the incumbents have to elicit suppression through regulators and law enforcement, if they can.

Google has deep pockets. But it could spend years trying to get access to poles. Incumbents don't just pay to oppose; they pay to slow things down as much as possible. Good luck deploying, when you have to wait 6 months for a court case, that's then going to get appealed.

Stay under the radar. That's a whole different approach.

And communities are getting desperate enough to try anything that might actually work.

While I cheer them on, I simultaneously hope the press coverage doesn't bring the corporate mercenaries down on their heads.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by paulific on Thursday May 03, 2018 @12:37AM (#16983380)
Reminds me of the Corey Doctorow/Charles Stross short story Unwirer: https://craphound.com/unwirer/
Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by Retric on Wednesday May 02, 2018 @07:22PM (#16982023)
FTE can work a few extra hours a week at a non profit more easily than they can make more money at their job and donate that. They may also enjoy their time and learn valuable skills which make the time vs money argument misleading, because for most tech workers they are very different pools.
Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by nycmesh on Thursday May 03, 2018 @01:16AM (#16983516)
Volunteers donate their time because they enjoy it. We have so many volunteers it is hard to keep track of them all. Obviously it is more efficient to use volunteers if there are enough. We are considering using contractors for some huge projects we are planning.
Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by pdkl95 on Wednesday May 02, 2018 @05:56PM (#16981502)
> they should treat it like a serious business and not a nonprofit.

A non-profit organization can be as "serious" as any for-profit business. The only difference is the non-profit might, for example, be required by their charter to re-invest income back into the business instead of paying it out as profit to the owners/shareholders/etc.

The level of "seriousness" of an organization should be determined by their business plan (is it profitable? And sustainable?), not their profit motive or financial footprint.

> if they're going to invest significant amounts of their personal time/engineering resources

First, it's their time and resources to spend as they wish.

Additionally, a suggestion I've heard is that pursuing anything that requires a serious investment of your time and resources (e.g. a job/internship, school, non-paid opportunities for "exposure") is worth pursuing if and only if it at lest two of these are true: [pays well/profitable, will improve future opportunities, you really enjoy it]. (If you love doing something and you're learning a lot or gaining experience that looks great on a CV, it can be worth a serious pay cut)

> build a network that actually does meet five nines through sufficient diversity in topology and routing

That is a good goal, but it isn't strictly necessary at first. Limit growth to what is possible at the time, and grow into a larger organization if it's successful.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by Retric on Wednesday May 02, 2018 @05:21PM (#16981252)
Google fiber had over 450,000 customers and was thus several times Webpass size when they acquired them. They have continually grown over time and by any reasonable measure very successful.
Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by jimktrains2 on Wednesday May 02, 2018 @05:27PM (#16981296)
> Google Fiber to cut jobs and halt expansion of US internet service

> Ambitious – and expensive – high-speed internet program will stop plans to expand to new cities as business reportedly under pressure to cut costs

https://www.theguardian.com/technology/2016/oct/26/google-fi...

Although it looks like they've started expanding again as per https://www.techrepublic.com/article/the-rise-and-fall-and-r.... So perhaps I counted them out too early. However, even with the backing of Google their success isn't guaranteed and they've had many non-technical, not-purely-financial problems along the way.

I still think it's a good example of why a "plucky upstart [with VC funding]" isn't able to waltz in and build a large, traditional ISP in the current environment.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by Retric on Wednesday May 02, 2018 @07:12PM (#16981969)
There are zero nationwide high speed ISP's. So, it's kind of a question of how do you measure success.

Yea, google paused expansion for ~3 months but building a nationwide network is a ~100 billion dollar investment. The only option to get there quickly is a LEO satellite constellation, otherwise it's going to be a really long slog with lot's of local governments getting involved.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by jimktrains2 on Wednesday May 02, 2018 @07:52PM (#16982177)
I hadn't heard they started expanding again, hence my not changed original stance.
Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by jimktrains2 on Wednesday May 02, 2018 @05:19PM (#16981241)
> Google did not fail at starting an ISP. They decided that the ROI for doing individual residential home FTTH was not worth it, considering the cost per house passed.

~Kind of sounds like it failed.~ Apparently Google Fiber is starting to expand again. I still think it serves as a good case against why the current climate makes it near-impossible for a "plucky upstart [with vc funding]" to start a large, traditional ISP.

> Starting a small ISP based out of extreme dissatisfaction with huge, evil incumbents is something that I encourage any plucky upstart to do. ... Just don't try to do it with $20,000 and some good wishes, because something with much deeper pockets like Webpass will come along and obliterate you.

You can't be a plucky upstart while simultaneously having pockets deep enough to fend against the current major ISP players.

I'm not going to sit here and reply line-by-line to your post. The issue isn't that people want a free network. The issue is that people are fed up with the large ISPs and that the large ISPs have made it all-but-impossible to start an ISP.

You seem to think that VC-sized money is the only thing people need to enter the market. You're entirely wrong. Entrenched players have made it next-to-impossible to compete in terms of capital and regulatory hurdles.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by rayiner on Thursday May 03, 2018 @12:38AM (#16983386)
Google is just a terrible example. On net, Google got incredibly preferential regulatory treatment. https://arstechnica.com/tech-policy/2012/09/how-kansas-city-...

They just realized it makes no sense for them to be in a low ROI industry: https://www.mckinsey.com/~/media/McKinsey/Business%20Functio...

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by walrus01 on Wednesday May 02, 2018 @05:23PM (#16981274)
> The issue is that people are fed up with the large ISPs and that the large ISPs have made it all-but-impossible to start an ISP.

that's the thing, they haven't. Not in the service territory this new nonprofit is operating in. There's something like 14 different ISPs which overlap service area with them in their part of New York City, none of which are big nationwide Comcast or Centurylink sized entities.

What they're trying to do on a shoestring is on super hard mode, they've picked the single most competitive market in the entire USA to try to compete with independent, non-LEC, non-major-telco companies that do have a few million dollars to throw at their own facilities-based network infrastructure.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by jimktrains2 on Wednesday May 02, 2018 @05:35PM (#16981357)
NYC, and potentially a few other urban areas, have the capability to run fiber easily.

I'm also just not sure why you're acting like this. You're being simply dismissive of the project. There are many successful meshnets.

You've yet to give a specific complaint about this project and have simply ranted about how they're just small, underfunded, and not a real ISP without ever actually substantiating any of that or specifically what they're doing wrong.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by walrus01 on Wednesday May 02, 2018 @09:33PM (#16982664)
NYC is actually the textbook example of most expensive places to run new fiber, due to needs for nearly 100% underground, and extensive pre existing utilities. I'm going to guess you've never been involved with an outside plant fiber project.
Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by stephengillie on Wednesday May 02, 2018 @11:54PM (#16983244)
With your knowledge of the subject, how would you suggest they overcome this challenge?

Do they need to run their own fiber - is there a way they could rent fiber space from an existing line? At least for a little while?

Don't be worried that a new player has entered the space; be worried that the existing ones haven't improved the space. The total market space can be increased by creating new connections, and someone has to do it.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by nycmesh on Thursday May 03, 2018 @12:47AM (#16983415)
There are about 10 fiber providers in the city. We've spoken to most of them. It's pretty much impossible for us to lay fiber, so we have to lease it. We'd like to eventually lease a fiber ring across the city so we can cut costs when connecting multiple buildings. If you don't have a ring then you have to pay for the run all the way back to a coloc.
Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by stephengillie on Thursday May 03, 2018 @11:23AM (#16986815)
That's a good start to having "your own" network, and then you can choose to have packets traverse your network or the peering networks. You could shift egress points around, based on cost and other factors as well. Each link you add can multiply your routing options.

Get large enough, and you can start carrying other traffic too, and eventually start buying/laying fiber. But you know this, it's probably part of your business plan.

Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by dzonga on Thursday May 03, 2018 @01:19AM (#16983525)
African Internet is really slow. Being a noob and knowing we can access the undersea cable. How would you go about building such a mesh network for an medium densely populate african city i.e 3+ million people ?
Re:'Anti-authority' tech rebels take on ISPs, connect NYC wi... by rayiner on Thursday May 03, 2018 @12:34AM (#16983373)
Do you have any experience with either ISP level network engineering or regulation? Because you speak awfully authoritatively.