Our uptime when connecting to supernode1 is significantly better than Time Warner Cable as many of our members have reported.
We are already established. This is something we have already done! It isn't impossible.
Our IXP is DE-CIX, the worlds biggest IXP. We are currently in one data-center/IXP and talking to a few more colocs.
Yep, DE-CIX is great and 32 Ave of the Americas is one of the most wired buildings in NYC. Great choice. Kudos on you folks, this is a great initiative. Reading this article made me happy. And I'm glad you are getting some press!
Same answer.
They are cheap as hell and awesome.
Actually - what you should do, is have a customer pay a fee of $100 to connect to the system - and take that $100 and buy another mesh was with each customer on boarding.
You announce a whopping total of two v4 /24.
Kudos for having enough clue to know that you needed to establish a presence at a major IX, and actually doing it, because it looks like you're adding peers. But your actual network presence is minuscule.
How do you intend to compete with the six NYC based companies I can think of off the top of my head that are putting fiber fed, $9,000 to $20,000 5 to 10Gbps 71-86 GHz PTP links on rooftops to build their own backbones, when you're playing around with 1000BaseT to the roof and AF24s?
You have high uptime? Do you have any of the following, because some of your much larger competitors sure do:
sites with parallel A and B side power systems
-48VDC rectifier + battery systems sized for 24 hour runtime at load
diesel generators
propane generators
generator resupply contracts
chassis-based routers with hotswap fan, N+1 power supplies, dual redundant routing engines
pair of identical core routers
singlemode fiber to the roof
-48vdc power to the roof
ironclad rooftop lease agreements with building owners, drawn up by professional telecom/real estate lawyers that run for 5+ year terms
a 24x7x365 NOC staffed by live humans
This follow up comes across as though you've drawn a conclusion and are now arguing towards that predetermined conclusion while ignoring the additional information provided. Plus your argument has kind of devolved from talking about specific concerns to throwing criticisms at the wall to see what sticks.
You don't seem to be arguing in good faith.
FOSS zealots like to point to the success of Linux over Windows (which is restricted to the server market, I might add), but there's little evidence that the FOSS philosophy is effective when it comes to physical infrastructure. Software requires bytes and labor. Bytes are cheap and labor can be donated or paid for by companies. Hardware requires fabs and factories, which are expensive.
So I guess the points you brought up don't matter.
We compete on price (donation only)
What is your problem??
What ISP are you affiliated with?
So then you must intimately understand the depth of the lies and anti-consumer business practices that spur such networks.
> Real five nines network infrastructure takes significant capital investment.
5-9s is 5 minutes of downtime a year. I've never had a commercial, let alone residential ISP hit that mark.
> Actual, reliable internet costs real money, both in the salaries of people to engineer and architect it, salaries for 24x7x365 clued-in NOC staff, equipment, salaries and expenses for field technicians to build it. ...
Going back to my first point, almost no one is unwilling to pay for their internet connection.
They're unwilling to pay less to get TV service they have no use for and internet than they are internet alone.
They're unwilling to look at the cost of network access in other countries and be happy when comparing that to the cost of internet in a major, densely populated area in the US.
They're unwilling to accept transfer caps and accept the lies they they help the network by making people share when they know that transfer is irrelevant and bandwidth is the limited commodity.
They're unwilling to accept lies of bandwidth speed that isn't attainable even under the best of conditions.
They're unwilling to accept shady and underhanded business practices such as random, "accidental" charges.
> It just needs to be channeled the right way so that they can figure out what it actually costs to run a reliable ISP. I'm all in favor of new startup ISPs.
However, as an "actual ISP network engineer", you must understand the (very arguably unfair) regulatory issues that the big players have put in place to make this impossible to do. Google failed at starting an ISP, and it's not like they are tight on either cash or lawyers.
As an enterprise customer, I only get 99% commitment, and frequently get credits. They’ve improved it in the last 5 years, but our whole region had an outage when the single fiber connecting to their core routers was cut by a house fire.
Starting a small ISP based out of extreme dissatisfaction with huge, evil incumbents is something that I encourage any plucky upstart to do. If you don't like Comcast, Charter, RCN, etc, if you have sufficient clue, go do it yourself. Just don't try to do it with $20,000 and some good wishes, because something with much deeper pockets like Webpass will come along and obliterate you.
The point that I am rather bluntly trying to get across to these folks is that if they're going to invest significant amounts of their personal time/engineering resources into it (unpaid labor?), that they should treat it like a serious business and not a nonprofit. Raise sufficient funds, somewhere in the $500,000+ range to start, to build a network that actually does meet five nines through sufficient diversity in topology and routing.
Compared to random YC companies that seem to raise $1m to $5m to do some vaguely defined web2.0 SAAS thing, the amount of money that a group of people need to raise to build the core of a serious ISP is not huge. Elsewhere in this thread I threw out a figure like $750,000 in beginning CapEx if they're serious. For some good reasons, however, it's easier to get $5m of series A funding from a VC for a pure software development business than a wholly-owned-facilities-based startup ISP.
Even if you completely ignore things like colocation costs, IP transit costs, IX/peering costs, office rental, office utilities and overhead, running a serious ISP with live, clued in humans watching the network takes a lot of money per year for salaries.
3 to 5 NOC persons for 24x7x365 NOC phone coverage. At a certain point if you peer with the big boy ASNs they will expect a live human to answer the phone at 3am when something unusual happens in traffic between your mutual networks. That live human needs to be somebody who has the equivalent of 'enable' on your peering router and sufficient clue to wield it.
2 network engineering/architecture positions, multiply by expected fully loaded yearly salary and benefits package for each person.
at least 2 field technicians
vehicles, vehicle insurance, tools for technicians
At NYC salaries? In a greenfield scenario, I could easily spend $900,000 a year in just payroll and benefits, just to begin to attempt to duplicate what some other serious ISPs already have in place for manpower.
And again that is before you spend one dollar on equipment.
I kind of don't understand what you're getting at here. They're a nonprofit. The majority of their work is coming from volunteers. They aren't paying New York salaries.
This is like going to Linus and saying "You can't possibly expect Linux to be sustainable - do you have any idea how much money Microsoft pays its developers?"
Nonprofit/open models aren't less work than commercial ventures; the difference is that people and companies donate their time/money because they want the project to succeed. If they end up with a competitive enough product, then businesses will start sponsoring more development, because it'll be a cheap alternative to dealing with Comcast.
At that point, large businesses operating in the city will pay/sponsor New York salaries to maintain the parts of the project they find useful. Exactly like every other Open Source or community run project on the planet.
I have plenty of doubts about volunteer mesh networks, but I'm not so skeptical to say they literally can't work. I think you're looking at one particular model of the market and assuming that anything deviating from that model is broken, rather than just targeting different segments or trying a different strategy or even just accomplishing different goals.
> the difference is that people and companies donate their time/money because they want the project to succeed.
And that volunteering time and energy are probably valuable then, especially if they can do the work of a real ISP.
I think one real question to ask though is what is actually more efficient—donating volunteer time and energy to a nonprofit, or donating money and having that nonprofit hire professionals instead?
GP doesn't sound like he is saying a volunteer mesh network literally cannot work, just pointing out that a network is inherently expensive to run, whether it is 'fully accounted for' on a payroll, or merely hidden in the cost of 'free' volunteering effort. It's an open question of whether asking for people's time or their money is better in the long run. Capitalism tends to suppose the latter.
That's a good question. Judging from software development (which may or may not map perfectly to hardware infrastructure) both seem to work pretty well. Depending on who you ask you'll get different answers about whether closed (purely paid) or open (either volunteer or mixed) is "winning".
But certainly they seem to both work, on the whole. Neither strategy is so radically inefficient that it's safe to dismiss a company that relies on one over the other.
One response/workaround: Guerrilla networking.
Why, in part? Because there's nothing much to target, financially. In some ways, legally. Instead, the incumbents have to elicit suppression through regulators and law enforcement, if they can.
Google has deep pockets. But it could spend years trying to get access to poles. Incumbents don't just pay to oppose; they pay to slow things down as much as possible. Good luck deploying, when you have to wait 6 months for a court case, that's then going to get appealed.
Stay under the radar. That's a whole different approach.
And communities are getting desperate enough to try anything that might actually work.
While I cheer them on, I simultaneously hope the press coverage doesn't bring the corporate mercenaries down on their heads.
A non-profit organization can be as "serious" as any for-profit business. The only difference is the non-profit might, for example, be required by their charter to re-invest income back into the business instead of paying it out as profit to the owners/shareholders/etc.
The level of "seriousness" of an organization should be determined by their business plan (is it profitable? And sustainable?), not their profit motive or financial footprint.
> if they're going to invest significant amounts of their personal time/engineering resources
First, it's their time and resources to spend as they wish.
Additionally, a suggestion I've heard is that pursuing anything that requires a serious investment of your time and resources (e.g. a job/internship, school, non-paid opportunities for "exposure") is worth pursuing if and only if it at lest two of these are true: [pays well/profitable, will improve future opportunities, you really enjoy it]. (If you love doing something and you're learning a lot or gaining experience that looks great on a CV, it can be worth a serious pay cut)
> build a network that actually does meet five nines through sufficient diversity in topology and routing
That is a good goal, but it isn't strictly necessary at first. Limit growth to what is possible at the time, and grow into a larger organization if it's successful.
> Ambitious – and expensive – high-speed internet program will stop plans to expand to new cities as business reportedly under pressure to cut costs
https://www.theguardian.com/technology/2016/oct/26/google-fi...
Although it looks like they've started expanding again as per https://www.techrepublic.com/article/the-rise-and-fall-and-r.... So perhaps I counted them out too early. However, even with the backing of Google their success isn't guaranteed and they've had many non-technical, not-purely-financial problems along the way.
I still think it's a good example of why a "plucky upstart [with VC funding]" isn't able to waltz in and build a large, traditional ISP in the current environment.
Yea, google paused expansion for ~3 months but building a nationwide network is a ~100 billion dollar investment. The only option to get there quickly is a LEO satellite constellation, otherwise it's going to be a really long slog with lot's of local governments getting involved.
~Kind of sounds like it failed.~ Apparently Google Fiber is starting to expand again. I still think it serves as a good case against why the current climate makes it near-impossible for a "plucky upstart [with vc funding]" to start a large, traditional ISP.
> Starting a small ISP based out of extreme dissatisfaction with huge, evil incumbents is something that I encourage any plucky upstart to do. ... Just don't try to do it with $20,000 and some good wishes, because something with much deeper pockets like Webpass will come along and obliterate you.
You can't be a plucky upstart while simultaneously having pockets deep enough to fend against the current major ISP players.
I'm not going to sit here and reply line-by-line to your post. The issue isn't that people want a free network. The issue is that people are fed up with the large ISPs and that the large ISPs have made it all-but-impossible to start an ISP.
You seem to think that VC-sized money is the only thing people need to enter the market. You're entirely wrong. Entrenched players have made it next-to-impossible to compete in terms of capital and regulatory hurdles.
They just realized it makes no sense for them to be in a low ROI industry: https://www.mckinsey.com/~/media/McKinsey/Business%20Functio...
that's the thing, they haven't. Not in the service territory this new nonprofit is operating in. There's something like 14 different ISPs which overlap service area with them in their part of New York City, none of which are big nationwide Comcast or Centurylink sized entities.
What they're trying to do on a shoestring is on super hard mode, they've picked the single most competitive market in the entire USA to try to compete with independent, non-LEC, non-major-telco companies that do have a few million dollars to throw at their own facilities-based network infrastructure.
I'm also just not sure why you're acting like this. You're being simply dismissive of the project. There are many successful meshnets.
You've yet to give a specific complaint about this project and have simply ranted about how they're just small, underfunded, and not a real ISP without ever actually substantiating any of that or specifically what they're doing wrong.
Do they need to run their own fiber - is there a way they could rent fiber space from an existing line? At least for a little while?
Don't be worried that a new player has entered the space; be worried that the existing ones haven't improved the space. The total market space can be increased by creating new connections, and someone has to do it.
Get large enough, and you can start carrying other traffic too, and eventually start buying/laying fiber. But you know this, it's probably part of your business plan.
And that's true even assuming just bare costs without markups. Obviously when you take into account how much profit & graft is attached to every single level of that stack, it really starts to add up. Throw in the absolutely, inexcusably bad, awful experience nearly everyone has with the big telecom companies and their employees, and that's enough ill will to propel many an alternative DIY network.
Metaphorically speaking, you just told a bicycle owner what a terrible car he has. Or rather, warned him how much the car is going to cost that he doesn't want in the first place. Is it possible the bicyclist can actually tell the difference between a bike and a car? Maybe he even sees some advantage you don't? (As opposed to the implausible but apparently favored hypothesis by car-owners that no, he just likes punishing himself while unsuccessfully pretending the bike is a car. When we don't know a stranger's rationale we glibly tend to assume there is none, and they're just cray-cray.)
To make yet another metaphor, when all you know is hammers, the whole world looks like it's full of 1) nails, 2) hammers, and 3) things that suck at being hammers.
A phone network that objectively sucked compared to AT&T and Verizon but was boatloads cheaper seems to have worked out just fine for T-Mobile. I can see the same thing working for this. $20/mo for an internet connection that works most of the time would have plenty of customers. I'm not familiar with the wireless tech involved so no clue how well this scales outside of high density cities but within some place like NYC there's plenty of people that would pay for this over $50+/mo (min!) to the local cable companies.
>"Actual ISP network engineer here: I'm sure these people are having fun and all, but it's insignificant in the larger scheme of things"
No it is actually very significant. In a past life I was also an "Actual ISP network engineer" at a large ISP. And what I can say is that large ISPs aren't exactly wellsprings of innovation. However these folks are hacking on things, solving an actual problem and seem to be having fun in the process. It's exactly this ethos that usually ends up moving the needle. Sometimes by accident. It's curious that you are so keen to poo-pooh them.
>"Real five nines network infrastructure takes significant capital investment"
There is no ISP that provides customers 5 9's of uptime. Not even for business customers. With a Time Warner or Comcast you are lucky if you get 3 9's from their network. This is what you get from "the duopoly."
>"And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each (do you really want to deploy something in the year 2018 to take several full BGP tables that doesn't have a 4 million FIB capacity?), etc."
There a clue in the article about which IXP they're - 32 Ave of the Americas(DE-CIX.) It's one of the 3 of the most "wired" buildings in NYC. You can peer with everyone there. These people seem to have done their homework. And you don't need to buy $15K Cisco ASR any more. You can take full tables on commodity hardware running Linux and ExaBGP or BIRD. There are many CDNs and IXP's doing just that.
I imagine consumers would be fine with 95% uptime of a gigabit connection compared to 99,99999% uptime of 16mbit.
I can guarantee that nobody would be okay with 95% uptime. That means that your Internet access is down for over an hour every day. Given that people go berserk when their Internet is down for even a few minutes, 95% uptime will be unacceptable to consumers.
(Actually, if I were an evil ISP lawyer/marketer, I'd try to claim a "Maintenance window of 1hr downtime sometime between 4am and 3am", then ignore any downtime of less than an hour during the day and if you complained about not getting yur expensive five nines I'd laugh and say "published schedule maintenance not included!"...)
"Real five nines network infrastructure takes significant capital investment."
No, it doesn't. Overly complex, corporatized infrastructures designed by committees in meetings - that takes significant capital investment. Simple infrastructure designed by hackers can achieve bullshit metrics like "five nines" without breaking the bank.
"And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each"
Establishing colo at a "major IX point" is a few emails and phonecalls away. Your salesperson will fall all over him or herself and waive the NRCs to get you in there. It doesn't cost much, even for a full rack with 20 or 30 amps. If you're just doing networking, you won't even need that. Then you get a fiber cross connect to he.net for 10 or 20 or 40 gbps and you're in business.
You can buy very, very nice used routers and since you're a hacker and not Mr. Enterprisey IT guy, you can run it and support it yourself.
We had many iterations of this exact critique in our early days - explaining how we couldn't possibly deploy this volume of storage without fancy bullshit enterprise solutions from EMC or Compaq and how we couldn't possibly run a global storage network without fancy routers (we currently own zero routers) and fantastically complex clustering and failover ... and 17 years later we're still making it all work just fine.
Fuck those people. Fuck those school trained, passionless, non-hacker, no-talent-pussies.
Good luck with your mesh network!
---
> In the mid-90s, I got a call from some friends at ATT, asking me to help them research the nascent web-hosting business. They thought ATT’s famous “five 9’s” reliability (services that work 99.999% of the time) would be valuable, but they couldn’t figure out how $20 a month, then the going rate, could cover the costs for good web hosting, much less leave a profit.
...
> The ATT guys had correctly understood that the income from $20-a-month customers wouldn’t pay for good web hosting. What they hadn’t understood, were in fact professionally incapable of understanding, was that the industry solution, circa 1996, was to offer hosting that wasn’t very good.
...
> For a century, ATT’s culture had prized—insisted on—quality of service; they ran their own power grid to keep the dial-tone humming during blackouts. ATT, like most organizations, could not be good at the thing it was good at and good at the opposite thing at the same time. The web hosting business, because it followed the “Simplicity first, quality later” model, didn’t just present a new market, it required new cultural imperatives.
Oh, and they'd need peering: more $$$$$
I don't think this is a helpful reply.
Since it sounds like you're associated with the project, can you address some of the specific concerns mentioned, like reliability? Do you disagree that hundreds of thousands of dollars are required for five nines connectivity? Do you disagree that that level of reliability is required for your customers etc.
Do you know what the cost per user is currently?
What frequency do the point-to-point links run at? Have you considered using some sort of lower frequency packet radio for longer distance links, perhaps to other nearby nodes?
DE-CIX, our IXP, donated bandwidth to us, and we have transit also donated from Packet Host and WebAir. We actually pay nothing for bandwidth. Probably one day we will pay but it is not that expensive if you do this at an IXP and use peering.
We have an AirFiber pair that is 24Ghz, but all of our sectors are 5Ghz wifi.
What is the incentive for them to donate bandwidth/transit to you? Your non-profit status? Do they get something else in return?
DE-CIX is the biggest IXP and they measure their network in terabits/sec. It is not a big deal for them to donate us a 1 gig connection. (it's a big deal for us!) Eventually we may upgrade it to 10 gig.
How often do you see full saturation on that link? I'm guessing if all 600 of your users all tried to stream one stream of $TV-show-de-jour at once they'd barely get 16Mbps each? Can one user's home office full of Apple gear flood the bandwidth downloading a dozen Apple software updates simultaneously?
Cable companies consider these high ratios to be quite embarrassing actually, as it shows how crummy their networks are: https://www.dslreports.com/forum/r31258251-Speed-Issues-Char...
We've never come anywhere near saturating our gig connection. We monitor it. Each building is limited to their own rooftop connection which is an average 100Mbps, so they can't do more than that if they tried. The thing is not everyone is downloading a file at the same time, and streaming uses much less bandwidth than downloading. From Netflix site-
0.5 Megabits per second - Required broadband connection speed
1.5 Megabits per second - Recommended broadband connection speed
3.0 Megabits per second - Recommended for SD quality
5.0 Megabits per second - Recommended for HD quality
25 Megabits per second - Recommended for Ultra HD quality
I have to assume another cost that's not noted is roof rights. That's not free - most people either have to pay or give the property management a kickback.
NYC is an interesting place for this - if you live in Manhattan, you clearly can afford the $80 for 1Gb/s FiOS or $40 for 100Mb/s FiOS or whatever Spectrum is charging. It would be way more interesting to plop this in a rust-belt city where people are on a paycheck-to-paycheck salary...
Not a single one of them has ever signed up for anything more expensive than three nines. Only a few have even discussed the differences between 3 and 4 nines solutions.
I'm reasonably sure I'd choose the same for my home internet connection - if offered representative pricing based on costs of providing 3, 4, or 5 nines, or possibly even 2 nines - I'd choose the least expensive because losing a few minutes or even occasionally 15mins a day of connectivity at home (or maybe more realistically an ~8 hour outage per month) really isn't going to bug me greatly - not if it's two or more orders of magnitude cheaper than a four of five nines connection.
(correction to original post: I did not at first see an ASN for them, or announcements of their own IP space. They have an ASN and announce two /24s, and peer at one physical location in metro NYC).
Real five nines network infrastructure takes significant capital investment.
This part is so much unadulerated bullshit:
""The internet doesn't really cost you anything, it's just the connection [that has a fee]. So however you can get plugged in — then you're on the internet. Nobody owns the internet, there's no one to pay.""
Actual, reliable internet costs real money, both in the salaries of people to engineer and architect it, salaries for 24x7x365 clued-in NOC staff, equipment, salaries and expenses for field technicians to build it. And that's before you get into things like establishing colo at major IX points, serious core routers that cost $15k each (do you really want to deploy something in the year 2018 to take several full BGP tables that doesn't have a 4 million FIB capacity?), etc. The Internet is a significant construction project at OSI layer 1, whether you're putting PTP radios on rooftops, running aerial fiber, or underground fiber. Otherwise you're just piggybacking on something that another, larger entity has already built.
I give them an A for enthusiasm and effort. It just needs to be channeled the right way so that they can figure out what it actually costs to run a reliable ISP. I'm all in favor of new startup ISPs.