News for hackers. Stuff that matters.
Re:Twitter ‘smytes’ customers by jarfil on Friday June 22, 2018 @12:21AM (#17371279)
"trust and safety as a service"

So... what was the Service Level Agreement? and the Terms Of Service?

People get all hyped by -aaS stuff, but services do go down, and if you aren't the one controlling when and how they will go up, then you better have a contract with the Service Provider on some terms you like.

Even then, prepare for when they go down, because they will sooner or later, for shorter or longer periods of time.

I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.

Re:Twitter ‘smytes’ customers by pvg on Friday June 22, 2018 @12:34AM (#17371336)
I find it silly to rely on some service, any external service

This is always trotted out and it's, by this point, a completely content-free thing to say. Short of removing yourself from the economy and society in general to live off potatoes grown in your own dung, you're going to rely on some external services and the fact that, for the most part, the parties you do business with will behave responsibly. When they behave very irresponsibly, unpleasant things happen and people quite reasonably complain.

Re:Twitter ‘smytes’ customers by bastawhiz on Friday June 22, 2018 @01:06AM (#17371485)
Exactly. You want reliable CAPTCHAs? CDN service? Malware detection? Customer support software? Website uptime monitoring? Accounting software? Analytics? Email delivery? Good luck building and running all that yourself.
Re:Twitter ‘smytes’ customers by im3w1l on Friday June 22, 2018 @01:13AM (#17371511)
All of those can be bought and then self hosted except for CDN.
Re:Twitter ‘smytes’ customers by laumars on Friday June 22, 2018 @05:31AM (#17372659)
I don't know why you're getting downvoted because you're absolutely right. Perhaps most of the people in this thread haven't managed servers for more than a decade - because it used to be (and still is in some organisations) the norm to self-host half the stuff that was listed.

  * Malware detection?
Isn't that pretty must how most malware detection works? Sure they use the cloud to pull periodic updates but the software itself is self hosted

  * Customer support software?
I've lost count of the number of solutions available to self host. Both free and commercial.

  * Website uptime monitoring?
While there is a strong argument for hosting that in the cloud, there are umpteen solutions for website monitoring.

  * Accounting software?
I didn't even realise running accounting software as a service was now a thing!

  * Email delivery?
Fair enough, self hosting email can be a massive headache. Less so if you run Microsoft Exchange (one of the few things Windows actually makes easier than it's Linux / UNIX counterparts). You can even go for a hybrid approach and self host the mail server but have a mail proxy between your server and the outside world - so you benefit from spam detection as a service while still hosting your own mail.

I'm not going to argue that one is better than the other when it comes to SaaS vs self hosting. But people in here seem to have very short memories when arguing that self-hosting isn't at all viable. In fact I have personally supported all of the above in self hosted versions in the last couple of years.

Re:Twitter ‘smytes’ customers by tomhoward on Friday June 22, 2018 @06:53AM (#17372989)
I don't think anyone is disputing that it's possible, but rather how cost-effective and time-efficient it is [1].

I also have the experience of managing many of these services, and I therefore know just how much more productive you can be when you don't need to do it yourself.

Of course there are trade-offs; business is nothing if not a game of tradeoffs. We're just talking about what is optimal.

[1] Not to mention how much more stable and secure; Self-hosting can involve huge security and reliability risks.

Re:Twitter ‘smytes’ customers by laumars on Friday June 22, 2018 @07:06AM (#17373040)
> We're just talking about what is optimal.

I don't believe what is optimal can be generalised like that. "Optimal" is going to be specific to the business in question. Which is why I'm a firm believer of leaving all the options on the table.

> Self-hosting can involve huge security and reliability risks.

In fairness so can the cloud. I personally don't see difference as 'risk' but more 'responsibility'. Some of the places I've worked have been PCI DSS and Gambling Commission audited so we had that responsibility already. Self hosting a few other resources didn't really add much extra in terms of our security responsibilities. But the case would be very different of lots of other different types of businesses.

Re:Twitter ‘smytes’ customers by tomhoward on Friday June 22, 2018 @07:37AM (#17373169)
I think we're agreed that one shouldn't be absolutist about these things and that different approaches will be optimal depending on the circumstances.

It's worth reminding ourselves that the root comment was an absolutist assertion about the folly of outsourcing.

So the discussion is easily resolved: don't be absolutist :)

Re:Twitter ‘smytes’ customers by mmt on Friday June 22, 2018 @09:14PM (#17378837)
> It's worth reminding ourselves that the root comment was an absolutist assertion about the folly of outsourcing.

Even that strikes me as a mischaracterization, or, at best, taking just one statement out of context.

It was an assertion about the folly of relinquishing control, not being prepared for that lack of control, and thereby subsequently suffering a prod outage.

Re:Twitter ‘smytes’ customers by tripzilch on Sunday June 24, 2018 @11:54AM (#17386977)
To be fair a couple of those don't qualify as "if it goes down it would cause a prod outage".
Re:Twitter ‘smytes’ customers by jtms on Friday June 22, 2018 @01:29AM (#17371583)
hosted where? On a VPS, in a data center, or in your closet? All of these depend on third party services and good luck keeping them up and running as well as someone who makes just that ONE little thing their entire reason for existing.
Re:Twitter ‘smytes’ customers by jarfil on Friday June 22, 2018 @04:37AM (#17372441)
> On a VPS, in a data center, or in your closet?

Yes, any of those sounds like a good enough plan B. You don't need it to be perfect, just good enough to avoid getting stranded in a pinch. Then you can start shopping around for someone who does that one little thing much better than you do. That way it no longer is a show stopper and rather just an optimization problem.

Re:Twitter ‘smytes’ customers by y4mi on Friday June 22, 2018 @02:16AM (#17371800)
I don't agree with the parent either (I'd rather use a SaaS if it makes sense) its also not as easy as you make it sound.

For starters, most smallish companies I know of use housing for their servers. So, they actually do own the hard metal servers and have full access to them. Both over SSH and on the console for debugging purposes. The housing provider generally replaces HDDs in case of disk failures. naturally, you're still dependent on electricity and internet, but even that can be mitigated by spreading your servers across several locations.

All of that is obviously possible with AWS, GCP or other VPS hosts, but doing it yourself is not that much harder if you've got an experienced ops person that wants to bother with all of that.

but i repeat... Its often not worth the time investment to get that system properly set up. Just renting VMs or going straight into dockerized microservices is an increasingly better alternative

Re:Twitter ‘smytes’ customers by mortenjorck on Friday June 22, 2018 @01:26AM (#17371571)
That definitely solves the building part.
Re:Twitter ‘smytes’ customers by hackerpacker on Friday June 22, 2018 @08:12AM (#17373330)
but real-time promiscuity where you spread your users info everywhere and have no backup plan is much more FUN. (/s)
Re:Twitter ‘smytes’ customers by jarfil on Friday June 22, 2018 @04:32AM (#17372421)
No. Whether it's potatoes or servers, having a single point of failure with no plan B in place, is silly.

Your local restaurant going out of business is no reason for you to starve to death, you just go to another one, or to a grocery store, or indeed grow your own food if you're in some post-apocalyptic world or got no money... but somehow a service provider going out of business is reason enough for your production to stop? No, just no.

Re:Twitter ‘smytes’ customers by JdeBP on Friday June 22, 2018 @03:57AM (#17372281)
You've constructed a straw man, by removing an important part of that sentence and indeed the rest of what you are replying to.
Re:Twitter ‘smytes’ customers by mmt on Friday June 22, 2018 @04:34AM (#17372431)
Indeed, it's spurred a discussion of an over-simplification of the issues.

I don't think anyone is suggesting never to outsource, or even that the answer to "build vs. buy" is always "build".

The point is not having a critical dependency on a single vendor.

I don't think anyone is even saying "never", either.

The question is one of risk.

The risk of your tax accountant "going down" (or being crooked or something) is sufficiently low, and the cost of making that redundant is sufficiently high, that I don't think anybody does that. This could even apply to general accounting software, though one might argue that's not critical in the same way and/or more easily replaceable.

There's a sibling sub-thread that essentially implies, if not outright states that there's no choice but to "buy" in "build vs. buy". Although it may be true that building is no longer a practical choice due to time/cost, under certain circustances, it by no means eliminates the risk if there's only one vendor.

It's also a decision that can end up having tremendous costs at scale, if not re-evaluated, especially with even a soft version of vendor lock-in, like AWS.

Re:Twitter ‘smytes’ customers by pvg on Friday June 22, 2018 @06:07AM (#17372811)
That's not what the comment I replied to said, though. It just said it's 'silly'. If they had said something specific about, let's say, npm and made some reasonable and knowledgable comment or even speculation about how in this particular case they'd mis-assessed risk, that would be a pretty interesting comment. As it is, it's just a clone of the same tedious comment we get in droves in every thread about some multi-party outage. They're generic and free of insight.
Re:Twitter ‘smytes’ customers by abiox on Friday June 22, 2018 @05:43PM (#17377660)
> It just said it's 'silly'.

you did not include "so much that if it goes down it would cause a prod outage" in your quote, which is why the strawman claim was put forth.

Re:Twitter ‘smytes’ customers by pvg on Friday June 22, 2018 @05:55PM (#17377757)
It doesn't really change the quality of the comment. There is nothing inherently and outright wrong with such a dependence, whether it causes an outage in prod or not. In some cases it matters (maybe you're making a nuclear reactor) in many, it's a completely sensible tradeoff. Reflexively saying 'zomg you depend on something else and this is bad' every single time something is affected by a problem in another thing is 100% uninteresting. Including or not including 'in prod' or 'in bed' doesn't make it any more interesting or clever. It's just lazy grousing.
Re:Twitter ‘smytes’ customers by mmt on Friday June 22, 2018 @09:08PM (#17378812)
> There is nothing inherently and outright wrong with such a dependence

This is yet another strawman. The original commenter found it "silly", which, I agree, was a poor choice of wording.

Perhaps "overly risky" would have been better. I don't know. I just take the most charitable reading, per the guidelines.

It also comes after an exhortation to have a contingency plan, so it's at least implied that the "silliness" isn't inherent merely to the dependence but to the dependence without such a plan.

> in many, it's a completely sensible tradeoff

Is the tradeoff actually considered, though? Or is there just an automatic "we're not a nuclear reactor" decision process?

> is 100% uninteresting

Were that true, I doubt there would be replies. This is not one of those traditionally emotional/political issues.

> Including or not including 'in prod'

I (again, charitably) read "in prod" as a metaphor for "business critical". I'm sure we can all come up with examples, even in Internet companies, where one does not necessarily mean the other (or vice-versa). In the instant case, it seems to apply, so it made sense as shorthand.

> It's just lazy grousing.

I would agree if there were no built-in suggestion on how to avoid the problem at all, but there were.

Just because the point has been made before doesn't make it any less valid, until it has been refuted.

Re:Twitter ‘smytes’ customers by pvg on Friday June 22, 2018 @11:24PM (#17379350)
This is yet another strawman.

Only if 'strawman' means 'thing I disagree with'. The comment says such a dependence is bad and how you have to prepare for it with SLAs or whatever. This isn't universally true at all.

I doubt there would be replies.

A lot of really boring, trite things generate piles of replies. That's why they should be avoided.

Just because the point has been made before doesn't make it any less valid

The stated goal of the forum is not 'an exhaustive, if repetitive collection of generically valid things'. Remember that time you wanted to print something and the thing just wouldn't print? That's bad and annoying. It's valid that it's bad and annoying. We probably don't need to talk about it in the general sense every time it happens, though.

Re:Twitter ‘smytes’ customers by tripzilch on Sunday June 24, 2018 @11:52AM (#17386973)
> This is always trotted out and it's, by this point, a completely content-free thing to say.

Only if you pretend there are no companies that do in fact responsibly nail this stuff down. There's plenty. Your argument seems to be that nobody does it because it's hard.

Re:Twitter ‘smytes’ customers by lainga on Friday June 22, 2018 @12:28AM (#17371301)
Welcome to the world of third-party Twitter clients. Not content with screwing over businesses relying on Twitter's API, it seems, they have now set out to screw over any company relying on any other third-party API as well.
Re:Twitter ‘smytes’ customers by consp on Friday June 22, 2018 @03:45AM (#17372224)
> I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.

There are pretty solid SLA's for that. We provide some services as kind-of-saas but the SLAs are watertight.

Usually in those kind of sla's you can't sell the company (or move assets) without guarantee of delivery of the contract. In some cases they (the contract holders) even own you if you screw up and go under.

Re:Twitter ‘smytes’ customers by BinaryIdiot on Friday June 22, 2018 @04:33AM (#17372428)
> I find it silly to rely on some service, any external service, so much that if it goes down it would cause a prod outage.

I'm curious; what would you have these companies do in such a scenario? If 100% uptime is impossible then what you seem to be suggesting is that people should never use third party services or if they do use them they must not be in a critical path.

Are you suggesting everyone should do everything in house?

Re:Twitter ‘smytes’ customers by lambda on Friday June 22, 2018 @10:24AM (#17374162)
Use free (FLOSS) software.

If you have to use non-free software, buy software, not services.

If you have to buy services, always have a well-tested fallback path for when that service is not available.

Re:Twitter ‘smytes’ customers by s73v3r_ on Friday June 22, 2018 @12:30PM (#17375280)
Care to point out the FLOSS equivalent of what Smyte did?
Re:Twitter ‘smytes’ customers by lambda on Saturday June 23, 2018 @11:25AM (#17381680)
You missed two thirds of my comment.

This was in order of preference. If you can, use free software that you can fix and support yourself if need be. If not, buy the software and run it yourself, so you can at least control its operation. The last option is to by SaaS, and if you do, you need to have contingency plans in case it goes away. You should take this risk seriously, and factor it in to your decision when buying SaaS, and invest the engineering resources to make sure you aren't overly dependent on it.

Re:Twitter ‘smytes’ customers by jarfil on Friday June 22, 2018 @10:52AM (#17374384)
For any critical service, they should have an alternate provider ready at all times so that they could instantly switch to it. Whether it's another external provider or in house, is not really relevant, although having a bare bones in house alternative for the critical parts is a good idea.

What I'm suggesting is that people should use external services for their lower costs, better performance, or extra features, not to 100% depend on any of them.

Re:Twitter ‘smytes’ customers by scarface74 on Saturday June 23, 2018 @09:05AM (#17381088)
No. Smyte wasn't so business critical that they should consider a failure in the service a reason to stop the site. Netflix had an example where if thier authentication service was down, they didn't stop people from watching videos.

There is a popular C# package called Polly that has all sorts of fault tolerant strategies.

https://github.com/App-vNext/Polly/wiki/Transient-fault-hand...

In this case, use a Fallback strategy that responds to an API failure with some type of generic success response.