News for hackers. Stuff that matters.
Re:Twitter ‘smytes’ customers by shakna on Friday June 22, 2018 @12:03AM (#17371189)
> Indiegogo, GoFundMe, npm, Musical.ly, TaskRabbit, Meetup, OLX, ThredUp, YouNow, 99 Designs, Carousell, and Zendesk

Half an hour of warning, to screw over all of these?

They broke npm's user sign-ups, and publishing of packages, with half an hour of warning.

I can't imagine the havoc over at Zendesk either.

That is not 'winding down'. That's ghosting.

Re:Twitter ‘smytes’ customers by murukesh_s on Friday June 22, 2018 @01:53AM (#17371689)
It's worse for companies that have shipped mobile apps with the API embedded in the client. Web apps can at least try an alternative or disable the affected features until they work around this mess..
Re:Twitter ‘smytes’ customers by maxxxxx on Friday June 22, 2018 @04:30AM (#17372410)
This confirms my belief that I don't want to ship anything that uses directly a third party API. Instead always go through your own server which then calls the API so at least you have a chance to replace that.
Re:Twitter ‘smytes’ customers by chatmasta on Friday June 22, 2018 @06:08AM (#17372816)
That works for a lot of cases, but for many tracking APIs you want the request to come from the end user, so the tracking service can get their IP and network metadata. That would certainly apply to a fraud detection API.

Tbh that has always frustrated me though. I should be able to pipe those requests and just add an X-Forwarded-For header.

Re:Twitter ‘smytes’ customers by nmg on Friday June 22, 2018 @06:50AM (#17372979)
This is not my area of experience, but is it feasible to capture all raw tracking data from the client devices and then just pass it on to the API?
Re:Twitter ‘smytes’ customers by jdironman on Friday June 22, 2018 @07:24AM (#17373110)
That's basically a proxy or MITM, no?
Re:Twitter ‘smytes’ customers by reading-at-work on Friday June 22, 2018 @11:14AM (#17374582)
Yeah, but it sounds like it would satisfy the concern of being able to replace the 3rd party API internally without the mobile apps having to change.
Re:Twitter ‘smytes’ customers by throw_away2 on Friday June 22, 2018 @02:23PM (#17376132)
But it will have the cost of 2x API bandwidth (vs 0) and another point of failure that you are responsible for. A DNS CNAME probably wouldn't work if you had to go over SSL. Maybe a 30x redirector (still have the SPoF, but simpler and much less bandwidth)? Except in the past, I've found that most clients react poorly to redirects for anything other than GETs.

Is there any indirection that would avoid having to walk the traffic over my own network?

Re:Twitter ‘smytes’ customers by chatmasta on Saturday June 23, 2018 @07:48AM (#17380818)
The way I would implement it is as a “proxy” on my own network, eg trackingservice.myapp.com, that is just a web server listening and forwarding requests (technically a MITM). Override the request method of the client SDK to hit my endpoint instead of the tracking service.

Obviously you then pay the bandwidth, but it’s likely negligible compared to your app traffic. As a bonus you’ll probably circumvent adblockers.

If you want to avoid traffic forwarding, but keep flexibility over the endpoint, you can override the init of the sdk to first query your server for which endpoint to use. That way if the third party service goes down, you just need to change the config on the server.

Re:Twitter ‘smytes’ customers by lolc on Friday June 22, 2018 @09:15AM (#17373689)
Damn right I'm the man in the middle if I'm the SSL-offloader. It actually strengthens security because there's one less party to trust for the client.
Re:Twitter ‘smytes’ customers by cinquemb on Friday June 22, 2018 @04:45AM (#17372472)
When I was developing mobile apps, this is what we did. Too many times something changes and trying to update the code in the mobile app is way more of a pain than updating the server.
Re:Twitter ‘smytes’ customers by Bartweiss on Friday June 22, 2018 @11:50AM (#17374915)
> Clients had multi-year contracts in some cases.

I wonder how much Twitter is going to pay on top of the acquisition cost of what I can only imagine constitutes breach of contract with all of these companies. (If it doesn't, why even bother having a contract with a duration?)

Re:Twitter ‘smytes’ customers by hodgesrm on Friday June 22, 2018 @12:32PM (#17375300)
Kind of depends on the terms of the deal, namely whether it's an asset deal as opposed to a stock deal. In the latter Twitter would assume liability for contracts. In the former they just acquire IP and hire away the employees.

Acquirers usually prefer asset deals for this reason as it allows them to leave unpleasant liabilities behind. There's generally some sort of shell left that goes through wind down but it has few/no assets attached. In this case you can try to go after the original investors including founders or other shareholders (the IRS may do this if there tax issues) but since there's nobody home you are less likely to get much out of a favorable judgement.

Re:Twitter ‘smytes’ customers by heavenlyhash on Friday June 22, 2018 @03:24PM (#17376636)
This seems like the kind of thing a legal system should be designed to help defend the market against. It's a net negative to the economy as a whole to let torpedoes like this fly without consequence; and evidently, a totally Laissez-faire market will engineer ways to insulate the perpetrators against the damages rather than the insulate economy as a whole against the splash.
Re:Twitter ‘smytes’ customers by hodgesrm on Friday June 22, 2018 @04:47PM (#17377212)
On the flip side asset deals also let acquisitions proceed that would otherwise be blocked due to uncertainty about liability, which is another way of saying that the acquirer can't properly assess value of the investment. I have been through this process and have some experience with the type of issues that can arise--in our case they had nothing to do with screwing customers. It's possible Smyte would have just disappeared anyway.

Moreover recovering damages in this kind of case is time-consuming, doubtful, and too late to fix anything. (Like closing the door after the horse is gone.)

What's left is reputation--the names of the founders of Smyte are listed at https://www.crunchbase.com/organization/smyte. I would have some pretty hard questions if they ever showed up in a deal or looking for work.

p.s., According to the Techcrunch page "Smyte stops bad actors on marketplaces and social networks." You can't make this stuff up.

Re:Twitter ‘smytes’ customers by toephu2 on Friday June 22, 2018 @01:03PM (#17375553)
I'm pretty sure their contracts covered their ass. Hence why lawyers write up contracts in the first place.
Re:Twitter ‘smytes’ customers by jacques_chester on Friday June 22, 2018 @01:21PM (#17375686)
You can write just about any words you like into a contract. But that doesn't mean they are enforceable and doesn't mean a legislature or court can't add or modify clauses. Then there's the whole doctrine of equity thing over the top of it. In law there are always a whole bunch of fine-grained questions that can change the whole outcome.

I expect one or more lawsuits to emerge, maybe as a class action. It might not come to trial and simply wash up as a settlement.

I am not now, nor have I ever been, a lawyer.

Re:Twitter ‘smytes’ customers by Bartweiss on Friday June 22, 2018 @01:42PM (#17375847)
Both parties presumably had lawyers writing ass-covering contracts, so who loses? This isn't the usual "Wells Fargo screws you over then forces arbitration where you lose again" imbalance, there's no particular reason to think Smyte could have swung a favorable contract with e.g. ZenDesk.
Re:Twitter ‘smytes’ customers by hodgesrm on Friday June 22, 2018 @04:54PM (#17377255)
It really depends on who was negotiating, whose paper was used to write the contract, and how much each side wanted the deal. Legacy enterprises like Disney often force vendors to use the legacy company's paper, which case substantial protections would be written in or the deal would not happen.

Self-service web services on the other hand generally come with take-it-or-leave-it conditions of use. When's the last time you redlined Github terms of use when starting a project? ;)

Re:Twitter ‘smytes’ customers by jquery on Friday June 22, 2018 @01:27AM (#17371572)
More companies than that, and large ones. These are just the ones that went public... some of them don't want you to know their trust/safety/moderation features just went dark. Smyte just ruined a lot of people's day(s).
Re:Twitter ‘smytes’ customers by TheSpiceIsLife on Friday June 22, 2018 @03:51AM (#17372256)
Twitter. Twitter just ruined a lot of people’s days.
Re:Twitter ‘smytes’ customers by kentonv on Friday June 22, 2018 @04:32AM (#17372420)
No. Smyte had a responsibility to its customers, and should not have entered into any deal that did not include some reasonable wind-down plan.
Re:Twitter ‘smytes’ customers by munificent on Friday June 22, 2018 @01:04PM (#17375560)
Had.

Twitter owns Smyte. When you buy a company, you acquire all of its responsibilities. There is no longer a Smyte to point the finger at. It's all Twitter.

Re:Twitter ‘smytes’ customers by kentonv on Friday June 22, 2018 @01:33PM (#17375770)
This is pedantic. "Smyte" means "the people who formerly made up Smyte, especially the founders".
Re:Twitter ‘smytes’ customers by ShabbosGoy on Friday June 22, 2018 @02:20PM (#17376112)
Legally, the onus is on Twitter for pulling the plug on an API that was being used actively without much notice.

I’m actually surprised that it was done in this manner, as many startups have a “business continuity” section in whatever contracts are drawn up with customers, detailing the specific steps and timeframe of retiring a particular service. I would have thought this is standard practice for a SaaS company.

Re:Twitter ‘smytes’ customers by munificent on Friday June 22, 2018 @04:56PM (#17377273)
My point was that we shouldn't point the finger at Smyte because that lets the blame drop on the floor when the "Smyte" brand evaporates.

Twitter owns this now and the way to ensure it's their reputation that is affected over the long term is by correctly calling this a "Twitter" failure.

Re:Twitter ‘smytes’ customers by TheSpiceIsLife on Friday June 22, 2018 @04:57AM (#17372525)
Have we read the contract? Do we know what happened?
Re:Twitter ‘smytes’ customers by s73v3r_ on Friday June 22, 2018 @11:56AM (#17374974)
Twitter is the one responsible now.
Re:Twitter ‘smytes’ customers by PurpleBoxDragon on Friday June 22, 2018 @11:21AM (#17374647)
What about blaming the companies who decided to depend upon a service that they have no control over? I find arguing against using third party APIs to save a few days of development a constantly losing battle, with it being presented as a benefit with no costs or risks. Do those pushing/agreeing with such a view not share some blame?
Re:Twitter ‘smytes’ customers by paranoidrobot on Friday June 22, 2018 @12:51PM (#17375472)
> What about blaming the companies who decided to depend upon a service that they have no control over?

Unless you're advocating against any kind of *aaS, this is kind of a silly argument.

A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.

Obviously when you make your product/company depend upon these things, you're taking a risk that they could go away, but you protect yourself against that by having robust contracts.

Outages are different, but for a significant outage you expect to see some kind of RFO and explanation of how they're going to mitigate it.

Deliberately withdrawing service for all customers with 30 minutes notice? That's entirely the fault of whomever is in charge at Smyte.

Even 30 days, while it'd probably ruin some existing plans, would at least allow people to have a chance to migrate without things just breaking.

Re:Twitter ‘smytes’ customers by PurpleBoxDragon on Friday June 22, 2018 @01:49PM (#17375886)
>A lot of these services are not "a few days of development" worth of work. The value in these is that they have dedicated teams of people who's entire job is to run that specific targeted thing. Unless you're willing to also invest in major development, it's going to be difficult to match that functionality.

For the cases I was involved in, there were options of buying the service but hosting it internally with some update pipeline. So even if the updates were suddenly turned off, you have a far longer time to swap to a different system. Generally deploying these aren't to much time, but what is really the issue to me isn't that the options that were chosen were the ones which were chosen, but that the process of choosing them did not evaluate the risks of creating such a strong external dependency at all. Its one thing if it is a risk taken with full knowledge of the potential costs and benefits, it is another when people do it because they think there is no possible downside.

Re:Twitter ‘smytes’ customers by wizzard on Friday June 22, 2018 @02:47PM (#17376354)
Are you saying that:

* No website or app should ever call out to any third party API

* Every single piece of functionality a website or app has should be developed in-house

* The functionality third party APIs provide would often take "a few days" to reproduce in-house

Re:Twitter ‘smytes’ customers by scarface74 on Saturday June 23, 2018 @08:49AM (#17381022)
No website or app should ever call out to any third party API

Definitely. Your website and app should only call your APIs. Your API can then call the third party API from the server. It's a lot easier to change something on the server than having to update the app.

Re:Twitter ‘smytes’ customers by ruskerdax on Saturday June 23, 2018 @01:50PM (#17382385)
I wanted to argue with you for posting this, but you're right, this is a smart way of handling API calls.

It's not really addressing the whole issue of depending on third party APIs if your API is calling on them, but you are technically correct, which I've heard is the best kind of correct.

Re:Twitter ‘smytes’ customers by kentonv on Friday June 22, 2018 @01:15PM (#17375656)
> What about blaming the companies who decided to depend upon a service that they have no control over?

No. Delegation is the cornerstone of civilization. Trying to build everything yourself is a terrible idea.

Re:Twitter ‘smytes’ customers by Bartweiss on Friday June 22, 2018 @11:54AM (#17374955)
> that they have no control over

Per the article, they did at least have long-term contracts locked in with Smyte. That's not the same as control over the feature, obviously, but plenty of companies do business on the strength of contracts instead of vertically integrating their entire product.

Obviously we don't have many details yet, but I think there's a real difference between simply using a third party API that works for the moment, and buying a service from a third party. If Twitter/Smyte simply broke contract with all their existing customers, that's very much their responsibility.

Re:Twitter ‘smytes’ customers by brazzledazzle on Friday June 22, 2018 @12:19PM (#17375162)
This isn’t really a leftpad issue. This was for fraud prevention.
Re:Twitter ‘smytes’ customers by chris_wot on Friday June 22, 2018 @05:35AM (#17372679)
Twitter has ruined a lot of people's days for a long time now. Especially in the Trump era. Nothing new here!
Re:Twitter ‘smytes’ customers by workinthehead on Friday June 22, 2018 @07:30AM (#17373129)
Not sure why you are being downvoted since Twitter has a well-established history of pulling the rug out from under people who depend on it.
Re:Twitter ‘smytes’ customers by pkaye on Friday June 22, 2018 @02:32AM (#17371874)
What would be these businesses backup plans if Smyte went bankrupt instead being acquired?
Re:Twitter ‘smytes’ customers by JauntyHatAngle on Friday June 22, 2018 @02:48AM (#17371943)
A company going bankrupt usually doesn't give 30 minutes warning that they are shutting down services though. They give more warning than that.
Re:Twitter ‘smytes’ customers by shakna on Friday June 22, 2018 @02:38AM (#17371901)
Presumably you would get more than half an hour to migrate.
Re:Twitter ‘smytes’ customers by tinus_hn on Friday June 22, 2018 @04:30AM (#17372406)
Your supplier going bankrupt is a well known risk that you cover by checking its credit rating and by paying on or after delivery.
Re:Twitter ‘smytes’ customers by sverhagen on Friday June 22, 2018 @05:13AM (#17372591)
That's why often startups can't do business with big companies that hold to rules like that.

Would Smyte's rating have cut it for people?

Re:Twitter ‘smytes’ customers by fauigerzigerk on Friday June 22, 2018 @05:51AM (#17372742)
Your supplier getting acquired is also a well known risk that you cover by reading/negotiating the contract.

The problem is, if all suppliers use the same contract clauses you may not have a choice.

Re:Twitter ‘smytes’ customers by AznHisoka on Friday June 22, 2018 @06:37AM (#17372919)
How do you check the credit rating of a startup?

and how would their credit rating be affected if they pay all their bills but just reduce the number of bills gradually over time?

Re:Twitter ‘smytes’ customers by PeterisP on Friday June 22, 2018 @07:11AM (#17373061)
One way that's sometimes done is that you require them to submit an independent respectable audit of their financials, or in case of confidentiality issues, not submit the actual financials but a statement from a respectable auditor that they have verified the suppliers financials and their cashflow/"runway"/etc satisfies the criteria that you required.
Re:Twitter ‘smytes’ customers by snowwolf on Friday June 22, 2018 @07:00AM (#17373017)
I'm not denying that what they've done is really bad, but who puts a 3rd party API in a critical path without having any fallback (alternative service, reasonable timeouts, etc.) for when it goes down (as almost any service will eventually do)?
Re:Twitter ‘smytes’ customers by mickronome on Friday June 22, 2018 @07:38AM (#17373172)
Everyone that does online transaction of money, and many more. Like all those who use AWS services on AWS.
Re:Twitter ‘smytes’ customers by striking on Friday June 22, 2018 @07:15AM (#17373073)
People who have multi-year contracts with those API providers, probably with an SLA detailed in the contract. Especially since this API call would have to be on the critical path; you don't want people signing up when the anti-spam service isn't running.
Re:Twitter ‘smytes’ customers by snowwolf on Friday June 22, 2018 @07:20AM (#17373088)
1. You have an SLA guaranteeing 100% uptime?

2. If it's that critical, you have redundancy (an alternative provider, or a naive implementation that maybe causes more false positives but that will cover you during an outage)

Re:Twitter ‘smytes’ customers by hluska on Friday June 22, 2018 @10:35AM (#17374237)
I might be wrong, but I'm getting the sense that you don't have much actual experience.

I've been a senior developer in an organization and known that a solution isn't perfectly robust. Clearly I want to fix it because I (like most other developers) like building robust solutions, but I have six weeks of tasks to finish in two weeks, and that isn't one of them. What do I do?

You might say "do it anyways" and, if so, that's a junior developer attitude. I'd strongly urge you not to do that as shipping big projects is really more about choreography than engineering.

Or, you might say "fight for permission" and that's not a bad idea, as long as you accept that you'll likely lose.

You'll likely lose for business reasons. On some level, we all know that a SAAS on a critical path is a bad idea, but we do math. Our SLAs are not 100%, but we can accept a few hours of downtime a year. And, the probability of a service you've seemingly vetted being acquired and shutting down this quickly is low. Is the probability of a shut down like this high enough to justify adding and testing redundancy?

The math usually works out in favour of bug fixes and new features > redundancy.

Re:Twitter ‘smytes’ customers by AndrewUnmuted on Friday June 22, 2018 @02:57PM (#17376439)
In the first place, it was an awful business decision to take the route of trusting a SaaS provider with such an important task.

It is extremely irresponsible to forward sensitive user data and site interactions to a third party, even if it is in the name of spam/abuse/scam filtering. That its implementation brings down production websites in the case of service failure only adds insult to injury, originally caused by plain-awful product design.

If there's one thing that you should be doing in-house when you run a large online community of privacy-conscious users, it's the kind of service that Smyte provided. If sound business reasoning had prevailed, npm would have suffered no downtime.

Re:Twitter ‘smytes’ customers by hluska on Saturday June 23, 2018 @11:38AM (#17381733)
It's easy to say that now that everything has gone to hell, but in business, we usually have to make decisions with imperfect knowledge.

I don't agree that it's irresponsible to forward sensitive data to third parties. Rather, I believe that intelligent companies have intelligent policies around sharing data with third parties. Some third parties provide an amazing service that would be extremely expensive for companies to mimic in-house. And, you cannot convince me that the probability of a well vetted service shutting down as fast as Smyte did is high enough to justify bringing this in-house when you have other things to work on.

Re:Twitter ‘smytes’ customers by striking on Friday June 22, 2018 @09:43AM (#17373875)
No, but if I were npm, I'd have an SLA guaranteeing three-and-a-half or four nines of uptime (which is far from unreasonable) and just accept that there might be times that signup is not accessible because that'd only cost me a couple hours of signups a year.

And no, I wouldn't have an alternative provider. It would not make sense to, if I'm paying a company to provide a service. Why should I pay another to do nothing?

Re:Twitter ‘smytes’ customers by workinthehead on Friday June 22, 2018 @07:28AM (#17373124)
I know for a fact that you've done worse in your own life. Spare us the moral posturing, please.
Re:Twitter ‘smytes’ customers by teddyh on Saturday June 23, 2018 @01:55PM (#17382420)
What you just did is called a “Tu quoque” argument, and it is a type of Ad Hominem attack; i.e. a fallacy.

https://en.wikipedia.org/wiki/Tu_quoque

Re:Twitter ‘smytes’ customers by workinthehead on Sunday June 24, 2018 @02:35PM (#17387714)
Good effort, buddy, but no.
Re:Twitter ‘smytes’ customers by snowwolf on Friday June 22, 2018 @07:35AM (#17373155)
This whole thread is moral posturing about how unprofessional Twitter/Smyte have been, but if we want to talk about professionalism we should look at ourselves.
Re:Twitter ‘smytes’ customers by ajeet_dhaliwal on Friday June 22, 2018 @08:39AM (#17373470)
If you upload to AWS S3 you put in a fallback to another service? And any other AWS services you use? That's a lot of fall backs. Is the second fallback good enough? Should we add a third? In any practical sense development would grind to a halt. Twitter/Smyte are totally in the wrong for not giving a reasonable notice. I think whoever made this decision should actually be removed from their post and the servers should powered back on again with an apology to all of their clients. If they still want to shut it down later then they should provide a reasonable time frame.