News for hackers. Stuff that matters.
Re:The Amazon Premium by kugelblitz on Thursday December 19, 2019 @10:25AM (#21835647)
The reason I use AWS is because I can be fairly certain, that their uptime is quite high (not 100%, but it's more than I can probably guarantee if I were to run on an own server), that their eco-system is huge and they connect to each other.

Being a "solo hacker" developing multiple websites, I need to prioritize where I put my time in. Thus my "default" choice is using Elastic Beanstalk to deploy websites, be it PHP, Python or something else. I can easily add environment variables, let it connect to RDS (database), scale it up (more servers or bigger servers), I can add SSL certificates, I can add a CDN (Cloudfront), I can add separate data storage (S3) all with a few clicks and from one (albeit super-huge and need-to-get-used-to) dashboard.

Alternatives: I don't trust Google services for anything critical (they tend to shut down easily or they might just change their pricing drastically - e.g. in Google Maps), Azure might be an alternative, just never had the time to check it out intensively.

For "fixed ceiling" projects, I use Digital Ocean (because unfortunately AWS mostly doesn't offer a budget ceiling, just alarms).

Re:The Amazon Premium by mattmanser on Thursday December 19, 2019 @11:12AM (#21836167)
I've used various hosts over the last 15 years in the UK and I honestly can't remember any downtime from anyone else. I do remember lots of AWS + Azure downtime though. Remember Netflix going down?

I'm not really sure where you're coming from.

As someone who's been doing this for years, the old days when you had a £10 virtual machine that could handle tens of concurrent requests don't seem to be handled by the low price AWS/Azure servers. Let's not forget, tens of concurrent requests is 100,000s of monthly users, a major win for most startups.

And even the expensive cloud options are utterly rubbish compared to what you can get for a bare metal, I like to compare the two as you can get a Ferrari for £1000 a year, or you can get a bicycle for £1000 but don't worry, it's a "cloud" bicycle so definitely go with that.

I am constantly having problems with an Azure app at the moment timing out because of some poorly optimized queries that are being throttled because they somehow exceed the ever mysterious DTU. On my laptop, a 5 year old laptop, the queries take 10ms. So it's is impossible to run on Azure because "reasons" unless I pay a ridiculous amount of money to upgrade the SQL server. Yes, I could fix the query, but why should I be spending time on that? They're part of the admin system and are simple reports.

Another app I worked on we ended up having to pay £1000 a year just so we could have multiple-domains + SSL, when we expecting to pay £80 p/y.

Personally, every time I use cloud servers I regret it. I keep trying or clients really want it, and can understand the lower devops cost and that with bare metal, every 5-10 years I need to migrate the app to get a better server, but I honestly find the cloud servers so completely underpowered for the cost. I don't understand how they've got so popular. Basic devops for most startups is a day or two task once every few years. Write a deploy script, backup script once, and almost forget about it.

(I absolutely love s3 though for backups though)

Re:The Amazon Premium by ClumsyPilot on Thursday December 19, 2019 @12:35PM (#21837128)
I have recently started a project of deploying kubernetes cluster at home and hosting some personal projects there. My top findings are:

1) bare metal is dramatically faster than most cloud instances. Many of us might have collection of old hardware that, by performance, would be equivalent to >$1,000 in monthly cloud fees.

2) If from the start of your career you have only dealt with modern cloud/Azure/AWS, there is a whole body of knowledge you are missing. Acquiring it from scratch is time consuming, and you are never quite sure you are doing the right thing. As a developer, I don't pay the AWS bill, so it would be more rational to spend my time on learning ML or something else that will be reflected in my paycheck.

Re:The Amazon Premium by sdan on Thursday December 19, 2019 @03:34PM (#21838964)
So how do you do routing? I have an old Linux machine that I haven’t been using and don’t know how I can securely and efficient route that into the internet (don’t want to use my ip for connecting to Redis for example) which is why I as thinking of using some vpn hosted on gcp.

Any advice would be greatly appreciated. Can’t handle the monthly payments to gcp only to get a raspberry pi level of compute.

Re:The Amazon Premium by whiteandnerdy on Thursday December 19, 2019 @05:50PM (#21840085)
I host a Plex instance on a server on my home network, and struggled with setting up external access because my ISP at the time had a double NAT. What I ended up doing was renting a super cheap VPS from TINYKVM (https://tinykvm.com/) which I pay $15 a year for.

I point my DNS at the VPS, then I use the Zerotier VPN to tunnel traffic to my home server (Zerotier is embarrassingly easy to set up - I struggled for a while with OpenVPN but gave up). I use iptables packet forwarding to route traffic on port 443 or 80 from the VPS over the VPN to my local server, which is running an Nginx instance to serve the traffic.

This works perfectly for me 99% of the time. Main pain points have been the unreliability of hosting a server locally where Internet can die and power drop off, and a small amount of sysadmining on the VPS, which is so tiny that I have to be mindful of disk space (e.g. removing old Linux upgrade images every so often, etc). All told though, it works surprisingly seamlessly, and there's a certain smugness to knowing traffic from my smart TV goes all the way round the world and back through a VPN just to talk to my server sitting one foot below the TV.

Re:The Amazon Premium by sdan on Thursday December 19, 2019 @06:26PM (#21840338)
Interesting setup. Have you heard of Cloudflare Argo by chance? Was thinking of using that, although this seems like a perfectly fine setup itself.
Re:The Amazon Premium by ClumsyPilot on Thursday December 19, 2019 @08:55PM (#21841142)
I am not entirely clear on your question, but I am avoiding reliance on any third-party service to keep it a clean experiment.

I have it easy for external access -> static IP and a Gigabit connection without NAT, so all I have to do is setup ports on my router.

For my services like databases and Redit, I expose them only on Cluster-IP, and them you use kubeDNS and get something like 'redis.default.svc.cluster.local'.

If I need to access them for development purposes, I use kubectl-proxy, but a VPN is also an option.

Re:The Amazon Premium by Aeolun on Thursday December 19, 2019 @07:29PM (#21840738)
If your DNS service has an API, you can just have your home server update it’s own record when it detects a change. It may be down for a little while while DNS propagates though.

Edit: I think I misunderstood, but will leave this up anyway.

Re:The Amazon Premium by gtycomb on Thursday December 19, 2019 @01:28PM (#21837691)
> started a project of deploying kubernetes cluster at home and hosting

I am also thinking along these lines but not sure about reasonable paths to try. Care to share some pointers on your setup? Basically looking to serve web pages from a server cluster at home.

Re:The Amazon Premium by dgemm on Thursday December 19, 2019 @12:02PM (#21836723)
If you can run your app on a £10 VPS then you are not the target market for the big clouds. That's why it doesn't make sense for you. Larger users have very different priorities.
Re:The Amazon Premium by mattmanser on Thursday December 19, 2019 @12:11PM (#21836825)
This really isn't true. People use them for everything. Tiny little enterprise apps with 10s of users. Startups with mere 1,000s of uniques per month. Side projects. Everything.

If you look at Microsoft's strategy with C#/Azure you'll see that every single tutorial uses deploying on Azure. Their instructions for how to deploy to a real server in their tutorial is pretty much "we're not telling you, figure it out yourself, sucker".

Their new business strategy is to get the entire of their userbase paying massively over the odds for hosting small apps.

And, in reality, almost every app by any startup you could go and look at on producthunt (and probably most of ycombinator startups) have such low volumes of usage they could run on a $5 VPS. But I bet all of them are on AWS (there's even a few comments on this thread that basically say "start on AWS, then move").

Everyone is using cloud now, go to any meetup and ask. Or simply look at the sales figures.

Re:The Amazon Premium by ivalm on Thursday December 19, 2019 @12:43PM (#21837224)
The thing is, maybe you're overpaying spending $100/month instead of $10/month, but those are both small numbers. Simply not having to standup a DB/server/getting certs/debug if something goes wrong somewhere/etc is worth more in my time than a few $100. If you're very small scale you can also use the "always free" tier of VMs.

I really like that for my side projects/solo hacks I don't have to think about anything other "here is my code that does the thing that I think is cool" and all the deployment/serving/data layer/ml model inference/etc just kind of happens.

For small/very early startups it's also similar. At least in the US, it's not the extra $5k/year you spend on cloud hosting vs doing it yourself that kills you; for many engineers that less than a week of comp.

At the same time, when you are large and spending $100s of k on your hosting then sure, look into alternatives and migrate if needed. This is where "start on AWS, then move" comes from.

Re:The Amazon Premium by mattmanser on Thursday December 19, 2019 @01:14PM (#21837553)
I guess the point I was trying to make is that all that "extra" stuff isn't really much extra work at all and the same magic deploy tools could be just as easy on a VPS/bare-metal, but aren't because of $$profit$$ making it better that they don't make it super-easy to deploy on VPS/bare-metal.

I can make a script to do it all for me (and have in the past), but it's a bit of work.

That's really why the title of the article is apt being "the Amazon Premium", and not "the Amazon Rip-off".

Re:The Amazon Premium by ClumsyPilot on Thursday December 19, 2019 @12:18PM (#21836907)
Honestly if you are deploying an application to production, hopefully you have a level of knowledge beyond "click here, then click here'.

The other side of this is that documentation is hard, and Microsoft documentation is frequently lacking or out of date, especially for anything off the beaten path. I use IoT Hub for messaging, and it's documentation is scant, many limitations and failure modes are found by trial and error.

Re:The Amazon Premium by mattmanser on Thursday December 19, 2019 @12:34PM (#21837121)
I don't understand where you would get this info from then? Tutorials provided by the framework makers themselves are pretty standard.

Magically absorb the knowledge in an incantation to the Redmond Gods in the ether?

Re:The Amazon Premium by ClumsyPilot on Thursday December 19, 2019 @09:37PM (#21841323)
haha, sometimes it feels that way, for their less-popular commercial services documentation is lacking and you can't even read the source!

On reflection my comment about level of skill is unfair, what I should have said is: generally the information is there, but it's not always in the form of follow-able 'How-to' guides, often it's a dry detailed documentation. If you need something other than the standard setup, it is up to you to know what you are looking for and put 2 and 2 together. Of course it is by no means easy.

They do have reasonable guidance on how to deploy ASP.net core on Linux, but it can be lacking for lesser used stuff

Re:The Amazon Premium by scarface74 on Thursday December 19, 2019 @04:04PM (#21839285)
If you’re that small, you can probably stay under the always free lambda tier.
Re:The Amazon Premium by mattmanser on Thursday December 19, 2019 @07:03PM (#21840561)
Honestly, why chime in with that? Not only is it sort of sneering (if you're that small), it's wrong and counter to the most practical advice we give out here, use what you know. It's hard enough starting a business without learning a whole new tech stack.

Ironically, we were having the discussion here yesterday about a guy with a failed startup saying he'd wasted the opportunity because he used too much new stuff and spent all his time on that instead of iterating the actual product. Several comments focused on it'd just be better to start off with a Rails app, it's quick, it's easy and it's understood.

And really, is lambda appropriate for anything more than a very focused niche app? I mean, try writing all the usual things you need to slot into a SaaS on it, billing, user management, signup, admin reports, admin functionality, emails, multi-tenancy, custom domains, etc. etc. Even a 20 user, internal, enterprise app needs most of that functionality. Put on top of that, you're forced to use a particular, proprietary tech stack and lock yourself into something that might very well die in a couple of years.

Re:The Amazon Premium by scarface74 on Thursday December 19, 2019 @07:27PM (#21840721)
Not only is it sort of sneering (if you're that small)

Lambdas always free tier is 1 million requests per month and 400Gb/seconds per month.

https://aws.amazon.com/lambda/pricing/

It takes a lot to have any meaningful lambda bill. That in no way is meant to be an insult. If you’re spending even $100/month on lambda, you’re running a successful website.

It's hard enough starting a business without learning a whole new tech stack.

If you know how to write a program and create a function that takes two parameters - a payload and context argument - you know lambda. You can write it in the console if you’re using Python or Node and just run it from there.

Even if you don’t want to learn that, you can host your standard C#/WebAPI, Python/Django/Flask, Node/Express API on lambda just by adding code and changing your endpoint.

Several comments focused on it'd just be better to start off with a Rails app, it's quick, it's easy and it's understood.

There is also code to wrap a Rails app and run it on lambda....

And really, is lambda appropriate for anything more than a very focused niche app? I mean, try writing all the usual things you need to slot into a SaaS on it, billing, user management, signup, admin reports, admin functionality, emails, multi-tenancy, custom domains, etc. etc. Even a 20 user, internal, enterprise app needs most of that functionality

Yes. Especially in today’s world of client side frameworks and microservices, you host your static assets like Javascript, css, and HTML in S3 and call server side APIs hosted on lambda.

Put on top of that, you're forced to use a particular, proprietary tech stack and lock yourself into something that might very well die in a couple of years.

Well, two things. What’s more likely to die in two years / your startup or AWS?

Second, you can host your standard API just by adding four lines of code and you don’t have to make any changes to host it in Docker or on a VM.

There is a way to do this in every language that lambda natively supports. For instance for Node:

https://github.com/awslabs/aws-serverless-express

I have a Node/Express app that gets deployed to both lambda and Fargate (Serverless Docker) when I push it. The only difference is that lambda uses one entry point and Docker uses another.

I’ve done something similar with C#

https://aws.amazon.com/blogs/developer/deploy-an-existing-as...

Where it is deployed to both lambda and a Windows server running IIS.

I can be a lot more productive with all the resources that AWS has to offer when all I have to do is provision resources with a CloudFormation template and instead of doing the grunt work of infrastructure administration I can concentrate on writing software.

Re:The Amazon Premium by halbritt on Thursday December 19, 2019 @11:15AM (#21836188)
> Alternatives: I don't trust Google services for anything critical

In the case of GCP, the product is the product rather than the user, so there's a greater expectation that it'll not get shut down or that pricing will change easily.

Further, they're obviously committed to being competitive with AWS which requires that they behave similarly.

Re:The Amazon Premium by scarface74 on Thursday December 19, 2019 @11:47AM (#21836546)
On the other hand.... see maps.
Re:The Amazon Premium by halbritt on Thursday December 19, 2019 @01:31PM (#21837718)
On the other hand of what?

Maps is a consumer facing service for which revenue is derived from the data collected and advertising. That business model is nothing like selling commodity compute services to enterprises.

If anything, I think GSuite is a portent of how Google will treat GCP. That is, the set of products has been relatively and consistently stable for quite some time, which has driven fairly decent adoption into the enterprise market.

There've a couple of product shifts, specifically around voice, hangout, and chat, but that seems to settled at this point.

I speculate that the changes that have raised the most ire have been around gmail, which I suspect has a fragmented product management team as it's both a free consumer service getting monetized through ad revenue as well as an enterprise product sold directly to customers.

Re:The Amazon Premium by scarface74 on Thursday December 19, 2019 @01:48PM (#21837877)
Maps is a consumer facing service for which revenue is derived from the data collected and advertising. That business model is nothing like selling commodity compute services to enterprises.

So selling access to an API that businesses depend on is nothing like selling access to other resources that businesses depend on?

https://gadgets.ndtv.com/apps/features/google-maps-apis-new-...

Prices have gone up as much as 14 times in certain scenarios

Re:The Amazon Premium by halbritt on Thursday December 19, 2019 @11:21PM (#21841695)
That's correct.

Selling access to the Google maps API is wholly superfluous to Google's business model. Certainly, it doesn't clock anywhere near a billion in revenue.

Prices have gone up because Google doesn't have any interest in sustaining the product. The total addressable market is miniscule by comparison to cloud or even enterprise productivity apps.

Re:The Amazon Premium by scarface74 on Friday December 20, 2019 @06:28AM (#21843316)
Shouldn’t that also give you pause that Google is known not to care about products that are not its main revenue driver and will abandon them?

It doesn’t matter how large the “addressable market” is if Google doesn’t have the will or talent to compete. Social is also a large market, how did that work out for Google?

Isn’t that the entire pitch of every startup? “If we just get 10% of the market, we will be huge”.

And “revenue” is meaningless is GCP profitable? There was a submission on the front page of HN yesterday about Google’s management not being happy about how GCP waS doing and they have three years to turn it around or “lose funding”.

Re:The Amazon Premium by halbritt on Friday December 20, 2019 @01:26PM (#21846286)
The lack of ease with which one gets into and out of the cloud computing business will preclude Google from killing off the GCP products with the same wanton disregard for customers as they demonstrated with social and others.

At worst, they'll sell off or divest the business entirely. They'll expose themselves to tremendous liability if they attempt to wind it down like their other properties.

Re:The Amazon Premium by scarface74 on Friday December 20, 2019 @03:07PM (#21846981)
It doesn't even matter as much if Google gets out of "cloud" as it matters if Google decides to kill one of its cloud offering that you depend on. If I were on GCP and building a service that is using 5 Google features one of them happened to be the Maps API and the other 4 came under the "Cloud" umbrella, it would be little solace that Maps wasn't part of GCP when they raised the prices 15x.

On the other hand, AWS announced something as minor as that they were discontinuing one form of accessing publicly available S3 urls that was deprecated in 2010 for the newer one. After a vocal outcry from the internet about how many websites that it would break, they backtracked:

https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-...

On the otherhand, how many Urls were broken when Google discontinued it's link shorterning service?

Google has shown time and time again that it can't be relied upon for any important infrastructure.

Re:The Amazon Premium by halbritt on Tuesday December 24, 2019 @01:41PM (#21874021)
I get your point, but you keep supporting it with examples of products that are well outside the auspices of the GCP business unit.

Yes, it's true that Google capriciously kills products for various business reasons without any regard for customer sentiment. My counter-point to that is that GCP products exist under different conditions, specifically legal and business realities. They have contractual obligations to corporations with very large legal departments. There is also the business reality that if they're going to compete with AWS and Azure then to be competitive they have to behave in similar ways. If they piss on customer sentiment as they have with their b2c services, they'll fail and I think this is very clear to them.

That's not to say that they aren't already failing. As I mentioned up thread, their support is laughably bad and when pressed about that fact, they don't seem to actually acknowledge how that could be a problem.

Re:The Amazon Premium by calpaterson on Thursday December 19, 2019 @02:26PM (#21838295)
(Article author here) Unfortunately my experiences with GCP have been fairly mixed. One place I worked had a number of problems with managed databases going AWOL on GCP.

Also the GCP panel is truly awful, much much worse than the AWS console.

Re:The Amazon Premium by abawany on Thursday December 19, 2019 @02:49PM (#21838513)
I have to wonder if this depends on one's familiarity with the respective products. I started with GCP and then used AWS and found the latter to be way more confusing and convoluted than GCP. The forced daily logouts on AWS added another element of hassle to the mix.
Re:The Amazon Premium by halbritt on Friday December 20, 2019 @12:02AM (#21841818)
My experiences with GCP have been pretty good, most of those experiences were with GKE. In many ways, their products are superior to AWS. At the very least, they have a different approach to offering cloud services rather than simply attempting to copy AWS offerings.

However, their support is absolute shit and they're absolutely devoid of the customer first mentality that AWS demonstrates.

Re:The Amazon Premium by bcrosby95 on Thursday December 19, 2019 @04:13PM (#21839376)
https://www.crn.com/news/cloud/google-reportedly-set-ambitio...

Doesn't matter what is the product. They're ready to kick GCP to the curb if it doesn't perform up to expectations. I wouldn't host anything important on it.

Re:The Amazon Premium by Roark66 on Thursday December 19, 2019 @11:08AM (#21836119)
I mostly work in AWS. I was recently creating an environment and a deployment mechanism for a fairly simple app that was initially developed in Azure. The client was surprised because the cost estimates previous devs gave him (based on Azure) were a lot lower. I don't know what their Azure architecture was - I had to come up with AWS stuff from scratch. Maybe they omitted certain things like backup, DR, auto scaling etc from their initial estimate or maybe it is worth looking at Azure. It is not the first time I'm hearing something along the lines of "Azure is cheaper".
Re:The Amazon Premium by halbritt on Thursday December 19, 2019 @11:17AM (#21836213)
I frequently see folks assert cloud vendor X is cheaper and present data comparing long-term reserved instances to on-demand.
Re:The Amazon Premium by the_gipsy on Thursday December 19, 2019 @12:55PM (#21837350)
How does your development cycle look like? For example, you decide you want to add S3 or Redis. So you add it to some yaml, or do it in the amazon console. How do you develop/test/debug the code that would use these services, locally, before pushing that to production?
Re:The Amazon Premium by kugelblitz on Thursday December 19, 2019 @02:15PM (#21838196)
I try to abstract something like the filesystem. So I would use something like Flysystem for PHP, or pyfilesystem for Python where an environment variable would decide whether it's saved to local disk or S3.

For some more difficult use cases I have to be pragmatic and decide on the circumstances. E.g. when needing presigned URLs (which is an S3 feature) I will use the native S3 SDK, it's not the cleanest way, but it gets the job done. Only when it starts getting annoying / slowing the development speed will I abstract it away. Security is important and I keep a high standard there, but when it comes to clean abstraction I will occasionally make compromises, knowing that often we will only need one implementation or that feature might be thrown away (I work freelance mostly with startups or SMEs).

Re:The Amazon Premium by the_gipsy on Friday December 20, 2019 @05:41AM (#21843108)
Thank you. BTW I agree with starting simple and only abstracting if necessary.
Re:The Amazon Premium by potta_coffee on Thursday December 19, 2019 @07:31PM (#21840746)
Amazon recommends "testing at scale", basically spin up a non-production version of your architecture and try it out.
Re:The Amazon Premium by anaganisk on Thursday December 19, 2019 @10:31AM (#21835700)
Checkout Lightsail, been using it past 6 months. Power and freedom of digital ocean in AWS ecosystem.
Re:The Amazon Premium by jacurtis on Thursday December 19, 2019 @02:01PM (#21838033)
To be clear, Lightsail is only "sorta" in the AWS ecosystem.

It is in the ecosystem in the sense that it ultimate boots up an EC2 instance. It is essentially a simplified skin on top of EC2 where AWS makes most of the decisions for you in regards to server setup, with simplified pricing. And it shows up as AWS on your bill, and can be billed along side other AWS services you use.

However, it isn't really in the ecosystem as I would argue. You can't connect it to networks and subnets you create. You don't have control over availability zones, so essentially everything (like talking to S3) is considered egress traffic. It is like a sandboxed AWS service, that sits out there on its own. It really isn't any different than using Linode or Digitial Ocean for your compute resource and using AWS for block storage, queuing, DNS, etc.. Other AWS services can't "see" your Lightsail instance, like they can see your EC2 instances. You can't connect EBS volumes to it, you can't take advantage of elastic IPs, or anything of that sort. It really is a separate service that sits on its' own, under the comfort of the AWS name.

If I remember right, even tags don't carry over between Lightsail and the rest of your AWS account. Again, Lightsail is fine as a service, but it is only "sort of" in the AWS ecosystem. It mostly sits on its own as an isolated service, backed by AWS.

Re:The Amazon Premium by pixelperfect on Thursday December 19, 2019 @11:07AM (#21836113)
Looks like you get charged if you go over the transfer allowance. Is there a way to automatically shut everything down to prevent that from happening? I like to go months without thinking about my server, knowing there's no way for a bot army to come around and ramp up my charges. I realize that's paranoid, but Amazon's competitors give me that option.
Re:The Amazon Premium by donarb on Thursday December 19, 2019 @01:09PM (#21837506)
The other cool thing about Lightsail is that you can upgrade to EC2 should you need to go to the next level. I assume that the Lightsail instances are really some sort of EC2 instance that has a user friendly wrapper around it.
Re:The Amazon Premium by maest on Thursday December 19, 2019 @10:41AM (#21835800)
And none of the cost control.
Re:The Amazon Premium by kugelblitz on Thursday December 19, 2019 @10:33AM (#21835738)
Thanks! I had taken a look when it first launched, but haven't looked at it since. Will try it out for the next (low-risk) project.