Exactly! I'm a sysadmin (sometimes a Dev), but sysadmin is what i'm really good at. The most important thing is to see a solution from another perspective.
Sometimes, and sometimes we have to ask the developer of that exact OS-subsystem/driver/firmware(AARGH!!) because it fails us too...and here we have a closed circle ;)
I was in ops for 13 years, and if you were to talk to any of my former coworkers they would bury you with praise for the quality of my work. Yet, I eventually chose to move into a management track because I had peaked my career about 7 years in and didn't realize it until later. There was nowhere I could go up, because I wasn't a software developer. Now I'm a manager that has technology understanding, which has a high value prop for many orgs all on its own, but I do sometimes miss "getting my hands dirty".
I've worked with a lot of software developers over the years, and while there are a handful who are really incredible, the majority of people are just mediocre. That's expected and okay. The same is true for Ops folks, as it happens, although generally it takes more competence to rise to "Senior" on the Ops side vs software. The thing is, "Senior" is as high as it goes for Ops folks. So you might meet really stellar Ops folks who are effectively titled and paid the same as a mediocre developer with 3 years of work experience. It's simply not sustainable, and the push towards moving everything to the cloud and off-premise is probably a symptom of this (not enough quality Ops folks to keep things on-prem) and exacerbates this (reducing need for quality Ops folks, driving down market demand, unless you want to work at a cloud provider).
Pretty much all of the other Ops people I've respected and admired over the years have moved into different career paths. I find the same is not true for software developers. So when younger people ask me about career paths, I always recommend software over Ops, if they are adamant they never want to go into management.
It's kind of sad, I suppose, but that's the way of it. I appreciate that there's a subthread on HN where folks recognize and respect the value of competent Ops folks, but I think you'll find that most are being pushed out of that career path.
As soon as you start using the full suite of cloud tools, it's impossible to not give the provider the encryption key...
I'll be honest - you guys need to find a new technology partner then. Creating a private cloud that's reliable and offers the basic services that the major providers have is not difficult in 2020. Some of the aaS stuff can get tricky but is still entirely do-able.
Regardless, if they found a way to make your infrastructure MORE expensive than the public clouds they either have no idea how to negotiate or are really, really bad at their jobs. The public cloud is a lot of things - but cheap isn't one of them.
If your business involves any amount of low priority bulk compute, this gets much much easier. You simply let the low priority stuff fall behind while your order to Dell for new servers is being delivered...
Also, if you have compute that could be on-prem or could be in the cloud, you can set up a kubernetes cluster spanning both and let non-privacy-sensitive overflow to GCP as needed.
All of the above rarely comes out cost-effective though, because while the raw compute is cheaper to DIY-it, when you factor in the staff time to build, maintain, and deal with the shortcomings of your bodged-together on-site solution, it's going to come out much more expensive.
We've built out something homebrew like this using cheap compute desktops but you're still going to pay a lot upfront.
Meaning, e.g. there are specific, rigid rules regarding how Postman can be used while developing backend services, to avoid that customer info is inadvertently transmitted during testing.
Of course, it’s a PITA, but it serves its purpose.
But the idea that Postman is actually git in that analogy is hogwash and a way to wrest data out of people.
It's done on a budget as well so we're kinda forced to use open source software.
Weirdly enough we use Skype for work chat and Zoom for video meetings, so it feels a bit inconsistent.
It's like as if somehow, in many eyes, that's impossible now. Depending on your scale, it may be impractical given external hosting options but it's certainly not impossible. Lots of data centers rose and started automated processes of shared hosting and co-locating hardware which was a step forward. I remember working with Rackspace, Equinix, The Planet, etc. which further automated quicker server deployment, had applications for UPS resets/interrupts, etc. The more you moved towards a specific business's automation services, the less portable your infrastructure was outside that environment.
That continued on until you now have more sophisticated hosting like what AWS and GC provide. Now, abstractions exist for about everything in a data center and the trade off is that you now have to manage all that complexity through proprietary APIs, consoles, and so forth.
In addition, the tradeoff here is the more complex the infrastructure, the less easy it is to shift it to another provider. That may be fine for you, it may also not be. It's all definitely possible though.
We have a gitlab server, and connecting requires being on VPN, which requires 2FA, and then ssh, which requires your keys to be properly set up.
As long as you have decent hardware (cpu/ram mostly) and reliable storage (netapp or something similar) you can get stuff done very easily.
Also, most people seems to not have noticed how fast computers and disks got recently and how resources you can pack into a single physical machine: you can nowadays fill a 2u, 2socket machine with 128c/256t (2x amd epyc) and literally terabytes of ram...
Good luck finding one. The world is divided in 3 spheres of influence: US, China and Russia. To make business you must obey one of them.
For most practical purposes, the EU is nested within the US sphere of influence.
Thats not a insignificant cost.
Nobody attacks code repositories of non-software companies anyway, people are after CRM and ERP data. There is the occasional issue with malware from mails and special users, but a backup solution with 15min snapshots solves that issue. Although the latter can cost a bit and might be too expensive for smaller companies.
And honestly moving to these tools from slack, confluence, etc. has been awesome.
Zulip threading model is great. So much better than slack.
And using markdown and jupyter notebooks for documentation on gitlab? Damn awesome.
- Is your company mainly a software company?
- How much time do you spend on a week on maintaining your servers?
- How do you make sure that your servers are secure? Maybe you are being hacked every night, does your company have the means to check if there has been a security breach?
- Do you follow/apply the security patches for the OS you are using on the server and all the software you are using on the server?
- Do you have regular offline backups? What would happen if there is a fire in your offices?
These are some of the reasons to go for a cloud solution, especially when you are not a software company (hence you don't have many people who have the knowledge for setting up/maintaining such stuff) or when you don't have the resources to hire dedicated sysadmins.
Mostly yes. We obviously have sales, marketing, etc. as well.
> - How much time do you spend on a week on maintaining your servers?
I don't do devops. There is a team of people that works full-time on IT infrastructure. No idea how time they spend. Gitlab and Zulip servers are updated every couple of weeks. No idea how much time these cost.
> - How do you make sure that your servers are secure? Maybe you are being hacked every night, does your company have the means to check if there has been a security breach?
There is a team of people that work on cybersecurity monitoring. No idea what they do. Normal IT people just make sure that everyone's computer is encrypted, setting up people's credentials, etc.
> Do you follow/apply the security patches for the OS you are using on the server and all the software you are using on the server?
I don't do anything, somebody does this for me.
> Do you have regular offline backups? What would happen if there is a fire in your offices?
We have multiple locations and the backups are replicated across our own locations.
Obviously we're not all going to build our own CPUs from sand at a local beach. So there is a balance between DIY and vetted suppliers
I mean I like to think the data has no value to e.g. the US or competitors, and that the sheer volume makes it worthless, but I suspect that's just a lack of imagination on my part.
Just because we (Germans, Europeans) are culturally closer to Americans and share certain values does not mean that we should have a double standard on our external affairs.
We are kicking out Huawei. When will we kick out Amazon, Google, Apple and Cisco?
This law, still on the books, theoretically allows the president to order military action against the ICC in Den Haag (The Hague) should they ever try an American Service Member.
> The act also prohibits U.S. military aid to countries that are party to the court. However, exceptions are allowed for aid to NATO members, major non-NATO allies, Taiwan, and countries that have entered into "Article 98 agreements", agreeing not to hand over U.S. nationals to the court.
[1]: https://en.wikipedia.org/wiki/American_Service-Members%27_Pr...
Nonetheless that does not make the US forces in Germany an occupational force.
A stick is a stick regardless of whether it has been used to thrash someone lately.
Maybe Germany (and the EU as a whole) should start paying for its own defense, and paying their agreed to share of GDP into NATO while they are at it
Huh, and here I thought it was 2020 and not 2024. Weird.
> Allies whose current proportion of GDP spent on defence is below this level [2% of GDP] will […] aim to move towards the 2% guideline within a decade.
From the 2014 NATO summit: https://www.gov.uk/government/uploads/system/uploads/attachm...
It works the other way around. We cannot have an European industry meanwhile USA government backed-up companies are competing with our startups.
> collection of small countries
Europe is bigger than the USA. And the division within the USA is as big or bigger than Europe. And, we cannot talk about China.
OK, on some metrics. But I think nobody questions which country is still leading the world.
> And the division within the USA is as big or bigger than Europe. And, we cannot talk about China.
I don't agree with this or we're talking about different things.
The grip of Washington DC on the states of the USA is much stronger than Brussels' on the countries of the EU. The degree of sovereignty is very different. No matter the internal divisions inside the USA, first of all there is one USA. On the other side of the ocean first of all there are multiple competing EU countries, each of them trying to exploit the others and the EU and with different economics and foreign policy goals. About that, do (random picks among large states) California, Texas and New York have a foreign policy worth talking about?
If you're in a security alliance with countries you share values and goals with, how can you not apply different standards in terms of data sharing? And how can you not react somewhat differently to any transgressions, mistakes and imperfections?
I know Trump has made this very difficult by making his personal whims indistinguishable from the interests and commitments of his country. In my view this is a form of corruption that does massive damage to the U.S. And yes, democratic control over security services has been rather tenuous at the best of times.
But I still see many good reasons not to impose broad economic sanctions on allied democracies. And I do see good reasons to side with them against completely illegitimate regimes that use their security services to keep themselves in power without a democratic mandate.
Of course there are many grey areas and a lot of valid criticism. But asking for Europe to officially impose economic sanctions on the U.S is not a proportionate reaction to that. The economic damage would far outweigh any additional freedom or security.
If you refuse to apply a degree of pragmatism and proportionality, other countries would then have to impose economic sanctions on Germany for their use of Staatstrojaner. Essentially, all cross border trade in digital goods and services would have to end globally. That can't be a good idea.
We've had to warn you about this before: https://news.ycombinator.com/item?id=23645175.
Yes I do. It follows from the simple fact that there are no free and fair elections on those countries. Freedom of speech in combination with elections gives us some power to change things. People in dictotorships don't have those powers, regardless of whether or not they are homeless.
Homelessless exists in Europe as well by the way.
Umm... the British monarchy hasn't held executive authority in centuries. They're window dressing like almost all European monarchs.
Is the "spying" by the US on Germany as damaging to Germany's future as spying by the Chinese?
If you answer these questions honestly, you'll understand why you're making a false equivalence.
What are the realistic odds that someone internally at Amazon is going to break into your instances to look at your data if you're just a regular business?
If you're in a German cloud, those risks are probably higher since your business has local competition, and if you're self-hosting, then your overall security risks are even higher.
This seems like a emotional choice, not based on realistic business or security sense.
I'm pretty sure you can think of stronger business cases against Amazon/US-Milspec involvement in a non-American business.
Amazon, in the hands of Americas spooks, can do a lot of damage to the world of non-Americans; i.e. anyone the spooks decide to hate/target for usurpation. You think Germany is inured from such attention?
I run a simply mid-level business. Amazon doesn't give a crap about my data - that, above all, is what keeps it safe. I'm just not important enough to bother.
Well, you should understand this a little better, then.
Okay try this scenario: lets say I work for a German company that is kicking ass in a market that a spook-at-Amazon has heavily invested in - lets just say "personal transportation system X".
Because its Germany, the tech is amazing and consequently I'm going to sell this tech to Africa, because I like doing that, and not in America - because fuck America, that's why. Also, maybe China, maybe Kazakhstan. My tech, my choice.
Should I succeed in this endeavour, I'm immediately under threat of being NSA'ed by the provider of service I'd rely on, ordinarily, to render my research.
This isn't paranoia, it is due diligence.
Yeah, no thanks, Amazon. This heralds a watershed moment: a return to local services.
Just better to avoid the American spook-o'-sphere entirely, and play a local hand. Maybe even take the data centre guys out for grünkohl and senf some time, whatever.
Look - to understand why the hate for all the spook/control paranoia, Americans/Brits/5-eyes/et al., all you have to do is try to see it from the other side of the national identity for a second.
Germans are very, very rightly on the side of data protection.
The holes in the ground that used to be Gestapo buildings are not all parking lots! Some of them are memorials.
Also not paranoia: diligence.
(^WThis comment brought to you by the ACME-HNnnn-comment-slowdown-negotiator-system™ .. time wasted, is creativity! Disclaimer: Not German, am applying hypotheticals. Still not paranoia!)
I mean, there's this: https://www.wsj.com/articles/amazon-scooped-up-data-from-its...
> The online retailing giant has long asserted, including to Congress, that when it makes and sells its own products, it doesn’t use information it collects from the site’s individual third-party sellers—data those sellers view as proprietary.
> Yet interviews with more than 20 former employees of Amazon’s private-label business and documents reviewed by The Wall Street Journal reveal that employees did just that. Such information can help Amazon decide how to price an item, which features to copy or whether to enter a product segment based on its earning potential, according to people familiar with the practice, including a current employee and some former employees who participated in it.
If you're a regular business and Amazon doesn't want to launch a new hosted product competing with you, then maybe you're safe. But if they want your data, why not just take it while repeatedly insisting that they don't?
It is wildly different to scoop up data from your own Amazon.com platform vs BREAKING IN to a private AWS instance. They aren't even the same divisions in Amazon Inc, so it would be extremely obvious to everyone, very inappropriate, and most likely get leaked that it was happening.
Going to go out on a limb and say that the NSA is going to have an easier time breaking into your on-prem servers.
Hiring a retired General for influence is as old as this town. People are reading too much into this. This doesn't make it more or less likely for Amazon/NSA/CIA/whatever to steal your data.
Perhaps Amazon did this on an abstract level to understand video streaming and the Netflix business to create Amazon Video but I doubt Amazon would have cared for the tiny startup I worked at here in Hamburg.
But the hysteria and the “but what if?” thinking is real and it was really codified in bank contracts. So much so that BaFin stipulated that the tech stacks could be audited and there was no way that Amazon was going to allow anyone in to look at it and such.
On the whole if it can be afforded my team is so much more productive being able to build on AWS than to spend all that time both building our product and the infrastructure to go with it. I’m just not that smart.
It's hard to imagine that this includes the code within a Cloud provider. If you're using a Windows server, are they expecting Microsoft to fork over MS source code?
This just sounds like an overly-conservative interpretation of the rules.
It basically rules out ANY cloud solutions - which is unreasonable and wouldn't ever hold up under scrutiny even if the Bank or regulators explicitly asked for it.
It feels like the rampant capitalism in the United States, and lack of regulations, fuels the VC economy. Making startups like Google, Amazon and others possible.
I can't imagine a company running on hundreds of millions in dollars of VC money with no real profit coming in, just hoping for an exit.
That's a very strange phenomenon, sort of like the .com bubble never burst but just got bigger.
I'm afraid that in order to see the same rapid development of cloud companies here in Europe we'd have to adopt the lax corporate regulations of the US. Which in turn would lead to other issues like workers losing rights.
Google, Apple, Amazon, ... are not startups, they are behemoths backed up by the USA government. Europe has many very good startups that are just purchased with USA dollars once they are successful.
> we'd have to adopt the lax corporate regulations of the US
That would only ruin Europe standards of living and open us more to be purchased by USA companies. The other way around is the way. It worked for China. Forbid non-European companies to purchase key software companies.
USA is not successful because capitalism, USA is successful were the government puts a lot of money: Military, I+D.
What I meant was that they were startups at one point.
I could have given other examples but there are so many that it's hard to decide. Dropbox, Reddit, imgur just to name a few. This site should know plenty of VC fueled startups.
The point was not which startups to name as examples, the point was that lack of regulations make this whole VC system go around.
I could never see that system working here because bureaucracy slows everything down and forces corporations to do things like care for employees and pay taxes.
But with that said, recent news shows that Sweden has given tax exemptions to Facebook and Amazon just like everyone else. To curry their favor. It's frankly disgusting.
Again, you are getting it the other way around. Taxes to invest in education and Development and Research and high qualified citizens is what allows innovation and good companies.
This is incorrect. The current big tech companies became successful and massive without any special government deals. Google dominated search, Microsoft dominated all business and home PCs for decades, Apple and Google dominated mobile for consumers and businesses, Amazon beat them all to cloud and ate online retail with free fast shipping.
It’s only recently with JEDI that Amazon/Microsoft tried to attach directly to the government’s wallet in a big way. And even with losing that, it will be a drop in the bucket for either of them.
These companies were successful because they were allowed to move quickly and beat out competitors. The entire Internet industry blossomed and gave birth to these companies before the government even took the Internet very seriously.
Where did their finance came from: previous SF companies and angel investors
Where did these companies get their funding from: US government
The current companies are only the "daughters and granddaughters" of massively government funded companies in Silicon Valley
Very good point. Many people forgets how much money the state invest, as it should, on technology and to create an industry. From ARPANET to the WWII computers (or NASA), technology needs a level of investment that only states can afford.
Cloud Act and National Security Letters really fuck things up for American cloud providers with regard to doing business with European customers.
Never trusted Bezos and the Amazon offerings are too expensive anyway.
There is an AWS Region in Germany. Your data will stay in Germany unless you specifically decide to move it elsewhere.
AWS also provide you with the tools to encrypt everything and if done correctly means that Amazon no matter how "evil" they are cannot decrypt the data. Not only that, there is such a huge separation in access and rigour around governance that there is no way anyone within Amazon can simply login and see your data even if unencrypted.
Every single case of data being leaked from AWS is because the people working for the company that manage the data literally checked a box to make the contents public. Contrast this to "on-prem" where physical security can get compromised or the vendors of the physical hardware/software leave gaping holes or maintenance backdoors that get exploited.
Honestly these types of views are no more grounded in reality that flat earth conspiracy type views.
The measures you've enumerated (EU zones, encryption, etc) are mitigations for working with a potentially compromised vendor and should be done anyway. Not using the vendor is such a blindingly obvious countermeasure that one has to be either unprofessional or have a hidden interest to dismiss out of hand.
I personally choose not to believe a company which puts on its board of directors a well-known perjurer [1].
[1] https://www.theguardian.com/commentisfree/2013/sep/25/nsa-re...
Prove it
They say "We are sorry", give you some free credit, and that's it ?
"Sorry not sorry" and no credit.
You seem to suggest that every single employee that works at AWS is aware of everything that goes on with the company.
Isn't just possible that despite of what we know about the architecture, this might not be the whole truth and concerns about involuntary inter region transfers might just be warranted?
The problem I see with the DIY-attitude is that security is easy to get wrong and is moving target. The other opinion here is "keep it in the country". If someone really wants your data, the locale won't save you. And yet, if there is a breach, I could see an foreign company like AWS trying to hush it, where a German company would make a bigger fuss (diplomatic issue).