News for hackers. Stuff that matters.
Re:Why I Won't Use Rimuhosting by SwellJoe on Thursday August 27, 2009 @03:11PM (#789815)
Several of our customers are hosted with Rimuhosting, and we've interacted with them on a number of occasions because of that. In my experience they are very responsive, knowledgeable, and helpful. That's not to say this isn't a pretty irritating set of events...and silently shutting down Apache shouldn't be anyone's first instinct.

But, I've had bad experiences with literally every single host I've ever used (dozens of them), and the longer you stay with a provider, the more likely you are to find a tech having a bad day, or their network fiber getting cut by the utility company, or various other demonstrations of man's fallibility.

I think the good probably still outweighs the bad with Rimuhosting...as Aaron mentioned, they do have a good reputation among many people who ought to know. The horror stories I hear about really bad hosts leave this event in the dust in terms of sheer stupidity and lack of good intentions. If this is the worst you've ever experienced with a hosting provider, count yourself very, very lucky. The data loss stories I've heard, in particular, are enough to keep me awake at night.

In the interest of full disclosure, I'll mention that Rimuhosting is a customer of ours (along with thousands of others). So, if you reckon a commercial relationship worth a few hundred bucks a year would sway my opinions, feel free to take this comment with a grain of salt or not at all.

Re:Why I Won't Use Rimuhosting by _pius on Thursday August 27, 2009 @05:04PM (#790047)
The longer you stay with a provider, the more likely you are to find a tech having a bad day ...

I'm sorry, but the behavior described in the article is completely unacceptable. I don't care how bad of a day anyone's having, you don't break into your customer's server and shut off Apache without notice.

Re:Why I Won't Use Rimuhosting by SwellJoe on Thursday August 27, 2009 @05:38PM (#790140)
It's not "breaking in". They own the server, and pretty much every host's terms of service allow them to do pretty much anything they deem necessary to maintain service for their other customers, or in response to abuse or law enforcement requests (read the terms of service at your favorite hosting provider; you'll find that you've agreed to this kind of intervention when you signed up).

So, this particular event shows a lack of good judgment on the part of the tech in question, and a lack of common courtesy in not notifying the customer of the issue in advance of taking action...but it is not an illicit act. No "break in" happened here. The options a hosting provider has when things are behaving badly are pretty much all things that a customer will not like. When a dedicated server is compromised and is being used to send spam, for example, the host will simply firewall the whole thing off from the world, shutting down all services. This happens all the time; the effected customer blames the host, sometimes. Sometimes there are other options. In this case, the host opted to turn off one (disputably) offending service rather than kill the whole host. I think we can all agree this wasn't the best way to handle it. Thus my assertion that this is a tech having a bad day (or simply one bad support tech at Rimuhosting, among mostly good ones).

Actually, I think the one thing I'm uncomfortable with is the illusion of propriety (ownership) Rimuhosting has given by saying "You are able to remove that if you are not comfortable with that.", in reference to the ssh key. This is a bit of a disconnect from the reality that under some circumstances, they will have to exert their ownership rights, and there's nothing you can do about it. At some point you just have to trust in the good faith of your provider. But the only alternative to doing what they did (assuming they, in good faith, thought Aaron's VM was damaging service for others), assuming they don't have the ability to throttle Xen instances on this particular machine (maybe it's too old to have that capability, I dunno, or maybe the tech in question didn't actually understand those capabilities; the tech could be new...everybody starts somewhere), would be to turn his VM off completely. From the techs perspective he may have thought he was doing Aaron a favor by leaving mail and other services running.

Again, I'm not arguing it was good judgment on the part of the tech. But to say that Rimuhosting broke into their own machine is ridiculous.

Re:Why I Won't Use Rimuhosting by mbreese on Thursday August 27, 2009 @09:34PM (#790533)
But to say that Rimuhosting broke into their own machine is ridiculous.

I'm not sure about that. Since the VM is virtual, you could say that they don't own the VM, they own the machine the VM runs on. They can control and manipulate the VM via the container for running them (Xen in this case). But since the VM doesn't physically exist, it could be argued that they don't actually own it, they only rent server capacity to run the VM.

Now, I know that you could similarly argue that they were only reading/modifying a disk image that existed on their server, but that isn't as much fun to argue.

I think that everyone here can agree that this was a dumb move by Rimuhosting. And I'm not sure what the contractual arrangement is when you buy a VPS from them... but what they have posted online doesn't make this very clear.

The question of ownership is interesting in this era of cloud computing (and not at all clear). I think it's clear that they own the bare metal. I think it's also clear that you own your data. However, where those mix is an interesting question. At what point is their (uninvited) intervention in a customer's VM warranted outside of their normal management options in the VM container (Xen)? At what point is the intervention too much?

Does the company matter? I can imagine everyone being up in arms if Amazon tried the same thing with an EC2 instance.

Of course this could all be alleviated by using a dedicated server... but even then, if the hosting company owns the hardware, would it be okay for them to pull a hard drive, and add themselves as root? Never. Ever. But this is effectively what we're talking about. The only difference is that with a dedicated server your actions don't affect others. In a VPS environment, your actions can affect others. But I think that there are far better methods for controlling this than "breaking in" to a customer's VM.

Re:Why I Won't Use Rimuhosting by astrec on Thursday August 27, 2009 @06:44PM (#790298)
If my VPS was on the same host I'd pretty much demand they do exactly that.

Actually I'd probably demand they stop the instance, so I reckon he got a free kick if they only stopped the offending service.

Re:Why I Won't Use Rimuhosting by lsc on Thursday August 27, 2009 @08:15PM (#790425)
CPU is one of the easiest things to throttle in Xen. there's no reason why the provider should care at all that you are running as much cpu as they let you. Proper use of the credit scheduler means this happens without administrator intervention.

You can also gracefully limit disk bandwidth and network bandwidth from the dom0, without ever getting a shell in the DomU.

Re:Why I Won't Use Rimuhosting by Andys on Thursday August 27, 2009 @08:36PM (#790467)
Yes, and this would have been the correct way to fix the situation. Gracefully reduce the available CPU for the offending VPS, and notify the customer.

Breaking in is OK as a last resort.

Re:Why I Won't Use Rimuhosting by lsc on Thursday August 27, 2009 @08:53PM (#790489)
Personally, I think freezing the domain is preferable to logging in and killing a (likely rather important) process.

But then, the discussion is academic. I've never had to shut down a domain to prevent heavy legitimate or runaway process use from harming other users.

I have shut down many domains to stop abuse, but that's a very different thing. If you want to examine a compromised disk, you are best off doing so booting off of read-only media then carefully examining the compromised partition, so you want it shut down anyhow.

Re:Why I Won't Use Rimuhosting by SwellJoe on Thursday August 27, 2009 @07:00PM (#790331)
I think this is actually the heart of the matter. No one (or almost no one) is arguing that a hosting provider shouldn't be able to take steps to prevent one customer from negatively impacting others.

What is at issue is whether the tech should have stopped the instance, or logged in and located and stopped the offending service. The privacy nuts among us (I'm one of them, but with a healthy dose of "hosting industry reality" thrown in for good measure) are crying foul that anyone should ever be able to login to "your" server under any circumstances. The other side of the coin is that a large percentage of hosting customers have no idea what they're doing, and rely on the host to occasionally step in and set things right.

Hosting providers tend to default to stepping in, because that's what the majority of hosting customers want and expect. Rackspace has built a ~$1 billion business on this premise. "Managed hosting" is more expensive because you get this kind of intervention (hopefully in a more positive direction most of the time) on a regular basis.

The only wrong, I think, in this case is that the tech at Rimuhosting did not respect Aaron's specific wish to not allow him to access his box. And, since I know Rimuhosting has a good reputation that spans a decade or so, I must assume it was a mistake made in good faith. It is entirely possible the tech didn't reread the full conversation about the problem, and missed the bit about "over my dead body", and just assumed Aaron would want to keep email spinning while he figured out why his website was causing problems.

To turn this into a huge political struggle is pointless, and casts aspersions on a company that is by most accounts, one of the really good ones in their industry.

Re:Why I Won't Use Rimuhosting by astrec on Thursday August 27, 2009 @07:42PM (#790392)
The only wrong, I think, in this case is that the tech at Rimuhosting did not respect Aaron's specific wish to not allow him to access his box.

Do you think this is more or less wrong than simply halting the instance?

Re:Why I Won't Use Rimuhosting by SwellJoe on Thursday August 27, 2009 @08:05PM (#790416)
Do you think this is more or less wrong than simply halting the instance?

If they had reason to believe his instance was impacting other users or their network, then halting the instance (preferably after contacting him about the problem, which seems to have happened) would not have been wrong at all. So, logging in after he'd asked them not to is definitely the "more wrong" of the two options. I tend to think it was an honest mistake made by a support tech who was having a bad/busy day rather than an attempt to violate any (real or imagined) privacy pact that the company had made with Aaron.

As I've mentioned a few times here, the default expectation of hosting customers is that the host will login to find and fix problems. From the perspective of most hosting customers, that is part of the service they are paying for (and they pay more to get more of it, as in the case of RackSpace), and if it were missing they would be angry and leave. My guess is that the tech at Rimu simply didn't remember or notice that Aaron had requested no one ever login to his box.

Basically, what I'm trying to say is that, as someone that is deeply familiar with this industry, when I read the shitstorm that has been kicked up here over this (extremely minor) incident, I'm just stunned and baffled. This is seriously a tempest in a teapot, if ever I saw one.

Yes, your hosting provider should respect your privacy, to the best of their ability to do so while shepherding their shared resources responsibly to provide the best service to each of their customers. But, there is no accusation of theft of data; no reason to believe Rimuhosting acted inappropriately with the access they had; no reason to think it was done maliciously or completely without warning (Aaron admits he'd talked with them in the past about the issue); and no reason to think Rimuhosting was trying to piss anyone off.

Not being thoroughly informed about a customer's wishes is pretty much all one can accuse them of here.

Re:Why I Won't Use Rimuhosting by sho on Thursday August 27, 2009 @07:15PM (#790358)
The only wrong, I think, in this case is that the tech at Rimuhosting did not respect Aaron's specific wish to not allow him to access his box.

The thing is, that's just Aaron that thinks that. I'm sure other customers would be delighted for tech support to access their boxes to fix problems, indeed annoyed if they didn't.

Aaron pays $20 for a cheap-ass VPS, uses too much CPU, and they go out of their way to fix the specific problem instead of just nuking his account, and he complains on some BS privacy grounds. You just can't please some people.

Re:Why I Won't Use Rimuhosting by apotheon on Thursday August 27, 2009 @08:40PM (#790475)
Yeah! You tell 'em! Nobody has any grounds to complain when a specific fucking request is ignored! Who cares if the customer gets what he wants? He should be happy he's getting what thirty other people want instead!

Criminy. Did you even read what you quoted before you responded to it?

Re:Why I Won't Use Rimuhosting by sho on Thursday August 27, 2009 @08:52PM (#790487)
You sound like a person at McDonalds. I SPECIFICALLY ASKED FOR A QUARTER POUNDER WITHOUT PICKLES AND YET I FOUND A PICKLE IN MY QUARTER POUNDER AND I WOULD LIKE TO SPEAK TO THE MANAGER
Re:Why I Won't Use Rimuhosting by apotheon on Friday August 28, 2009 @01:03AM (#790780)
It's more like he went to McDonald's and specifically requested that they not give him a colonoscopy -- but they did so anyway.
Re:Why I Won't Use Rimuhosting by sho on Friday August 28, 2009 @05:48PM (#792354)
You often get colonoscopies at your local McDs do you?
Re:Why I Won't Use Rimuhosting by apotheon on Friday August 28, 2009 @08:26PM (#792554)
Just about as often as my service providers break in to my servers after asking whether I wanted them to have access to those systems and I said "no".
Re:Why I Won't Use Rimuhosting by DocSavage on Thursday August 27, 2009 @05:17PM (#790076)
The data loss stories I've heard, in particular, are enough to keep me awake at night.

Over four years ago, I was helping my father-in-law establish a web presence and e-mail system for his small business. Things were fine on my personal dedicated server but I decided to move their stuff to Yahoo's "proven" business service. One day, I found out everything was removed -- email accounts, web site -- for no explainable reason. The whole sordid affair was documented here:

http://billkatz.com/2005/6/Run-Dont-Walk-from-Yahoo-Small-Bu...

Re:Why I Won't Use Rimuhosting by apotheon on Thursday August 27, 2009 @05:28PM (#790105)
Gaining root in such an underhanded manner against the client's wishes is bordering on criminal behavior. It doesn't get much worse than that.

Of course, I write about security professionally. Maybe I have a different perspective on what constitutes unacceptable behavior than you have.

Re:Why I Won't Use Rimuhosting by SwellJoe on Thursday August 27, 2009 @05:55PM (#790187)
Gaining root in such an underhanded manner against the client's wishes is bordering on criminal behavior.

No. Not even close. Rimuhosting owns the box in question. You can't possibly break into your own box.

Of course, I write about security professionally. Maybe I have a different perspective on what constitutes unacceptable behavior than you have.

I work in the hosting industry professionally. I've never been or worked at a service provider, but I've worked for the industry for 12 years as a vendor.

As a security professional, perhaps you can take the other side's view for a moment...

What would you, as a security professional who was dumped into the role of running a hosting data center, do when a box on your network (which has thousands of other boxes) has been rooted and is sending out spam at a rate of a million messages per day, even if you know that your paying customer is not responsible for sending those messages? This much traffic is impacting network performance for others, is spewing filth into the mailboxes of innocent people all over the world, and your customer is unable to stop it or cannot be reached in a reasonable time frame. What do you believe your responsibility would be in such a case?

How about if it is (innocently) DoSing other systems on the network? This happens a lot more than you'd think.

Many people here are suggesting that simply applying resource limits would solve this problem...but that exhibits a lack of understanding of how such resource limits work, and the likely end result for the customer of such resource limiting in the event something has gone wrong. To the customer, a limit on resource usage could very well also result in services becoming unavailable. If you cap CPU at 10%, and your website gets enough traffic to need 90% of the CPU, you'll only be able to serve a small percentage of your clients (and they will wait a long time). The same is true of nearly every resource limit in a hosting environment. If you are using more than your fair share (which I know Aaron has explained he was not; again, probably poor judgment and problem assessment on the part of the tech) then when the limits are imposed, you will be "turned off" for some of your users. You needed those resources to serve all of your requests...you don't have those resources anymore, so you can't serve all of those requests anymore, or if you do, you do so very slowly.

As programmers, you all know there are no silver bullet for solving hard, complex problems. Resource usage in a virtualized environment with untrusted users is a hard, complex problem. No amount of hand-waving about your "rights" makes that less true.

Re:Why I Won't Use Rimuhosting by apotheon on Thursday August 27, 2009 @06:14PM (#790238)
> No. Not even close. Rimuhosting owns the box in question. You can't possibly break into your own box.

I don't think you're familiar with the common uses of "bordering on".

> What would you, as a security professional who was dumped into the role of running a hosting data center, do when a box on your network (which has thousands of other boxes) has been rooted and is sending out spam at a rate of a million messages per day, even if you know that your paying customer is not responsible for sending those messages?

I'd stop the server without gaining root against the client's wishes, immediately inform the client, and try to get things cleaned up to everyone's satisfaction (except the security cracker who rooted the box, of course). I wouldn't gain root on the box in violation of the client's obvious wishes, particularly after articulating specific policy saying that you can remove my initial root access to the box if you like, and shut down specific services that are making my life difficult for reasons entirely unrelated to the client's running of the site, then send an email later pointing out my policy that fails to mention I might do this.

Privacy is security, after all: http://blogs.techrepublic.com.com/security/?p=293

Nothing you say in your hypothetical examples in any way justifies accessing the customer's data, particularly by breaking in to the system. Shut it down? Sure, if circumstances call for that. Start meddling with server configuration and establish what amounts to a rootkit on the system (in effect, if not in technical truth) without the client's permission? That's just shady.

> No amount of hand-waving about your "rights" makes that less true.

Perhaps you should stop waving your hands about that straw man, then.

Re:Why I Won't Use Rimuhosting by SwellJoe on Thursday August 27, 2009 @07:23PM (#790369)
I don't think you're familiar with the common uses of "bordering on".

I'll promise to read up on it, if you'll learn the meaning of "not even close".

Nothing you say in your hypothetical examples in any way justifies accessing the customer's data, particularly by breaking in to the system. Shut it down? Sure, if circumstances call for that. Start meddling with server configuration and establish what amounts to a rootkit on the system (in effect, if not in technical truth) without the client's permission? That's just shady.

I will merely mention that most hosting provider customers not only accept this kind of thing, they demand it. If you spend a little time on web hosting forums (as I must because it is my industry) you will notice a very strong tendency for complaints to be system administration related. The customer expected more involvement than the hosting provider offered, and thus things went horribly awry.

I agree with you, entirely, that if you ask the host to never login to your system, they should respect that wish. But, I can also state without hesitation that you and I (and most people here at HN) are thoroughly in the minority in wanting our hosting provider to never login to our hosting systems. The default mode for hosting providers is to drop in on the box within a couple of comments in their ticketing system...if it can't be solved with one or two replies, then it's safest to simply drop in and fix it. For most hosting customers this is not an invasion of privacy or "breaking in", it is "great support".

Finally, as a security professional, I'm sure you're also aware that with access to the hardware, your host has root all the time. There is nothing you or I can do about it. Even more interestingly, the host also has the ability to login, poke around, and never tell you about it (and not leave a trail...just boot up a live CD, mount up the disk read only, and poke around til their heart's content). Also nothing you or I can do about that. With someone else having access to the hardware, you have nothing but good faith on the part of the hosting provider.

Re:Why I Won't Use Rimuhosting by apotheon on Thursday August 27, 2009 @08:37PM (#790468)
> I'll promise to read up on it, if you'll learn the meaning of "not even close".

I don't see how "I know you are, but what am I?" is a very strong argument.

> I will merely mention that most hosting provider customers not only accept this kind of thing, they demand it.

Give it to customers who want it. Don't ask someone if he wants it, get a "no" answer, then turn around and do it anyway. The former is good customer service. The latter is shady and underhanded.

> The default mode for hosting providers is to drop in on the box within a couple of comments in their ticketing system...if it can't be solved with one or two replies, then it's safest to simply drop in and fix it. For most hosting customers this is not an invasion of privacy or "breaking in", it is "great support".

Most hosting customers don't have explicit suggestions from the host that if they don't want the hosting provider logging in to the system they can remove their SSH keys -- and, more to the point, most hosting customers don't do that then find out the hosting provider's support personnel have been rooting around (pardon the pun) in their data anyway.

It's "breaking in" in this case only because the SSH key for access was removed, with the hosting provider's blessing, and they basically leveraged a local access vulnerability to give themselves root access.

If the hosting provider had a policy that forbade customers from obstructing host administrators from logging in to the machine, articulated in the terms and conditions of use, I'd say go for it -- but that's not the situation in this case at all.

> Finally, as a security professional, I'm sure you're also aware that with access to the hardware, your host has root all the time.

In principle, sure -- but when there's a clearly encouraged expectation of privacy, it's really bad form to break in to the system against the customer's wishes by virtue of having access to the hardware. That's a betrayal of trust.

> Even more interestingly, the host also has the ability to login, poke around, and never tell you about it (and not leave a trail...just boot up a live CD, mount up the disk read only, and poke around til their heart's content).

Indeed. The fact it wasn't kept secret in this case just shows how little they value the customer's request that they don't log in to the system with root privileges and muck about with the customer's data. I didn't say they were necessarily malicious about it -- but that doesn't mean it's not bad.

> With someone else having access to the hardware, you have nothing but good faith on the part of the hosting provider.

. . . or, in this case, bad faith.

Re:Why I Won't Use Rimuhosting by lsc on Thursday August 27, 2009 @08:29PM (#790456)
xm sched-credit -d username -w 10

would have solved the CPU problem without doing anything more than slowing down the heavy user. Giving each guest a single CPU rather than all CPUs on the box also helps a lot. In fact, if your 'bad' user bought the same percentage of the box as everyone else, if you left the weights at their default levels, that user would get approximately their fair share of CPU.

Tip: if you don't dedicate a core to the Dom0, you will have problems if anyone else uses much cpu, disk, or network.

If you want to do the same for network, simply use tc: http://wiki.xen.prgmr.com/xenophilia/2008/06/traffic-shaping... - if your problem is disk, use ionice: http://wiki.xen.prgmr.com/xenophobia/2008/06/using-ionice1-t...

The whole point of providing Xen instances is that I don't want to muck around in your instance. It pisses you off, and it wastes my time. That's why I'm using xen, and not OpenVZ or some other glorified chroot. From the point of view of a provider, handling everything within the Dom0 not only makes customers more comfortable, it's also less work for you, the provider.

Re:Why I Won't Use Rimuhosting by astine on Thursday August 27, 2009 @07:12PM (#790353)
You can't possibly break into your own box.

Yes you can; I've broken into my own house before.

Personally, I'd like it if they gave you notice before doing anything to a VPS you may be using so that you have the opportunity to shut things down gracefully.

Even a sudden CPU throttle would be better than shutting down services on the customer's VPS.

Edit: spelling

Re:Why I Won't Use Rimuhosting by ErrantX on Thursday August 27, 2009 @06:08PM (#790223)
as another security professional I'm inclined to agree with Swelljoe.

(mostly because it's clearly going to be in the T&C's)

Re:Why I Won't Use Rimuhosting by apotheon on Thursday August 27, 2009 @06:20PM (#790249)
If there's something in the terms and conditions that says they reserve the right to change the configuration on your system because their explicitly identified policies for dealing with high CPU usage aren't working, I might back off from the "borderline criminal" descriptor -- but I'd still say that's a great way to make customers feel violated and run like the wind to other service providers. As many others have pointed out, a much less invasive approach would have been to simply pause the server and get in touch with the client -- especially after the service provider asked for permission to get root access, and was turned down.
Re:Why I Won't Use Rimuhosting by ErrantX on Thursday August 27, 2009 @06:58PM (#790329)
yep, I'd agree entirely with that :)
Re:Why I Won't Use Rimuhosting by mbreese on Thursday August 27, 2009 @09:42PM (#790537)
In this case their posted terms and conditions aren't very clear on the extent that they can muck around with a customer's VM. It's quite clear that they can suspend or kill it, but it's unclear how far they can go to "restrict" the customer's VM.
Re:Why I Won't Use Rimuhosting by rbanffy on Thursday August 27, 2009 @09:44PM (#790539)
Rimuhostinh owns the box. The client rents it, but the owner never gives up the right to step in and fix something if it is impacting the service they provide for other clients.

If I require this level of privacy, I will go with a co-location contract, not a virtual machine. Those are different products with different features. A VM on a shared chassis is hardly a private computer.

Oddly, I have some experience with virtual servers and never had much trouble throttling down an instance that wanted too much resources.