No SANs, anywhere! :)
> Are you guys using Eucalyptus?
It's our own stack. Eucalyptus afaik doesn't handle zones as geographically distinct datacentres and, when we looked at it a long time ago, had some pretty worrying SPOFs.
> Why is your outbound data more expensive then?
Transit is symmetrical, and incoming bandwidth is less utilised compared to outgoing so we charge less for it.
> How are you securing KVM?
In what sense?
(Co-founder at Brightbox)
Ephemeral disks? Or persistent local?
> Our own stack
Very cool ^_^ How do you deal with geographic zones? Are they silod?
> Bandwidth
Ok, makes perfect sense. Thanks.
> Securing KVM
Do you use cfgroups/selinux to deal with compromise of a kvm domain? I've seen quite a few vulnerabilities coming out on the debian/etc. security mailing lists.
Persistent local disks (hardware raid6 15k rpm). More storage options on the roadmap too.
> Very cool ^_^ How do you deal with geographic zones? Are they silod?
Our zones are different datacenters in different buildings, with completely different power supplies, UPSes and backup generators.
> Do you use cfgroups/selinux to deal with compromise of a kvm domain?
cgroups currently, selinux in development.
(full disclosure: I'm a Brightbox bod too!)
> cgroups currently How do you protect the kernel from something like CVE-2011-2212?
Quite cool otherwise :)
http://security-tracker.debian.org/tracker/source-package/qe... http://security-tracker.debian.org/tracker/CVE-2011-2212 https://rhn.redhat.com/errata/RHSA-2011-0919.html
So much of this can be automated now that it is not a problem. As a provider myself, I allow customers to pick their patch day/time. They can even manually push patches themselves and be present to test when the service comes back up. Proactive maintenance(datacenter, networking, hardware, OS, and appptack/utils) should be considered a way of life these days if you're a provider. If customers don't understand or agree with that, then there are plenty of providers who don't keep up-to-date offerings that they can migrate to.
We protect against those types of problems at the moment with keeping patched up to date. Xen doesn't solve this problem either, it has had it's own share of vulnerabilities with these kinds of repercussions. Even selinux only mitigates some of the risks - not all. A combination of mandatory access control and a good update, audit and monitoring strategy is the best approach imo.
Interesting. I agree with your approach. When I was looking at KVM, I noticed its rather insane surface area (Everything is in the kernel as a kernel module) and that almost all of the vulnerabilities found seem to stem from that arch decision. As an example have a look at this: nelhage.com/talks/kvm-defcon-2011.pdf
I was just wondering if you knew of any way to secure that down, or any way to patch quickly enough that you don't break SLA by having to forever reboot people.
Just generally given that we've had CVE-2011-2212 CVE-2011-2527 CVE-2011-1751 CVE-2011-0011 CVE-2011-1750 for kvm itself in the last few months, so say you have 5 critical bugs in kvm a year. If you need to restart everybody to patch and it takes a minute or so to restart per vm and you have say 40 vms/box (random numbers for the sake of argument), that's 3 hours of downtime/year not counting kernel upgrades/etc. That means the best you can do is 99.9% uptime not even considering transit failure/dc power failure/etc.
So my question is how do you deal with that?
Remember that all of these fixes are upgrades to userspace - which means there are many more upgrade options than if they were kernel or hypervisor (think the equivalent of a live migrate to the same host, as an example).
Btw, Brightbox happened to be the original reporters of CVE-2011-0011 :)
There are a few interesting questions raised, though.
Are you guys using local disks or SAN?
Are you guys using Eucalyptus?
Also, it's interesting that you charge less for incoming data then outgoing. I understand that standard asym links are cheaper upload then download, due to them being basically "slack" cap from Adsl tails/etc. Why is your outbound data more expensive then?
How are you securing KVM?
(assuming somebody from there is around, or that anybody else would have answers....)