camuel

Exactly!

Some are role playing forensic investigators here. The guy admitted that he got overexcited and submitted unfinished work which he doesn't even remembers which build it was exactly and he published his current source code as OSS which scores even better and fixes all the accusations.

It's the exact opposite.

The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.

Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.

Computer chess is the canonical example of this.

Will be super cool to see SlateDB instead RocksDB

Perhaps we just need an S3 implementation specifically built from the ground up for the data-intensive workloads? Such implementation might use conventional S3 underneath for cost and reliability advantages. Such implementation can also deliver a lot of additional functionality which will also be uniform across cloud vendors like enforcing Iceberg conventions on the server-side rather than on the client side and those pre-conditions uniformly implemented or even a full blown transactions but without breaking S3 semantics? Metering and billing can be also made friendly to those data intensive workloads. The question is: if the overhead of such indirection level, right on the data path, can be made reasonable? This is what we are trying to figure out at Embucket.com

I wonder why no one mentions bitonic sort? If you want to do anything in SIMD you better avoid branching as much as possible... and ideally altogether. Here is an implementation I co-authored some 10 years ago: https://github.com/zerovm/zerovm-samples/blob/master/disort/...

Sorting-networks which were already mentioned seems similar but a bit too abstract.

My code above doesn't contains values but those are easy to add I think. Of course it is better to permute fixed size pointers / offsets and not the entire blobs which can be of variable size and then it will complicate everything beyond feasible for SIMD

So what is exactly Azure or Amazon offering for "AutoML Tables" or for "End-to-end AI Platform that runs ALSO on premises"?

How does this compare to KubeFlow?

It is claimed to be also an open-source project and meritocracy is common with open-source projects and github has the required automatic infrastructure for this "keeping track".

I think by "something" he means "phenomenon" and by "re-creating" he means "observing". So I think he is saying: if it is claimed that phenomenon X exists but there is no way to ever observe it for entity Y then X doesn't exists for Y.

But I'm not sure, it's my interpretation. The write-up resonates with me but I can see it as a bit ambiguous and inviting many incompatible interpretations.

Proper treatment of the subject in a book format would be great.

Google is very different from Intel on these fronts at least: 1. Google's open framework TensorFlow is exploding in popularity effectively moving the common interface up from CUDA. 2. Google has a huge need for internal AI/DeepLearning 3. Google has cloud and AI/DeepLearning as a service business 4. Google doesn't sells hardware

However, Intel dominance is not to be underestimated, they definitely can make industry wide impact quickly. Just saying you can not easily draw parallels here

we are heads down coding and haven't got time yet to update ZeroVM website. If you want to contribute contact us now please, if you to use ZeroVM hypervisor stay tuned...

but instance store is transient! You may argue if you do triple replicated in different availability zones then you are ok. Well, in this case it would be very costly as you will end up with constantly spinning EC2 cluster. Even if you don't do any computation you must keep it all spinning. And see what happened to elasticity... you end up paying inflated cloud prices for constantly spinning fixed size EC2 cluster! Instead of being able to rapidly roll out large cluster, make the computation and fold it back and pay only for what you have used - isn't it the true promise of cloud?

If all is that simple & rosy why no hosting/cloud service uses it?

Why entire shared hosting industry custom-patches their kernel constantly with OpenVZ and stuff?

Why shared hosting has such a bad reputation for being insecure (despite all efforts).

Every system might have weaknesses. What matters is:

1. Small surface for attack. With NaCl it is all concentrated in single tiny validator module. The model is also simple and mathematically proven to be secure. 2. Prior testing. It is especially hard for security product. Establishing motivating prizes is good way to ensure it is not easily breakable. 3. The speed with which patch is made available 4. Defense-in-depth, ability to have multiple levels of defense cheaply.

1) there is $100K bounty on each Chrome/NaCl exploit and we have only one ZeroVM 'syscall' that we allow with a lot of attention put how to make it easily secure. The situation is not same on Linux. First of all kernel exploits by process are not really considered severe in Linux and for sure it is not top priority to anyone. Linux built to be secure from outside not from inside.

2) All these is impossible in ZeroVM except accessing memory randomly and thrashing caches and TLB tables. Hm... that could work, I guess. For the first time in this forum we talk about real vulnerability. However, I think the problem exists also in KVM/XEN (will do a proper research now, Googling EC2 TLB thrashing doesn't yield anything interesting), no access to other tenant data just temporarily slowing down specific processor chip.

Well... this echoes ZeroVM ideas but so is PiCloud and a few others mentioned here. I think it goes without doubt that current OSes and VMs are not best suited for cloud technologies. How they can be? The were designed to completely different requirements.

I assume by VM you mean ZeroVM. Well, ZeroVM currently doesn't allow self modifying code at all. Nice try.... We haven't touched Google's provided validator in order not to break anything security-related. And if you think you have a good idea for vulnerability then you can claim some Google prizes.

If you meant more practical uses for it then unfortunately modern JIT would be difficult to support efficiently as they constantly recompile and with ZeroVM it is not only recompilation but also validation. However, JIT that recompile only once, on loading, is easy to support. In fact, next version of NaCl dumps GNU toolchain in favor for JITy LLVM, but then recompilation is happening only once.

you got (2) wrong, the reason is running code on S3 is insecure and would not be allowed and spawning VM inside S3 for some local calculation would be cumbersome at best as it is too bulky for such acrobatics.

If you use EMR or just roll your own Hadoop in EC2 then:

1. Hadoop runs on EC2 2. Data is stored on S3 3. Intermediary results stored in EC2 4. Hadoop loads the data from S3 to EC2 5. EC2<->S3 bandwidth is not that fast or efficient (S3 proxy, network contention, TCP/IP processing)

Hypothetical MapReduce/ZeroVM/Swift scenario:

1. Data is stored on S3/Swift 2. Map and Reduce functions are run inside S3/Swift secured by ZeroVM in majority of cases accessing data locally without networking/proxies getting in the way. 3. Intermediate and final results are also stored within S3/Swift. 4. Local data access is efficient, fast and predictable 5. Local networking within S3/Swift is more efficient, fast and predictable than S3<->EC2 / Swift<->Nova

Accelerated Hadoop scenario:

Exactly as in #1, just Hadoop makes "predicate pushdown optimization" into S3/Swift secured by ZeroVM.

Regarding 'due to security restrictions' I meant that cloud vendor would not let you run your own code in S3 or CloudFiles. Why? Because you could mess up other people data and storage system itself. Why not run in VM inside S3? well I guess it would be impractical due to long provisioning time of conventional VM.

What if I DoS attack some syscall? Or create zillions of files with 1 byte size driving crazy file-system or anything else.

Kernel is such vast area vulnerable for an attack that it is scary even to think about securing all of it and not leaving a single weak point. Moreover, you will screw your syscall API to the point that it will become unusable. At bare least we need standard for the syscall capping and etc... so programmer will know what to expect.

And thanks for the link, will check them and what solution they use and whether they are happy with it.

Constructive criticism is always welcome.

The answer is that multi-tenant is different from multi-user on so many aspects.

Multi-user: 1. Usually closed institution or even single group within single institution where all know each other. 2. Usually a lot of shared data and application. 3. The risk of malicious activity is low. 4. The consequences of malicious activity are not severe/fatal. 5. Auditing followed by identification and penal actions are an effective strategy to stop malicious activity.

Multi-tenant: 1. Usually open to anyone without "background checks". 2. Single instance of malicious activity is severe, repeated malicious activity is fatal. 3. Auditing, identifying the intruder and suing him in court is not an effective strategy to stop malicious activities.

There are two techniques for resources isolation:

1. Filtering all resources accesses, letting some pass and others denied.

2. Enforcing different abstraction and then unwanted resources accesses become impossible as they are not even addressable.

Filtering is by definition less secure. As filtering get more complicated there would be false negatives and false positives. Both are harmful.

Enforcing different abstraction is usually less efficient as there is a need to simulate hardware devices. However, some devices have hardware support for virtualization as with Intel CPUs and MR-IOV devices and then enforcing abstraction is free.

LXC/OpenVZ uses mostly #1

XEN/KVM uses mostly #2

ZeroVM is not a lightweight container so all the above doesn't hold. ZeroVM doesn't use any "syscall firewalling" techniques. ZeroVM efficiently emulates new hardware platform just as XEN/KVM.

Now let me address your concerns one by one:

1. (Host) Kernel exploits cannot happen as no syscalls to host OS are allowed. ZeroVM app doesn't even have such a concept as host OS.

2. DoS attacks impossible on host OS as there are no access to host OS. The interface between ZeroVM application and ZeroVM itself is specifically designed to be impossible to DoS attack. It consists now of 4 functions. Setup and exit are callable only once during lifetime of app. Message queue read/write could be repeatedly called but the it intentionally designed to be synchronous, so throttling mechanism could be transparently implemented.

3. We intentionally haven't hacked NaCl validator as it the only component in the system that guaranties security. Google established 5-6 digit monetary prizes for any exploits in Chrome/NaCl and heavily invest in security. For many customers it is enough.

4. No one asserted that ZeroVM is mature and right now it is not! So this piece of advice is correct. If you need security now for production usage - KVM/XEN is the only way to go.

5. OS security... So much work was done to secure OS from outside... not from inside. I would appreciate description how it is easy to lock-down OS process in multi-tenant sense. Also when you start "syscall firewalling" and draconian restriction it would be very hard to program such as system. Just think for a moment. You take the whole syscall list, for every syscall you decide on restrictions (I haven't found any document on web on that). Now how you work with such API? How you enforce yourself before you issue a syscall, what to do when third-party code is causing violations? Syscall API is not built for such draconian capping....

Thanks for that, I feel the same way for process isolation. In theory that should be a solution but in practice doesn't work for historic reasons.

Just one comment. ZEROVM IS NOT NACL. It uses NaCl, moreover we explicitly refrained to touch validator in order to remain under its proven security blanket (Google established hefty monetary prizes for each found exploit). However, except of validator it is heavely refactored and rewritten.

MAIN DIFFERENCE:

NaCl has "syscall firewalling" feature that is called Pepper. ZeroVM forbids all host syscalls. In fact ZeroVM is a new virtual hadware architectures (a subset of x86 and subset of ARM and new ones in future) so there is no such concept for code running inside as "host syscalls".

1. ZeroVM abstraction is C including full support for native inline assembly/intristics. Make your own conclusions about portability with regards to platform. OS portability could/will be provided and it is piece-of-cake (for geeky overoptimism of course)

2. Don't want to go into JIT / C flamewar... but let's take a sorting example. In samples directory you can find my hand-coded assembly code (thank you Takeshi Yamamura for the help) for sorting. GB is sorted within seconds. I am sure no JIT comes even close and I tried it hard. I my self SCEA and Javaist and JIT/LLVM enthusiast. If you need my help with compiling the sample let me know I'll gladly help... I know it is not ready for the prime time, very very cryptic badly commented too... which should not be for a sample. So you are warned. But it I never ever saw in my life a single threaded sorting performance that comes even close. Let me know if I mistaken.

3. Why not just use process isolation.... Well this is right way to go and zerovm would be a solution looking for a problem if and only if OS process would be secure in multi-tenant sense. However, there is no such OS. The only way is to use KVM/XEN and this is just too heavy... Another issue: why I use NaCl and not native hardware support for memory protection? The answer is two fold:

a) I was afraid of the amount of work needed to write my own complete OS from scratch (if I use existing stuff it would have most of shortcomings I am trying to solve in the first place)

b) It would be useless in its initial years if not decades as there are would be not enough applications for it to justify complete hardware dedication for zerovm. I thought about writing it as lightweight real OS running under some quickly spawned KVM processes, but still was afraid I and my team just had no spare engineering capacity for that and missing knowledge also that would be fun to acquire but just could take too long and even then it would be heavily linux-specific.

Thanks for this.

I'm indeed not a native English speaker. And I apologies for not having diagrams, in fact I was working on this part right now... And planned HN exposure little later after more diagrams are added to website, VirtualBox image ready with whole environment to save time for folks who want to experiment and a well tested initial version separated from development branch.

However it went out earlier, uncontrollably, while I'm travelling... and I will play now a catch-game...

ZeroVM uses both. Well, not now, but one contributor took the clustering thing on himself and implementing it right now with zeromq while another is developing a client library with collection of serdeses verified to compile and work on zerovm.

We decided on micro-kernel design with zerovm so all data-crunching libraries will go into untrusted code part.