News for hackers. Stuff that matters.
Re:DreamWorks Animation to release MoonRay as open source by miohtama on Friday August 05, 2022 @11:59AM (#32357992)
Would be interesting to know what business considerations lead to the decision to open source MoonRay.
Re:DreamWorks Animation to release MoonRay as open source by thebeardisred on Friday August 05, 2022 @12:40PM (#32358545)
DreamWorks Animation (aka DWA) is a huge advocate of open source.

Over the years they've bankrolled a lot of development around Linux on the desktop and have one of the most astonishing NFS infrastructures I've ever directly layed hands on.

They've talked about their love of open source over the years, especially at Red Hat summit.

(I spent quite a bit of time working with them and their former CTO is currently a teammate of mine).

Re:DreamWorks Animation to release MoonRay as open source by albatross13 on Friday August 05, 2022 @01:12PM (#32358975)
Can you expound a bit on the NFS infrastructure? I'm curious.

If you can't, legally, nbd.

Re:DreamWorks Animation to release MoonRay as open source by newman314 on Friday August 05, 2022 @01:47PM (#32359438)
I was curious, did a quick search and found this OLD deck. Would be interesting to see how it has evolved since.

https://www.usenix.org/legacy/event/lisa10/tech/slides/kamat...

Re:DreamWorks Animation to release MoonRay as open source by thebeardisred on Friday August 05, 2022 @10:33PM (#32364279)
Fortunately, Mike (Cutler) & Sean (Kamath) documented many things in the USENIX presentation cited by newman314 that I'm still under NDA about. I worked with both of them and they're extremely knowledgeable folks.

Now, switching gears to talk about that presentation. ;)

I'd call out a few things.

1) Note on slide 17 the call out to _multiple sites_. Yup. You got it. There is a single view of the file store. At a practical level to the end user the sync time between sites is trivial to non-existent.

2) automount, automount, automount (Slide 25). The automount configuration is stored in LDAP providing a convenient way to correlate and sync data which is already integrated with the name service switch/NSS (man 5 nss).

3) Caching (Slides 22-24). This is where a lot of the magic happens. It's this cache tier which _presents_ as if it's a single file store.

One conceit I will make (which is glossed over a bit) is that the use of DNS and suffix searching is critical in a situation like this. It allows each site to use configuration management to push out relevant config files (e.g. /etc/resolv.conf) which can then present a search order like "site1.us.example.com us.example.com example.com". Thus, when you do a query for the unqualified name "ldap" (or even better "IN SRV _ldap._tcp") NSS will hand that off to the resolver which will dutifully check for the following (in order) until it gets a positive match or fails: ldap.site1.us.example.com, ldap.us.example.com ldap.example.com. This allows for a DNS based fall back as well as the added benefit of always hitting the local system (instead of a remote site).

Also, make sure you check out the stats on Slide 28.... 4 PB... in 2008... before they had to render a movie for each eye... (3D animation!)

Re:DreamWorks Animation to release MoonRay as open source by munificent on Friday August 05, 2022 @12:54PM (#32358736)
There is a lot of staffing churn in the film world and many films are built by farming out work to dozens of separate VFX studios.

I speculate that having this open source increases its popularity, which makes more likely that any given potential hire or VFX company will have experience with it, which makes it easier for DreamWorks to ramp up new hires or contract out to third party companies.

When I was at EA, we switched from a proprietary UI tool (which was quite well suited for consoles) to Flash (which wasn't) entirely because it eased staffing problems even though the technology itself was a worse fit. The best tool is often the one that people you can hire already know.

Re:DreamWorks Animation to release MoonRay as open source by egypturnash on Friday August 05, 2022 @02:04PM (#32359668)
god, Flash wasn't suited for anything and yet somehow it got used for everything until Apple managed to kill it by keeping it the hell off the iPhone. It was kind of okay for so many things and actually great at none of them.

It might even still be in use at Cartoon Network, it's been a while since I last heard from my former co-worker in the dialup-delivered Flash cartoon mines who ended up as the animation director for Foster's and Titans but he sure was still using an old version of Flash/Animate long after the browser plugin was retired.

There's probably a lot of lessons in Flash's status as a tool that did a ton of different things half-assedly.

Re:DreamWorks Animation to release MoonRay as open source by munificent on Friday August 05, 2022 @03:08PM (#32360546)
It was like the JavaScript of UX and 2D games for a while. It was trash as a technology, but it was a technology that millions of UI designers and indie game developers cut their teeth on and already knew deeply, so it stuck around much longer than it should have. Like grown-ups trying to ride tricycles to work.

I remember someone at EA talked to Adobe about getting access to the Flash (.fla) file format so that our UI tools could read it directly to export into the in game format instead of reading the already compiled and compressed .swf format. And they were like, "Uh... you really don't want to do that. It's like a memory dump of the entire application state. It's a giant mess. We're really sorry."

By all accounts, it seemed like the Flash codebase was a dumpster fire. But so many amazing little games and UIs were built using it.

Re:DreamWorks Animation to release MoonRay as open source by Jasper_ on Saturday August 06, 2022 @01:04PM (#32369746)
The .fla format was a OLE compound file. It wasn't that bad -- I reverse engineered how it worked in the Flash MX 2004 days, and it was mostly editor data structures formatted like how you'd expect.
Re:DreamWorks Animation to release MoonRay as open source by egypturnash on Friday August 05, 2022 @05:58PM (#32362445)
hahahha, "It's a memory dump of the entire app state" explains SO MUCH.
Re:DreamWorks Animation to release MoonRay as open source by game-of-throws on Friday August 05, 2022 @06:27PM (#32362705)
To be fair that's how a lot of software worked in that era. MS Office worked that way too (doc files, not docx files).
Re:DreamWorks Animation to release MoonRay as open source by erichocean on Friday August 05, 2022 @12:21PM (#32358302)
Even small companies can create competitive high-end renderers today so it's not much of a differentiator. Filmmaking is about human talent mostly, not technology. (Same thing happened in live-action filmmaking, anyone can afford the equipment today.)

But there's a huge advantage to being the first high-end renderer to go open source, so lots of small studios will adopt it if it's as solid as it looks.

It'll become available in Blender.

Maintenance will then get extended to the community, including (most likely) support for alternate shading schemes (MaterialX/Lama has decent momentum right now, converging with Sony's OpenShadingLanguage).

Apache 2.0 is an ideal license for this kind of project.

Re:DreamWorks Animation to release MoonRay as open source by berkut on Friday August 05, 2022 @12:07PM (#32358075)
Yeah. They've open sourced OpenVDB and other smaller things before, and have contributed things to Embree and OpenSubDiv, but those were libraries/storage formats, not entire production-capable renderers.

At the same time however, does their renderer give them that many advantages? As someone who works on a (sort of) competing proprietary renderer, it's a lot of work and effort to do it and support it, and maybe they want to build a community around that from smaller studios and compete with Renderman a bit for mindshare?

Re:DreamWorks Animation to release MoonRay as open source by dimatura on Friday August 05, 2022 @01:23PM (#32359129)
OpenVDB is a really cool tool and a nice C++ codebase. During my PhD I adapted it to manage volumetric data extracted from LiDAR measurements and found it to be much more efficient than more popular alternatives in the robotics world, such as Octomap (which is also a great tool in its own right!).
Re:DreamWorks Animation to release MoonRay as open source by wlesieutre on Friday August 05, 2022 @12:40PM (#32358542)
If they can get students or freelance animators to learn it, that's a potential hiring pool with a lot less onboarding
Re:DreamWorks Animation to release MoonRay as open source by desindol on Friday August 05, 2022 @12:39PM (#32358540)
Students get used to your pipeline before you hire them. The technology is nothing groundbreaking but the best that is available for free. Most students will take that opportunity.
Re:DreamWorks Animation to release MoonRay as open source by bombcar on Friday August 05, 2022 @12:43PM (#32358584)
This is huge and people underestimate how valuable it can be for a company. DreamWorks Animation's product is not MoonRay, it is movies and they're commodizing their complement.

If every animation studio started using MoonRay because of this, they'd have won.

Re:DreamWorks Animation to release MoonRay as open source by pwdisswordfish9 on Friday August 05, 2022 @01:04PM (#32358868)
Good point in your second sentence, but hard see where you're coming from in the last one.

Surely the win for DreamWorks is the competitive advantage of having aspiring creators be familiar with their pipeline. If all animation studios use MoonRay, then that advantage gets erased, and the ultimate impact is null. (The only way this isn't true is if you subscribe to the belief that more MoonRay use → more MoonRay experts → more DreamWorks movies → more profit. That's probably not sound. This probably won't impact how many movies they put out, just how good they look (after benefiting from open source work) and how easy it is to hire for the movies that they put out.)

Re:DreamWorks Animation to release MoonRay as open source by XorNot on Friday August 05, 2022 @01:40PM (#32359337)
The estimate would be that their tooling investment is no longer a major competitive advantage, compared to other studios. But, if other studios or new studios switch to their tooling or start using it, then functionally they're subsidizing skilling up animators who they can later hire as contractors or full-time onto their projects.

They're in the movie business after all - speed of production saves a lot of money.

Re:DreamWorks Animation to release MoonRay as open source by kanzure on Friday August 05, 2022 @01:22PM (#32359126)
> If all animation studios use MoonRay, then that advantage gets erased, and the ultimate impact is null.

Other studios could also contribute improvements, which the original studio could benefit from. That's one of my motivations for open-source: I'm selfish and I enjoy the benefits from the efforts of others making my software better, should they find it useful.

Re:DreamWorks Animation to release MoonRay as open source by bombcar on Friday August 05, 2022 @03:16PM (#32360664)
And it would mean that they're all on a level playing field - everyone would be using MoonRay so there wouldn't be an obvious technological advantage.

(Note that this could mean that if you thought your software was putting you at an advantage, you might not want to open-source it, but you could also believe that even if it is better, your competitors won't / can't switch to it).

Re:DreamWorks Animation to release MoonRay as open source by com2kid on Friday August 05, 2022 @03:57PM (#32361207)
> And it would mean that they're all on a level playing field - everyone would be using MoonRay so there wouldn't be an obvious technological advantage.

But if the improvements to MoonRay drive down production costs for movies, the entire industry wins.

Re:DreamWorks Animation to release MoonRay as open source by berkut on Friday August 05, 2022 @01:11PM (#32358965)
This won't be their entire pipeline though. People will have to use the open source version in vanilla DCCs without all of Dreamworks' integrations, so will only be exposed to a bit of it in terms of configuring the renderer's options.
Re:DreamWorks Animation to release MoonRay as open source by ziddoap on Friday August 05, 2022 @02:22PM (#32359888)
Which is still more familiarity than before.
Re:DreamWorks Animation to release MoonRay as open source by Someone on Friday August 05, 2022 @12:58PM (#32358791)
Can students realistically run this kind of software on the hardware they have? I would guess DreamWorks only runs it on fairly expensive networks, and would not have prioritized keeping it usable on single-machine setups.
Re:DreamWorks Animation to release MoonRay as open source by mkaic on Friday August 05, 2022 @01:55PM (#32359547)
I disagree -- a large part of the lookdev process at animation studios involves single artists working on single workstations with single GPUs while they work on their part of the pipeline. I'd wager that Dreamworks almost certainly does big distributed renders for the final frames of the movie, but a large number of the in-between steps and still-frame test renders probably happen in individual employee's machines -- not to mention that having an engine that can preview the render in near-realtime is a staple of modern 3D pipelines now that we have such beefy graphics cards.
Re:DreamWorks Animation to release MoonRay as open source by ArtWomb on Friday August 05, 2022 @12:25PM (#32358354)
If we're speculating, Pixar is also announcing it's Renderman 25 XPU cloud native render tech. If gpu cluster & virtualization continues to explode and transform the effects industry. I imagine all the major studios would seek ancillary revenue, farming out their cloud investments during idle hours, to third parties in gov research and academics to use for large scale data viz ;)
Re:DreamWorks Animation to release MoonRay as open source by samstave on Friday August 05, 2022 @02:32PM (#32360029)
An interesting tidbit of history:

When I was on the design team for the Lucas Presidio Campus, we were collapsing all the various divisions to the new campus, aside from Skywalker...

I was the LV/Datacenter/network designer for the project.

One of the early design requirements was to 1) have fiber to the desktop and 2) have every machine on an interconnect matrix which brought the edge desktop connections as close to the core as possible such that every workstation would become a part of the ender cluster when idle/at night/on-demand etc...

This didnt happen but it was a goal back in ~2004

--

I recall them telling us when Jobs came to visit and they had the "Death Star" <-- render cluster they were working on... Lucas apparently saw no future and sold it to Jobs for ~$10mm?? and thats where Pixar got it start.

That was an amazing fun project to work on.

---

Oh - and during the design sessions, Cisco borked the design review for their (joke of a response) to the RFP, the then CIO for LucasArts in this design meeting with ILM, Lucas, Cisco others -- he says to the entire team in dead seriousness

"When can I have power-over-fiber to the desktop" with a look like "Ha! GotchA!

And everyone just blinked..

Re:DreamWorks Animation to release MoonRay as open source by erichocean on Friday August 05, 2022 @12:32PM (#32358447)
> during idle hours

Studios don't really have idle hours on their render farms.

Re:DreamWorks Animation to release MoonRay as open source by samstave on Friday August 05, 2022 @02:34PM (#32360053)
This was 2004. The concepts were still being figured out WRT to making a huge campus for one of the most creative companies...

EDIT: I thought you were replying to my comment about the Lucas Presidio Campus

Re:DreamWorks Animation to release MoonRay as open source by a_e_k on Friday August 05, 2022 @05:39PM (#32362280)
My experience from when I was in the industry was the opposite. Studios would generally maintain a solid baseline renderfarm and then send overflow to the major cloud providers.

They also weren't really set up for the kind of strict isolation and security that offering resources to external parties would require.

Re:DreamWorks Animation to release MoonRay as open source by KittyCatCuddler on Friday August 05, 2022 @12:07PM (#32358087)
It kinda seems like they might be in cahoots with Microsoft to compete with AWS Nimble.

https://aws.amazon.com/nimble-studio/

Re:DreamWorks Animation to release MoonRay as open source by bogwog on Friday August 05, 2022 @02:19PM (#32359858)
This is released under Apache 2.0 though, so Amazon could just do what they always do and repackage/sell it under a different name as an AWS product.
Re:DreamWorks Animation to release MoonRay as open source by berkut on Friday August 05, 2022 @12:13PM (#32358176)
Interesting, seeing as at least on Trolls they were using AWS for some of their compute for rendering. Some Dreamworks employees on the tech side left for AWS as well.
Re:DreamWorks Animation to release MoonRay as open source by mchusma on Friday August 05, 2022 @12:09PM (#32358122)
This is not really like Nimble, it's more like Picard renderman
Re:DreamWorks Animation to release MoonRay as open source by khazhoux on Friday August 05, 2022 @12:14PM (#32358197)
Just FYI, Nimble was former Dreamworks animators and engineers.
Re:DreamWorks Animation to release MoonRay as open source by colechristensen on Friday August 05, 2022 @12:26PM (#32358362)
>We expect to see the code base grow stronger with community involvement as DreamWorks continues to demonstrate our commitment to open source

Translation: we think making this open source will bring in significant outside contribution so we can get a better renderer with less investment ourselves while not having to pay external providers

Open source is a good strategy for the runners up trying to compete, you get the good will from giving something away in exchange for other people improving your product for free.

Re:DreamWorks Animation to release MoonRay as open source by ziddoap on Friday August 05, 2022 @12:43PM (#32358587)
Why are you framing this in such a negative light?

They open something up, let people use it, and hope that some people might contribute back. Cool, sounds great to me. They aren't forcing me to contribute back to the project. Seems pretty win-win.

Re:DreamWorks Animation to release MoonRay as open source by colechristensen on Saturday August 06, 2022 @03:14PM (#32370756)
It’s clearly business, not self sacrifice. Your interpretation of that as negativity is a reflection of your own judgement not mine.
Re:DreamWorks Animation to release MoonRay as open source by tpmx on Friday August 05, 2022 @01:21PM (#32359111)
That's not how it works. What they're after is mindshare, establishing their codebase as a standard etc.

You're not going to get a free work force just by open sourcing your very context-specific code. Most companies/people learned this ~22 years ago.

Re:DreamWorks Animation to release MoonRay as open source by erichocean on Friday August 05, 2022 @12:31PM (#32358426)
> runners up trying to compete

MoonRay likely has the best volume rendering in the world, certainly that's open source.

Only Weta's internal renderer is in the same league.

Re:DreamWorks Animation to release MoonRay as open source by berkut on Friday August 05, 2022 @12:45PM (#32358622)
Not necessarily: it would depend on what Dreamworks' requirements have been up to now as to the current state: obviously, they use volumetrics, but almost all their stuff is CG and not necessarily photo real, being more artistic-driven.

That fact might have allowed them to take short-cuts.

How do you know what Manuka's capable of? :)

Re:DreamWorks Animation to release MoonRay as open source by mkaic on Friday August 05, 2022 @01:58PM (#32359585)
I'd guess that the volume rendering in MoonRay is pretty top notch considering Dreamworks was also responsible for creating OpenVDB. Furthermore, while their characters and environments are often stylized, watch a few minutes of How To Train Your Dragon 3 and you'll see their volumetric rendering is stunningly believable.