Birch-san

“Windows Subsystem for Linux” is probably the most confusing example of this (an environment subsystem which provides a Linux userspace to a Windows NT kernel). more intuitive would be to call it a Linux Subsystem for Windows, but presumably for branding purposes they wanted Windows in front.

I think they meant scrollback. as conventionally a backlog would evoke "work yet to be done", whereas in this context we're talking about a conversation history one can revisit.

feels like it could be nice to abide by the license terms https://bria.ai/bria-huggingface-model-license-agreement/

> 1.1 License. > BRIA grants Customer a time-limited, non-exclusive, non-sublicensable, personal and non-transferable right and license to install, deploy and use the Foundation Model for the sole purpose of evaluating and examining the Foundation Model. > The functionality of the Foundation Model is limited. Accordingly, Customer are not permitted to utilize the Foundation Model for purposes other than the testing and evaluation thereof.

> 1.2.Restrictions. Customer may not: > 1.2.2. sell, rent, lease, sublicense, distribute or lend the Foundation Model to others, in whole or in part, or host the Foundation Model for access or use by others.

> The Foundation Model made available through Hugging Face is intended for internal evaluation purposes and/or demonstration to potential customers only.

> Is it super resolution?

nope, we don't do Imagen-style super-resolution. we go direct to high resolution with a single-stage model.

FID doesn't reward high-resolution detail. the inception feature size is 299x299! so we are forced to downsample our FFHQ-1024 samples to compute FID.

it also doesn't punish poor detail either! this advantages latent diffusion, which can claim to achieve a high resolution but without actually needing to have correct textures to get good metrics.

the FFHQ-1024 examples shouldn't be blurry. you can download the originals from the project page[0] — click any image in the teaser, or download our 50k samples.

the ImageNet-256 examples also aren't typically blurry (but they are 256x256 so your viewer may be bicubic scaling them or something). the ImageNet dataset _can_ have blurry, compressed or low resolution training samples, which can afflict some classes more than others, and we learn to produce samples like the training set.

[0] https://crowsonkb.github.io/hourglass-diffusion-transformers...

cross-attention doesn't need to involve NATTEN. there's no neighbourhood involved because it's not self-attention. so you can do it the stable-diffusion way: after self-attention, run torch sdp with Q=image and K=V=text.

I tried adding "stable-diffusion-style" cross-attn to HDiT, text-conditioning on small class-conditional datasets (oxford flowers), embedding the class labels as text prompts with Phi-1.5. trained it for a few minutes, and the images were relevant to the prompts, so it seemed to be working fine.

but if instead of a text condition you have a single-token condition (class label) then yeah the adanorm would be a simpler way.

ah, originally lstein/stable-diffusion? yeah that was an important fork for us Mac users in the early days. I have to confess I've still never used a UI. :)

this year I'm hoping for efficiency and small models! even if it's proprietary. if our work can reduce some energy usage behind closed doors that'd still be a good outcome.

regarding ControlNet: we have a UNet backbone, so the idea of "make trainable copies of the encoder blocks" sounds possible. the other part, "use a zero-inited dense layer to project the peer-encoder output and add it to the frozen-decoder output" also sounds fine. not quite sure what they do with the mid-block but I doubt there'd be any problem there.

regarding IPAdapter: I'm not familiar with it, but from the code it looks like they just run cross-attention again and sum the two attention outputs. feels a bit weird to me, because the attention probabilities add up to 2 instead of 1. and they scale the bonus attention output only instead of lerping. it'd make more sense to me to formulate it as a cross-cross attention (Q against cat([key0, key1]) and cat([val0, val1])), but maybe they wanted it to begin as a no-op at the start of training or something. anyway.. yes, all of that should work fine with HDiT. the paper doesn't implement cross-attention, but it can be added in the standard way (e.g. like stable-diffusion) or as self-cross attention (e.g. DeepFloyd IF or Imagen).

I'd recommend though to make use of HDiT's mapping network. in our attention blocks, the input gets AdaNormed against the condition from the mapping network. this is currently used to convey stuff like class conditions, Karras augmentation conditions and timestep embeddings. but it supports conditioning on custom (single-token) conditions of your choosing. so you could use this to condition on an image embed (this would give you the same image-conditioning control as IPAdapter but via a simpler mechanism).

I'm one of the authors; happy to answer questions. this arch is of course nice for high-resolution synthesis, but there's some other cool stuff worth mentioning..

activations are small! so you can enjoy bigger batch sizes. this is due to the 4x patching we do on the ingress to the model, and the effectiveness of neighbourhood attention in joining patches at the seams.

the model's inductive biases are pretty different than (for example) a convolutional UNet's. the innermost levels seem to train easily, so images can have good global coherence early in training.

there's no convolutions! so you don't need to worry about artifacts stemming from convolution padding, or having canvas edge padding artifacts leak an implicit position bias.

we can finally see what high-resolution diffusion outputs look like _without_ latents! personally I think current latent VAEs don't _really_ achieve the high resolutions they claim (otherwise fine details like text would survive a VAE roundtrip faithfully); it's common to see latent diffusion outputs with smudgy skin or blurry fur. what I'd like to see in the future of latent diffusion is to listen to the Emu paper and use more channels, or a less ambitious upsample.

it's a transformer! so we can try applying to it everything we know about transformers, like sigma reparameterisation or multimodality. some tricks like masked training will require extra support in [NATTEN](https://github.com/SHI-Labs/NATTEN), but we're very happy with its featureset and performance so far.

but honestly I'm most excited about the efficiency. there's too little work on making pretraining possible at GPU-poor scale. so I was very happy to see HDiT could succeed at small-scale tasks within the resources I had at home (you can get nice oxford flowers samples at 256x256px with half an hour on a 4090). I think with models that are better fits for the problem, perhaps we can get good results with smaller models. and I'd like to see big tech go that direction too!

-Alex Birch

Most things require workarounds, some things aren't possible (or we haven't found workaround yet) and it's not as fast as CUDA. But stable-diffusion inference works, and so does textual inversion training. I was also able to run training of a T5 model with just a couple of tweaks.

I'd stick with PyTorch 1.12.1 for now. 1.13 has problems with backpropagation (I get NaN gradients now when I attempt CLIP-guided diffusion -- I think this applies to training too), and some einsum formulations are 50% slower (there is a patch to fix this; I expect it'll be merged soon), making big self-attention matmuls slow and consequently making stable-diffusion inference ~6% slower.

It's probably the same thing prestodb encountered: https://github.com/prestodb/presto/issues/8993

For performance reasons, glibc may not return freed memory to the OS. You can increase the incentive for it to do so, by reducing MALLOC_ARENA_MAX to 2. https://github.com/prestodb/presto/issues/8993

stable-diffusion supports k-diffusion just fine on M1. You just have to detach a tensor in to_d() to stop the values exploding to infinity. https://twitter.com/Birchlabs/status/1563622002581184517?s=2...

9.3 secs if you sample for 7 steps. 11 secs if you sample for 8 steps. Heun sampler, M1 Max. https://twitter.com/Birchlabs/status/1564792349221330944?s=2...

Neither of these should take minutes. Try Heun sampler, 8 steps, Karras noise schedule. Should be possible to get good images in 11 secs (or 10 secs if you go down to 7 steps). measurements admittedly from M1 Max. https://twitter.com/Birchlabs/status/1565029734865584143?s=2...

About 10 secs per image on M1 Max with the right noise schedule and sampler. https://twitter.com/Birchlabs/status/1565029734865584143?s=2...

You might be eligible to use Google's TPU Research Cloud for free, provided you publicize your results? https://sites.research.google/trc/about/

Otherwise, perhaps you could ask LAION on #compute-allocation?

Even without mods, Minecraft supports programming mechanisms (albeit not robots). redstone blocks can be used like logic gates, to construct traps, play music, open doors or build factories. People have implemented ALUs in redstone.

yes, that demo has very impressive shaders. it models the physics in a more simple way though (i.e. as waves), rather than as thousands of particles, so it can't generalize to more complex scenarios.

network inspector says 2.1MB. but that's dominated by a 1.3MB image.

the main assets of the library are:

- Box2D.simd.js (422kB)

- Box2D.simd.wasm (266 kB)

a minimal demo that uses the library can be created in just a few kB:

https://github.com/Birch-san/box2d-wasm/tree/master/demo/mod...

it's all modelled using physical laws; see slides 16 onwards of the "Inside Liquidfun" presentation:

http://docs.google.com/presentation/d/1fEAb4-lSyqxlVGNPog3G1...

okay, have another look now[0]. I've updated it to use a consistent number of particle iterations regardless of how much time is being simulated.

in fact, you've caught that it was using as many as 16 iterations (my intention was 3), so this change should make it substantially more performant and less stiff.

moreover, I found that some objects were being allocated by my perf-measurement code. ironically making it slower. fixed now.

if it needs to simulate large stretches of time: I now split that into multiple 1/60th-second world.Step() calls, instead of attempting to scale up the time and number of iterations.

as a result of these changes: Wave Machine now looks a bit more like the original liquidfun.js demo. Gravity looks a bit more stable now (doesn't bounce when the scheduler throttles us).

I've also added a safeguard to help underpowered devices. if we determine that we're not simulating fast enough to finish in a timely manner: we settle for simulating just 1/60th of a second (i.e. even if 1/20th of a second elapsed since we were last scheduled). this admittedly results in slow-mo, but is preferable to exacerbating the perf problem (and has a chance of getting us back-on-track).

[0] https://birchlabs.co.uk/liquidfun-wasm/

yes, I think so! some examples here[0][1].

[0] https://www.youtube.com/watch?v=2Bw5f4vYL98

[1] https://www.youtube.com/watch?v=atcKO15YVD8

yeah, I've played around with a few approaches for running the timestep and for some reason I don't feel like I get the same results as liquidfun.js.

their loop [0] is pretty simple; it's scheduled by `requestAnimationFrame`, advances time by 1/60th of a second, and runs their default of 3 particle iterations. it completes the physics simulation within 3.9–5.5ms, which is easily in time for the 16ms deadline. the rendering is WebGL, which I assume fits easily into that 16ms budget too.

my loop [1] is more complicated; I don't hardcode the timestep to 1/60 seconds, because requestAnimationFrame may be scheduled less frequently than that. so instead I advance time by the time elasped since I was last scheduled. hm, I think there's a mistake there — `lastMs = nowMs` is probably on the wrong side of the physics calculation.

there's an additional technique I use: I put a `Math.min()` over the simulation interval, so that I don't attempt to simulate more than 20ms (this can happen if you get scheduled infrequently due to hot CPU or backgrounding the app) — simulating too much time will make us fail our frame deadline anyway.

furthermore, if we are calculating more than 1/60th of a second, I employ more particle iterations (i.e. 3 particle iterations for every 1/60th of a second that passes). this gave me good results, but turns out it is based on incorrect assumptions (iterations are unrelated to timestep)[3]. moreover, I may be making mistakes in my decision of whether to round this fraction up/down.

if too few particle iterations for a timestep: the particles will bounce. if too many: the particles will look too incompressible[4]. I think that's the "solid-like" structure you're describing.

the main reason I complicated this is because the last one I did[5] made me feel motion-sick. I think if "every 1/60th, or 1/30th, or 1/20th of a second: you simulate a 1/60th of a second of time": the result (if you're not scheduled consistently) is that the world speed keeps changing. I think liquidfun.js's approach should be vulnerable to this, but for some reason it looks fine to me. maybe they get scheduled more consistently than me (even though by my measurements, my physics runs slightly faster, so should be able to achieve similar results).

I think I need to remind myself of what happens if I program the timestep in the simple way that liquidfun.js did. will try that out at some point.

[0] https://github.com/google/liquidfun/blob/master/liquidfun/Bo...

[1] https://github.com/Birch-san/liquidfun-play-2/blob/master/sr...

[2] https://github.com/Birch-san/liquidfun-play-2/blob/master/sr...

[3] http://google.github.io/liquidfun/Programmers-Guide/html/md_...

[4] http://google.github.io/liquidfun/Programmers-Guide/html/md_...

[5] https://birchlabs.co.uk/box2d-wasm-liquidfun/

wow! thanks for your contributions. :) would you mind giving some background on how LiquidFun came into existence? is it a 20% time project, or a weekend thing made by Googlers? was there ever a business case for it?

yes. the fastest "regular JS" version achievable would be via asm.js, and WebAssembly can go faster[0].

you can compare the performance of this demo's Wave Machine mode against the one used on the liquidfun.js frontpage[1]: in my crude measurement just now, a `Step()` took liquidfun.js 3.9–5.52ms, whereas it took 3.6–3.7ms in liquidfun-wasm. take with a pinch of salt since I just sampled a few random frames of each experiment at random CPU temperatures.

[0] https://hacks.mozilla.org/2017/03/why-webassembly-is-faster-...

[1] http://google.github.io/liquidfun/

Author here. This demo showcases liquidfun-wasm[0], my effort to revive liquidfun[1] (a fork which adds fluid simulation and soft-body physics to Box2D[2]).

to make liquidfun-wasm, I repurposed my existing box2d-wasm[3] and pointed it at a different release of Box2D — a commit obtained by rebasing liquidfun over 7 years of upstream Box2D changes[4]. the end result is that liquidfun is now distributed in WebAssembly and with TypeScript typings for the first time. The TypeScript typings are generated from WebIDL bindings via my webidl-to-ts[5] compiler.

this demo in particular aims to bring to the Web the shaders from the liquidfun EyeCandy demo[6], and show how fast JS can run if you avoid incurring the garbage collector (the main loop tries not to allocate objects). the demo repurposes gravity and drag calculations that I'd used previously in my Lunar Survey experiment[7] (a Mario Galaxy homage).

[0] https://github.com/Birch-san/box2d-wasm/releases/tag/liquidf...

[1] http://google.github.io/liquidfun/

[2] https://github.com/erincatto/box2d

[3] https://github.com/Birch-san/box2d-wasm

[4] https://github.com/Birch-san/box2d-wasm/releases/tag/v4.0.0-...

[5] https://github.com/Birch-san/box2d-wasm/tree/master/webidl-t...

[6] https://github.com/google/liquidfun/blob/master/liquidfun/Bo...

[7] https://birchlabs.co.uk/lunar-survey/

yes, I compiled with -msimd128 to enable LLVM's auto-vectorization. I distribute both SIMD and non-SIMD, and the entrypoint picks whichever distribution your browser supports. for box2d-wasm, SIMD acceleration resulted in a 0.6–0.9% performance boost [0] when simulating a pyramid of boxes.

liquidfun-wasm is a fork with additional algorithms for performantly simulating particles. I have not yet built a benchmark to measure the particle code, but do intend to. I am optimistic that liquidfun's particle code could auto-vectorize better than the general Box2D code.

the Google engineers considered how to take advantage of SIMD, to the extent that they even ship a NEON SIMD algorithm[1]. I don't believe my compiler config will use that NEON algorithm (and will instead fallback to the general algorithm [2]). that's probably not a missed opportunity; many NEON features are not supported[3]. but since the engineers were thinking about SIMD, hopefully the non-NEON algorithm will try to make good use of the CPU and memory layout too, and auto-vectorize well.

[0] https://github.com/Birch-san/box2d.ts/pull/1

[1] https://github.com/google/liquidfun/blob/master/liquidfun/Bo...

[2] https://github.com/google/liquidfun/blob/master/liquidfun/Bo...

[3] https://emscripten.org/docs/porting/simd.html#compiling-simd...