teendifferent

OP here. I wanted to test whether the vision encoder's pre-training strategy matters when you stitch it into an LLM. So I froze three encoders (CLIP, I-JEPA, supervised ViT), stitched each into Qwen2.5 with a small trainable projector + LoRA (~3M params), and compared.

Key findings: - CLIP dominates on average (language-aligned embeddings make the projector's job trivial). - But I-JEPA — which has never seen text during pre-training — ties CLIP on compositional reasoning (CLEVR). And scaling the LLM from 0.5B to 1.5B helped more than swapping any encoder.

Code, trained weights, and eval scripts are all open: https://github.com/REDDITARUN/CLIP-ViT-IJEPA-VLM/tree/main

Blog: https://teendifferent.substack.com/p/stitching-vision-into-l...

Curious what others think about I-JEPA-style representations for VLMs — the spatial reasoning results surprised me.

OP here. I spent the weekend red-teaming small-scale open weights models (Qwen2.5-1.5B, Qwen3-1.7B, Gemma-3-1b-it, and SmolLM2-1.7B).

I found a consistent vulnerability across all of them: Safety alignment relies almost entirely on the presence of the chat template.

When I stripped the <|im_start|> / instruction tokens and passed raw strings:

Gemma-3 refusal rates dropped from 100% → 60%.

Qwen3 refusal rates dropped from 80% → 40%.

SmolLM2 showed 0% refusal (pure obedience).

Qualitative failures were stark: models that previously refused to generate explosives tutorials or explicit fiction immediately complied when the "Assistant" persona wasn't triggered by the template.

It seems we are treating client-side string formatting as a load-bearing safety wall. Full logs, the apply_chat_template ablation code, and heatmaps are in the post.

Read the full analysis: https://teendifferent.substack.com/p/apply_chat_template-is-...

I miss the era of deep learning where we actually diagnosed failure modes instead of just scaling compute.

I spent the weekend auditing ViT vs. CNN decision boundaries using a custom perturbation pipeline. I bypassed the standard LIME segmentation (Quickshift produces too much high-variance noise) and injected a custom SLIC pipeline to force semantic superpixels.

The results show that "Clever Hans" is still very much an issue.

Spurious Correlations: The ViT predicted "Jeep" (p=0.99) purely by overfitting on the muddy terrain texture. The attention map showed it ignored the vehicle geometry entirely.

Hallucinations: EfficientNet hallucinated a "toaster" solely because it detected a white counter + flowers context.

Accuracy metrics are masking the fact that our models are just exploiting dataset biases. Full write-up on the surrogate loss implementation and visual audits here.