Augustinian BabyLM

Vision-seeded word embeddings for a BabyLM-scale masked language model

Artifacts for Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model (BabyLM Workshop 2026) · code on GitHub

A language model normally begins training with random word embeddings. Here, words that label regions in visual grounding data start from a frozen vision encoder's features over those regions — this ostension is applied once, before training on text. Everything after that is ordinary text-only masked language modelling.

Models

DeBERTa-v3-base, trained on ~9.9M words. Every vision-init model has a same-vocabulary baseline that differs only in the input-embedding initialization — same data order, same random seed, same hyperparameters.

Intermediate checkpoints are stored as branches (step0, then chck_1M through chck_100M), so training dynamics can be studied without retraining:

from transformers import AutoModelForMaskedLM
AutoModelForMaskedLM.from_pretrained(
    "augustinian-babylm/deberta-base-75k-sam", revision="chck_10M")

Datasets

• region-embeddings — per-region visual features over 563k region annotations from Flickr30k Entities, RefCOCO+, RefCOCOg and THINGS, encoded with DINOv3, iBOT and SAM. All three encoders are trained on images alone, so the features carry a purely perceptual prior.
• token-embeddings — the [V, 768] seeding tables themselves. This is what you need to reuse the intervention on a different model.
• vpswap-checkpoint-scores — per-item VP-Swap correctness for 9 models at 20 points in training, with pseudo-log-likelihoods. Enough to ask when an effect appears and which words carry it, without retraining anything.
• synthetic-grounding-images — the 3,162 SDXL-Turbo images used to extend grounding to concrete words no photograph dataset covers, with a manifest of which target words each image was written to contain.

Result summary

Vision seeding produces a consistent object-property gain (COMPS, +1.30 averaged over nine encoder-vocabulary configurations, positive in all nine) while grammar-focused benchmarks stay flat. On a corpus-tailored Visual-Property Swap probe the advantage is word-specific: it's constrained to the words that were actually seeded, and synthetically grounding previously unseeded words transfers it to exactly those words.

Function words receive strong visual seeds too, retain them to the end of training, and the training objective is shown to use them — but no benchmark registers that.

License and citation

Models and datasets are released under CC BY 4.0.

@inproceedings{bylinina2026augustinian,
  title     = {Augustinian BabyLM: What Ostensive Definition Can and Cannot
               Teach a Small Language Model},
  author    = {Bylinina, Lisa},
  booktitle = {Proceedings of the BabyLM Workshop},
  year      = {2026}
}

Note: these models are trained on bb24.train (Edman et al. 2024), not the official 2026 strict-small distribution, so leaderboard comparisons are indirect. They are research artifacts, not intended for deployment.