← back to the notebook Research · paper iii of iii
Manuscript · July 2026 · Gunvir Singh Dhesi, University of Waterloo

Cost-Aware Language Adaptation in Large Language Models

A Framework for Low-Resource Languages

some languages pay a token tax. this is about lowering it.

Abstract

LLMs have brought big gains to speakers of high resource languages while thousands of languages stay poorly served. This synthesis frames the problem around three linked costs: tokenization inefficiency, adaptation compute and catastrophic forgetting. Instead of one recipe, it proposes a conditional framework that treats adapted tokenizers, parameter efficient fine tuning, curriculum ordered data and retention aware updates as complementary tools, and argues that typological distance, script and pretraining exposure should shape the plan alongside data and compute.

The token tax Tokens per unit of content, relative to English
same sentence, up to 10× the tokens so audit the tokenizer first

The three variable framework

DExpected transfer challenge, from typology, script and pretraining exposure.
VHow much monolingual data you actually have.
CYour compute budget, which decides LoRA, QLoRA or full continual pretraining.

Retention aware updates matter too: in one study, shielding source critical parameters cut source language degradation from 20.3% to 3.4% for 7B models.

What's inside

  1. 01The token tax. Fertility across languages and when vocabulary expansion pays off.
  2. 02Adaptation strategies. Continual pretraining, LoRA, curriculum training and synthetic data, by cost.
  3. 03Catastrophic forgetting. How much source ability gets lost, and replay and shielding as fixes.
  4. 04Open problems. Layer selection, MoE routing, a real cost benchmark and oral languages.