Cost-Aware Language Adaptation in Large Language Models

Abstract

Large language models (LLMs) have created significant capability gains for speakers of high-resource languages, while thousands of the world's languages remain poorly served by these systems. This targeted literature synthesis examines how to extend linguistic coverage under practical constraints. It organizes the problem around three linked concerns: tokenization inefficiency (the token tax), adaptation compute, and catastrophic forgetting.

Rather than presenting a universal recipe, it develops a conditional framework for choosing and evaluating adaptation strategies. The synthesis highlights language-adapted tokenizers, parameter-efficient fine-tuning, curriculum-ordered data, and retention-aware updates as complementary tools. It also argues that typological distance, script, and pretraining exposure should inform planning alongside data volume and compute budget.

Keywords: multilingual NLP, low-resource language adaptation, parameter-efficient fine-tuning, tokenizer fertility, continual pretraining, catastrophic forgetting, cross-lingual transfer

Introduction

Of the approximately 7,159 living languages in the world, modern NLP research systematically focuses on a small subset (van Esch et al., 2022). The resulting disparity is not merely academic. Speakers of underserved languages are denied access to educational tools, healthcare information, legal assistance, and economic opportunity that AI systems now routinely provide to English speakers.

Low-resource adaptation should not be treated as a single fine-tuning decision. It begins with a fertility audit, then selects an adaptation path based on available data, compute, and the expected transfer challenge. The purpose is not to estimate a universal cost for every language, but to make the main trade-offs explicit before expensive experiments begin.

Prior surveys of multilingual LLMs have catalogued methods and benchmarks but have not systematically integrated cost into their analysis (Liu et al., 2024). This paper develops a cost-aware framework that maps adaptation strategies to their likely resource requirements across linguistic contexts. Its contributions are: a synthesis of the token-tax literature that connects tokenizer design to deployment cost; a comparison of adaptation strategies across data and budget regimes; a review of approaches that mitigate catastrophic forgetting; and a decision framework using linguistic distance, data volume, and compute budget as practitioner inputs. This is a targeted narrative synthesis, not a systematic review or meta-analysis. The recommendations should therefore be treated as planning hypotheses that require target-language validation.

Background and Problem Formulation

The Multilingual Coverage Gap

The NLP community has long acknowledged that linguistic diversity in research lags behind the true distribution of human language (Bender, 2019; Joshi et al., 2020). Joshi et al. (2020) proposed a six-tier taxonomy of language resourcefulness ranging from languages with billions of tokens and mature NLP ecosystems (Tier 5: English, Chinese) down to entirely undocumented languages (Tier 0). The vast majority of the world's languages fall into Tiers 0 through 2. Research at ACL 2021 found that 70 percent of papers evaluated only in English, a proportion that has not changed substantially despite growing awareness (LoResLM Workshop, 2025).

Multilingual models trained on more balanced data, such as BLOOM (BigScience Workshop, 2023) and EuroLLM (Martins et al., 2024), demonstrate that more equitable pretraining distributions yield more consistent performance across languages. However, from-scratch multilingual pretraining at competitive scale requires massive compute budgets unavailable to most researchers. The practical question is therefore not how to pretrain better multilingual models from scratch, but how to adapt existing high-quality English-centric models to additional languages at the lowest possible cost.

Framing the Cost Problem

We treat total adaptation burden as three interlocking components. The tokenization overhead factor (T) describes how many more tokens a model produces per comparable unit of target-language content than it does for English. Adaptation compute (C) captures the accelerator time and engineering effort required to reach a stated target-language threshold. The forgetting penalty (F) records degradation on source-language or general capabilities after adaptation. These are operational planning variables rather than a single universal cost equation.

These components interact in non-obvious ways. Vocabulary expansion can reduce T but introduces a one-time adaptation overhead. Continual pretraining can improve language representations but usually raises C and may amplify F. PEFT methods reduce C and can limit unwanted updates, but they do not remove T when the tokenizer is poorly matched to the target language. An integrated strategy must weigh all three axes jointly.

The Token Tax: Tokenizer Fertility and Its Costs

Fertility Values Across Languages

The reported fertility ranges below are illustrative rather than a common benchmark: fertility depends on the tokenizer, the corpus, and the unit of analysis. Their value here is to show why a preliminary measurement can change an adaptation plan.

Petrov et al. (2024) document substantial disparities in tokenization efficiency across languages. Lundin et al. (2025) further show that high fertility is associated with both higher compute requirements and lower accuracy in a multilingual evaluation setting. At the extreme, poorly matched tokenizers can approach byte-level behavior for particular scripts and morphological systems.

The economic consequences compound at scale. Extra tokens increase inference charges and latency directly. At long context lengths, the sequence-length cost of attention can magnify the burden further; Lundin et al. (2025) report that a doubling in tokens can translate into roughly four times the training cost and time in their analysis. The exact multiplier remains model- and workload-dependent, so deployment estimates should be measured with representative prompts rather than inferred from a single language average.

Vocabulary Expansion as a Mitigation Strategy

The EEVE method (Efficient and Effective Vocabulary Expansion) demonstrated that vocabulary expansion with parameter freezing and subword-based embedding initialization can improve non-English proficiency using substantially fewer tokens than earlier approaches (Kim et al., 2024). Csaki et al. (2024) extended this direction to Thai, reporting a reduction in fertility from 4.8 to 1.1 tokens per word.

Rana et al. (2026) report that the MUTANT-Indic tokenizer improves average fertility by 39.5 percent over Llama 4 and inference throughput by 44 percent while maintaining comparable English and Indic benchmark performance. These results make tokenizer evaluation a sensible early diagnostic, particularly for non-Latin-script targets. Whether vocabulary expansion pays off, however, depends on the target workload, expected volume, and the quality of the adaptation data.

Adaptation Strategies: A Cost-Performance Analysis

Once tokenizer efficiency is addressed, the next question is how to teach a model a new language. The qualitative positioning below is a planning aid rather than a benchmark: the ordering can change with model family, data quality, and the evaluation target.

Full Continual Pretraining

CPT can improve a model's handling of low-level phonological, morphological, and syntactic patterns in a target language without requiring labeled examples. Language-specific CPT has been explored for Japanese (Fujii et al., 2024), German and Arabic (Dobler & de Melo, 2024), and Korean (Kim et al., 2024). Its central limitation is the compute required to update a full model. Even modest runs can require substantial accelerator time, and the price depends on hardware, sequence length, model architecture, and provider. Extended pretraining can also make subsequent instruction tuning more difficult (Springer et al., 2025).

Springer et al. (2025) also show that additional pretraining need not improve downstream utility: overtraining can make later instruction tuning harder. This makes it important to define a target metric, a token budget, and a source-language retention check before scaling a CPT run.

Parameter-Efficient Fine-Tuning (PEFT) and LoRA

PEFT methods hold most base parameters fixed and train a small set of added parameters, reducing memory pressure and making systematic experimentation more feasible (Hu et al., 2021). This can be valuable for multilingual adaptation, but performance and retention must still be evaluated on both target-language and source-language tasks.

Middle-layer representation alignment offers a useful rationale for testing layer-targeted adapters. Liu and Niehues (2025) find that middle layers show particularly strong potential for cross-lingual alignment and that an alignment objective can improve transfer across several tasks. This evidence supports comparing targeted adapters with all-layer baselines, rather than assuming that one placement is optimal across architectures.

Curriculum-Based Training

CSCL (Yoo et al., 2025) orders training data from token-level code switching to sentence-level code switching and finally to monolingual corpora. In experiments with Qwen 2 and additional models, the authors report gains in transfer to Korean and other languages over monolingual continual pretraining. Because it changes the training schedule rather than model architecture, it is a practical candidate for a controlled ablation.

Synthetic Data and Cross-Lingual Distillation

When native-language data is extremely scarce, machine-translated synthetic data provides a practical alternative. Wang et al. (2025) show that documents machine-translated from a high-quality English corpus can make a useful contribution to multilingual pretraining. Self-distillation from resource-rich languages, where a high-resource model generates training signals for target-language adaptation without human annotation, offers another low-cost path (Zhang et al., 2024).

Catastrophic Forgetting in Language Adaptation

The Scope of the Problem

Yamaguchi et al. (2025) report that full fine-tuning on target-language data degraded source-language performance by 20.3 percent for 7B models and 22.3 percent for 13B models in their setting. The mechanism is parameter overlap: updates that improve one language can overwrite representations that also support source-language capabilities. The comparison below reports the full-fine-tuning and Source-Shielded Updates (SSU) results from that study; it does not estimate the effect of every mitigation method.

Mitigation Techniques

Several techniques have been developed to address this. Experience replay mixes a sample of source-language data into target-language batches, anchoring the model to its original distribution. SSU takes a proactive approach: before target-language training begins, it uses a small set of source-language data to score parameter importance, then applies column-wise freezing to protect the most critical parameters (Yamaguchi et al., 2025). In that study, SSU reduced source-task degradation to 3.4 percent for 7B models and 2.8 percent for 13B models while retaining competitive target-language performance.

Recent work on linguistic neuron overlap suggests that cross-lingual transfer can depend on shared activation patterns across languages (Xu et al., 2025). That finding motivates targeted analyses of adapter placement and source-language retention, but it does not yet prescribe a single layer-selection rule across architectures.

Instruction-tuned models present a specific complication: naively adapting them to new languages causes loss of multi-task instruction-following ability, not just language performance (InstructAlign, 2023). The InstructAlign framework addresses this using alignment-based cross-lingual instruction tuning with experience replay, enabling effective new-language learning from limited parallel data while preventing forgetting of the instruction-following capability.

A Cost-Aware Decision Framework

Typological Distance as a Cost Predictor

Typological similarity is a useful, but task-dependent, proxy for cross-lingual transfer. Eronen et al. (2026) show that linguistic similarity can predict transfer performance across several tasks and languages, and can help with source-language selection. In practice, typology should be considered with script, tokenizer behavior, pretraining exposure, and the task itself.

Formal language family is not the whole story. Pretraining exposure and the distribution of text available to a model can also influence transfer. For that reason, typological distance should be treated as a planning variable rather than a deterministic estimate of adaptation cost.

The Three-Variable Framework

We propose a decision framework with three practitioner-controlled inputs: (D) expected transfer challenge, informed by typology, script, and preliminary tokenizer measurements; (V) available monolingual data volume; and (C) compute budget. The categories below are deliberately coarse so that they can guide an initial experimental plan rather than replace target-specific measurement.

Two cross-cutting practices apply across the framework. First, measure tokenizer fertility before committing to an adaptation strategy, then model the payback of vocabulary expansion using the intended workload. Second, evaluate source-language retention during training; experience replay and SSU are candidate mechanisms when retention matters, not universal defaults.

Open Problems and Future Directions

Despite the progress synthesized above, several important problems remain unsolved. First, the optimal layer selection strategy for multilingual PEFT has not been rigorously characterized across all model architectures. The middle-layer finding is robust for LLaMA 3 and Mistral (Liu & Niehues, 2025) but has not been validated for Mixture-of-Experts models or state-space architectures.

Second, sparse Mixture-of-Experts models enable parameter scaling at fixed inference cost by routing each token through only a subset of expert sub-networks. Research on multilingual routing in MoE models finds that large performance gaps remain for non-English languages, suggesting that routing mechanisms do not naturally distribute multilingual representations without explicit intervention (Bandarkar et al., 2025). Whether language-specific expert specialization can be induced cheaply through targeted fine-tuning is an important open question.

Third, almost no published work quantifies the end-to-end cost of the adaptation pipeline in actual monetary terms across a representative set of target languages. A standardized cost benchmark would substantially improve the ability of practitioners to compare methods and make evidence-based resource allocation decisions. Without such benchmarks, cost claims in the literature remain incomparable.

Fourth, the majority of low-resource language adaptation work focuses on written language. For the hundreds of languages with oral tradition and minimal digital text, the adaptation pathway necessarily involves speech-first approaches or aggressive synthesis of written resources. The interaction between tokenizer fertility, speech-LLM architectures, and adaptation cost in these settings is largely unexplored.

Conclusion

This paper has argued that extending LLM linguistic coverage is a cost-aware design problem spanning tokenization, adaptation compute, and retention of existing capabilities. Progress on one axis alone is rarely enough; practical deployment requires explicit trade-offs among all three.

The literature supports several cautious conclusions. Tokenizer design deserves early attention because excess tokens increase both latency and cost. Parameter-efficient methods, including LoRA, make adaptation experiments more accessible, while curriculum design and retention-aware updates can help protect existing capabilities. The relative value of each intervention depends on the language, data, model, task, and budget. Typological distance is a useful planning signal, but not a substitute for direct evaluation.

The global AI language gap is not merely a question of technical capability; it is also a question of resource allocation and incentive design. Methods that reduce the cost of language adaptation can lower the threshold for serving another language. The framework in this paper is intended as a practical starting point for experiment design. It should be tested and revised against the needs of each target language and community.

References

1. Ahia, O., Marchetti, S., Kuzmin, K., Pinter, Y., & Goldwater, S. (2023). Do all languages cost the same? Tokenization in the era of commercial language models. In Proceedings of EMNLP 2023.
2. Lundin, J. M., Zhang, A., Karim, N., Louzan, H., Wei, V., Adelani, D., & Carroll, C. (2025). The token tax: Systematic bias in multilingual tokenization. arXiv:2509.05486.
3. AIoT-MLSys-Lab. (2024). Efficient large language models: A survey. Transactions on Machine Learning Research.
4. Balachandran, V., et al. (2026). Tokenizer fertility and zero-shot performance of foundation models on Ukrainian legal text. arXiv:2605.14890.
5. Bender, E. M. (2019). The #BenderRule: On naming the languages we study and why it matters. The Gradient.
6. BigScience Workshop. (2023). BLOOM: A 176B-parameter open-access multilingual language model. Transactions on Machine Learning Research.
7. Cahyawijaya, S., Lovenia, H., Yu, T., Chung, W., & Fung, P. (2023). InstructAlign: High-and-low resource language alignment via continual cross-lingual instruction tuning. arXiv:2305.13627.
8. Christophe, C., et al. (2024). Beyond fine-tuning: Unleashing the potential of continuous pretraining for clinical LLMs. arXiv:2409.14988.
9. Dobler, K., & de Melo, G. (2024). Language adaptation on a tight academic compute budget. arXiv:2408.15793.
10. Kim, S., Choi, S., & Jeong, M. (2024). Efficient and effective vocabulary expansion towards multilingual large language models. arXiv:2402.14714.
11. Fujii, K., et al. (2024). Continual pre-training for cross-lingual LLM adaptation: Enhancing Japanese language capabilities. arXiv:2404.17790.
12. Hu, E. J., et al. (2021). LoRA: Low-rank adaptation of large language models. arXiv:2106.09685.
13. Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020). The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of ACL 2020.
14. Yoo, H., Park, C., Yun, S., Oh, A., & Lee, H. (2025). Code-switching curriculum learning for multilingual transfer in LLMs. Findings of the Association for Computational Linguistics: ACL 2025, 7816-7836.
15. Eronen, J., Ptaszynski, M., Wicherkiewicz, T., Borges, R., Janic, K., Liu, Z., Mahmud, T., & Masui, F. (2026). Language models are polyglots: Language similarity predicts cross-lingual transfer learning performance. Machine Learning and Knowledge Extraction, 8(3), 65. https://doi.org/10.3390/make8030065
16. Xu, Y., Xu, K., Zhou, J., Hu, L., & Gui, L. (2025). Linguistic neuron overlap patterns to facilitate cross-lingual transfer on low-resource languages. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
17. Liu, F., & Niehues, J. (2025). Middle-layer representation alignment for cross-lingual transfer in fine-tuned LLMs. arXiv:2502.14830.
18. Liu, J., et al. (2024). A survey on large language models with multilingualism: Recent advances and new frontiers. arXiv:2405.10936.
19. LoResLM Workshop. (2025). Overview of the first workshop on language models for low-resource languages. arXiv:2412.16365.
20. Martins, P., et al. (2024). EuroLLM: Multilingual language models for Europe. arXiv.
21. Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A., Ganu, T., Segal, S., Axmed, M., Bali, K., & Sitaram, S. (2023). MEGA: Multilingual evaluation of generative AI. Proceedings of EMNLP 2023.
22. Yamaguchi, A., Morishita, T., Villavicencio, A., & Aletras, N. (2025). Mitigating catastrophic forgetting in target language adaptation of LLMs via Source-Shielded Updates. arXiv:2512.04844.
23. Wang, J., Lu, Y., Weber, M., Ryabinin, M., Adelani, D. I., Chen, Y., Tang, R., & Stenetorp, P. (2025). Multilingual language model pretraining using machine-translated data. Proceedings of EMNLP 2025.
24. Rana, S., Menezes, A., Kulkarni, A., Khatri, C., & Agarwal, S. (2026). MUTANT: A recipe for multilingual tokenizer design. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, 46261-46277.
25. Özeren, E., Liu, Y., & Schütze, H. (2025). HYPEROFA: Expanding LLM vocabulary to new languages via hypernetwork-based embedding initialization. arXiv:2504.21018.
26. Petrov, A., La Malfa, E., Torr, P. H. S., & Bibi, A. (2024). Language model tokenizers introduce unfairness between languages. arXiv:2305.15425.
27. Rust, P., Pfeiffer, J., Vulic, I., Ruder, S., & Gurevych, I. (2021). How good is your tokenizer? On the monolingual performance of multilingual language models. In Proceedings of ACL 2021.
28. Csaki, Z., Li, B., Li, J., Xu, Q., Pawakapan, P., Zhang, L., Du, Y., Zhao, H., Hu, C., & Thakker, U. (2024). SambaLingo: Teaching large language models new languages. Proceedings of the Fourth Workshop on Multilingual Representation Learning, 1-21.
29. Springer, J. M., et al. (2025). Overtrained language models are harder to fine-tune. arXiv:2503.19206.
30. van Esch, D., et al. (2022). Writing system and speaker metadata for 2,800+ language varieties. In Proceedings of LREC 2022.
31. Zhang, Y., et al. (2024). Enhancing multilingual capabilities of large language models through self-distillation from resource-rich languages. In Proceedings of ACL 2024.
32. Zhou, Z., Zhang, Q., Kumbong, H., & Olukotun, K. (2025). LowRA: Accurate and efficient LoRA fine-tuning of LLMs under 2 bits. arXiv:2502.08141.
33. Bandarkar, L., et al. (2025). Multilingual routing in mixture-of-experts. arXiv:2510.04694.