Large Language Models for Linguists : Building Culturally Sensitive AI

This book bridges linguistics and technology, offering an in-depth exploration of Large Language Models (LLMs) tailored for linguists.It examines how Artificial Intelligence (AI) is trained using language as data, often overlooking linguistic nuance, and empowers linguists to shape AI development to preserve diverse linguistic and cultural heritages. Key Features: Accessible to All: No prior knowledge of technology or coding required, making it ideal for anyone transitioning into AI for languages. Demystifies LLMs: Covers essential concepts like Natural Language Processing (NLP), Named Entity Recognition (NER), hallucinations, fine-tuning, and custom LLMs for languages. Global Scope: Includes research, datasets, and LLM models for languages such as Portuguese, Yoruba, Tamil, Hindi, Swahili, Spanish, Korean, Arabic, Indigenous languages, and sign languages. Cultural Context: Explores how historical trauma, colonialism, migration, and resilience shape language, emphasizing the need for AI to respect this complexity. Practical Playbook: Guides linguists and cultural stewards in creating datasets and benchmarks that honor their communities, showcasing how vocabulary reflects cultural identity. Online Resources: Offers tools, datasets, and community support for building AI tailored to specific languages at LLMLinguist.org. Ideal for humanities scholars, computational linguists, and technologists seeking to develop language models that preserve the cultural richness of the world's voices.