wtf is text-llm-training-from-scratch?

This project builds an entire text generating AI system from the ground up, in Python, without relying on the popular libraries that usually hide the details. Instead of using ready made frameworks like transformers, it writes every piece of the training process itself, so someone learning how these systems work can read through the code stage by stage and understand what is actually happening at each step. The pipeline covers four main phases: turning text into a numbered vocabulary the model can read, training the model to predict the next piece of text over and over, teaching it to follow instructions through supervised fine tuning, and then refining its behavior further using preference based methods that reward better responses over worse ones. The underlying model is a modern text transformer built with the same kinds of components used in current well known models, and it runs on a regular CPU, an Apple Silicon Mac, or an Nvidia GPU without needing separate code for each. Once installed, the project gives you a single command called llm that can train a tokenizer, prepare training data, pretrain a model, generate text from it, and more. It even includes a small sample text dataset so you can run through the whole process quickly, and an interactive menu that walks you through every step by arrow keys, always showing you the exact command it is about to run. As a concrete demonstration, the repo includes a ready made configuration for a small 17 million parameter model trained on a dataset of short children's stories, and shows a real sample of the coherent short stories that model can produce after training. The tokenizer works at the level of raw text bytes, so it can handle any language without ever failing to recognize a character, and training data is stored efficiently on disk so datasets larger than your computer's memory can still be used. This project is best suited for people who want to genuinely understand how language models are trained rather than just use one, and it is released under the MIT license.