wtf is transformer-decoder-from-scratch?

This project is a personal learning exercise where the author built a decoder only Transformer, the type of architecture behind modern language models, almost entirely by hand instead of using pre built libraries. PyTorch is used only for basic building blocks like automatic differentiation, tensor math, and running computations on the GPU. Everything else, including the attention mechanism, positional encoding, normalization layers, the optimizer, the loss function, and the training loop, was written from scratch. The resulting model has about 30 million trainable parameters and can both be trained and used to generate text one word at a time. The README is written as a personal account of what the author learned along the way, rather than as a polished product description. They describe debugging issues like their custom optimizer accidentally staying on the CPU while the model ran on the GPU, and several bugs in the positional encoding caused by reshaping tensors incorrectly. They also describe efficiency improvements they made, such as combining three separate calculations into one, and building then abandoning a custom tokenizer in favor of an existing one called tiktoken because their own version was too slow. Training the model required adjusting the learning rate schedule and using a technique called gradient accumulation to work around limited GPU memory on the author's own graphics card. The model was trained on a public dataset called WikiText-103, and its training loss dropped steadily over time before leveling off. Sample output included in the README shows the model producing text that is grammatically correct and readable, but which tends to drift away from the original topic over longer passages and does not stay factually accurate. The author attributes this mainly to the size of the training data and the amount of compute used rather than to the architecture itself, and plans to continue training on a larger dataset using rented GPUs to improve results.