gitwtfhub

wtf is gpt-neox?

blinkdl/gpt-neox — explained in plain English

Analysis updated 2026-07-18 · repo last pushed 2022-02-25

PythonAudience · researcherComplexity · 5/5DormantSetup · hard

TL;DR

A research toolkit for training billion-parameter GPT-style language models across many GPUs, built on NVIDIA Megatron and Microsoft DeepSpeed.

Mindmap

mindmap
  root((gpt-neox))
    What it does
      Trains large LLMs
      Splits work across GPUs
      Configured via YAML
    Tech stack
      Python
      Megatron
      DeepSpeed
    Use cases
      Train custom LLM
      Fine-tune models
      Evaluate benchmarks
    Audience
      Researchers
      GPU cluster owners

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

Why would anyone build with this?

REASON 1

Train a large language model from scratch across multiple GPUs using YAML configs.

REASON 2

Fine-tune an existing GPT-NeoX model on custom data.

REASON 3

Evaluate a trained model's performance on standard benchmarks.

REASON 4

Download and run inference on the pretrained GPT-NeoX-20B weights.

What's in the stack?

PythonMegatronDeepSpeedPyTorch

How it stacks up

blinkdl/gpt-neox0xallam/my-recipe0xhassaan/nn-from-scratch
Stars——0
LanguagePythonPythonPython
Last pushed2022-02-252022-11-22—
MaintenanceDormantDormant—
Setup difficultyhardmoderatemoderate
Complexity5/52/54/5
Audienceresearchergeneraldeveloper

Figures from each repo's GitHub metadata at analysis time.

How do you spin it up?

Difficulty · hard Time to first run · 1day+

Requires multiple high-end GPUs with substantial VRAM and deep familiarity with distributed training concepts.

Yoink these prompts

Prompt 1
Walk me through configuring a YAML file in gpt-neox to train a smaller model on my own GPU cluster.
Prompt 2
Explain how gpt-neox splits both the model and data across multiple GPUs during training.
Prompt 3
Show me how to download and run inference with the pretrained GPT-NeoX-20B weights using deepy.py.
Prompt 4
Help me convert and tokenize my own text dataset into the format gpt-neox expects for training.

Frequently asked questions

wtf is gpt-neox?

A research toolkit for training billion-parameter GPT-style language models across many GPUs, built on NVIDIA Megatron and Microsoft DeepSpeed.

What language is gpt-neox written in?

Mainly Python. The stack also includes Python, Megatron, DeepSpeed.

Is gpt-neox actively maintained?

Dormant — no commits in 2+ years (last push 2022-02-25).

How hard is gpt-neox to set up?

Setup difficulty is rated hard, with roughly 1day+ to a first successful run.

Who is gpt-neox for?

Mainly researcher.

View the repo → Decode another repo

This repo across BitVibe Labs

Don't trust strangers blindly. Verify against the repo.

Other repos in this lane