wtf is pim-hbm-bypass?

pim-hbm-bypass is an experimental hardware and software prototype aimed at reducing memory copy overhead and improving fault tolerance in large scale GPU clusters running JAX and XLA, the machine learning framework and compiler used for training models like Llama-3-8B. The README describes it as a research prototype exploring whether these techniques can work reliably, rather than a finished, production-ready product. The project's stated goal is to connect low-level CUDA memory addresses directly to JAX's tensor handling, aiming to avoid the usual copying of data between host and device memory. It also describes writing its core CUDA kernel without conditional branching, instead using techniques meant to keep the GPU's compiler generating simple conditional-move instructions, which the README frames as a way to avoid slowdowns from unpredictable branches. Another described feature scans for out-of-bounds memory accesses at the GPU warp level to try to prevent crashes in edge cases like uneven data batch sizes. The repository also includes code that isolates hardware fault signals from JAX's automatic differentiation process, so that a hardware problem does not corrupt gradient calculations during training. A separate piece attempts to detect a failed memory bank in a cluster and reroute to a backup address in real time, without stopping the distributed communication used for multi-GPU training or forcing a recompilation of the computation graph. The repository is organized into a CUDA kernel file, a CMake build script, and several Python files handling JAX compiler warmup, distributed memory sharding, fault detection, and a Llama-3-8B specific adapter. Setup requires an NVIDIA Ampere or Hopper GPU cluster, a CMake and pybind11 build step to compile a shared library, and then running several Python scripts in sequence. The README notes this is intended for high-performance cluster environments rather than typical single-machine use. The project is licensed under Apache License 2.0.