Process Archive
Blogs
Notes on CUDA, Triton, LLM inference, profiling, DeepSpeed, clusters, and practical sandboxing.
Claude Code in a Docker sandbox (kept alive with tmux)
I wanted a YOLO sandbox for Claude Code: isolated dependencies, optional CUDA/PyTorch, and a way to keep the agent running even when my SSH session drops. My setup is: Docker for i...
Dec 12, 2024Efficient Gather-and-scatter Feed-forward Network Kernel with Triton
In our recent work Learn to be efficient: Build structured sparsity in large language modelshttps://arxiv.org/pdf/2402.06126, we propose a novel method to build structured sparsity...
Nov 7, 2024Use Nsight System to Profile a Model Training with DeepSpeed on Multi-Node Cluster
This post is to log how I managed to profile a model training running on multiple nodes in a cluster with DeepSpeed and Nsight System. Click herefinal-implementation to jump to the...
Jun 25, 2024Training Custom Mixtral Model with DeepSpeed
Placeholder for the blog logging how I trained a custom Mixtral model with DeepSpeed.
Jun 19, 2024Custom Gather-scatter Operator by CUTLASS
This blog is to log my experience of building efficient custom operator based on CUTLASShttps://github.com/NVIDIA/cutlass. Jump to the final implementationfinal-implementation of g...
May 27, 2024Compact Inference with CUDA graph and StaticCache
This post is to log a minimum prototype of LLM inference with CUDA graph to eliminate bubbles between kernel launches. Click herefinal-implementation to jump to the final implement...
Apr 24, 2024Efficient Gather-and-scatter Matrix Multiplication Kernel with Triton
This post is to log my implementation of gather-and-scatter matrix multiplication operation with Triton. Click herefinal-implementation to jump to the final implementation code. If...
Mar 31, 2024Understand CUDA Unified Memory
This post is to log my experiments with CUDA unified memory and some innovative and interesting application of UVM in large language model LLM.
Mar 25, 2024Profile CUDA UVM Performance
This post is to log my profile of CUDA unified virtual memory.