Efficient Gather-and-scatter Feed-forward Network Kernel with Triton
In our recent work Learn to be efficient: Build structured sparsity in large language models, we propose a novel method to build structured sparsity in large...
Recent Posts
Use Nsight System to Profile a Model Training with DeepSpeed on Multi-Node Cluster
This post is to log how I managed to profile a model training running on multiple nodes in a cluster with DeepSpeed and Nsight System. Click here to jump to ...
Training Custom Mixtral Model with DeepSpeed
Placeholder for the blog logging how I trained a custom Mixtral model with DeepSpeed.
Custom Gather-scatter Operator by CUTLASS
This blog is to log my experience of building efficient custom operator based on CUTLASS. Jump to the final implementation of gather and scatter matrix multi...
Compact Inference with CUDA graph and StaticCache
This post is to log a minimum prototype of LLM inference with CUDA graph to eliminate bubbles between kernel launches. Click here to jump to the final implem...