Use Nsight System to Profile a Model Training with DeepSpeed on Multi-Node Cluster
This post is to log how I managed to profile a model training running on multiple nodes in a cluster with DeepSpeed and Nsight System. Click here to jump to ...
These blogs are written to log my study during research for future reference. Hope you can also enjoy!
2024 Nov
2024 Aug
Profiling of different implementation of attention modules
Placeholder for the blog logging the results of profiling vllm
Profiling of different implementation of attention modules
Placeholder for the blog logging the results of profiling different implementations of attention modules.
2024 Jun
Build an App with OpenAI API
Placeholder for the blog logging how I build an app with OpenAI API.
Training Custom Mixtral Model with DeepSpeed
Placeholder for the blog logging how I trained a custom Mixtral model with DeepSpeed.
Custom Gather-scatter Operator by CUTLASS
This blog is to log my experience of building efficient custom operator based on CUTLASS. Jump to the final implementation of gather and scatter matrix multi...
2024 May
Compact Inference with CUDA graph and StaticCache
This post is to log a minimum prototype of LLM inference with CUDA graph to eliminate bubbles between kernel launches. Click here to jump to the final implem...
2024 Apr
Efficient Gather-and-scatter Matrix Multiplication Kernel with Triton
This post is to log my implementation of gather-and-scatter matrix multiplication operation with Triton. Click here to jump to the final implementation code.
2024 Mar
Understand CUDA Unified Memory
This post is to log my experiments with CUDA unified memory and some innovative and interesting application of UVM in large language model (LLM).
Understand CUDA PTXAS
Placeholder for my blog
Profile CUDA program with Nsight
Placeholder for my blog