Tags

Attention

• August 01, 2024 » Profiling of different implementation of attention modules
• August 01, 2024 » Profiling of different implementation of attention modules

CUDA

• June 19, 2024 » Custom Gather-scatter Operator by CUTLASS
• May 27, 2024 » Compact Inference with CUDA graph and StaticCache
• April 24, 2024 » Efficient Gather-and-scatter Matrix Multiplication Kernel with Triton
• March 31, 2024 » Understand CUDA Unified Memory
• March 25, 2024 » Understand CUDA PTXAS
• March 25, 2024 » Profile CUDA program with Nsight

CUDA Graph

• May 27, 2024 » Compact Inference with CUDA graph and StaticCache

CUTLASS

• June 19, 2024 » Custom Gather-scatter Operator by CUTLASS

Cluster

• November 07, 2024 » Use Nsight System to Profile a Model Training with DeepSpeed on Multi-Node Cluster

Compiler

• March 25, 2024 » Understand CUDA PTXAS

Deepspeed

• November 07, 2024 » Use Nsight System to Profile a Model Training with DeepSpeed on Multi-Node Cluster
• June 25, 2024 » Training Custom Mixtral Model with DeepSpeed

Flash-attn

• August 01, 2024 » Profiling of different implementation of attention modules
• August 01, 2024 » Profiling of different implementation of attention modules

GEMM

• April 24, 2024 » Efficient Gather-and-scatter Matrix Multiplication Kernel with Triton

GPT

• June 25, 2024 » Build an App with OpenAI API

Huggingface

• August 01, 2024 » Profiling of different implementation of attention modules
• August 01, 2024 » Profiling of different implementation of attention modules
• May 27, 2024 » Compact Inference with CUDA graph and StaticCache

LLM

• August 01, 2024 » Profiling of different implementation of attention modules
• August 01, 2024 » Profiling of different implementation of attention modules
• May 27, 2024 » Compact Inference with CUDA graph and StaticCache

Mixtral

• June 25, 2024 » Training Custom Mixtral Model with DeepSpeed

MoE

• June 25, 2024 » Training Custom Mixtral Model with DeepSpeed

Multi-GPU

• November 07, 2024 » Use Nsight System to Profile a Model Training with DeepSpeed on Multi-Node Cluster

Multi-Node

• November 07, 2024 » Use Nsight System to Profile a Model Training with DeepSpeed on Multi-Node Cluster

Node.js

• June 25, 2024 » Build an App with OpenAI API

Nsight

• November 07, 2024 » Use Nsight System to Profile a Model Training with DeepSpeed on Multi-Node Cluster

OpenAI

• June 25, 2024 » Build an App with OpenAI API

Profile

• August 01, 2024 » Profiling of different implementation of attention modules
• August 01, 2024 » Profiling of different implementation of attention modules

Profiler

• March 25, 2024 » Profile CUDA program with Nsight

PyTorch

• June 25, 2024 » Training Custom Mixtral Model with DeepSpeed
• June 19, 2024 » Custom Gather-scatter Operator by CUTLASS

Python

• June 25, 2024 » Build an App with OpenAI API
• June 19, 2024 » Custom Gather-scatter Operator by CUTLASS

Pytorch

• August 01, 2024 » Profiling of different implementation of attention modules
• August 01, 2024 » Profiling of different implementation of attention modules
• May 27, 2024 » Compact Inference with CUDA graph and StaticCache
• April 24, 2024 » Efficient Gather-and-scatter Matrix Multiplication Kernel with Triton

Training

• November 07, 2024 » Use Nsight System to Profile a Model Training with DeepSpeed on Multi-Node Cluster
• June 25, 2024 » Training Custom Mixtral Model with DeepSpeed

Triton

• April 24, 2024 » Efficient Gather-and-scatter Matrix Multiplication Kernel with Triton

Xformers

• August 01, 2024 » Profiling of different implementation of attention modules
• August 01, 2024 » Profiling of different implementation of attention modules