Inside PyTorch: How Tensors are Created and Stored
From dtype and device selection, through allocators and storage, into TensorImpl, and finally the user-facing tensor, we will walk through the complete lifecycle of a PyTorch tensor.
Read moreI like building and tuning language models - figuring out how they think and how to make them better. I code in Python and C++, use JAX and PyTorch a lot, and know my way around GPUs, CUDA and ROCm.
I write to give structure to my thinking. So here's some of my writings.
From dtype and device selection, through allocators and storage, into TensorImpl, and finally the user-facing tensor, we will walk through the complete lifecycle of a PyTorch tensor.
Read moredesign inspired by jhey