I think depending on your perspective it might be even better to eschew PyTorch and implement with pure NumPy. Sure it will be slow, but perhaps as a pedagogical exercise it is more worthwhile.