How to assign a name for a pytorch layer?

Written by - Aionlinecourse2167 times views

There are mostly two ways to assign a name for a PyTorch layer. The first way is to use the torch.nn package's naming convention, where each value in Bidirectional(...) is replaced with underscore( _ ) underscores. This way, the layered module would be called FuBLongShortTermMemory .

The second way uses prefixes or keywords to group layers by type. "Local" for example could be used as the keyword if adding redundant layers to localize receptive fields of convolutional neural networks, which can be useful in pedestrian detection applications that require high precision on pedestrian boundaries. "Generative" could serve as keyword if adding generative neural network layers that produce new content at each timestep.

How to assign a name for a pytorch layer:

Sequential

Pass an instance of collections.OrderedDict. Code below gives conv1.weights, conv1.bias, conv2.weight, conv2.bias (notice lack of torch.nn.ReLU(), see end of this answer).

import collections

import torch

model = torch.nn.Sequential(
    collections.OrderedDict(
        [
            ("conv1", torch.nn.Conv2d(1, 20, 5)),
            ("relu1", torch.nn.ReLU()),
            ("conv2", torch.nn.Conv2d(20, 64, 5)),
            ("relu2", torch.nn.ReLU()),
        ]
    )
)

for name, param in model.named_parameters():
    print(name)

Dynamic

Use ModuleDict instead of ModuleList:

class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.whatever = torch.nn.ModuleDict(
            {f"my_name{i}": torch.nn.Conv2d(10, 10, 3) for i in range(5)}
        )

Will give us whatever.my_name{i}.weight (or bias) for each created module dynamically.

Direct

Just name it however you want and that's how it will be named

self.my_name_or_whatever = nn.Linear(7, 8)

You didn't think about

If you want to plot weights, biases and their gradients you can go along this route
You can't plot activations this way (or output from activations). Use PyTorch hooks instead (if you want per-layer gradients as they pass through network use this also)

For last task you can use third party library torchfunc (disclaimer: I'm the author) or go directly and write your own hooks.

Thank you for reading the article. If you face any problem, please comment below.

Recommended Projects