March 15, 2021
Machine Learning

# Let’s begin our TorchAdventure in AWS!

with this 11 basic functions distributed in the following sections:

• Creating tensors: Empty() and Zeros()
• Measuring tensors: Size() and Shape(), Sum() and Dimensions
• Changing and copying tensors: Reshape(), view() and randn()
• Modifying tensors: Unsqueeze()
• Comparing tensors: Element-Wise Equality: Eq

First lets spin up a new sagemaker instance:

Now go to Jupyter Lab and import or create a new Notebook with this kernel for Pytorch:

Select the appropiate kernel, for this case it could be: conda_pytorch_p36

Here is where I will start from scratch.

Before we begin, let’s install and import PyTorch

{% c-block language="js" %}
# Uncomment and run the appropriate command for your operating system, if required# Linux / Binder

# !pip install numpy torch torchvision torchaudio
{% c-block-end %}

Import Pytorch from the Notebook instance
{% c-block language="js" %}
import torch

{% c-block-end %}

# Function 1 — Empty and Zeros — how to initialize tensors

We need to start working with the basics pytorch functions, and the first thing is create our matrix

{% c-block language="js" %}
# Creates a 3 x 2 matrix which is empty

a = torch.empty(3, 2)
print(a)tensor([[1.5842e-35, 0.0000e+00],
[4.4842e-44, 0.0000e+00],
[       nan, 0.0000e+00]])
{% c-block-end %}

Here is how PyTorch is allocating memory for this tensor. Whatever, it will not erase anything previous content in the memory.

by default, when you initializes a tensor is used the float32 dtype. you can review it here: https://pytorch.org/docs/stable/generated/torch.set_default_tensor_type.html#torch.set_default_tensor_type

But you can also start working with torch.zeros

# Applying the zeros function and
# storing the resulting tensor

{% c-block language="js" %}
a = torch.zeros([3, 5])
print("a = ", a)
b = torch.zeros([2, 4])
print("b = ", b)
c = torch.zeros([4, 1])
print("c = ", c)
d = torch.zeros([4, 4, 2])
print("d = ", d)
{% c-block-end %}

the result will be:

{% c-block language="js" %}
a =  tensor([[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.]])
b =  tensor([[0., 0., 0., 0.],
[0., 0., 0., 0.]])
c =  tensor([[0.],
[0.],
[0.],
[0.]])
d =  tensor([[[0., 0.],
[0., 0.],
[0., 0.],
[0., 0.]],[[0., 0.],
[0., 0.],
[0., 0.],
[0., 0.]],[[0., 0.],
[0., 0.],
[0., 0.],
[0., 0.]],[[0., 0.],
[0., 0.],
[0., 0.],
[0., 0.]]])
{% c-block-end %}

this tensor is filled with zeros, so PyTorch allocates memory and zero-initializes the tensor elements inside

You cannot change the way a tensor is created, if you create a zeros tensor, make sure is not referenced to any other.

{% c-block language="js" %}
# correctly initialized

a = torch.empty(3,3)
print(a)#also correct
b = torch.empty(1,2,3)print(b)
{% c-block-end %}

the results will be:

{% c-block language="js" %}
tensor([[2.4258e-35, 0.0000e+00, 1.5975e-43],
[1.3873e-43, 1.4574e-43, 6.4460e-44],
[1.4153e-43, 1.5274e-43, 1.5695e-43]])
tensor([[[2.3564e-35, 0.0000e+00, 1.4013e-45],
[1.4574e-43, 6.4460e-44, 1.4153e-43]]])
{% c-block-end %}

Lets see now, how you cannot use the zeros function:

{% c-block language="js" %}
# incorrect reference, you must create a new one

c = torch.zeros(b,1)
print("c = ", c)
{% c-block-end %}

output:

{% c-block language="js" %}
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-35-c044e2995879> in <module>()
1 # incorrect reference, you must create a new one
----> 2 c = torch.zeros(b,1)
3 print("c = ", c)TypeError: zeros() received an invalid combination of arguments - got (Tensor, int), but expected one of:
* (tuple of ints size, *, tuple of names names, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
* (tuple of ints size, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
{% c-block-end %}

lets review the zero method closely:

Our documentation says:

Syntax: torch.zeros(size, out=None)Parameters:
size: a sequence of integers defining the shape of the output tensor
out (Tensor, optional): the output tensorReturn type: A tensor filled with scalar value 0, of same shape as size.

torch.zeros and torch.empty are the first functions to start working with pytorch tensors and learning a little bit of matrix and vectors

# Function 2 — Tensor Size, Shape and Dimension Operations

Lets understand dimensions in Pytorch.

Now lets create some tensors and determine the size of every one

{% c-block language="js" %}
import torch
# Create a tensor from data
c = torch.tensor([[3.2 , 1.6, 2], [1.3, 2.5 , 6.9]])
print(c)print(c.size())
{% c-block-end %}

output[]:

{% c-block language="js" %}
tensor([[3.2000, 1.6000, 2.0000],
[1.3000, 2.5000, 6.9000]])
torch.Size([2, 3])
{% c-block-end %}

Lets see torch.shape and take a closer look at how size is given here

Now in the next example lets use shape functions to get the size of a tensor

In [ ]:
{% c-block language="js" %}
x = torch.tensor([
[1, 2, 3],
[4, 5, 6]
])
x.shape
torch.Size([2, 3])
{% c-block-end %}

Out[ ]:
{% c-block language="js" %}
torch.Size([2, 3])
{% c-block-end %}

Shape and Size give us the same correct dimensions of the tensor.

in this case we have a 3D-tensor (with 3 dimensions)

Dimension 0 Dimension 1 and Dimension 2

lets create a new tensor:

In [ ]:
{% c-block language="js" %}
y = torch.tensor([
[
[1, 2, 3],
[4, 5, 6]
],
[
[1, 2, 3],
[4, 5, 6]
],
[
[1, 2, 3],
[4, 5, 6]
]
])y.shape
{% c-block-end %}

Out[ ]:
{% c-block language="js" %}
torch.Size([3, 2, 3])
{% c-block-end %}

Lets se how we can make operations using a 3d tensor now for every dimension layer and see how it behaves

In [ ]:
{% c-block language="js" %}
sum1 = torch.sum(y, dim=0)
print(sum1)sum2 = torch.sum(y, dim=1)
print(sum2)sum3 = torch.sum(y, dim=2)
print(sum3)tensor([[ 3,  6,  9],
[12, 15, 18]])
tensor([[5, 7, 9],
[5, 7, 9],
[5, 7, 9]])
tensor([[ 6, 15],
[ 6, 15],
[ 6, 15]])
{% c-block-end %}

we can see now a 3d tensor is more complicated as we advance, and we can perform custom operations within every dimension

its limited right now to 3 dims so we cannot perform this:

In []
{% c-block language="js" %}
sum2 = torch.sum(y, dim=3)
print(sum1)
{% c-block-end %}

Out [ ]:
{% c-block language="js" %}
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-56-7caf723c6102> in <module>()
----> 1 sum2 = torch.sum(y, dim=3)
2 print(sum1)IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)
{% c-block-end %}

to accomplish adding more layers of dimensions we can review the unsqueeze functions → .unsqueeze() but for now let’s go to the next basic function

# Function 3 — Reshape, View and Random

There is a function in NunPy called ndarray.reshape() for reshaping an array.

Now in pytorch, there is torch.view(tensor) for the same purpose, but at the same time, there is also a torch.reshape(tensor).

let’s figure out the differences between them and when you should use either of them.

First of all we are going to use a new functions for randomize our tensor.

In [ ]:
{% c-block language="js" %}
import torch
x = torch.randn(5, 3)print(x)
{% c-block-end %}

Out []
{% c-block language="js" %}
tensor([[-0.5793,  0.6999,  1.7417],
[-0.9810,  0.0626,  0.4100],
[-0.6519, -0.0595, -1.2156],
[-0.3973, -0.3103,  1.6253],
[ 0.2775, -0.0045, -0.2985]])
{% c-block-end %}

this is basic usage of torch.randm, so now lets use View from another variable “y” and describe all elements

In [ ]:
# Return a view of the x, but only having
# one dimension and max number of elements

{% c-block language="js" %}
y = x.view(5 * 3)
#lets see the size of every tensor
print("lets see the size of every tensor")
print('Size of x:', x.size())
print('Size of y:', y.size())
#and the elements of very tensor to compare
print("and the elements of very tensor to compare")
print("X:", x)
print("Y:", y)
{% c-block-end %}

Out []:
{% c-block language="js" %}
lets see the size of every tensor
Size of x: torch.Size([5, 3])
Size of y: torch.Size()
and the elements of very tensor to compare
X: tensor([[-0.5793,  0.6999,  1.7417],
[-0.9810,  0.0626,  0.4100],
[-0.6519, -0.0595, -1.2156],
[-0.3973, -0.3103,  1.6253],
[ 0.2775, -0.0045, -0.2985]])
Y: tensor([-0.5793,  0.6999,  1.7417, -0.9810,  0.0626,  0.4100, -0.6519, -0.0595,
-1.2156, -0.3973, -0.3103,  1.6253,  0.2775, -0.0045, -0.2985])
{% c-block-end %}

take a look at Y tensor, it only has 1 dimension. so Viewing another tensor may be difficult for some operations

Now lets use Reshape to replicate the exact dimensions of X

In [ ]:
# Get back the original tensor with reshape()

{% c-block language="js" %}
z = y.reshape(5, 3)
print(z)tensor([[-0.2927,  0.0329,  0.8485],
[ 1.9581,  0.8313, -0.1529],
[-0.2330, -0.1887,  1.8206],
[ 1.5252,  1.0909,  0.0547],
[-0.1231, -0.4238, -0.6724]])
{% c-block-end %}

we cannot only reshape the original, we can also change the dimensions with some limited actions related to the maximum elements:

first lets reshape to 1 more dimension

In [ ]:
# Get back the original tensor with reshape()

{% c-block language="js" %}
z = y.reshape(15)
print(z)#reshaping 15 elements, 1 dim
z = y.reshape(3*5)
print(z)#reshaping in different order, 3 dimensions
z = y.reshape(3,5)
print(z)#Reshaping with more dimensions but its 15 elements always
z = y.reshape(3,5,1)
print(z)tensor([-0.5793,  0.6999,  1.7417, -0.9810,  0.0626,  0.4100, -0.6519, -0.0595,
-1.2156, -0.3973, -0.3103,  1.6253,  0.2775, -0.0045, -0.2985])
tensor([-0.5793,  0.6999,  1.7417, -0.9810,  0.0626,  0.4100, -0.6519, -0.0595,
-1.2156, -0.3973, -0.3103,  1.6253,  0.2775, -0.0045, -0.2985])
tensor([[-0.5793,  0.6999,  1.7417, -0.9810,  0.0626],
[ 0.4100, -0.6519, -0.0595, -1.2156, -0.3973],
[-0.3103,  1.6253,  0.2775, -0.0045, -0.2985]])
tensor([[[-0.5793],
[ 0.6999],
[ 1.7417],
[-0.9810],
[ 0.0626]],        [[ 0.4100],
[-0.6519],
[-0.0595],
[-1.2156],
[-0.3973]],        [[-0.3103],
[ 1.6253],
[ 0.2775],
[-0.0045],
[-0.2985]]])
{% c-block-end %}

Now lets reshape exceeding the number of elements in the tensor:

In [ ]:
{% c-block language="js" %}
z = y.reshape(16)
print(z)
{% c-block-end %}

Out []:
{% c-block language="js" %}
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-78-c7ae174fce73> in <module>()
----> 1 z = y.reshape(16)
2 print(z)RuntimeError: shape '' is invalid for input of size 15
{% c-block-end %}

it will fail also for z = y.reshape(3*6) or putting more elements that does not exist in the tensor.

Now let’s keep going to the next section.

# Function 4 — Unsqueeze()

Mainly, it allows us to add more dimensions at specific index you define.

lets take a look:

In [ ]:
{% c-block language="js" %}
import torch
#dim=1, that is (3)
x = torch.tensor([1, 2, 3])
print('x: ', x)
print('x.size: ', x.size())#x1 becomes a matrix of (3,1)
x1 = torch.unsqueeze(x, 1)
print('x1: ', x1)
print('x1.size: ', x1.size())
{% c-block-end %}

Out []:
{% c-block language="js" %}
x:  tensor([1, 2, 3])
x.size:  torch.Size()
x1:  tensor([,
,
])
x1.size:  torch.Size([3, 1])
x2:  tensor([[1, 2, 3]])
x2.size:  torch.Size([1, 3])
{% c-block-end %}

Our initial tensor is Tensor([1,2,3]), and the output size is .

And then we proceed with adding 1 dimenson with unsqueeze operation, namely torch.unsqueeze(x, 1), the size of x1 is [3,1].

In [ ]:
{% c-block language="js" %}
x2 = torch.unsqueeze(x, 0)
#x2 becomes a matrix of (1,3)
print('x2: ', x2)
print('x2.size: ', x2.size())
{% c-block-end %}

Out []:
{% c-block language="js" %}
x2:  tensor([[1, 2, 3]])
x2.size:  torch.Size([1, 3])
{% c-block-end %}

When we perform torch.unsqueeze(x, 0), the size of x2 is [1,3].

Example 3 - breaking (to illustrate when it breaks)

In [ ]:
{% c-block language="js" %}
print(x.unsqueeze())
{% c-block-end %}

Out []:
{% c-block language="js" %}
---------------------------------------------------------------------------
TypeError  Traceback (most recent call last)
<ipython-input-89-5a320a828907> in <module>()
2
3
----> 4 print(x.unsqueeze())TypeError: unsqueeze() missing 1 required positional arguments: "dim"
{% c-block-end %}

We must specified the dimension correctly, although we are just adding 1 dim, it is necessary to put like this: x.unqueeze(0)

# Function 5 — Torch Eq (Element Wise equality)

This function is under comparison category and it computes equality in element-wise and returns a boolean tensor. True if equal, False otherwise.

Lets review how we can operate with different sizes of tensors:

Example 1 - working

In [ ]:
{% c-block language="js" %}
x1 = torch.tensor([[1, 2], [3, 4.]])
x2 = torch.tensor([[2, 2], [2, 5]])
x3 = torch.randn(3,5)#size x1 and z2
print(x1.size())
print(x2.size())# tensors 1 and 2
print(x1)
print(x2)
#size x3
print(x3.size())#tensors 3
print(x3)torch.eq(x1,x2)torch.Size([2, 2])
torch.Size([2, 2])
tensor([[1., 2.],
[3., 4.]])
tensor([[2, 2],
[2, 5]])
torch.Size([3, 5])
tensor([[-1.3040, -0.4658, -0.5269,  0.7409,  0.9135],
[ 1.0780,  2.0584, -0.9629, -1.1412, -0.3105],
[ 0.3613, -1.4196,  2.1145,  0.3649,  0.2037]])
{% c-block-end %}

Out[ ]:
{% c-block language="js" %}
tensor([[False,  True],
[False, False]])
{% c-block-end %}

x1 and x3 have the same size, but x3 is [3,5], has bigger size.

comparing x1 and x2 is Ok.

Example 2 - working (with broadcasting)

In [ ]:
{% c-block language="js" %}
x4 = torch.tensor([[1, 2], [3, 4]])
print(x4.size())
x5 = torch.tensor([2, 5])
print(x5.size())
torch.eq(x4, x5)torch.Size([2, 2])
torch.Size()
{% c-block-end %}

Out[ ]:
{% c-block language="js" %}
tensor([[False, False],
[False, False]])
{% c-block-end %}

we can also compare, different sizes only if the seconf value, in this case x5, that has size of  is broadcastable with the frist one thst is [2,2]

Example 3 - breaking (to illustrate when it breaks)

In [ ]:
{% c-block language="js" %}
x6 = torch.tensor([[0, 2, 4], [3, 4, 5]])
print(x6.size())
x7 = torch.tensor([[2, 3], [2, 4]])
print(x7.size())torch.eq(x6, x3)torch.Size([2, 3])
torch.Size([2, 2])---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
5 print(x7.size())
6
{% c-block-end %}

----> 7 torch.eq(x6, x3)RuntimeError: The size of tensor a (3) must match the size of tensor b (5) at non-singleton dimension 1

finally, we cant compare different sizes if the second arguments shape is not broadcastable with the first argument.

# Conclusion

we review 5 basic topics covering more than 10 PyTorch functions.

in the next post i’ll talk about Linear Regression.

#aws #reinvent2020 #awsperu #awsugperu #awscloud

Carlos Cortez — AWS UG Perú Leader / AWS ML Community Builder
ccortez@aws.pe
@ccortezb
Podcast: imperiocloud.com @imperiocloud
twitch.tv/awsugperu
cennticloud.thinkific.com Carlos Cortez

Founder of AWS UG Perú official community. Certified With 7+ years of experience in AWS, Educating people in Cloud computing is my passion so I’m creating different ways to enhance cloud skills and have fun at the same time in Perú. Host of my own Podcast Imperio Cloud and DeepFridays for AI Learning video series. I’m a System Engineer and Data Analytics Certified at MIT Sloan Global Program in Boston. Selected to be part of AWS Community Builder for ML. Founder of CENNTI Cloud to help peruvian companies in their difficult journey to the cloud and developing ML/AI solutions to fight COVID-19 and health related applications. Now Working as Senior Cloud Architect at DB Solutions in Chile.