Skip to content

================

by Jawad Haider

01 - Tensor Operations

Image
Copyright Qalmaqihir
For more information, visit us at www.github.com/qalmaqihir/

Tensor Operations

This section covers: * Indexing and slicing * Reshaping tensors (tensor views) * Tensor arithmetic and basic operations * Dot products * Matrix multiplication * Additional, more advanced operations

Perform standard imports

import torch
import numpy as np

Indexing and slicing

Extracting specific values from a tensor works just the same as with NumPy arrays


Image source: http://www.scipy-lectures.org/_images/numpy_indexing.png

x = torch.arange(6).reshape(3,2)
print(x)

tensor([[0, 1],
        [2, 3],
        [4, 5]])

# Grabbing the right hand column values
x[:,1]

tensor([1, 3, 5])

# Grabbing the right hand column as a (3,1) slice
x[:,1:]

tensor([[1],
        [3],
        [5]])

Reshape tensors with .view()

view() and reshape() do essentially the same thing by returning a reshaped tensor without changing the original tensor in place.
There’s a good discussion of the differences here.

x = torch.arange(10)
print(x)

tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

x.view(2,5)

tensor([[0, 1, 2, 3, 4],
        [5, 6, 7, 8, 9]])

x.view(5,2)

tensor([[0, 1],
        [2, 3],
        [4, 5],
        [6, 7],
        [8, 9]])

# x is unchanged
x

tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Views reflect the most current data

z = x.view(2,5)
x[0]=234
print(z)

tensor([[234,   1,   2,   3,   4],
        [  5,   6,   7,   8,   9]])

Views can infer the correct size

By passing in -1 PyTorch will infer the correct value from the given tensor

x.view(2,-1)

tensor([[234,   1,   2,   3,   4],
        [  5,   6,   7,   8,   9]])

x.view(-1,5)

tensor([[234,   1,   2,   3,   4],
        [  5,   6,   7,   8,   9]])

Adopt another tensor’s shape with .view_as()

view_as(input) only works with tensors that have the same number of elements.

x.view_as(z)

tensor([[234,   1,   2,   3,   4],
        [  5,   6,   7,   8,   9]])

Tensor Arithmetic

Adding tensors can be performed a few different ways depending on the desired result.

As a simple expression:

a = torch.tensor([1,2,3], dtype=torch.float)
b = torch.tensor([4,5,6], dtype=torch.float)
print(a + b)

tensor([5., 7., 9.])

As arguments passed into a torch operation:

print(torch.add(a, b))

tensor([5., 7., 9.])

With an output tensor passed in as an argument:

result = torch.empty(3)
torch.add(a, b, out=result)  # equivalent to result=torch.add(a,b)
print(result)

tensor([5., 7., 9.])

Changing a tensor in-place

a.add_(b)  # equivalent to a=torch.add(a,b)
print(a)

tensor([5., 7., 9.])
NOTE: Any operation that changes a tensor in-place is post-fixed with an underscore *.
In the above example: a.add*(b) changed a.

Basic Tensor Operations

Arithmetic
OPERATION FUNCTION DESCRIPTION
a + b a.add(b) element wise addition
a - b a.sub(b) subtraction
a \* b a.mul(b) multiplication
a / b a.div(b) division
a % b a.fmod(b) modulo (remainder after division)
ab a.pow(b) power
 
Monomial Operations
OPERATION FUNCTION DESCRIPTION
\|a\| torch.abs(a) absolute value
1/a torch.reciprocal(a) reciprocal
$\sqrt{a}$ torch.sqrt(a) square root
log(a) torch.log(a) natural log
ea torch.exp(a) exponential
12.34 ==\> 12. torch.trunc(a) truncated integer
12.34 ==\> 0.34 torch.frac(a) fractional component
Trigonometry
OPERATION FUNCTION DESCRIPTION
sin(a) torch.sin(a) sine
cos(a) torch.sin(a) cosine
tan(a) torch.sin(a) tangent
arcsin(a) torch.asin(a) arc sine
arccos(a) torch.acos(a) arc cosine
arctan(a) torch.atan(a) arc tangent
sinh(a) torch.sinh(a) hyperbolic sine
cosh(a) torch.cosh(a) hyperbolic cosine
tanh(a) torch.tanh(a) hyperbolic tangent
Summary Statistics
OPERATION FUNCTION DESCRIPTION
$\sum a$ torch.sum(a) sum
$\bar a$ torch.mean(a) mean
amax torch.max(a) maximum
amin torch.min(a) minimum
torch.max(a,b) returns a tensor of size a
containing the element wise max between a and b
NOTE: Most arithmetic operations require float values. Those that do work with integers return integer tensors.
For example, torch.div(a,b) performs floor division (truncates the decimal) for integer types, and classic division for floats.

Use the space below to experiment with different operations

a = torch.tensor([1,2,3], dtype=torch.float)
b = torch.tensor([4,5,6], dtype=torch.float)
print(torch.add(a,b).sum())

tensor(21.)

Dot products

A dot product is the sum of the products of the corresponding entries of two 1D tensors. If the tensors are both vectors, the dot product is given as:

\(\begin{bmatrix} a & b & c \end{bmatrix} \;\cdot\; \begin{bmatrix} d & e & f \end{bmatrix} = ad + be + cf\)

If the tensors include a column vector, then the dot product is the sum of the result of the multiplied matrices. For example:
\(\begin{bmatrix} a & b & c \end{bmatrix} \;\cdot\; \begin{bmatrix} d \\ e \\ f \end{bmatrix} = ad + be + cf\)

Dot products can be expressed as torch.dot(a,b) or a.dot(b) or b.dot(a)

a = torch.tensor([1,2,3], dtype=torch.float)
b = torch.tensor([4,5,6], dtype=torch.float)
print(a.mul(b)) # for reference
print()
print(a.dot(b))

tensor([ 4., 10., 18.])

tensor(32.)
NOTE: There’s a slight difference between torch.dot() and numpy.dot(). While torch.dot() only accepts 1D arguments and returns a dot product, numpy.dot() also accepts 2D arguments and performs matrix multiplication. We show matrix multiplication below.

Matrix multiplication

2D Matrix multiplication is possible when the number of columns in tensor A matches the number of rows in tensor B. In this case, the product of tensor A with size \((x,y)\) and tensor B with size \((y,z)\) results in a tensor of size \((x,z)\)



$\begin{bmatrix} a & b & c \\ d & e & f \end{bmatrix} \;\times\; \begin{bmatrix} m & n \\ p & q \\ r & s \end{bmatrix} = \begin{bmatrix} (am+bp+cr) & (an+bq+cs) \\ (dm+ep+fr) & (dn+eq+fs) \end{bmatrix}$

Matrix multiplication can be computed using torch.mm(a,b) or a.mm(b) or a @ b

a = torch.tensor([[0,2,4],[1,3,5]], dtype=torch.float)
b = torch.tensor([[6,7],[8,9],[10,11]], dtype=torch.float)

print('a: ',a.size())
print('b: ',b.size())
print('a x b: ',torch.mm(a,b).size())

a:  torch.Size([2, 3])
b:  torch.Size([3, 2])
a x b:  torch.Size([2, 2])

print(torch.mm(a,b))

tensor([[56., 62.],
        [80., 89.]])

print(a.mm(b))

tensor([[56., 62.],
        [80., 89.]])

print(a @ b)

tensor([[56., 62.],
        [80., 89.]])

Matrix multiplication with broadcasting

Matrix multiplication that involves broadcasting can be computed using torch.matmul(a,b) or a.matmul(b) or a @ b

t1 = torch.randn(2, 3, 4)
t2 = torch.randn(4, 5)

print(torch.matmul(t1, t2).size())

torch.Size([2, 3, 5])

However, the same operation raises a RuntimeError with torch.mm():

print(torch.mm(t1, t2).size())

RuntimeError: matrices expected, got 3D, 2D tensors at ..\aten\src\TH/generic/THTensorMath.cpp:956

Advanced operations

L2 or Euclidian Norm

See torch.norm()

The Euclidian Norm gives the vector norm of \(x\) where \(x=(x_1,x_2,...,x_n)\).
It is calculated as

\({\displaystyle \left\|{\boldsymbol {x}}\right\|_{2}:={\sqrt {x_{1}^{2}+\cdots +x_{n}^{2}}}}\)

When applied to a matrix, torch.norm() returns the Frobenius norm by default.

x = torch.tensor([2.,5.,8.,14.])
x.norm()

tensor(17.)

Number of elements

See torch.numel()

Returns the number of elements in a tensor.

x = torch.ones(3,7)
x.numel()

21

This can be useful in certain calculations like Mean Squared Error:
def mse(t1, t2):
    diff = t1 - t2
    return torch.sum(diff * diff) / diff.numel()

Great work!