Provide an automated and accurate way to locate where people live using satellite images.
Work through tutorials for deep learning packages.
Roy Hyunjin Han
When we first started this project in 2007, the leading convolutional neural network framework was a Lisp package called Lush written by Yann LeCun and his graduate students. Subsequent versions of our remote sensing system used Alex Krizhevsky's cuda-convnet and cuda-convnet2. At the time we also considered using theano or caffe. Since then, there have been several new options, including TensorFlow, Keras and PyTorch. We would like to re-evaluate the deep learning landscape by trying each package's tutorial.
20170720-0930 - 20170720-1000: 30 minutes
I remember during my last few weeks of volunteering at the CBLL, a doctoral student named Koray was working on a Lush replacement called Torch. At the 2016 NIPS conference, the Facebook announced the release of PyTorch. I think PyTorch is a wrapper around Torch. At any rate, it looks like they have been very good about their documentation and tutorials.
We are going to start with the Deep Learning 60 Minute Blitz on the PyTorch website.
import torch
But first, it looks like we are going to have to install it. I wonder if Torch/PyTorch works on CPUs as well as GPUs. We don't quite have full GPU support on our website yet. It looks like the latest Torch was written in C/C++. I can just imagine all the headaches those Torch developers had with void pointers haha. Too bad they didn't have Rust at the time.
So I am attempting to use the pip version of the installation.
pip install http://download.pytorch.org/whl/cu80/torch-0.1.12.post2-cp35-cp35m-linux_x86_64.whl
pip install torchvision
Whoa, it looks like the installation finished! There weren't any errors! Let's not get too excited and make sure we can import the package.
import torch
Okay, we can import the package. Let's go back to the tutorial.
torch.Tensor(5, 3)
It looks like it is filled with junk numbers of epsilon value.
x = torch.rand(5, 3)
x
x.size()
torch.rand(5, 3) + torch.rand(5, 3)
torch.rand(5, 3) + torch.rand(3, 5)
from pytest import raises
with raises(RuntimeError):
torch.rand(2, 2) + torch.rand(10, 10)
x = torch.rand(4, 2)
y = torch.rand(4, 2)
torch.add(x, y)
It looks like torch is following the convention of m rows by n columns.
z = torch.rand(4, 2)
x = torch.rand(4, 2)
y = torch.rand(4, 2)
torch.add(x, y, out=z)
z
I'm not sure what is the point of the out argument. Maybe it's just consistency with the C/C++ code.
y.add_(x)
Maybe the add_ and out are speed and memory optimizations so that the data doesn't have to get passed back and forth between the C code and Python, which can get expensive for huge datasets.
x
x[:1]
x[:, 1]
a = torch.ones(5)
a
b = a.numpy()
b
b[2] = 10
b
a
It looks like changing the numpy array will not result in a change in the tensor. What about the other way around?
Trying the same procedure again a few days later, it seems as though they fixed it. We can change the numpy array and see the change reflected in the tensor and vice versa.
a[2] = 5
a
b
a.add_(1)
b
Hmm, the behavior seems to differ from the way it is promised in the documentation. Let's try this again.
x = torch.ones(5)
y = x.numpy()
x.add_(1)
print(x)
print(y)
Well, that seems to work. Maybe using the direct array indexing notation causes the numpy array to disconnect from the tensor. We should stick to using the tensor functions then.
import numpy as np
import torch
a = np.ones(5)
b = torch.from_numpy(a)
np.add(a, 1, out=a)
print(a)
print(b)
a[3] = 10
a
b
But the index change seems to work from the numpy to tensor side. Maybe this is a bug.
torch.cuda.is_available()
from pytest import raises
with raises(AssertionError):
b.cuda()
Great, we finished the first page in the first PyTorch tutorial.
20170726-1515 - 20170726-1615: 60 minutes
+ Update jupyter notebook so that setup.sh renders properly
_ File issue that using direct index on tensor does not change numpy array
+ Process completed tasks
To review, installing the various packages is extremely easy.
pip install \
http://download.pytorch.org/whl/cu80/torch-0.1.12.post2-cp35-cp35m-linux_x86_64.whl \
torchvision
pip install \
tensorflow \
keras
Today, let's work through the automatic differentiation tutorial of PyTorch. I think it might be useful to implement the system using both PyTorch and Keras, especially if it is not that difficult.
import torch
from torch.autograd import Variable
x = Variable(torch.ones(2, 2), requires_grad=True)
print(x)
x + 2
y = x + 2
y.creator
z = y * y * 3
out = z.mean()
print(z, out)
y
(y * y).creator
a = torch.ones(2, 2) * 2
b = torch.ones(2, 2) * 3
print(a, b, a * b)
This looks like element-wise multiplication rather than matrix multiplication.
out.creator
out.backward()
print(out)
y
z = y * y * 3
out = z.mean()
print(z, out)
print(out)
out.backward()
x
x.grad
x = Variable(torch.ones(2, 2), requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
print(out)
out.backward()
print(x.grad)
print(z.grad)
print(y.grad)
print(x.grad)
Working through the differentiation manually makes sense if we understand the notation that we are evaluating the gradient at $x_1 = 1$.
x = torch.randn(3)
x = Variable(x, requires_grad=True)
y = x * 2
while y.data.norm() < 1000:
y = y * 2
print(y)
gradients = torch.FloatTensor([0.1, 1.0, 0.0001])
y.backward(gradients)
print(x.grad)
It looks like backward is a function that populates grad in the originating variable.
+ Work through [Autograd: automatic differentiation](http://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html)from torch import Tensor
x = Variable(torch.ones(2, 2), requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
out.backward(Tensor([2])) # Evaluate the gradient at x_1 = 2
print(x.grad)
"""
from torch import Tensor, ones
from torch.autograd import Variable
x = Variable(ones(2, 2), requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
out.backward(Tensor([[2, 2], [2, 2]])) # Evaluate the gradient at x_1 = 2
print(x.grad)
""";
# Raises TypeError
# fill_ received an invalid combination of arguments - got (!torch.FloatTensor!), but expected (float value)
import torch
torch.randn(3)
torch.ones(2, 2)
I've been trying to understand why backward in the first example does not require specifying an argument, but backward in the second example does. So far, I've figured out that it is because the first example computes the backward on a scalar, while the second example computes the backward on a non-scalar.
Update links in README
Enable GPU support