# Satellite Image Recognition Research Log

 Pay Notebook Creator: Roy Hyunjin Han 10 Set Container: Numerical CPU with TINY Memory for 10 Minutes 0 Total 0

# Vision¶

Provide an automated and accurate way to locate where people live using satellite images.

# Mission¶

Work through tutorials for deep learning packages.

Roy Hyunjin Han

# Context¶

When we first started this project in 2007, the leading convolutional neural network framework was a Lisp package called Lush written by Yann LeCun and his graduate students. Subsequent versions of our remote sensing system used Alex Krizhevsky's cuda-convnet and cuda-convnet2. At the time we also considered using theano or caffe. Since then, there have been several new options, including TensorFlow, Keras and PyTorch. We would like to re-evaluate the deep learning landscape by trying each package's tutorial.

# Timeframe¶

20170720-0900 - 20170803-0900: 2 weeks estimated

20170720-0900 - ?: X weeks actual

# Objectives¶

• Identify which deep learning packages we should try.
1. Work through tutorials for each package.
2. Select a package.

# Log¶

20170720-0930 - 20170720-1000: 30 minutes

I remember during my last few weeks of volunteering at the CBLL, a doctoral student named Koray was working on a Lush replacement called Torch. At the 2016 NIPS conference, the Facebook announced the release of PyTorch. I think PyTorch is a wrapper around Torch. At any rate, it looks like they have been very good about their documentation and tutorials.

We are going to start with the Deep Learning 60 Minute Blitz on the PyTorch website.

In [1]:
import torch


But first, it looks like we are going to have to install it. I wonder if Torch/PyTorch works on CPUs as well as GPUs. We don't quite have full GPU support on our website yet. It looks like the latest Torch was written in C/C++. I can just imagine all the headaches those Torch developers had with void pointers haha. Too bad they didn't have Rust at the time.

So I am attempting to use the pip version of the installation.

pip install http://download.pytorch.org/whl/cu80/torch-0.1.12.post2-cp35-cp35m-linux_x86_64.whl
pip install torchvision



Whoa, it looks like the installation finished! There weren't any errors! Let's not get too excited and make sure we can import the package.

In [2]:
import torch


Okay, we can import the package. Let's go back to the tutorial.

In [3]:
torch.Tensor(5, 3)

Out[3]:
 1.5725e+22  4.5659e-41  2.6372e+36
3.0841e-41  0.0000e+00  0.0000e+00
0.0000e+00  0.0000e+00  0.0000e+00
0.0000e+00  0.0000e+00  0.0000e+00
0.0000e+00  0.0000e+00  0.0000e+00
[torch.FloatTensor of size 5x3]

It looks like it is filled with junk numbers of epsilon value.

In [4]:
x = torch.rand(5, 3)
x

Out[4]:
 0.3391  0.2732  0.6154
0.7070  0.0998  0.9780
0.3929  0.3662  0.9639
0.4660  0.9596  0.5121
0.6269  0.5718  0.6123
[torch.FloatTensor of size 5x3]
In [5]:
x.size()

Out[5]:
torch.Size([5, 3])
In [6]:
torch.rand(5, 3) + torch.rand(5, 3)

Out[6]:
 0.8743  1.3435  0.7877
1.1953  0.8484  1.8045
1.5132  0.9887  1.5274
1.3090  0.8338  0.7291
1.2235  1.0730  1.0407
[torch.FloatTensor of size 5x3]
In [7]:
torch.rand(5, 3) + torch.rand(3, 5)

Out[7]:
 0.9438  1.3795  0.7975
0.3456  1.3028  0.5725
0.7174  1.1847  0.3910
1.8609  0.3182  0.2913
0.7402  0.2958  0.2261
[torch.FloatTensor of size 5x3]
In [11]:
from pytest import raises

with raises(RuntimeError):
torch.rand(2, 2) + torch.rand(10, 10)

In [12]:
x = torch.rand(4, 2)
y = torch.rand(4, 2)

Out[12]:
 1.4064  1.1901
1.1512  1.3390
0.8880  1.7907
0.7088  1.3874
[torch.FloatTensor of size 4x2]

It looks like torch is following the convention of m rows by n columns.

In [14]:
z = torch.rand(4, 2)

x = torch.rand(4, 2)
y = torch.rand(4, 2)

z

Out[14]:
 0.8179  1.2970
0.9149  0.7656
1.7340  1.6352
1.5919  0.3606
[torch.FloatTensor of size 4x2]

I'm not sure what is the point of the out argument. Maybe it's just consistency with the C/C++ code.

In [15]:
y.add_(x)

Out[15]:
 0.8179  1.2970
0.9149  0.7656
1.7340  1.6352
1.5919  0.3606
[torch.FloatTensor of size 4x2]

Maybe the add_ and out are speed and memory optimizations so that the data doesn't have to get passed back and forth between the C code and Python, which can get expensive for huge datasets.

In [16]:
x

Out[16]:
 0.5383  0.5456
0.0927  0.6682
0.8381  0.8466
0.8108  0.0233
[torch.FloatTensor of size 4x2]
In [17]:
x[:1]

Out[17]:
 0.5383  0.5456
[torch.FloatTensor of size 1x2]
In [18]:
x[:, 1]

Out[18]:
 0.5456
0.6682
0.8466
0.0233
[torch.FloatTensor of size 4]
In [19]:
a = torch.ones(5)
a

Out[19]:
 1
1
1
1
1
[torch.FloatTensor of size 5]
In [20]:
b = a.numpy()
b

Out[20]:
array([ 1.,  1.,  1.,  1.,  1.], dtype=float32)
In [21]:
b[2] = 10
b

Out[21]:
array([  1.,   1.,  10.,   1.,   1.], dtype=float32)
In [22]:
a

Out[22]:
  1
1
10
1
1
[torch.FloatTensor of size 5]

It looks like changing the numpy array will not result in a change in the tensor. What about the other way around?

Trying the same procedure again a few days later, it seems as though they fixed it. We can change the numpy array and see the change reflected in the tensor and vice versa.

In [27]:
a[2] = 5
a

Out[27]:
 2
2
5
2
2
[torch.FloatTensor of size 5]
In [28]:
b

Out[28]:
array([ 2.,  2.,  5.,  2.,  2.], dtype=float32)
In [29]:
a.add_(1)

Out[29]:
 3
3
6
3
3
[torch.FloatTensor of size 5]
In [30]:
b

Out[30]:
array([ 3.,  3.,  6.,  3.,  3.], dtype=float32)

Hmm, the behavior seems to differ from the way it is promised in the documentation. Let's try this again.

In [31]:
x = torch.ones(5)
y = x.numpy()
print(x)
print(y)

 2
2
2
2
2
[torch.FloatTensor of size 5]

[ 2.  2.  2.  2.  2.]


Well, that seems to work. Maybe using the direct array indexing notation causes the numpy array to disconnect from the tensor. We should stick to using the tensor functions then.

In [32]:
import numpy as np
import torch
a = np.ones(5)
b = torch.from_numpy(a)
print(a)
print(b)

[ 2.  2.  2.  2.  2.]

2
2
2
2
2
[torch.DoubleTensor of size 5]


In [33]:
a[3] = 10
a

Out[33]:
array([  2.,   2.,   2.,  10.,   2.])
In [34]:
b

Out[34]:
  2
2
2
10
2
[torch.DoubleTensor of size 5]

But the index change seems to work from the numpy to tensor side. Maybe this is a bug.

In [35]:
torch.cuda.is_available()

Out[35]:
False
In [37]:
from pytest import raises

with raises(AssertionError):
b.cuda()


Great, we finished the first page in the first PyTorch tutorial.

20170726-1515 - 20170726-1615: 60 minutes

+ Update jupyter notebook so that setup.sh renders properly
_ File issue that using direct index on tensor does not change numpy array
+ Process completed tasks



To review, installing the various packages is extremely easy.

pip install \
torchvision
pip install \
tensorflow \
keras



Today, let's work through the automatic differentiation tutorial of PyTorch. I think it might be useful to implement the system using both PyTorch and Keras, especially if it is not that difficult.

In [38]:
import torch
from torch.autograd import Variable

In [39]:
x = Variable(torch.ones(2, 2), requires_grad=True)
print(x)

Variable containing:
1  1
1  1
[torch.FloatTensor of size 2x2]


In [40]:
x + 2

Out[40]:
Variable containing:
3  3
3  3
[torch.FloatTensor of size 2x2]
In [42]:
y = x + 2
y.creator

Out[42]:
<torch.autograd._functions.basic_ops.AddConstant at 0x7f470a67e128>
In [43]:
z = y * y * 3
out = z.mean()
print(z, out)

Variable containing:
27  27
27  27
[torch.FloatTensor of size 2x2]
Variable containing:
27
[torch.FloatTensor of size 1]


In [44]:
y

Out[44]:
Variable containing:
3  3
3  3
[torch.FloatTensor of size 2x2]
In [46]:
(y * y).creator

Out[46]:
<torch.autograd._functions.basic_ops.Mul at 0x7f470a67e828>
In [47]:
a = torch.ones(2, 2) * 2
b = torch.ones(2, 2) * 3
print(a, b, a * b)

 2  2
2  2
[torch.FloatTensor of size 2x2]

3  3
3  3
[torch.FloatTensor of size 2x2]

6  6
6  6
[torch.FloatTensor of size 2x2]



This looks like element-wise multiplication rather than matrix multiplication.

In [48]:
out.creator

Out[48]:
<torch.autograd._functions.reduce.Mean at 0x7f470a67e2e8>
In [50]:
out.backward()

In [51]:
print(out)

Variable containing:
27
[torch.FloatTensor of size 1]


In [52]:
y

Out[52]:
Variable containing:
3  3
3  3
[torch.FloatTensor of size 2x2]
In [54]:
z = y * y * 3
out = z.mean()
print(z, out)

Variable containing:
27  27
27  27
[torch.FloatTensor of size 2x2]
Variable containing:
27
[torch.FloatTensor of size 1]


In [55]:
print(out)

Variable containing:
27
[torch.FloatTensor of size 1]


In [56]:
out.backward()

In [57]:
x

Out[57]:
Variable containing:
1  1
1  1
[torch.FloatTensor of size 2x2]
In [58]:
x.grad

Out[58]:
Variable containing:
9  9
9  9
[torch.FloatTensor of size 2x2]
In [15]:
x = Variable(torch.ones(2, 2), requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
print(out)
out.backward()

Variable containing:
27
[torch.FloatTensor of size 1]

Variable containing:
4.5000  4.5000
4.5000  4.5000
[torch.FloatTensor of size 2x2]


In [66]:
print(z.grad)

None
None
Variable containing:
4.5000  4.5000
4.5000  4.5000
[torch.FloatTensor of size 2x2]



Working through the differentiation manually makes sense if we understand the notation that we are evaluating the gradient at $x_1 = 1$.

In [14]:
x = torch.randn(3)
x = Variable(x, requires_grad=True)

y = x * 2
while y.data.norm() < 1000:
y = y * 2

print(y)

gradients = torch.FloatTensor([0.1, 1.0, 0.0001])


Variable containing:
975.8725
576.8318
-971.4985
[torch.FloatTensor of size 3]

Variable containing:
51.2000
512.0000
0.0512
[torch.FloatTensor of size 3]



It looks like backward is a function that populates grad in the originating variable.

+ Work through [Autograd: automatic differentiation](http://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html)
In [68]:
from torch import Tensor

In [74]:
x = Variable(torch.ones(2, 2), requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
out.backward(Tensor([2]))  # Evaluate the gradient at x_1 = 2

Variable containing:
9  9
9  9
[torch.FloatTensor of size 2x2]


In [8]:
"""
from torch import Tensor, ones
from torch.autograd import Variable
x = Variable(ones(2, 2), requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
out.backward(Tensor([[2, 2], [2, 2]]))  # Evaluate the gradient at x_1 = 2
""";
# Raises TypeError
# fill_ received an invalid combination of arguments - got (!torch.FloatTensor!), but expected (float value)

In [11]:
import torch
torch.randn(3)

Out[11]:
-0.3651
-3.3019
-0.8879
[torch.FloatTensor of size 3]
In [12]:
torch.ones(2, 2)

Out[12]:
 1  1
1  1
[torch.FloatTensor of size 2x2]

I've been trying to understand why backward in the first example does not require specifying an argument, but backward in the second example does. So far, I've figured out that it is because the first example computes the backward on a scalar, while the second example computes the backward on a non-scalar.

Update links in README
Enable GPU support