Torch Tips

=================================

 python -m SimpleHTTPServer

 

https://your-ip:8888 — this will work like network

 

 

========================================================
–list of screen
screen ls

–resume the screen
screen -r 18497.new_vision

–detach screen
cntl+A D

=============================================================

how to convert a table to tensor in torch

torch.Tensor(table)

The argument is assumed to be a Lua array of numbers. The constructor returns a new Tensor of the size of the table, containing all the table elements. The table might be multi-dimensional.

Example:

> torch.Tensor({{1,2,3,4}, {5,6,7,8}})
 1  2  3  4
 5  6  7  8
[torch.DoubleTensor of dimension 2x4]

p={0.3148,
0.3574,
0.3829,
0.3967,
0.4062,
0.4180,
0.4208,
0.4267,
0.4312,
0.4329,

}
–torch.Tensor(table)
–p is table and x is tensor. all the operation in torch are in tensor, not on table.
x=torch.Tensor(p)
print(x)
print(‘p’,p)
y=torch.Tensor(q)
q={0.2603,
0.3541,
0.3874,
0.4088,
0.4232,
0.4330,
0.4404,
0.4479,
0.4549,
0.4608,
0.4631,
0.4693,
0.4740,
0.4822,
}
print(y)
x=torch.mul(x,10000) — x should be tensor, if x is table then i will give error
y=torch.mul(y,10000)
print(y)
print(x)

————————-

[res] torch.mul([res,] tensor1, value)

Multiply all elements in the Tensor by the given value.

z = torch.mul(x, 2) will return a new Tensor with the result of x * 2.

torch.mul(z, x, 2) will put the result of x * 2 in z.

x:mul(2) will multiply all elements of x with 2 in-place.

z:mul(x, 2) will put the result of x * 2 in z.

 

ref

https://github.com/torch/torch7/blob/master/doc/maths.md

https://github.com/torch/torch7/blob/master/doc/tensor.md

 


for math operation

https://github.com/torch/torch7/blob/master/doc/maths.md

————————————————–

for adding table

https://github.com/torch/nn/blob/master/doc/table.md#nn.CAddTable

—————————————————–

to plot graph

https://github.com/torch/optim/blob/master/doc/logger.md

–[[ Logger: a simple class to log symbols during training,
and automate plot generation
Example:
logger = optim.Logger(‘somefile.log’) — file to save stuff
for i = 1,N do — log some symbols during
train_error = … — training/testing
test_error = …
logger:add{[‘training error’] = train_error,
[‘test error’] = test_error}
end
logger:style{[‘training error’] = ‘-‘, — define styles for plots
[‘test error’] = ‘-‘}
logger:plot() — and plot
—- OR —
logger = optim.Logger(‘somefile.log’) — file to save stuff
logger:setNames{‘training error’, ‘test error’}
for i = 1,N do — log some symbols during
train_error = … — training/testing
test_error = …
logger:add{train_error, test_error}
end
logger:style{‘-‘, ‘-‘} — define styles for plots
logger:plot() — and plot
———–
logger:setlogscale(true) — enable logscale on Y-axis
logger:plot() — and plot
]]

 

-------------------------------------

JoinTable

module = JoinTable(dimension, nInputDims)

Creates a module that takes a table of Tensors as input and outputs a Tensor by joining them together along dimensiondimension. In the diagram below dimension is set to 1.

+----------+             +-----------+
| {input1, +-------------> output[1] |
|          |           +-----------+-+
|  input2, +-----------> output[2] |
|          |         +-----------+-+
|  input3} +---------> output[3] |
+----------+         +-----------+

The optional parameter nInputDims allows to specify the number of dimensions that this module will receive. This makes it possible to forward both minibatch and non-minibatch Tensors through the same module.

Example 1

x = torch.randn(5, 1)
y = torch.randn(5, 1)
z = torch.randn(2, 1)

print(nn.JoinTable(1):forward{x, y})
print(nn.JoinTable(2):forward{x, y})
print(nn.JoinTable(1):forward{x, z})

gives the output:

 1.3965
 0.5146
-1.5244
-0.9540
 0.4256
 0.1575
 0.4491
 0.6580
 0.1784
-1.7362
[torch.DoubleTensor of dimension 10x1]

 1.3965  0.1575
 0.5146  0.4491
-1.5244  0.6580
-0.9540  0.1784
 0.4256 -1.7362
[torch.DoubleTensor of dimension 5x2]

 1.3965
 0.5146
-1.5244
-0.9540
 0.4256
-1.2660
 1.0869
[torch.Tensor of dimension 7x1]

Example 2

module = nn.JoinTable(2, 2)

x = torch.randn(3, 1)
y = torch.randn(3, 1)

mx = torch.randn(2, 3, 1)
my = torch.randn(2, 3, 1)

print(module:forward{x, y})
print(module:forward{mx, my})

gives the output:

 0.4288  1.2002
-1.4084 -0.7960
-0.2091  0.1852
[torch.DoubleTensor of dimension 3x2]

(1,.,.) =
  0.5561  0.1228
 -0.6792  0.1153
  0.0687  0.2955

(2,.,.) =
  2.5787  1.8185
 -0.9860  0.6756
  0.1989 -0.4327
[torch.DoubleTensor of dimension 2x3x2]

A more complicated example

mlp = nn.Sequential()         -- Create a network that takes a Tensor as input
c = nn.ConcatTable()          -- The same Tensor goes through two different Linear
c:add(nn.Linear(10, 3))       -- Layers in Parallel
c:add(nn.Linear(10, 7))
mlp:add(c)                    -- Outputing a table with 2 elements
p = nn.ParallelTable()        -- These tables go through two more linear layers
p:add(nn.Linear(3, 2))        -- separately.
p:add(nn.Linear(7, 1))
mlp:add(p)
mlp:add(nn.JoinTable(1))      -- Finally, the tables are joined together and output.

pred = mlp:forward(torch.randn(10))
print(pred)

for i = 1, 100 do             -- A few steps of training such a network..
   x = torch.ones(10)
   y = torch.Tensor(3); y:copy(x:narrow(1, 1, 3))
   pred = mlp:forward(x)

   criterion= nn.MSECriterion()
   local err = criterion:forward(pred, y)
   local gradCriterion = criterion:backward(pred, y)
   mlp:zeroGradParameters()
   mlp:backward(x, gradCriterion)
   mlp:updateParameters(0.05)

   print(err)
end

Ref:

https://github.com/torch/nn/blob/master/doc/table.md#nn.JoinTable

Concat

module = nn.Concat(dim)

Concat concatenates the output of one layer of “parallel” modules along the provided dimension dim: they take the same inputs, and their output is concatenated.

mlp=nn.Concat(1);
mlp:add(nn.Linear(5,3))
mlp:add(nn.Linear(5,7))
print(mlp:forward(torch.randn(5)))

which gives the output:

 0.7486
 0.1349
 0.7924
-0.0371
-0.4794
 0.3044
-0.0835
-0.7928
 0.7856
-0.1815
[torch.Tensor of dimension 10]

[res] torch.cat( [res,] {x_1, x_2, …}, [dimension] )

x = torch.cat(x_1, x_2, [dimension]) returns a Tensor x which is the concatenation of Tensors x_1 and x_2along dimension dimension.

If dimension is not specified it is the last dimension.

The other dimensions of x_1 and x_2 have to be equal.

Also supports arrays with arbitrary numbers of Tensors as inputs.

Examples:

> torch.cat(torch.ones(3), torch.zeros(2))
 1
 1
 1
 0
 0
[torch.DoubleTensor of size 5]

> torch.cat(torch.ones(3, 2), torch.zeros(2, 2), 1)
 1  1
 1  1
 1  1
 0  0
 0  0
[torch.DoubleTensor of size 5x2]

> torch.cat(torch.ones(2, 2), torch.zeros(2, 2), 1)
 1  1
 1  1
 0  0
 0  0
[torch.DoubleTensor of size 4x2]

> torch.cat(torch.ones(2, 2), torch.zeros(2, 2), 2)
 1  1  0  0
 1  1  0  0
[torch.DoubleTensor of size 2x4]

> torch.cat(torch.cat(torch.ones(2, 2), torch.zeros(2, 2), 1), torch.rand(3, 2), 1)
 1.0000  1.0000
 1.0000  1.0000
 0.0000  0.0000
 0.0000  0.0000
 0.3227  0.0493
 0.9161  0.1086
 0.2206  0.7449
[torch.DoubleTensor of size 7x2]

> torch.cat({torch.ones(2, 2), torch.zeros(2, 2), torch.rand(3, 2)}, 1)
 1.0000  1.0000
 1.0000  1.0000
 0.0000  0.0000
 0.0000  0.0000
 0.3227  0.0493
 0.9161  0.1086
 0.2206  0.7449
[torch.DoubleTensor of size 7x2]

ConcatTable

module = nn.ConcatTable()

ConcatTable is a container module that applies each member module to the same input Tensor or table.

                  +-----------+
             +----> {member1, |
+-------+    |    |           |
| input +----+---->  member2, |
+-------+    |    |           |
   or        +---->  member3} |
 {input}          +-----------+

Example 1

mlp = nn.ConcatTable()
mlp:add(nn.Linear(5, 2))
mlp:add(nn.Linear(5, 3))

pred = mlp:forward(torch.randn(5))
for i, k in ipairs(pred) do print(i, k) end

which gives the output:

1
-0.4073
 0.0110
[torch.Tensor of dimension 2]

2
 0.0027
-0.0598
-0.1189
[torch.Tensor of dimension 3]

Example 2

mlp = nn.ConcatTable()
mlp:add(nn.Identity())
mlp:add(nn.Identity())

pred = mlp:forward{torch.randn(2), {torch.randn(3)}}
print(pred)

which gives the output (using th):

{
  1 :
    {
      1 : DoubleTensor - size: 2
      2 :
        {
          1 : DoubleTensor - size: 3
        }
    }
  2 :
    {
      1 : DoubleTensor - size: 2
      2 :
        {
          1 : DoubleTensor - size: 3
        }
    }
}

ref:

https://github.com/torch/nn/blob/master/doc/containers.md#nn.Concat

https://github.com/torch/nn/blob/master/doc/table.md#nn.ConcatTable

————————————

Math Library Tutorial

lua-users home
wiki

The math library is documented in section 6.7 of the Reference Manual.[1] Below is a summary of the functions and variables provided. Each is described, with an example, on this page.

math.abs
math.acos
math.asin
math.atan
math.ceil
math.cos
math.deg
math.exp
math.floor
math.fmod
math.huge
math.log
math.max
math.maxinteger
math.min
math.mininteger
math.modf
math.pi
math.rad
math.random
math.randomseed
math.sin
math.sqrt
math.tan
math.tointeger
math.type
math.ult

math.abs

Return the absolute, or non-negative value, of a given value.

> = math.abs(-100)
100
> = math.abs(25.67)
25.67
> = math.abs(0)
0

math.acos , math.asin

Return the inverse cosine and sine in radians of the given value.

> = math.acos(1)
0
> = math.acos(0)
1.5707963267949
> = math.asin(0)
0
> = math.asin(1)
1.5707963267949

Ref:

http://lua-users.org/wiki/MathLibraryTutorial

————————————

Neural Network Package

This package provides an easy and modular way to build and train simple or complex neural networks using Torch:

——————————

Ref:

https://github.com/torch/nn

———————————–

Convolutional layers

A convolution is an integral that expresses the amount of overlap of one function g as it is shifted over another function f. It therefore “blends” one function with another. The neural network package supports convolution, pooling, subsampling and other relevant facilities. These are divided based on the dimensionality of the input and output Tensors:

——————————————————————-

REF:

https://github.com/torch/nn/blob/master/doc/convolution.md#nn.convlayers.dok

https://github.com/torch/nn/blob/master/doc/training.md#nn.traningneuralnet.dok

————————————

Simple layers

Simple Modules are used for various tasks like adapting Tensor methods and providing affine transformations :

  • Parameterized Modules :
    • Linear : a linear transformation ;
    • SparseLinear : a linear transformation with sparse inputs ;
    • Bilinear : a bilinear transformation with sparse inputs ;
    • PartialLinear : a linear transformation with sparse inputs with the option of only computing a subset ;
    • Add : adds a bias term to the incoming data ;
    • Mul : multiply a single scalar factor to the incoming data ;
    • CMul : a component-wise multiplication to the incoming data ;
    • Euclidean : the euclidean distance of the input to k mean centers ;
    • WeightedEuclidean : similar to Euclidean, but additionally learns a diagonal covariance matrix ;
    • Cosine : the cosine similarity of the input to k mean centers ;
  • Modules that adapt basic Tensor methods :
  • Modules that adapt mathematical Tensor methods :
    • AddConstant : adding a constant ;
    • MulConstant : multiplying a constant ;
    • Max : a max operation over a given dimension ;
    • Min : a min operation over a given dimension ;
    • Mean : a mean operation over a given dimension ;
    • Sum : a sum operation over a given dimension ;
    • Exp : an element-wise exp operation ;
    • Log : an element-wise log operation ;
    • Abs : an element-wise abs operation ;
    • Power : an element-wise pow operation ;
    • Square : an element-wise square operation ;
    • Sqrt : an element-wise sqrt operation ;
    • Clamp : an element-wise clamp operation ;
    • Normalize : normalizes the input to have unit L_p norm ;
    • MM : matrix-matrix multiplication (also supports batches of matrices) ;
  • Miscellaneous Modules :
    • BatchNormalization : mean/std normalization over the mini-batch inputs (with an optional affine transform) ;
    • Identity : forward input as-is to output (useful with ParallelTable) ;
    • Dropout : masks parts of the input using binary samples from a bernoulli distribution ;
    • SpatialDropout : same as Dropout but for spatial inputs where adjacent pixels are strongly correlated ;
    • VolumetricDropout : same as Dropout but for volumetric inputs where adjacent voxels are strongly correlated ;
    • Padding : adds padding to a dimension ;
    • L1Penalty : adds an L1 penalty to an input (for sparsity) ;
    • GradientReversal : reverses the gradient (to maximize an objective function) ;
    • GPU : decorates a module so that it can be executed on a specific GPU device.

 

Table Layers

This set of modules allows the manipulation of tables through the layers of a neural network. This allows one to build very rich architectures:

  • table Container Modules encapsulate sub-Modules:
    • ConcatTable: applies each member module to the same input Tensor and outputs a table;
    • ParallelTable: applies the i-th member module to the i-th input and outputs a table;
  • Table Conversion Modules convert between tables and Tensors or tables:
  • Pair Modules compute a measure like distance or similarity from a pair (table) of input Tensors:
  • CMath Modules perform element-wise operations on a table of Tensors:
  • Table of Criteria:

 

CmdLine

This class provides a parameter parsing framework which is very useful when one needs to run several experiments that rely on different parameter settings that are passed in the command line. This class will also override the default print function to direct all the output to a log file as well as screen at the same time.

A sample lua file is given below that makes use of CmdLine class.

cmd = torch.CmdLine()
cmd:text()
cmd:text()
cmd:text('Training a simple network')
cmd:text()
cmd:text('Options')
cmd:option('-seed',123,'initial random seed')
cmd:option('-booloption',false,'boolean option')
cmd:option('-stroption','mystring','string option')
cmd:text()

-- parse input params
params = cmd:parse(arg)

params.rundir = cmd:string('experiment', params, {dir=true})
paths.mkdir(params.rundir)

-- create log file

Torch Packages

  • Tensor Library
    • Tensor defines the all powerful tensor object that provides multi-dimensional numerical arrays with type templating.
    • Mathematical operations that are defined for the tensor object types.
    • Storage defines a simple storage interface that controls the underlying storage for any tensor object.
  • File I/O Interface Library
  • Useful Utilities
    • Timer provides functionality for measuring time.
    • Tester is a generic tester framework.
    • CmdLine is a command line argument parsing utility.
    • Random defines a random number generator package with various distributions.
    • Finally useful utility functions are provided for easy handling of torch tensor types and class inheritance.
    • Ref:

REF

https://github.com/torch/nn/blob/master/doc/simple.md#nn.Select

https://github.com/torch/nn/blob/master/doc/table.md

Blog : Deeplearning In torch

http://rnduja.github.io/2015/10/07/deep_learning_with_torch_step_4_nngraph/

https://github.com/torch/torch7/blob/master/doc/cmdline.md

https://github.com/torch/torch7

 

Deep Learning with Torch

REF

http://learning.cs.toronto.edu/wp-content/uploads/2015/02/torch_tutorial.pdf

http://hunch.net/~nyoml/torch7.pdf

 

http://atamahjoubfar.github.io/Torch_for_Matlab_users.pdf

http://ml.informatik.uni-freiburg.de/_media/teaching/ws1415/presentation_dl_lect3.pdf

A Torch autoencoder example

Extracting features from MNIST digits

http://rnduja.github.io/2015/11/06/torch-autoencoder/

http://rnduja.github.io/2015/10/13/torch-mnist/

Introduction to nngraph

https://github.com/torch/nngraph

http://kbullaughey.github.io/lstm-play/2015/09/18/introduction-to-nngraph.html

 

LSTM and Fast LSTM

http://www.humphreysheil.com/blog/getting-to-grips-with-lstm-part-one

http://christopher5106.github.io/deep/learning/2016/07/14/element-research-torch-rnn-tutorial.html

 

Torch Slide

http://hunch.net/~nyoml/torch7.pdf

https://moodle.technion.ac.il/mod/forum/discuss.php?d=293691&lang=en

https://github.com/torch/torch7/wiki/Cheatsheet

 


Module

Module is an abstract class which defines fundamental methods necessary for a training a neural network. Modules areserializable.

Modules contain two states variables: output and gradInput.

[output] forward(input)

Takes an input object, and computes the corresponding output of the module. In general input and output areTensors. However, some special sub-classes like table layers might expect something else. Please, refer to each module specification for further information.

After a forward(), the output state variable should have been updated to the new value.

It is not advised to override this function. Instead, one should implement updateOutput(input) function. The forward module in the abstract parent class Module will call updateOutput(input).

[gradInput] backward(input, gradOutput)

Performs a backpropagation step through the module, with respect to the given input. In general this method makes the assumption forward(input) has been called before, with the same input. This is necessary for optimization reasons. If you do not respect this rule, backward() will compute incorrect gradients.

In general input and gradOutput and gradInput are Tensors. However, some special sub-classes like table layers might expect something else. Please, refer to each module specification for further information.

A backpropagation step consist in computing two kind of gradients at input given gradOutput (gradients with respect to the output of the module). This function simply performs this task using two function calls:

It is not advised to override this function call in custom classes. It is better to override updateGradInput(input, gradOutput)and accGradParameters(input, gradOutput,scale) functions.

updateOutput(input)

Computes the output using the current parameter set of the class and input. This function returns the result which is stored in the output field.

updateGradInput(input, gradOutput)

Computing the gradient of the module with respect to its own input. This is returned in gradInput. Also, the gradInput state variable is updated accordingly.

accGradParameters(input, gradOutput, scale)

Computing the gradient of the module with respect to its own parameters. Many modules do not perform this step as they do not have any parameters. The state variable name for the parameters is module dependent. The module is expected toaccumulate the gradients with respect to the parameters in some variable.

scale is a scale factor that is multiplied with the gradParameters before being accumulated.

Zeroing this accumulation is achieved with zeroGradParameters() and updating the parameters according to this accumulation is done with updateParameters().

zeroGradParameters()

If the module has parameters, this will zero the accumulation of the gradients with respect to these parameters, accumulated through accGradParameters(input, gradOutput,scale) calls. Otherwise, it does nothing.

updateParameters(learningRate)

If the module has parameters, this will update these parameters, according to the accumulation of the gradients with respect to these parameters, accumulated through backward() calls.

The update is basically:

parameters = parameters - learningRate * gradients_wrt_parameters

If the module does not have parameters, it does nothing.

accUpdateGradParameters(input, gradOutput, learningRate)

This is a convenience module that performs two functions at once. Calculates and accumulates the gradients with respect to the weights after multiplying with negative of the learning rate learningRate. Performing these two operations at once is more performance efficient and it might be advantageous in certain situations.

Keep in mind that, this function uses a simple trick to achieve its goal and it might not be valid for a custom module.

Also note that compared to accGradParameters(), the gradients are not retained for future use.

function Module:accUpdateGradParameters(input, gradOutput, lr)
   local gradWeight = self.gradWeight
   local gradBias = self.gradBias
   self.gradWeight = self.weight
   self.gradBias = self.bias
   self:accGradParameters(input, gradOutput, -lr)
   self.gradWeight = gradWeight
   self.gradBias = gradBias
end

As it can be seen, the gradients are accumulated directly into weights. This assumption may not be true for a module that computes a nonlinear operation.

share(mlp,s1,s2,…,sn)

This function modifies the parameters of the module named s1,..sn (if they exist) so that they are shared with (pointers to) the parameters with the same names in the given module mlp.

The parameters have to be Tensors. This function is typically used if you want to have modules that share the same weights or biases.

Note that this function if called on a Container module will share the same parameters for all the contained modules as well.

Example:

-- make an mlp
mlp1=nn.Sequential();
mlp1:add(nn.Linear(100,10));

-- make a second mlp
mlp2=nn.Sequential();
mlp2:add(nn.Linear(100,10));

-- the second mlp shares the bias of the first
mlp2:share(mlp1,'bias');

-- we change the bias of the first
mlp1:get(1).bias[1]=99;

-- and see that the second one's bias has also changed..
print(mlp2:get(1).bias[1])

clone(mlp,…)

Creates a deep copy of (i.e. not just a pointer to) the module, including the current state of its parameters (e.g. weight, biases etc., if any).

If arguments are provided to the clone(...) function it also calls share(…) with those arguments on the cloned module after creating it, hence making a deep copy of this module with some shared parameters.

Example:

-- make an mlp
mlp1=nn.Sequential();
mlp1:add(nn.Linear(100,10));

-- make a copy that shares the weights and biases
mlp2=mlp1:clone('weight','bias');

-- we change the bias of the first mlp
mlp1:get(1).bias[1]=99;

-- and see that the second one's bias has also changed..
print(mlp2:get(1).bias[1])

type(type[, tensorCache])

This function converts all the parameters of a module to the given type. The type can be one of the types defined fortorch.Tensor.

If tensors (or their storages) are shared between multiple modules in a network, this sharing will be preserved after type is called.

To preserve sharing between multiple modules and/or tensors, use nn.utils.recursiveType:

-- make an mlp
mlp1=nn.Sequential();
mlp1:add(nn.Linear(100,10));

-- make a second mlp
mlp2=nn.Sequential();
mlp2:add(nn.Linear(100,10));

-- the second mlp shares the bias of the first
mlp2:share(mlp1,'bias');

-- mlp1 and mlp2 will be converted to float, and will share bias
-- note: tensors can be provided as inputs as well as modules
nn.utils.recursiveType({mlp1, mlp2}, 'torch.FloatTensor')

float([tensorCache])

Convenience method for calling module:type(‘torch.FloatTensor'[, tensorCache])

double([tensorCache])

Convenience method for calling module:type(‘torch.DoubleTensor'[, tensorCache])

cuda([tensorCache])

Convenience method for calling module:type(‘torch.CudaTensor'[, tensorCache])

State Variables

These state variables are useful objects if one wants to check the guts of a Module. The object pointer is never supposed to change. However, its contents (including its size if it is a Tensor) are supposed to change.

In general state variables are Tensors. However, some special sub-classes like table layers contain something else. Please, refer to each module specification for further information.

output

This contains the output of the module, computed with the last call of forward(input).

gradInput

This contains the gradients with respect to the inputs of the module, computed with the last call of updateGradInput(input, gradOutput).

Parameters and gradients w.r.t parameters

Some modules contain parameters (the ones that we actually want to train!). The name of these parameters, and gradients w.r.t these parameters are module dependent.

[{weights}, {gradWeights}] parameters()

This function should returns two tables. One for the learnable parameters {weights} and another for the gradients of the energy wrt to the learnable parameters {gradWeights}.

Custom modules should override this function if they use learnable parameters that are stored in tensors.

[flatParameters, flatGradParameters] getParameters()

This function returns two tensors. One for the flattened learnable parameters flatParameters and another for the gradients of the energy wrt to the learnable parameters flatGradParameters.

Custom modules should not override this function. They should instead override parameters(…) which is, in turn, called by the present function.

This function will go over all the weights and gradWeights and make them view into a single tensor (one for weights and one for gradWeights). Since the storage of every weight and gradWeight is changed, this function should be called only once on a given network.

training()

This sets the mode of the Module (or sub-modules) to train=true. This is useful for modules like Dropout orBatchNormalization that have a different behaviour during training vs evaluation.

evaluate()

This sets the mode of the Module (or sub-modules) to train=false. This is useful for modules like Dropout orBatchNormalization that have a different behaviour during training vs evaluation.

findModules(typename)

Find all instances of modules in the network of a certain typename. It returns a flattened list of the matching nodes, as well as a flattened list of the container modules for each matching node.

Modules that do not have a parent container (ie, a top level nn.Sequential for instance) will return their self as the container.

This function is very helpful for navigating complicated nested networks. For example, a didactic example might be; if you wanted to print the output size of all nn.SpatialConvolution instances:

-- Construct a multi-resolution convolution network (with 2 resolutions):
model = nn.ParallelTable()
conv_bank1 = nn.Sequential()
conv_bank1:add(nn.SpatialConvolution(3,16,5,5))
conv_bank1:add(nn.Threshold())
model:add(conv_bank1)
conv_bank2 = nn.Sequential()
conv_bank2:add(nn.SpatialConvolution(3,16,5,5))
conv_bank2:add(nn.Threshold())
model:add(conv_bank2)
-- FPROP a multi-resolution sample
input = {torch.rand(3,128,128), torch.rand(3,64,64)}
model:forward(input)
-- Print the size of the Threshold outputs
conv_nodes = model:findModules('nn.SpatialConvolution')
for i = 1, #conv_nodes do
  print(conv_nodes[i].output:size())
end

Another use might be to replace all nodes of a certain typename with another. For instance, if we wanted to replace allnn.Threshold with nn.Tanh in the model above:

threshold_nodes, container_nodes = model:findModules('nn.Threshold')
for i = 1, #threshold_nodes do
  -- Search the container for the current threshold node
  for j = 1, #(container_nodes[i].modules) do
    if container_nodes[i].modules[j] == threshold_nodes[i] then
      -- Replace with a new instance
      container_nodes[i].modules[j] = nn.Tanh()
    end
  end
end

listModules()

List all Modules instances in a network. Returns a flattened list of modules, including container modules (which will be listed first), self, and any other component modules.

For example :

mlp = nn.Sequential()
mlp:add(nn.Linear(10,20))
mlp:add(nn.Tanh())
mlp2 = nn.Parallel()
mlp2:add(mlp)
mlp2:add(nn.ReLU())
for i,module in ipairs(mlp2:listModules()) do
   print(module)
end

Which will result in the following output :

nn.Parallel {
  input
    |`-> (1): nn.Sequential {
    |      [input -> (1) -> (2) -> output]
    |      (1): nn.Linear(10 -> 20)
    |      (2): nn.Tanh
    |    }
    |`-> (2): nn.ReLU
     ... -> output
}
nn.Sequential {
  [input -> (1) -> (2) -> output]
  (1): nn.Linear(10 -> 20)
  (2): nn.Tanh
}
nn.Linear(10 -> 20)
nn.Tanh
nn.ReLU

clearState()

Clears intermediate module states as output, gradInput and others. Useful when serializing networks and running low on memory. Internally calls set() on tensors so it does not break buffer sharing.

apply(function)

Calls provided function on itself and all child modules. This function takes module to operate on as a first argument:

model:apply(function(module)
   module.train = true
end)

In the example above train will be set to to true in all modules of model. This is how training() and evaluate()functions implemented.

replace(function)

Similar to apply takes a function which applied to all modules of a model, but uses return value to replace the module. Can be used to replace all modules of one type to another or remove certain modules.

For example, can be used to remove nn.Dropout layers by replacing them with nn.Identity:

model:replace(function(module)
   if torch.typename(module) == 'nn.Dropout' then
      return nn.Identity()
   else
      return module
   end
end)






_______________________________________________________________________________

narrow, select and copy two type with example

https://github.com/torch/torch7/blob/master/doc/tensor.md#self-narrowdim-index-size
https://github.com/torch/nn/blob/master/doc/simple.md#nn.Cosine


Vision and language


http://visionandlanguage.net



http://handong1587.github.io/deep_learning/2015/10/09/nlp.html/
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s