Overview

Python-DNN uses 3 configuration files which are in json format.
  1. Model Configuration (model.conf)
  2. Data Configuration (data.conf)
  3. Network Configuration (nnet.conf)

Model Configuration

  • nnetType : (Mandatory) Type of Network (CNN/RBM/SDA/DNN)
  • train_data : (Mandatory) The working directory containing data configuration and output
  • wdir : (Mandatory) Working Directory.
  • data_spec : (Mandatory) The path of the data sepification relative to model_config.json

  • nnet_spec : (Mandatory) The path of network configuration specification relative to model_config.json

  • output_file : (Mandatory) The path of RBM network output file relative to wdir

  • input_file : The path of PreTrained/FineTuned network input file relative to wdir.(Mandatory for DNN)

  • random_seed : Random Seed used for initialization of weights.

  • logger_level : Level of Logger.Valid Values are “INFO”,”DEBUG” and “ERROR”

  • batch_size : specify the mini batch size while training, default 128
  • n_ins :Dimension of input (Mandatory for all except CNN)
  • n_outs :(Mandatory) Dimension of output (No: of Classes)

  • input_shape: The input shape of a given feature vector.(Mandatory For CNN).Should be an Array.

  • finetune_params : Configuration of finetune learning method.Contains a json object with following params:
  • momentum : The momentum factor while finetuning
  • method : Two methods are supported
  1. C: Constant learning rate(DEFAULT): run epoch_num iterations with learning_rate unchanged
  2. E: Exponential decay: we start with the learning rate of start_rate; if the validation error reduction between two epochs is less than min_derror_decay_start, the learning rate is scaled by scale_by during each of the remaining epoch. The whole traing terminates when the validation error reduction between two epochs falls below min_derror_stop. min_epoch_decay_star is the minimum epoch number after which scaling can only be performed.

Default value of other paramters

Param Default value Learning method
learning_rate 0.08 C
epoch_num 10 C
start_rate 0.08 E
scale_by 0.5 E
min_derror_decay_start 0.05 E
min_derror_stop 0.05 E
min_epoch_decay_start 15 E

These parameters are used by Constant learning rate or Exponential decay

  • pretrain_params: Configuration of pretraining method.Contains a json object with following params
Param Default value nnet Type Description
gbrbm_learning_rate 0.005 DBN Pretraining learning rate for gbrbm layer.
learning_rate 0.08 SDA,DBN Pretraining learning rate (DBN: for all layers except gbrbm layer)
epochs 15 DBN No of Pretraining epochs
initial_momentum 0.5 DBN The initial momentum factor while pre-training
final_momentum 0.9 DBN The final momentum factor while pre-training
initial_momentum_epoch 5 DBN No: of epochs with the initial momentum factor before switching to final momentum factor
keep_layer_num 0 SDA,DBN From which layer Pre-Trainig Should Start.If non-Zero layer is intilaized with weights from input_file
  • export_path : path (realative to wdir) for writting (bottleneck) features.
  • processes : Process should be run by program.Contains a json object with following params
  • pretraining : whether Pre-Training is needed.(invalid for DNN and CNN).(Default value = false)
  • finetuning : whether Fine Tuning is needed.(Default value = false)
  • testing : whether Fine Tuning is needed.(Default value = false)
  • export_data : whether extracted features should written to file.If true,export_path is required.(Default value = false).

Also See:

Data Configuration

Data Specification has 3 fields:

  1. training
  2. validation
  3. testing

Each one is a json object with following fields:

  • base_path :(Mandatory) Base path of data.
  • filename :(Mandatory) Filename,
  • partition :(Mandatory) Size of data which should be loaded to memory at a time (in MiB)
  • random : Whether to use random order (Default value = true)
  • random_seed : Seed for random numbers if random is true
  • keep_flatten : Whether to use data as flatten vector or reshape(Default Value = false)
  • reader_type : (Mandatory) Type of reader NP/T1/T2.
  • dim_shuffle : how to use reshape given fatten vector.Used only keep_flatten is false

Also See:

NNET Configuration

CNN Specification

It has 2 parts:

  1. cnn
  2. mlp

Each one contain a json object.cnn describes convolution layer configuration and mlp describes hidden layer configuration.

  • cnn contains a json object with following parameters:
  • layers: An Array of json objects.Each one decribes a convolution layer which contains:
  • convmat_dim : Dimension of Convolution Weight
  • num_filters : No. of Feature maps
  • poolsize : Dimension for Max-pooling
  • flatten : whether to flatten output or not(true for last layer else false)
  • update : true if weight need to updated during training.
  • activation : Activation function used by this layer, if not present global activation fuction is used.
  • activation : Activation function used by layers (global)
  • use_fast : if true program will use pylearn2 library for faster computation (Default Value = false)
  • mlp contains a json object with following parameters:
  • layers : An Array contain size of hidden layers.
  • adv_activation: if maxout/pnorm is used.
  • method : ‘maxout’,’pnorm’. In maxout, a pooling of neuron o/p is done based on poolsize. But in pnorm output is normalized after pooling.
  • pool_size: pool size
  • pnorm_order: order of normalization (in pnorm)
  • activation : Activation function used by layers. (if adv_activation is used, it sholud be either ‘linear’,’relu’ or ‘cappedrelu’)

Also See

DBN(RBM) Specification

  • hidden_layers : (Mandatory) An Array contain size of hidden RBM layer.
  • activation : Activation function used by layers.
  • pretrained_layers : Number of layers to be pre-trained.(Default Value = Size of hidden_layers)
  • first_layer_type : Type for the first layer.It should be either ‘bb’ (Bernoulli-Bernoulli) or ‘gb’ (Gaussian-Bernoulli).(Default Value = gb).

Also See

DNN Specification

  • hidden_layers : (Mandatory) An Array contain size of hidden RBM layer.

  • pretrained_layers : Number of layers to be pre-trained.(Default Value = Size of hidden_layers)

  • max_col_norm : The max value of norm of gradients; usually used in dropout and maxout.(Default Value = null)
  • l1_reg : l1 norm regularization weight.(Default Value = 0)

  • l2_reg : l2 norm regularization weight.(Default Value = 0)

  • adv_activation: if maxout/pnorm is used. It contains
  • method : Either ‘maxout’,’pnorm’. In maxout, a pooling of neuron o/p is done based on poolsize. But in pnorm output is normalized after pooling.
  • pool_size: The number of units in each max-pooling(or pnorm) group for maxout/pnorm (Default Value = 1)
  • pnorm_order: The norm order for pnorm.(Default Value = 1)

  • activation : Activation function used by layers. (if adv_activation is used, it sholud be either ‘linear’,’relu’ or ‘cappedrelu’)

  • do_dropout : whether to use dropout or not. (Default Value =false)
  • dropout_factor : the dropout factors for DNN layers.(One for each hidden layer)(Default Value =[0.0])
  • input_dropout_factor : The dropout factor for the input features.(Default Value =0.0)

Also See

SDA Specification

  • hidden_layers : (Mandatory) An Array contains size of hidden denoising autoencoder layers.
  • activation : Activation function used by layers
  • corruption_levels : (Mandatory) An Array contains corruption level for each layer.Size should be equal to size of hidden_layers

Also See

Activation Functions

Activation Function is a function used to transform the activation level of a unit (neuron) into an output signal. Typically, activation functions have a “squashing” effect.

Python-DNN currently support following Activation Functions:

  • sigmoid:

Sigmoid function with equation: f(x) = 1/(1 + e^(-x)).This is an S-shaped (sigmoid) curve, with output in the range (0,1).

  • tanh:

The Hyperbolic tangent function is a sigmoid curve, like the logistic function, except that output lies in the range (-1,+1).

  • relu:

The rectifier is an activation function defined as f(x) = max(0, x)

  • cappedrelu:

It is same as ReLU except we cap the units at 6.ie, f(x) = min(max(x,0),6)

Data File Formats

Python-dnn supports 3 file formats* for data:

  1. NP :Numpy Format.
  2. T1 :Text File With One level header structure.
  3. T2 :Text File With Two level header structure

Numpy Format

The dataset is stored as single file in binary format Data file has:

<json-header>
<structured numpy.array>
<structured numpy.array>
..
..

Fist Line is a json-header which Contains two parameters:

  • featdim : Dimension of input vector after flattening.
  • input_shape : Actual shape of input before flattening.

eg: {"featdim": 784, "input_shape": [28, 28, 1]}

Thereafter, each row contain a numpy structured Arrays (or Record Arrays) with vector as 'd' and label as 'l'.The “dtype” of this Record Arrays is given by:

dtype={'names': ['d','l'],'formats': [('>f2',header['featdim']),'>i2']}

Text File (One level header)

In this type,we use a file containing list of simple text file names,each corresponding to single class.It has Following Format:

<feat_dim> <num_classes>
<data_file1>
<data_file2>
..
..

Where <feat_dim> is Dimension of input vector and <num_classes> is No. of classes, <data_file1> is ‘simple text file’ of class 1,<data_file2> is ‘simple text file’ of class2 and so on.

The simple text file has the structure.

<feat_dim> <num_feat_vectors(optional)>
<feat_vector>
<feat_vector>
..
..

Where <feat_dim> is Dimension of input vector,<num_feat_vectors> is No. of Vectors in this file and <feat_vector> are features of a vector separated by spaces.Whole file contains only feature vector of single class.

Text File (Two level header)

In this type,we use a file containing list of filename with each with a list of simple text file names corresponding to a single class.It has Following Format:

<feat_dim> <num_classes>
<class_index_file1>
<class_index_file2>
..
..

Where <feat_dim> is Dimension of input vector and <num_classes> is No. of classes, <class_index_file1> is ‘class index file’ (File with list of files of a class) of class 1,<class_index_file1> is ‘class index file’ of class2 and so on.

Each <class_index_file> is a file with Following Format:

<td_data_file1>
<td_data_file2>
..
..

Each <td_data_file*> is name of a simple text file.The simple text file has the structure same as that of T1.

*READER TYPE in data_spec

Example

See: Sample Configuration Folder