Models

Seq2seq

class models.seq2seq.Seq2seq(encoder, decoder, function=<function log_softmax>)

Sequence to Sequence Model

Parameters:

• encoder (torch.nn.Module) – encoder of seq2seq
• decoder (torch.nn.Module) – decoder of seq2seq
• function (torch.nn.functional) – A function used to generate symbols from RNN hidden state

Inputs: inputs, targets, teacher_forcing_ratio, use_beam_search

• inputs (torch.Tensor): tensor of sequences, whose length is the batch size and within which each sequence is a list of token IDs. This information is forwarded to the encoder.
• targets (torch.Tensor): tensor of sequences, whose length is the batch size and within which each sequence is a list of token IDs. This information is forwarded to the decoder.
• teacher_forcing_ratio (float): The probability that teacher forcing will be used. A random number is drawn uniformly from 0-1 for every decoding token, and if the sample is smaller than the given value, teacher forcing would be used (default is 0.90)
• use_beam_search (bool): flag indication whether to use beam-search or not (default: false)

Returns: y_hats, logits

• y_hats (batch, seq_len): predicted y values (y_hat) by the model
• logits (batch, seq_len, vocab_size): logit values by the model

Examples::

>>> encoder = EncoderRNN(input_size, ...)
>>> decoder = DecoderRNN(class_num, ...)
>>> model = Seq2seq(encoder, decoder)
>>> y_hats, logits = model()

forward(inputs, targets, teacher_forcing_ratio=0.9, use_beam_search=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

EncoderRNN

class models.encoderRNN.EncoderRNN(in_features, hidden_size, dropout_p=0.5, n_layers=5, bidirectional=True, rnn_cell='gru')

Converts low level features into higher level features

Parameters:

• in_features (int) – size of input
• hidden_size (int) – the number of features in the hidden state h
• n_layers (int, optional) – number of recurrent layers (default: 1)
• bidirectional (bool, optional) – if True, becomes a bidirectional encoder (defulat: False)
• rnn_cell (str, optional) – type of RNN cell (default: gru)
• dropout_p (float, optional) – dropout probability for the output sequence (default: 0)

Inputs: inputs

• inputs: list of sequences, whose length is the batch size and within which each sequence is a list of token IDs.

Returns: output, hidden

• output (batch, seq_len, hidden_size): tensor containing the encoded features of the input sequence
• hidden (num_layers * num_directions, batch, hidden_size): tensor containing the features in the hidden state h

Examples:

>>> listener = Listener(in_features, hidden_size, dropout_p=0.5, n_layers=5)
>>> output, hidden = listener(inputs)

forward(inputs)

Applies a multi-layer RNN to an input sequence

DecoderRNN

class models.decoderRNN.DecoderRNN(class_num, max_len, hidden_size, sos_id, eos_id, n_layers=1, rnn_cell='gru', dropout_p=0.5, use_attention=True, device=None, use_beam_search=False, k=8)

Converts higher level features (from encoder) into output sequence.

Parameters:

• class_num (int) – the number of class
• max_len (int) – a maximum allowed length for the sequence to be processed
• hidden_size (int) – the number of features in the hidden state h
• sos_id (int) – index of the start of sentence symbol
• eos_id (int) – index of the end of sentence symbol
• layer_size (int, optional) – number of recurrent layers (default: 1)
• rnn_cell (str, optional) – type of RNN cell (default: gru)
• dropout_p (float, optional) – dropout probability for the output sequence (default: 0)
• use_attention (bool, optional) – flag indication whether to use attention mechanism or not (default: false)
• k (int) – size of beam

Inputs: inputs, encoder_outputs, function, teacher_forcing_ratio

• inputs (batch, seq_len, input_size): list of sequences, whose length is the batch size and within which each sequence is a list of token IDs. It is used for teacher forcing when provided. (default None)
• encoder_outputs (batch, seq_len, hidden_size): tensor with containing the outputs of the listener. Used for attention mechanism (default is None).
• function (torch.nn.Module): A function used to generate symbols from RNN hidden state (default is torch.nn.functional.log_softmax).
• teacher_forcing_ratio (float): The probability that teacher forcing will be used. A random number is drawn uniformly from 0-1 for every decoding token, and if the sample is smaller than the given value, teacher forcing would be used (default is 0).

Returns: y_hats, logits

• y_hats (batch, seq_len): predicted y values (y_hat) by the model
• logits (batch, seq_len, class_num): predicted log probability by the model

Examples:

>>> decoder = DecoderRNN(class_num, max_len, hidden_size, sos_id, eos_id, n_layers)
>>> y_hats, logits = decoder(inputs, encoder_outputs, teacher_forcing_ratio=0.90)

forward(inputs, encoder_outputs, function=<function log_softmax>, teacher_forcing_ratio=0.9, use_beam_search=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

forward_step(input, hidden, encoder_outputs=None, function=<function log_softmax>)

forward one time step

Beam

class models.beam.Beam(k, decoder, batch_size, max_len, function, device)

Applying Beam-Search during decoding process.

Parameters:

• k (int) – size of beam
• batch_size (int) – mini-batch size during infer
• max_len (int) – a maximum allowed length for the sequence to be processed
• function (torch.nn.Module) – A function used to generate symbols from RNN hidden state
• (default – torch.nn.functional.log_softmax)
• decoder (torch.nn.Module) – get pointer of decoder object to get multiple parameters at once
• beams (torch.Tensor) – ongoing beams for decoding
• probs (torch.Tensor) – cumulative probability of beams (score of beams)
• sentences (list) – store beams which met <eos> token and terminated decoding process.
• sentence_probs (list) – score of sentences

Inputs: decoder_input, encoder_outputs

• decoder_input (torch.Tensor): initial input of decoder - <sos>
• encoder_outputs (torch.Tensor): tensor with containing the outputs of the encoder.

Returns: y_hats

• y_hats (batch, seq_len): predicted y values (y_hat) by the model

Examples:

>>> beam = Beam(k, decoder, batch_size, max_len, F.log_softmax)
>>> y_hats = beam.search(inputs, encoder_outputs)

search(input, encoder_outputs)

Beam-Search Decoding (Top-K Decoding)

Attention

class models.attention.Attention(decoder_hidden_size)

Applies an dot product attention mechanism on the output features from the decoder.

\[egin{array}{ll} x = context*output \ attn = exp(x_i) / sum_j exp(x_j) \ output = anh(w * (attn * encoder_output) + b * output) \end{array}\]

Parameters:dim (int) – The number of expected features in the output

Inputs: decoder_output, encoder_output

• decoder_output (batch, output_len, hidden_size): tensor containing the output features from the decoder.
• encoder_output (batch, input_len, hidden_size): tensor containing features of the encoded input sequence.Steps to be maintained at a certain number to avoid extremely slow learning

Outputs: output, attn

• output (batch, output_len, dimensions): tensor containing the attended output features from the decoder.

forward(decoder_output, encoder_outputs)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.