Skip to contents

Initializes generators defined by generator_fasta_label_folder function

Source: R/generator_folder_collect.R
generator_initialize.Rd

Initializes generators defined by generator_fasta_label_folder function. Targets get encoded in order of directories. Number of classes is given by length of directories.

Usage

generator_initialize(
  directories,
  format = "fasta",
  batch_size = 256,
  maxlen = 250,
  max_iter = 10000,
  vocabulary = c("a", "c", "g", "t"),
  verbose = FALSE,
  shuffle_file_order = FALSE,
  step = 1,
  seed = 1234,
  shuffle_input = FALSE,
  file_limit = NULL,
  path_file_log = NULL,
  reverse_complement = FALSE,
  reverse_complement_encoding = FALSE,
  val = FALSE,
  ambiguous_nuc = "zero",
  proportion_per_seq = NULL,
  target_middle = FALSE,
  read_data = FALSE,
  use_quality_score = FALSE,
  padding = TRUE,
  added_label_path = NULL,
  add_input_as_seq = NULL,
  skip_amb_nuc = NULL,
  max_samples = NULL,
  file_filter = NULL,
  concat_seq = NULL,
  use_coverage = NULL,
  set_learning = NULL,
  proportion_entries = NULL,
  sample_by_file_size = FALSE,
  n_gram = NULL,
  n_gram_stride = 1,
  add_noise = NULL,
  return_int = FALSE,
  reshape_xy = NULL
)

Arguments

directories

Vector of paths to folder containing fasta files. Files in one folder should belong to one class.

format

File format, either "fasta" or "fastq".

batch_size

Number of samples in one batch.

maxlen

Length of predictor sequence.

max_iter

Stop after max_iter number of iterations failed to produce a new batch.

vocabulary

Vector of allowed characters. Characters outside vocabulary get encoded as specified in ambiguous_nuc.

verbose

Whether to show messages.

shuffle_file_order

Logical, whether to go through files randomly or sequentially.

step

How often to take a sample.

seed

Sets seed for set.seed function for reproducible results.

shuffle_input

Whether to shuffle entries in every fasta/fastq file before extracting samples.

file_limit

Integer or NULL. If integer, use only specified number of randomly sampled files for training. Ignored if greater than number of files in path.

path_file_log

Write name of files to csv file if path is specified.

reverse_complement

Boolean, for every new file decide randomly to use original data or its reverse complement.

reverse_complement_encoding

Whether to use both original sequence and reverse complement as two input sequences.

val

Logical, call initialized generator "genY" or "genValY" where Y is an integer between 1 and length of directories.

ambiguous_nuc

How to handle nucleotides outside vocabulary, either "zero", "discard", "empirical" or "equal".

  • If "zero", input gets encoded as zero vector.

  • If "equal", input is repetition of 1/length(vocabulary).

  • If "discard", samples containing nucleotides outside vocabulary get discarded.

  • If "empirical", use nucleotide distribution of current file.

proportion_per_seq

Numerical value between 0 and 1. Proportion of sequence to take samples from (use random subsequence).

target_middle

Split input sequence into two sequences while removing nucleotide in middle. If input is x_1,..., x_(n+1), input gets split into input_1 = x_1,..., x_m and input_2 = x_(n+1),..., x_(m+2) where m = ceiling((n+1)/2) and n = maxlen. Note that x_(m+1) is not used.

read_data

If true the first element of output is a list of length 2, each containing one part of paired read.

use_quality_score

Whether to use fastq quality scores. If TRUE input is not one-hot-encoding but corresponds to probabilities. For example (0.97, 0.01, 0.01, 0.01) instead of (1, 0, 0, 0).

padding

Whether to pad sequences too short for one sample with zeros.

added_label_path

Path to file with additional input labels. Should be a csv file with one column named "file". Other columns should correspond to labels.

add_input_as_seq

Boolean vector specifying for each entry in added_label_path if rows from csv should be encoded as a sequence or used directly. If a row in your csv file is a sequence this should be TRUE. For example you may want to add another sequence, say ACCGT. Then this would correspond to 1,2,2,3,4 in csv file (if vocabulary = c("A", "C", "G", "T")). If add_input_as_seq is TRUE, 12234 gets one-hot encoded, so added input is a 3D tensor. If add_input_as_seq is FALSE this will feed network just raw data (a 2D tensor).

skip_amb_nuc

Threshold of ambiguous nucleotides to accept in fasta entry. Complete entry will get discarded otherwise.

max_samples

Maximum number of samples to use from one file. If not NULL and file has more than max_samples samples, will randomly choose a subset of max_samples samples.

file_filter

Vector of file names to use from path_corpus.

concat_seq

Character string or NULL. If not NULL all entries from file get concatenated to one sequence with concat_seq string between them. Example: If 1.entry AACC, 2. entry TTTG and concat_seq = "ZZZ" this becomes AACCZZZTTTG.

use_coverage

Integer or NULL. If not NULL, use coverage as encoding rather than one-hot encoding and normalize. Coverage information must be contained in fasta header: there must be a string "cov_n" in the header, where n is some integer.

set_learning

When you want to assign one label to set of samples. Only implemented for train_type = "label_folder". Input is a list with the following parameters

  • samples_per_target: how many samples to use for one target.

  • maxlen: length of one sample.

  • reshape_mode: "time_dist", "multi_input" or "concat".

    • If reshape_mode is "multi_input", generator will produce samples_per_target separate inputs, each of length maxlen (model should have samples_per_target input layers).

    • If reshape_mode is "time_dist", generator will produce a 4D input array. The dimensions correspond to (batch_size, samples_per_target, maxlen, length(vocabulary)).

    • If reshape_mode is "concat", generator will concatenate samples_per_target sequences of length maxlen to one long sequence.

  • If reshape_mode is "concat", there is an additional buffer_len argument. If buffer_len is an integer, the subsequences are interspaced with buffer_len rows. The input length is (maxlen \(*\) samples_per_target) + buffer_len \(*\) (samples_per_target - 1).

proportion_entries

Proportion of fasta entries to keep. For example, if fasta file has 50 entries and proportion_entries = 0.1, will randomly select 5 entries.

sample_by_file_size

Sample new file weighted by file size (bigger files more likely).

n_gram

Integer, encode target not nucleotide wise but combine n nucleotides at once. For example for n=2, "AA" -> (1, 0,..., 0), "AC" -> (0, 1, 0,..., 0), "TT" -> (0,..., 0, 1), where the one-hot vectors have length length(vocabulary)^n.

n_gram_stride

Step size for n-gram encoding. For AACCGGTT with n_gram = 4 and n_gram_stride = 2, generator encodes (AACC), (CCGG), (GGTT); for n_gram_stride = 4 generator encodes (AACC), (GGTT).

add_noise

NULL or list of arguments. If not NULL, list must contain the following arguments: noise_type can be "normal" or "uniform"; optional arguments sd or mean if noise_type is "normal" (default is sd=1 and mean=0) or min, max if noise_type is "uniform" (default is min=0, max=1).

return_int

Whether to return integer encoding or one-hot encoding.

reshape_xy

Can be a list of functions to apply to input and/or target. List elements (containing the reshape functions) must be called x for input or y for target and each have arguments called x and y. For example: reshape_xy = list(x = function(x, y) {return(x+1)}, y = function(x, y) {return(x+y)}) . For rds generator needs to have an additional argument called sw.

Value

List of generator function.

Examples

if (FALSE) { # reticulate::py_module_available("tensorflow")
# create two folders with dummy fasta files
path_input_1 <- tempfile()
dir.create(path_input_1)
create_dummy_data(file_path = path_input_1, num_files = 2, seq_length = 5,
                  num_seq = 2, vocabulary = c("a", "c", "g", "t"))
path_input_2 <- tempfile()
dir.create(path_input_2)
create_dummy_data(file_path = path_input_2, num_files = 3, seq_length = 7,
                  num_seq = 5, vocabulary = c("a", "c", "g", "t"))

gen_list <- generator_initialize(directories = c(path_input_1, path_input_1),
                                        batch_size = 4, maxlen = 5)
z1 <- gen_list[[1]]()
z1[[1]]
z1[[2]]
}