Creates training batches from rds files. Rds files must contain a
list of length 2 (input/target) or of length 1 (for language model).
If target_len
is not NULL will take the last target_len
entries of
the first list element as targets and the rest as input.
Usage
generator_rds(
rds_folder,
batch_size,
path_file_log = NULL,
max_samples = NULL,
proportion_per_seq = NULL,
target_len = NULL,
seed = NULL,
delete_used_files = FALSE,
reverse_complement = FALSE,
sample_by_file_size = FALSE,
n_gram = NULL,
n_gram_stride = 1,
reverse_complement_encoding = FALSE,
add_noise = NULL,
reshape_xy = NULL
)
Arguments
- rds_folder
Path to input files.
- batch_size
Number of samples in one batch.
- path_file_log
Write name of files to csv file if path is specified.
- max_samples
Maximum number of samples to use from one file. If not
NULL
and file has more thanmax_samples
samples, will randomly choose a subset ofmax_samples
samples.- proportion_per_seq
Numerical value between 0 and 1. Proportion of sequence to take samples from (use random subsequence).
- target_len
Number of target nucleotides for language model.
- seed
Sets seed for
set.seed
function for reproducible results.- delete_used_files
Whether to delete file once used. Only applies for rds files.
- reverse_complement
Boolean, for every new file decide randomly to use original data or its reverse complement.
- sample_by_file_size
Sample new file weighted by file size (bigger files more likely).
- n_gram
Integer, encode target not nucleotide wise but combine n nucleotides at once. For example for
n=2, "AA" -> (1, 0,..., 0),
"AC" -> (0, 1, 0,..., 0), "TT" -> (0,..., 0, 1)
, where the one-hot vectors have lengthlength(vocabulary)^n
.- n_gram_stride
Step size for n-gram encoding. For AACCGGTT with
n_gram = 4
andn_gram_stride = 2
, generator encodes(AACC), (CCGG), (GGTT)
; forn_gram_stride = 4
generator encodes(AACC), (GGTT)
.- reverse_complement_encoding
Whether to use both original sequence and reverse complement as two input sequences.
- add_noise
NULL
or list of arguments. If notNULL
, list must contain the following arguments:noise_type
can be"normal"
or"uniform"
; optional argumentssd
ormean
if noise_type is"normal"
(default issd=1
andmean=0
) ormin, max
ifnoise_type
is"uniform"
(default ismin=0, max=1
).- reshape_xy
Can be a list of functions to apply to input and/or target. List elements (containing the reshape functions) must be called x for input or y for target and each have arguments called x and y. For example:
reshape_xy = list(x = function(x, y) {return(x+1)}, y = function(x, y) {return(x+y)})
. For rds generator needs to have an additional argument called sw.
Examples
if (FALSE) { # reticulate::py_module_available("tensorflow")
# create 3 rds files
rds_folder <- tempfile()
dir.create(rds_folder)
batch_size <- 7
maxlen <- 11
voc_len <- 4
for (i in 1:3) {
x <- sample(0:(voc_len-1), maxlen*batch_size, replace = TRUE)
x <- keras::to_categorical(x, num_classes = voc_len)
x <- array(x, dim = c(batch_size, maxlen, voc_len))
y <- sample(0:2, batch_size ,replace = TRUE)
y <- keras::to_categorical(y, num_classes = 3)
xy_list <- list(x, y)
file_name <- paste0(rds_folder, "/file_", i, ".rds")
saveRDS(xy_list, file_name)
}
# create generator
gen <- generator_rds(rds_folder, batch_size = 2)
z <- gen()
x <- z[[1]]
y <- z[[2]]
x[1, , ]
y[1, ]
}