rTASSEL Benchmarks

Methods

To achieve benchmarking results, several data sets were used. Genotypic and phenotypic maize data consisting of 279 samples, 3093 variant sites, and 1 measured trait were utilized for the analysis of variant call format (VCF) import, generalized linear model (GLM) association, mixed linear model (MLM) association, and kinship generation times (Flint-Garcia et al., 2005). To illustrate the effectiveness of the fast association method, 100 simulated RNA expression traits for the prior genotype data was used. Trait data was generated using the makeExampleDESeqDataSet() function from the R package DESeq2 (Love et al., 2014). A larger genotypic data set consisting of 1,210 samples and 2,255,405 variant sites was also utilized for large VCF import and kinship generation times. All benchmarks were generated using the microbenchmark() function from the R package microbenchmark (Mersmann, 2019).

All benchmarks sans large VCF import and kinship generation times were evaluated 100 times and recorded on a workstation running 16 GB of RAM and 4 cores on an Intel® CoreTM i5-6500 CPU with a clock speed of 3.20 GHz and. Large VCF import and kinship generation benchmarks were evaluated 10 times and recorded on a workstation running 256 GB of RAM and 12 cores on an Intel® Xeon® CPU E5-2643 v3 with a clock speed of 3.40GHz.

Feature comparisons

Feature comparisons of rTASSEL with other R packages. Features of rTASSEL (y-axis) are compared with other commonly-used R packages (x-axis). Packages that contain a specified feature are highlighted green (yes) and grey (no) if they do not contain a feature or are limited in scope. Association features for packages are based on if said package contains methods for generalized linear models, mixed linear models utilizing the “Q+K” method (Yu et al., 2006), or multi trait fast association methods (Shabalin, 2012). Kinship and distance matrix features denote if a package can return an n x n matrix of values for further use. Packages that contain plotting features indicate if the package contains an automated plot feature instead of using base or grid-based R graphics (R Core Team, 2020) in conjunction with data output. The packages used for this comparison are BGData (Grueneberg and Campos, 2019), GAPIT3 (Wang and Zhang, 2020), MatrixEQTL (Shabalin, 2012), rrBLUP (Endelman, 2011), SNPRelate (Zheng et al., 2012), statgenGWAS (Rossum and Kruijer, 2020), VariantAnnotation (Obenchain et al., 2014), VariantExperiment (Liu et al., 2020), and vcfR (Knaus and Grünwald, 2017).

Speed comparisons

VCF import

VCF import time comparisons of genotypic data. A distribution of replicated benchmark evaluations with recorded means (cross shapes) are plotted for rTASSEL and several R packages: SNPRelate (Zheng et al., 2012), VariantAnnotation (Obenchain et al., 2014), VariantExperiment (Liu et al., 2020), and vcfR (Knaus and Grünwald, 2017). Import times are recorded for 279 samples x 3093 variant sites (A) and 1,210 samples x 2,255,405 variant sites (B).

Association methods

GLM association time comparisons. A distribution of replicated benchmark evaluations with recorded means (cross shapes) are plotted for rTASSEL and the R packages GAPIT3 (Wang and Zhang, 2020) and BGData (Grueneberg and Campos, 2019). Import times are recorded for 279 samples x 3093 variant sites and 1 measured trait.

MLM association time comparisons. A distribution of replicated benchmark evaluations with recorded means (cross shapes) are plotted for rTASSEL and the R packages rrBLUP (Endelman, 2011) and GAPIT3 (Wang and Zhang, 2020). Import times are recorded for 279 samples x 3093 variant sites and 1 measured trait.

Fast association time comparisons. A distribution of replicated benchmark evaluations with recorded means (cross shapes) are plotted for rTASSEL and the R package MatrixEQTL (Shabalin, 2012). Import times are recorded for 279 samples x 3093 variant sites and 100 simulated RNA expression traits.

Kinship creation

Kinship matrix (IBS) generation time comparisons of genotypic data. A distribution of replicated benchmark evaluations with recorded means (cross shapes) are plotted for rTASSEL and the R packages statgenGWAS (Rossum and Kruijer, 2020) and SNPRelate (Zheng et al., 2012). Generation times are recorded for 279 samples x 3093 variant sites (A) and 1,210 samples x 2,255,405 variant sites (B).