Covariate data preprocessing

Covariate data preprocessing#

Combine known covariates with genotype principal components and optionally infer hidden factors for xQTL association testing.

Miniprotocol Timing#

This is the total duration for one complete toy-data route; module-specific timings appear on their respective pages.

Timing: <3 min (on the toy dataset)

Overview#

This mini-protocol walks through construction of an association-ready covariate matrix. covariate_formatting.ipynb first merges sample covariates with genotype principal components. covariate_hidden_factor.ipynb then residualizes the molecular phenotype against those observed covariates and optionally estimates latent factors using Marchenko–Pastur selection, a user-specified PCA dimension, PEER, or bi-cross-validation (BiCV).

Run step 1 once, then choose one hidden-factor method from steps 2–5. These alternatives are not a four-step chain. Use the resulting matrix as the covariate input to association testing.

Steps#

Choose a route before running commands; steps 2–5 are alternative hidden-factor methods.

Analysis goal	Commands to run, in order	Inputs
Observed covariates and genotype PCs only	1	`input/covariate/protocol_example.covariates.base.tsv`; `output/genotype/genotype_pca/protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.rds`; `output/genotype/genotype_pca/protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.scree.txt`
Automatically select PCA hidden factors	1 → 2	Step 1 inputs; `output/rnaseq/protocol_example.rnaseq.bed.bed.gz`
Use a fixed number of PEER factors	1 → 3	Step 1 inputs; `output/rnaseq/protocol_example.rnaseq.bed.bed.gz`
Use PCA factors with configurable dimension selection	1 → 4	Step 1 inputs; `output/rnaseq/protocol_example.rnaseq.bed.bed.gz`
Select hidden factors by bi-cross-validation	1 → 5	Step 1 inputs; `output/rnaseq/protocol_example.rnaseq.bed.bed.gz`

Run only the row matching the intended analysis goal.

1. Merge observed covariates and genotype PCs #

What it does: Combine the base covariate table with the selected genotype principal components.

sos run pipeline/covariate_formatting.ipynb merge_genotype_pc \
    --cwd output/covariate \
    --pcaFile output/genotype/genotype_pca/protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.rds \
    --covFile input/covariate/protocol_example.covariates.base.tsv \
    --name protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca \
    --tol-cov 0.4 \
    --k `awk '$3 < 0.8' output/genotype/genotype_pca/protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.scree.txt | tail -1 | cut -f 1`

2. Infer PCA factors with Marchenko–Pastur selection #

What it does: Residualize the phenotype and automatically retain PCA factors above the Marchenko–Pastur noise threshold.

sos run pipeline/covariate_hidden_factor.ipynb Marchenko_PC \
    --cwd output/covariate \
    --phenoFile output/rnaseq/protocol_example.rnaseq.bed.bed.gz \
    --covFile output/covariate/protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.gz \
    --mean-impute-missing

3. Infer PEER factors #

What it does: Residualize the phenotype and estimate the requested number of probabilistic PEER factors.

sos run pipeline/covariate_hidden_factor.ipynb PEER \
    --cwd output/covariate \
    --phenoFile output/rnaseq/protocol_example.rnaseq.bed.bed.gz \
    --covFile output/covariate/protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.gz \
    --N 3

4. Infer configurable PCA factors #

What it does: Residualize the phenotype and estimate PCA factors using the selected dimension rule.

sos run pipeline/covariate_hidden_factor.ipynb PCA \
    --cwd output/covariate \
    --phenoFile output/rnaseq/protocol_example.rnaseq.bed.bed.gz \
    --covFile output/covariate/protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.gz \
    --choose_k_method Marchenko \
    --mean-impute-missing

5. Infer factors with bi-cross-validation #

What it does: Residualize the phenotype and select latent structure using the BiCV workflow.

sos run pipeline/covariate_hidden_factor.ipynb BiCV \
    --cwd output/covariate \
    --phenoFile output/rnaseq/protocol_example.rnaseq.bed.bed.gz \
    --covFile output/covariate/protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.gz \
    --N 3

Output Files#

Route or step	Products and relative paths
Step 1	`output/covariate/protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.gz`
Step 2	`output/covariate/protocol_example.rnaseq.Marchenko_PC.gz`
Step 3	`output/covariate/protocol_example.rnaseq.bed.PEER.gz`; `output/covariate/protocol_example.rnaseq.bed.PEER.diag.pdf`; `output/covariate/protocol_example.rnaseq.bed.PEER_MODEL.hd5`
Step 4	`output/covariate/protocol_example.rnaseq.Marchenko_PC.gz` when `--choose_k_method Marchenko` is used
Step 5	`output/covariate/protocol_example.rnaseq.bed.BiCV.gz`

Anticipated Results#

Step 1 produces a sample-by-covariate matrix containing the observed covariates and selected genotype principal components. A selected hidden-factor route adds latent factors estimated after residualizing the molecular phenotype against those observed covariates.

Use exactly one final matrix in association testing; do not concatenate results from alternative hidden-factor methods.

Command interface#

List the workflows and parameters available in each module used by this mini-protocol.

sos run pipeline/covariate_formatting.ipynb -h
sos run pipeline/covariate_hidden_factor.ipynb -h

Covariate data preprocessing

Contents