Combine known covariates with genotype principal components and optionally infer hidden factors for xQTL association testing.
Miniprotocol Timing¶
This is the total duration for one complete toy-data route; module-specific timings appear on their respective pages.
Timing: <3 min (on the toy dataset)
Overview¶
This mini-protocol walks through construction of an association-ready covariate matrix. covariate_formatting.ipynb first merges sample covariates with genotype principal components. covariate_hidden_factor.ipynb then residualizes the molecular phenotype against those observed covariates and optionally estimates latent factors using Marchenko–Pastur selection, a user-specified PCA dimension, PEER, or bi-cross-validation (BiCV).
Run step 1 once, then choose one hidden-factor method from steps 2–5. These alternatives are not a four-step chain. Use the resulting matrix as the covariate input to association testing.
Steps¶
Choose a route before running commands; steps 2–5 are alternative hidden-factor methods.
| Analysis goal | Commands to run, in order | Inputs |
|---|---|---|
| Observed covariates and genotype PCs only | 1 | tests/fixtures/covariate_formatting/covariates.base.tsv; output/genotype/genotype_pca/protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.rds; output/genotype/genotype_pca/protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.scree.txt |
| Automatically select PCA hidden factors | 1 → 2 | Step 1 inputs; tests/fixtures/phenotype_formatting/protocol_example.rnaseq.bed.bed.gz |
| Use a fixed number of PEER factors | 1 → 3 | Step 1 inputs; tests/fixtures/phenotype_formatting/protocol_example.rnaseq.bed.bed.gz |
| Use PCA factors with configurable dimension selection | 1 → 4 | Step 1 inputs; tests/fixtures/phenotype_formatting/protocol_example.rnaseq.bed.bed.gz |
| Select hidden factors by bi-cross-validation | 1 → 5 | Step 1 inputs; tests/fixtures/phenotype_formatting/protocol_example.rnaseq.bed.bed.gz |
Run only the row matching the intended analysis goal.
1. Merge observed covariates and genotype PCs¶
What it does: Combine the base covariate table with the selected genotype principal components.
sos run pipeline/covariate_formatting.ipynb merge_genotype_pc \
--cwd output/covariate \
--pcaFile output/genotype/genotype_pca/protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.rds \
--covFile tests/fixtures/covariate_formatting/covariates.base.tsv \
--name protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca \
--tol-cov 0.4 \
--k `awk '$3 < 0.8' output/genotype/genotype_pca/protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.scree.txt | tail -1 | cut -f 1`
2. Infer PCA factors with Marchenko–Pastur selection¶
What it does: Residualize the phenotype and automatically retain PCA factors above the Marchenko–Pastur noise threshold.
sos run pipeline/covariate_hidden_factor.ipynb Marchenko_PC \
--cwd output/covariate \
--phenoFile tests/fixtures/phenotype_formatting/protocol_example.rnaseq.bed.bed.gz \
--covFile output/covariate/protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.gz \
--mean-impute-missing
3. Infer PEER factors¶
What it does: Residualize the phenotype and estimate the requested number of probabilistic PEER factors.
sos run pipeline/covariate_hidden_factor.ipynb PEER \
--cwd output/covariate \
--phenoFile tests/fixtures/phenotype_formatting/protocol_example.rnaseq.bed.bed.gz \
--covFile output/covariate/protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.gz \
--N 3
4. Infer configurable PCA factors¶
What it does: Residualize the phenotype and estimate PCA factors using the selected dimension rule.
sos run pipeline/covariate_hidden_factor.ipynb PCA \
--cwd output/covariate \
--phenoFile tests/fixtures/phenotype_formatting/protocol_example.rnaseq.bed.bed.gz \
--covFile output/covariate/protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.gz \
--choose_k_method Marchenko \
--mean-impute-missing
5. Infer factors with bi-cross-validation¶
What it does: Residualize the phenotype and select latent structure using the BiCV workflow.
sos run pipeline/covariate_hidden_factor.ipynb BiCV \
--cwd output/covariate \
--phenoFile tests/fixtures/phenotype_formatting/protocol_example.rnaseq.bed.bed.gz \
--covFile output/covariate/protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.gz \
--N 3
Output¶
| Route or step | Products and relative paths |
|---|---|
| Step 1 | output/covariate/protocol_example.covariates.protocol_example.genotype.merged.plink_qc.plink_qc.prune.pca.gz |
| Step 2 | output/covariate/protocol_example.rnaseq.Marchenko_PC.gz |
| Step 3 | output/covariate/protocol_example.rnaseq.bed.PEER.gz; output/covariate/protocol_example.rnaseq.bed.PEER.diag.pdf; output/covariate/protocol_example.rnaseq.bed.PEER_MODEL.hd5 |
| Step 4 | output/covariate/protocol_example.rnaseq.Marchenko_PC.gz when --choose_k_method Marchenko is used |
| Step 5 | output/covariate/protocol_example.rnaseq.bed.BiCV.gz |
Anticipated Results¶
Step 1 produces a sample-by-covariate matrix containing the observed covariates and selected genotype principal components. A selected hidden-factor route adds latent factors estimated after residualizing the molecular phenotype against those observed covariates.
Use exactly one final matrix in association testing; do not concatenate results from alternative hidden-factor methods.
Command Interface¶
List the workflows and parameters available in each module used by this mini-protocol.
sos run pipeline/covariate_formatting.ipynb -h
sos run pipeline/covariate_hidden_factor.ipynb -h