Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Quantifying expression from RNA-seq data

Takes raw FASTQ files through quality control, adaptor trimming, STAR alignment and duplicate marking to gene-level and transcript-level expression estimates.

Overview

This module follows the GTEx and TOPMed RNA-seq pipeline. Paired-end or single-end fastq.gz files go in; per-sample expression tables and a merged cohort matrix come out.

The recommended order is: fastqc to inspect read quality, fastp_trim_adaptor to remove adaptors if the reports call for it, STAR_align to map reads to the genome, then rnaseqc_call and rsem_call to quantify. Alignment and duplicate marking are combined: picard_qc is declared as the second step of STAR_align, so running the alignment also marks duplicates and collects Picard metrics. The alignment step reimplements the GTEx run_STAR.py wrapper inline rather than calling it, so that read length may differ between samples, and re-sorts the unsorted STAR output with samtools instead of letting STAR sort in memory.

Gene-level expression comes from RNA-SeQC v2.4.2 against a gene model collapsed so that each gene is a single feature rather than a set of transcripts DeLuca et al. (2012). Its read filters are strict: a read counts only if it is uniquely mapped (mapping quality 255 for STAR BAMs), aligned in a proper pair, carries no more than six non-reference bases, and falls entirely within exon boundaries. Reads overlapping introns are discarded. Exon-level counts are produced as well, and a read spanning several exons contributes to each in proportion to the share of the read it covers. Transcript-level expression comes from RSEM v1.3.0 against a transcript-level GTF.

Strandedness can be given per sample in the manifest as rf, fr, unstranded or strand_missing, following the library types of Signal et al. (2022); see the strand settings reference for what these mean. When it is not supplied, the strand_detected workflow infers it from the gene count table STAR emits. Read length is likewise per sample; a length of zero falls back to 100, which --sjdbOverhang can change.

Three workflows sit outside that main path. bam_to_fastq recovers FASTQ from aligned BAM when only BAM is available. trimmomatic_trim_adaptor is an alternative to fastp. filter_reads filters the alignments for allele-specific work: WASP correction drops reads carrying WASP flags, which mitigates reference-allele bias, and unique-read filtering keeps only alignments at mapping quality 255, following GTEx practice. Either can be applied alone or both together.

When to run it. At the start of the expression branch, on raw sequencing data, and before bulk_expression_QC and bulk_expression_normalization, which take the merged matrices produced here.

Input

  • --sample-list -- tab-delimited manifest with one line per sample and a header. ID and fq1 are required, fq2 is required for paired-end data, and strand and read_length are optional:

    ID	fq1	fq2	strand	read_length
    SAMPLE_001	SAMPLE_001_R1.fastq.gz	SAMPLE_001_R2.fastq.gz	rf	100
    SAMPLE_002	SAMPLE_002_R1.fastq.gz	SAMPLE_002_R2.fastq.gz	rf	100

    strand is rf, fr, unstranded or strand_missing; leaving it out lets strand_detected infer it. read_length of 0 means unknown and falls back to 100. If no sample has a known read length, omit the column entirely rather than filling it with zeros.

  • --data-dir -- directory holding the FASTQ files named in fq1 and fq2. The manifest carries bare filenames, so this has to point at the directory that actually contains them.

  • --cwd -- output directory.

Reference data, built by reference_data_preparation:

  • --STAR-index -- STAR genome index, required by STAR_align.

  • --RSEM-index -- RSEM reference, required by rsem_call.

  • --gtf -- gene model. STAR_align and picard_qc take the transcript-level GTF, while rnaseqc_call takes the collapsed gene-level one; passing the wrong one silently changes what is quantified.

  • --reference-fasta -- the genome FASTA the STAR index was built from. It has to be exactly that file; a different copy of the same assembly makes Picard fail.

  • --ref-flat -- refFlat annotation for Picard CollectRnaSeqMetrics, which reports how bases distribute across UTRs, introns, intergenic regions and coding exons. reference_data_preparation generates it in its RefFlat_generation step.

  • --bam-list -- the BAM manifest written by STAR_align, required by rnaseqc_call and rsem_call.

Adaptor trimming, for fastp_trim_adaptor and trimmomatic_trim_adaptor:

  • --min-len -- reads shorter than this after trimming are discarded; 15 under fastp, 50 under Trimmomatic.

  • --window-size -- sliding window for quality trimming, default 4.

  • --required-quality -- quality floor within the window, default 20.

  • --leading / --trailing -- mean quality below which the leading or trailing window is cut. fastp defaults to 20 and Trimmomatic to 3, so the fastp setting is by far the stricter, and matches cutadapt.

  • --fasta-with-adapters-etc -- adaptor reference. fastp infers adaptors from the reads and does not need one; Trimmomatic does, for instance TruSeq3-PE.fa from the Trimmomatic repository, chosen on the evidence of the FastQC overrepresented-sequences report.

  • --seed-mismatches, --palindrome-clip-threshold, --simple-clip-threshold -- Trimmomatic Illumina-clip settings.

Frequently adjusted:

  • --sjdbOverhang -- splice junction overhang, default 100, i.e. read length minus one.

  • --chimSegmentMin -- minimum chimeric segment length, default 15; 0 disables chimeric detection.

  • --mapping-quality -- uniqueness cutoff, default 255.

  • --detection-threshold (rnaseqc_call) and --max-frag-len, --estimate-rspd (rsem_call).

  • --optical-distance (picard_qc) -- default 100; patterned flowcells such as HiSeq X need 2500.

  • --zap-raw-bam -- delete the raw BAM once duplicates are marked.

  • --varVCFfile -- variant VCF used to prepare WASP filtering.

  • --uncompressed -- set when the FASTQ files are not gzipped.

The captured help below lists every option, including the fastp and Trimmomatic trimming settings and the full set of STAR arguments.

Runtime:

  • --numThreads, --job-size, --walltime, --mem -- threads and cluster resources.

  • --container, --entrypoint -- software environment.

  • --modular-script-dir -- location of the drivers, default code/script.

Output

  • <sample>_fastqc.html and <sample>_fastqc.zip -- per-sample FastQC reports.

  • <fastq>.trimmed.fq.gz with <fastq>.trimmed.fq.html and .json -- trimmed reads and the fastp report; <sample_list>.trimmed.txt is the manifest of trimmed files. A read whose mate did not survive trimming is discarded by default. trimmomatic_trim_adaptor instead keeps them, writing paired and unpaired FASTQ separately, four files per paired-end sample.

  • <sample>.Aligned.sortedByCoord.out.bam and <sample>.Aligned.toTranscriptome.out.bam -- STAR genome and transcriptome alignments. filter_reads writes a WASP-filtered ..._wasp_qc...bam alongside.

  • <sample_list>.bam_file_list -- the BAM manifest that rnaseqc_call and rsem_call consume through --bam-list:

    sample_id	strand	coord_bam_list	BW_list	SJ_list	trans_bam_list
    SAMPLE_001	rf	SAMPLE_001.Aligned.sortedByCoord.out_nowasp_noqc.md.bam	SAMPLE_001.Aligned.sor
  • <sample>.alignment_summary_metrics and the merged <sample_list>.picard.aggregated_quality.metrics.tsv -- Picard duplicate and alignment metrics; the marked BAM flags duplicates with 0x400, decimal 1024:

    Sample PF_READS PF_READS_ALIGNED PCT_PF_READS_ALIGNED PCT_RIBOSOMAL_BASES PCT_CODING_BASES P
    SAMPLE_002 200000 12498 0.06249 0.001012 0.039564 0.289782 0.270124 0.399518 0.329346 0.0184
  • <sample>.rnaseqc.gene_tpm.gct.gz, .gene_readsCount.gct.gz and .exon_readsCount.gct.gz -- RNA-SeQC gene- and exon-level quantifications, merged across samples into <bam_list>.rnaseqc.*.gct.gz, next to <sample>.metrics.tsv and per-transcript coverage statistics.

  • <sample>.rsem.genes.results and <sample>.rsem.isoforms.results -- RSEM per-sample estimates, with run statistics in <sample>.rsem.stat/ including the .cnt read-assignment counts. These merge into seven files, four from the isoform columns and three from the gene columns, among them <bam_list>.rsem_transcripts_expected_count.txt.gz and the matching TPM and gene-level matrices.

  • <sample_list>.multiqc_report.html -- MultiQC summary across all of the above.

Each step also writes .stdout and .stderr beside its output. Example results for the toy data are under output/rnaseq/, but the raw FASTQ files that produced them are not distributed in this repository.

Minimal Working Example

These commands use the two demo samples listed in tests/fixtures/rna_calling/protocol_example.rnaseq.fastq.list.txt, with the simplest alignment recipe, no WASP filtering and no unique-read filtering, which is what quantifying expression for eQTL analysis calls for. The main path is fastqc, fastp_trim_adaptor, STAR_align, rnaseqc_call, rsem_call, run in that order; the remaining blocks are optional or alternatives. The raw FASTQ files are not distributed in this repository, so --data-dir is shown as a placeholder and has to point at the directory holding the files the manifest names.

Convert BAM back to FASTQ (optional)

Only needed when the starting material is aligned BAM rather than FASTQ.

Timing: TBD (on toy dataset)

Quality control with fastqc

Timing: TBD (on toy dataset)

Trim adaptors with fastp (optional)

Generates trimmed FASTQ files and a new sample list pointing at them.

Timing: TBD (on toy dataset)

Trim adaptors with Trimmomatic (alternative)

An alternative to the fastp step above. Run one or the other, not both.

Timing: TBD (on toy dataset)

Align reads with STAR

Aligns reads and produces the per-run BAM manifest (<sample-list-stem>.bam_file_list) that the downstream RSEM step consumes. The recipe below adds no WASP/unique-filter tags (fastest; suitable for eQTL expression quantification). Other recipes are available by adding --wasp yes and/or --unique yes together with a --varVCFfile.

Timing: TBD (on toy dataset)

Detect strandedness

Infers the library strand from the STAR gene counts, for samples whose manifest leaves strand empty. Run it after STAR_align.

Timing: TBD (on toy dataset)

Mark duplicates and collect Picard metrics

picard_qc is also the second step of STAR_align, so the alignment above has already run it. This command runs it on its own, for instance on BAMs aligned outside this pipeline.

Timing: TBD (on toy dataset)

Gene-level expression with rnaseqc

Uses the collapsed gene-level GTF. Point --bam_list at the manifest produced by STAR_align and keep --cwd consistent with where the BAMs live.

Timing: TBD (on toy dataset)

Transcript-level expression with RSEM

Uses the transcript-level GTF and the RSEM reference. As above, --bam_list is the STAR_align manifest and --cwd must match the BAM location (the manifest stores bare filenames that are resolved relative to --cwd).

Timing: TBD (on toy dataset)

Filter reads with WASP

Applies WASP filtering to the alignments, producing the _wasp_qc BAMs used for allele-specific analysis.

Timing: TBD (on toy dataset)

Command Interface

usage: sos run pipeline/RNA_calling.ipynb
               [workflow_name | -t targets] [options] [workflow_options]
  workflow_name:        Single or combined workflows defined in this script
  targets:              One or more targets to generate
  options:              Single-hyphen sos parameters (see "sos run -h" for details)
  workflow_options:     Double-hyphen workflow-specific parameters

Workflows:
  bam_to_fastq
  fastqc
  fastp_trim_adaptor
  trimmomatic_trim_adaptor
  STAR_align
  strand_detected
  picard_qc
  rnaseqc_call
  rsem_call
  filter_reads

Global Workflow Options:
  --modular-script-dir code/script (as path)
  --cwd output (as path)
                        The output directory for generated files.
  --sample-list VAL (as path, required)
                        Sample meta data list
  --sample-name  (as list)
                        Sample names to analyze
  --data-dir  path(f"{sample_list:d}")

                        Raw data directory, default to the same directory as
                        sample list
  --job-size 1 (as int)
                        For cluster jobs, number commands to run per job
  --walltime 5h
                        Wall clock time expected
  --mem 16G
                        Memory expected
  --java-mem 6G
                        Memory for Java virtual mechine (`picard`)
  --numThreads 8 (as int)
                        Number of threads
  --varVCFfile ''
                        VarVCFfile for data preparation for wasp_filtering
  --[no-]uncompressed (default to False)
                        Whether the fasta/fastq file is compressed or not.

Sections
  bam_to_fastq:
  fastqc:
  fastp_trim_adaptor_1:
    Workflow Options:
      --window-size 4 (as int)
                        sliding window setting
      --required-quality 20 (as int)
      --leading 20 (as int)
                        the mean quality requirement option for cut_front
      --trailing 20 (as int)
                        the mean quality requirement option for cut_tail
      --min-len 15 (as int)
                        reads shorter than length_required will be discarded
      --fasta-with-adapters-etc . (as path)
                        Path to the reference adaptors
  fastp_trim_adaptor_2:
  trimmomatic_trim_adaptor:
    Workflow Options:
      --fasta-with-adapters-etc . (as path)
                        Illumina clip setting Path to the reference adaptors
      --seed-mismatches 2 (as int)
      --palindrome-clip-threshold 30 (as int)
      --simple-clip-threshold 10 (as int)
      --window-size 4 (as int)
                        sliding window setting
      --required-quality 20 (as int)
      --leading 3 (as int)
                        Other settings
      --trailing 3 (as int)
      --min-len 50 (as int)
  STAR_align_1:
    Workflow Options:
      --gtf VAL (as path, required)
                        Reference gene model
      --STAR-index VAL (as path, required)
                        STAR indexing file
      --outFilterMultimapNmax 20 (as int)
      --alignSJoverhangMin 8 (as int)
      --alignSJDBoverhangMin 1 (as int)
      --outFilterMismatchNmax 999 (as int)
      --outFilterMismatchNoverLmax 0.1 (as float)
      --alignIntronMin 20 (as int)
      --alignIntronMax 1000000 (as int)
      --alignMatesGapMax 1000000 (as int)
      --outFilterType BySJout
      --outFilterScoreMinOverLread 0.33 (as float)
      --outFilterMatchNminOverLread 0.33 (as float)
      --limitSjdbInsertNsj 1200000 (as int)
      --outSAMstrandField intronMotif
      --outFilterIntronMotifs None
      --alignSoftClipAtReferenceEnds Yes
      --quantMode TranscriptomeSAM GeneCounts (as list)
      --outSAMattrRGline ID:rg1 SM:sm1 (as list)
      --outSAMattributes NH HI AS nM NM ch (as list)
      --chimSegmentMin 15 (as int)
      --chimJunctionOverhangMin 15 (as int)
      --chimOutType Junctions WithinBAM SoftClip (as list)
      --chimMainSegmentMultNmax 1 (as int)
      --sjdbOverhang 100 (as int)
      --mapping-quality 255 (as int)
  strand_detected_1:
  strand_detected_2:
    Workflow Options:
      --strand ''
  picard_qc, STAR_align_2:
    Workflow Options:
      --gtf VAL (as path, required)
                        Reference gene model
      --ref-flat VAL (as path, required)
                        Path to flat reference file, for computing QC metric
      --reference-fasta VAL (as path, required)
                        The fasta reference file used to generate star index
      --optical-distance 100 (as int)
                        For the patterned flowcell models (HiSeq X), change to
                        2500
      --[no-]zap-raw-bam (default to False)
  STAR_align_3:
  rnaseqc_call_1:
    Workflow Options:
      --bam-list VAL (as path, required)
      --gtf VAL (as path, required)
                        Reference gene model
      --reference-fasta VAL (as path, required)
      --detection-threshold 5 (as int)
      --mapping-quality 255 (as int)
  rnaseqc_call_2:
    Workflow Options:
      --bam-list VAL (as path, required)
  rsem_call_1:
    Workflow Options:
      --bam-list VAL (as path, required)
      --RSEM-index VAL (as path, required)
      --max-frag-len 1000 (as int)
      --[no-]estimate-rspd (default to True)
  rsem_call_2:
    Workflow Options:
      --bam-list VAL (as path, required)
  rsem_call_3, rnaseqc_call_3:
  rsem_call_4, rnaseqc_call_4:
  filter_reads:
    Workflow Options:
      --unique ''
      --wasp ''
      --mapping-quality 255 (as int)

Workflow implementation

Step 0. Convert BAM back to FASTQ

Recovers paired FASTQ from an aligned BAM, for datasets that arrive as BAM.

Step 1. QC before alignment

fastqc on each FASTQ. Paired-end mates are assessed separately, so a paired sample yields two reports.

Step 2. Remove adaptors with fastp

Documentation: fastp. fastp infers the adaptor from the reads themselves: assuming a single adaptor confined to read tails, it counts 10-mers over the first million reads, keeps the frequent ones as seeds once low-complexity artefacts are dropped, and extends them by a tree search. No adaptor reference is required.

Step 2 alternative: remove adaptors with Trimmomatic

Documentation: Trimmomatic. Superseded by fastp, which catches adaptors Trimmomatic cannot detect and needs no adaptor reference. Kept for cases where a known reference is preferred.

Step 3. Alignment with STAR

Documentation: STAR, and the GTEx run_STAR.py wrapper this step reimplements inline. Produces both the genome-coordinate and the transcriptome BAM.

Step 4. Mark duplicates and collect metrics with Picard

The first QC pass after alignment: collects Picard RNA-seq metrics and rewrites the BAM with duplicates flagged.

Step 5. Post-alignment QC and quantification with RNA-SeQC

Documentation: RNA-SeQC, and the GTEx run_rnaseqc.py. The second QC pass, which also produces the gene- and exon-level quantifications.

The per-sample RNA-SeQC results are merged in the step below.

Step 6. Quantify expression with RSEM

Documentation: RSEM, and the GTEx run_RSEM.py. Estimates gene- and isoform-level expression from the transcriptome BAM.

The per-sample RSEM results are merged in the steps below.

Step 7. Summarize with MultiQC

MultiQC searches the output directory, recognises the logs and summary files the earlier steps left behind, and writes a single HTML report plus a directory of parsed data. Pointing it at the directory holding all outputs is enough.

Step 8. Filter reads with WASP

Optional. It reads the STAR_align_1 output directly, so no input has to be named.

Troubleshooting

StepSubstepProblemPossible ReasonSolution
rsem_call / rnaseqc_call-Target unavailable: .../<name>.Aligned.toTranscriptome.out_*.bam--bam_list points at a manifest whose bare filenames do not exist under the given --cwd (the step prepends {cwd}/ to each entry)Use the STAR_align-generated <sample-list-stem>.bam_file_list and set --cwd to the directory that actually holds those BAMs
References
  1. DeLuca, D. S., Levin, J. Z., Sivachenko, A., Fennell, T., Nazaire, M.-D., Williams, C., Reich, M., Winckler, W., & Getz, G. (2012). RNA-SeQC: RNA-seq metrics for quality control and process optimization. Bioinformatics, 28(11), 1530–1532. 10.1093/bioinformatics/bts196
  2. Signal, B., & Kahlke, T. (2022). how_are_we_stranded_here: quick determination of RNA-Seq strandedness. BMC Bioinformatics, 23(1). 10.1186/s12859-022-04572-7