RNA-seq workflow: from FASTQ to counts

What each stage hands to the next, and what to give DESeq2

13 min

Nine short sequencing reads piled along one transcript, with lines running from its ends to one lime column of a blank gene-by-sample grid, where nine tally ticks mark the count.

To turn a folder of .fastq.gz files into a gene count matrix, list each sample's files in a samplesheet and run nf-core/rnaseq. One nextflow run command does read QC, trimming, alignment, quantification and the gene-level summary, and writes the counts DESeq2 or edgeR expect. To quantify one sample against a transcriptome without installing anything, ProteinIQ's Salmon tool runs the quantification stage on its own.

The samplesheet is a CSV with one row per sequencing run and a column for library strandedness (which DNA strand each read comes from, explained below):

csv
sample,fastq_1,fastq_2,strandedness,seq_platform
CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz,auto,ILLUMINA
Bash
nextflow run nf-core/rnaseq \
    --input samplesheet.csv \
    --outdir results \
    --gtf annotation.gtf \
    --fasta genome.fa \
    -profile docker

Both snippets follow the pipeline's own usage page, which also says that rows sharing a sample value are merged, so one sample sequenced over three lanes becomes one library.[2] The pipeline is one of the community-curated workflows in the nf-core framework, and its latest release, 3.27.0, was published on September 23, 2026.[3][1]

SalmonUpload a transcript FASTA and reads to get quant.sf with TPM and estimated counts.Open tool

What happens between FASTQ and counts

An RNA-seq run does not count genes. It counts fragments: each read, or read pair, is traced back to the transcript it most likely came from, and a gene's count is the number of fragments assigned to it. Every stage below either cleans the reads, decides where they came from, or turns those decisions into a table. Figure 1 shows the stages with the nf-core/rnaseq default tools and what each hands to the next.[2][4]

Figure 1. The stages of nf-core/rnaseq and what each one produces. Tool names are the pipeline defaults. The dashed route is pseudo-alignment, which skips STAR and the genome BAM. MultiQC collects the reports of every stage into one file.

Quality control and trimming

Before anything else runs, the pipeline checks every FASTQ file for formatting errors and stops if one is malformed, which catches truncated downloads early.[2] FastQC then reads each file and reports per-base quality, adapter content and duplication.[5] Adapters are the short synthetic sequences attached to each fragment during library preparation; a read that runs past the end of a short fragment reads into them.

Trim Galore, the pipeline's default trimmer, removes adapters and low-quality read ends. Setting --trimmer fastp swaps in fastp, which does adapter trimming and quality filtering in one fast pass.[6][4] The FastQC reports in the fastqc/ folder describe the raw reads, so adapters and low-quality tails are expected there; the reports in trimgalore/fastqc/ show the reads after trimming.[4]

Alignment or pseudo-alignment

By default the pipeline aligns reads to the genome with STAR, projects the alignments onto the transcriptome, and quantifies them with Salmon (--aligner star_salmon).[2][7] STAR is fast but memory hungry: the pipeline documentation quotes about 38 GB for the human GRCh37 reference.[2] The payoff is a genome BAM file, which feeds the pipeline's alignment-based QC such as strandedness checks and mapping summaries.

The alternative is to skip the genome. With --pseudo_aligner salmon, reads go straight to a transcriptome index, and adding --skip_alignment drops STAR entirely.[2] Salmon's selective alignment assigns each fragment to the transcripts it is compatible with and then runs a statistical model to split ambiguous fragments between them.[8]

This route skips the memory-hungry STAR step, at the cost of the genome BAM and the QC that depends on it. A third option, --aligner hisat2, needs less memory, but the pipeline does not quantify HISAT2 alignments, so it serves alignment and QC only.[2]

Transcript quantification with Salmon

Salmon writes a quant.sf file per sample. It is a tab-separated table with the columns Name, Length, EffectiveLength, TPM and NumReads, one row per transcript in the reference FASTA you gave it.[9] A standalone run against a transcriptome index looks like this, with -l A asking Salmon to infer the library type:

Bash
salmon quant -i transcripts_index -l A -1 reads1.fq -2 reads2.fq --validateMappings -o transcripts_quant

The command matches the Salmon manual, which also documents -r for single-end reads and recommends a decoy-aware index.[10] The pipeline builds that decoy-aware index itself when you supply --fasta; without a genome FASTA or a prebuilt index it indexes the transcripts alone, which its documentation calls usually not recommended.[2]

NumReads is the estimated number of fragments for that transcript. It is often not a whole number, because a fragment that fits two isoforms equally well is divided between them rather than thrown away.[8]

Gene-level counts with tximport

Differential expression is usually run per gene, so transcript estimates have to be summed. The step needs a tx2gene table linking each transcript ID to a gene ID, which the pipeline derives from your GTF and publishes as salmon.merged.tx2gene.tsv.[4]

The pipeline runs tximport for you and writes matrices such as salmon.merged.gene_counts.tsv, salmon.merged.gene_tpm.tsv and salmon.merged.gene_counts_length_scaled.tsv, plus R objects that hold the same assays.[4] Running it yourself on the per-sample files takes a few lines of R:[11]

r
library(tximport)
txi <- tximport(files, type = "salmon", tx2gene = tx2gene)

Here files is a named vector of paths to quant.sf files and tx2gene is a two-column data frame of transcript and gene IDs.

Summing is not the only reason for the extra package. Soneson, Love and Robinson showed that summarizing transcript-level estimates to the gene level improves downstream inference over counting reads per gene directly, and nf-core stopped using featureCounts for quantification in v3.0, citing its limitations and that paper.[12][2]

Give DESeq2 and edgeR counts, not TPM

TPM and counts answer different questions. TPM, transcripts per million, divides each transcript's estimated count by its effective length (roughly, the stretch of the transcript where a fragment could start), then rescales the results so that they sum to one million in every sample. With xix_i as the estimated count and l~i\tilde{l}_i as the effective length of transcript ii:

TPMi=106⋅xi/l~i∑jxj/l~j\text{TPM}_i = 10^6 \cdot \frac{x_i / \tilde{l}_i}{\sum_j x_j / \tilde{l}_j}

A toy case shows what the length term does. Two transcripts with 300 estimated fragments each but effective lengths of 1,500 and 500 nucleotides give ratios of 0.2 and 0.6, which sum to 0.8, so their TPM values are 250,000 and 750,000. Equal counts give a threefold difference in TPM, because each fragment from the short transcript stands for more molecules in the pool. The numbers are invented to show the formula.

Figure 2. Equal counts, unequal TPM. Two transcripts with 300 estimated fragments each but effective lengths of 1,500 and 500 nucleotides. The ratios of count to length, 0.2 and 0.6, sum to 0.8, so the shorter transcript takes 750,000 of the million. The numbers are invented to show the formula.

Match the matrix to the question. DESeq2, edgeR and limma model counts and correct for library size themselves, so differential expression starts from counts. Plots, heatmaps and comparisons between genes within one sample start from gene_tpm.tsv, which is already normalized for length and depth.[4][11]

The tximport vignette names two accepted routes into a differential expression tool. The first keeps the estimated gene-level counts and supplies a length offset; DESeqDataSetFromTximport builds that offset for you, and the vignette shows the equivalent steps for edgeR.[11]

The second route sets countsFromAbundance to "lengthScaledTPM" or "scaledTPM" and uses the resulting count matrix directly; the pipeline's gene_counts_length_scaled.tsv is the lengthScaledTPM version.[11][4] What the authors advise against is passing raw gene-level counts with no offset.[11] The DESeq2 vignette uses the first route:[13]

r
dds <- DESeqDataSetFromTximport(txi, colData = sampleTable, design = ~ condition)

In the vignette's example, sampleTable holds the sample information and ~ condition is the design formula; yours will name your own experimental variables. Do not feed TPM into DESeq2: its vignette says the input should be un-normalized counts or estimated counts of reads or fragments.[13]

Two exceptions apply. For 3' tagged protocols, where reads do not cover the transcript length, length correction would introduce bias, and the authors recommend using the original counts without an offset.[11] For the pipeline's per-sample mode, the lengthScaledTPM matrix is not published, because those values are not comparable across samples; recompute it by passing all quant.sf files to a single tximport call.[2]

Check the report before trusting the counts

The pipeline collects every tool's output into a single multiqc_report.html.[14][4] Three sections decide whether the counts are usable.

Strandedness comes first. A stranded library preserves which DNA strand each transcript came from, and a library analyzed with the wrong setting quietly loses or misassigns reads. The samplesheet accepts unstranded, forward, reverse or auto.[2]

With auto, the pipeline subsamples 1 million reads and runs Salmon to infer the orientation. It calls a sample forward stranded when at least 80% of fragments are forward, and unstranded when the two fractions differ by under 10%.[2] Anything in between is labeled undetermined, which can point to genomic DNA contamination. A sample whose inferred strandedness disagrees with the expected one is marked as a fail in the report.[2]

Mapping rate is second. The STAR section of the MultiQC report is expected to show mostly uniquely mapped reads and few unmapped ones.[4] Salmon's manual suggests that a mapping rate lower than expected is a reason to build the index with a slightly smaller k-mer.[10]

Third comes sample-level similarity. The pipeline runs DESeq2 on all samples without using group labels and plots a PCA and a sample distance heatmap, which can reveal batch effects or a swapped sample.[4] Table 1 lists problems these checks can point back to, with a fix for each.

ProblemSymptomFix
Wrong strandednessThe MultiQC strandedness check marks the sample as failedRerun with the strandedness the check found, or with auto
Genome FASTA and GTF from different releases or naming stylesFewer reads assigned than expected, or genes absent from the count matrixTake both files from the same source and release, and check that chromosome names match
Transcriptome that lacks isoforms or contains redundant recordsAbundance spread over near-duplicate transcriptsQuantify against a complete, non-redundant annotation
Index k-mer longer than the readsVery low mapping rateRebuild the index with a smaller k
Re-run of lanes under different sample namesThe same library counted twice as separate samplesGive lanes of one library the same sample value so they merge
Table 1. Common RNA-seq quantification problems, how they show up and what to do. The strandedness, k-mer and lane rows follow the pipeline and Salmon documentation; the two reference rows describe general failure modes. Sources: nf-core/rnaseq usage, Salmon documentation.[2][10]

Before the run, check the reference rows by hand: take the genome FASTA and the GTF from the same release, and compare the read length with the index k-mer. A quantifier can only assign reads to the transcripts and genes it is given, so a gene missing from the annotation is missing from the count matrix however many reads it had.

Quantify one sample with Salmon on ProteinIQ

ProteinIQ's Salmon tool runs the quantification stage of this workflow alone. You upload a transcript FASTA file as the reference, one record per transcript, plus single-end or paired-end reads (split libraries can be uploaded as several files per mate). It returns the quant.sf table with cmd_info.json and meta_info.json.

The settings cover library type (with auto-detect), index k-mer size, bootstrap replicates, mapping validation, and sequence and GC bias correction. Library types use Salmon's codes, such as ISR and OSR, built from the relative orientation of the two reads, whether the library is stranded and which strand read 1 comes from.[15]

Everything around that stage is left to you. The tool does no FASTQ QC or trimming, no genome alignment, no tximport gene matrices and no multi-sample MultiQC report or DESeq2 QC, all of which nf-core/rnaseq runs. It fits trying Salmon on one sample, quantifying a small library without setting up Nextflow, and confirming library type before a large run.

SalmonOne library per job: quant.sf, cmd_info.json and meta_info.json.Open tool

Each job quantifies one library. To use the results for differential expression, run every sample, download the quant.sf files and pass them to tximport with a tx2gene table. We did not measure the tool's upload limits or runtime for deep libraries for this page, so check the tool page before planning a full experiment around it.

After you have counts

With a gene count matrix, DESeqDataSetFromTximport builds the object that DESeq2 needs for a differential expression test, which models the counts of each gene and estimates fold changes between conditions.[16][13] The annotation chosen at the start still shapes that result, because it decides which of the roughly 20,000 protein-coding genes and which part of the human transcriptome the matrix can contain at all.

Frequently asked questions

What is a count in RNA-seq?

A count is the number of sequenced fragments assigned to a gene or transcript. Salmon's counts can be fractional because ambiguous fragments are shared between candidates.[8]

Do I need to align to the genome to get counts?

No. Salmon quantifies directly against a transcriptome, and nf-core/rnaseq can skip alignment entirely with --pseudo_aligner salmon --skip_alignment.[2] You lose the genome BAM and the QC that relies on it.

Should I give DESeq2 counts or TPM?

Counts. Import Salmon output with tximport and use DESeqDataSetFromTximport, which keeps the counts and adds a length offset.[11][13] TPM suits comparisons within a sample.

What strandedness should I choose?

Use the protocol of your library kit if you know it. If not, auto lets the pipeline infer it from a 1 million read subsample, and the MultiQC strandedness section shows what it found.[2]

Is nf-core/rnaseq the only option?

No. You can run FastQC, a trimmer, STAR or Salmon and tximport yourself. The pipeline packages the same kinds of tools behind one samplesheet and one MultiQC report.[3]