What is a protein language model, and what is it used for

Hide one amino acid in a protein sequence and ask a model to guess it. Repeat that across hundreds of millions of sequences from every branch of life, and a model that gets good at the guessing ends up encoding things nobody labeled for it: which residues sit in the core, which pairs touch in three dimensions, which substitutions a protein tolerates.

That is a protein language model (PLM): a neural network trained on amino acid sequences alone, with each residue treated as a token, the way a text model treats words.^[1][2]

A trained model gives you two things. Every residue gets an embedding, a vector that summarizes it in the context of its sequence, and ESM-2 returns those for any protein you paste in. The model also assigns a probability to each possible amino acid at each position, which is what scores mutations, drives the structure predictor ESMFold, and lets autoregressive models such as ProGen2 write new sequences.

How a protein language model is trained

The vocabulary is tiny. Each of the 20 standard amino acids is one token, with a few extra tokens for ambiguous or rare residues, gaps and the mask symbol. A sequence of 300 residues is 300 tokens, and the model never sees a structure, an alignment or a label. Everything it learns comes from which residues occur together in natural sequences.

In masked language modeling, a random subset of positions is hidden and the model predicts them from everything else in the sequence. ESM-2 includes each token with 15% probability; an included token is replaced by the mask 80% of the time, by a random amino acid 10% of the time and left unchanged 10% of the time.^[2] Training minimizes the negative log-likelihood of the true residues at the masked positions:

$$\mathcal{L}_{\text{MLM}} = -\,\mathbb{E}_{x}\,\mathbb{E}_{M}\left[\sum_{i \in M} \log p_\theta\!\left(x_i \mid x_{\setminus M}\right)\right]$$

Here $x$ is a sequence, $M$ the set of masked positions, $x_{\setminus M}$ the residues left visible and $\theta$ the model weights. The exponential of the average loss per token is the perplexity. A model that guessed uniformly among the 20 amino acids would score 20, and lower means the model is less surprised by real proteins.

Autoregressive models such as ProGen and ProGen2 instead read left to right and predict each residue from the ones before it:

$$\mathcal{L}_{\text{AR}} = -\sum_{i=1}^{L} \log p_\theta\!\left(x_i \mid x_{<i}\right)$$

This makes generation direct, since you sample one residue at a time, but each position sees only its left context. Masked models see both sides, which suits embeddings. ProtTrans trained both kinds, two autoregressive and four autoencoder models, to compare them.^[4] Diffusion models such as EvoDiff and the multimodal ESM3 sit between the two: they mask residues in random order or at a varying rate and learn to fill them in.^[3]

Why would predicting a hidden residue teach a model anything about structure? Residues that touch in the folded protein tend to change together over evolution, so the best guess for a masked position often depends on its 3D neighbors. A model that predicts well has to pick up those contact patterns, and across hundreds of millions of sequences it does.^[1][9]

The main model families

Table 1 lists the models most often used, with sizes and training data taken from each model's paper or official repository.

Model	Training objective	Sizes (parameters)	Training data
ESM-1b (2021)	Masked	650M	250 million UniRef50 sequences, 86 billion amino acids
ESM-1v (2021)	Masked	650M, five models with different seeds	UniRef90 (March 2020), 98 million sequences
ESM-2 (2023)	Masked	8M, 35M, 150M, 650M, 3B, 15B	UniRef50 (September 2021), sampling UniRef90 members
ESM C (December 2024)	Masked	300M, 600M, 6B	UniRef, MGnify and JGI clustered at 70% identity; 6.2 trillion tokens
ESM3 (2025)	Masked, multimodal	1.4B, 7B, 98B	2.78 billion natural proteins, 771 billion tokens with structure and function tracks
ProtT5-XL (2022)	Masked denoising, encoder-decoder	About 3B	BFD, then UniRef50 (about 45 million sequences)
ProGen / ProGen2 (2023)	Autoregressive	ProGen2: 151M to 6.4B	ProGen: 280 million sequences from over 19,000 families. ProGen2: over a billion proteins
ProstT5 (2024)	Translation between sequence and 3Di structure tokens	Fine-tuned from ProtT5-XL-U50	17 million AlphaFold DB proteins
EvoDiff (2023)	Diffusion in sequence space	38M, 640M	About 42 million UniRef50 sequences
ProFam-1 (2025)	Autoregressive over sets of related sequences	251M	ProFam-Atlas protein families

ESM-1b, ESM-2, ESM C and ESM3

The ESM line came from Meta AI's research group. Several of its authors later founded EvolutionaryScale, which has since joined Biohub. ESM-1b showed that a single masked model trained on 250 million sequences encodes secondary structure, contacts and mutation effects in its representations.^[1] ESM-2 scaled the same recipe to 15 billion parameters, and the larger models fold more accurately, which is how ESMFold became possible.^[2]

ESM C, announced in December 2024, keeps the masked objective but trains on metagenomic sequences from MGnify and JGI as well as UniRef. EvolutionaryScale's own contact-precision benchmark on CASP15 reports that ESM C 300M performs similarly to ESM-2 650M and that ESM C 600M rivals ESM-2 3B.^[15] That is the developer's benchmark, not an independent one.

ESM3 changes the input rather than the size. It reasons over sequence, structure tokens, function keywords and surface accessibility in one masked model, so you can prompt it with a partial structure or function and let it fill in the sequence. In the paper, a fluorescent protein designed this way (esmGFP) differs from its closest known relative, tagRFP, at 96 positions, which is 58% sequence identity.^[3]

ProtTrans and ProstT5

ProtTrans, from the Rost lab, trained six models on UniRef and BFD with up to 393 billion amino acids, using the Summit supercomputer and TPU pods. Its ProtT5 encoder became a widely used embedding model for prediction tasks that skip alignments.^[4]

ProstT5 fine-tunes ProtT5-XL-U50 to translate between amino acids and 3Di tokens, the structural alphabet that Foldseek uses to compare structures.^[7] It derives 3Di strings from sequence alone, three orders of magnitude faster than computing them from a predicted structure, according to its authors. ProstT5 runs both directions on ProteinIQ.

ProGen, ProGen2 and the generative models

ProGen added control tags to an autoregressive model so that sampling can be steered toward a family or function. ProGen2 scaled it to 6.4 billion parameters and argued that the distribution of the training data deserves more attention than size alone.^[5][6]

EvoDiff generates in sequence space by diffusion and can scaffold a motif or inpaint a region.^[24] ProFam-1 conditions on a set of related sequences instead of a single prompt, so the family acts as the context.^[25]

What the outputs are used for

Embeddings for function and localization

An embedding is the vector a model's hidden layers assign to a residue. Averaging over residues gives one fixed-length vector per protein, which can feed a small classifier or a clustering analysis. The ProtT5-XL encoder produces 1,024 values per residue, ESM-2 650M 1,280, and ESM C 960, 1,152 or 2,560 depending on size.^[23]

The ProtTrans paper tested embeddings as the only input to simple predictors. Secondary structure reached three-state accuracy of 81 to 87%, ten-state subcellular localization 81%, and membrane versus water-soluble 91%, and ProtT5 embeddings were the first to beat the state of the art in secondary structure prediction without multiple sequence alignments.^[4]

Embeddings are not interpretable one dimension at a time; they work because a classifier can find the right combination.

Zero-shot mutation effect scoring

The same probabilities that train the model can rank mutations without any labeled data. For a masked model, hide position $i$, then compare the probability of the mutant residue with the wild type:

$$\Delta_i = \log p\!\left(x_i = a_{\text{mut}} \mid x_{\setminus i}\right) - \log p\!\left(x_i = a_{\text{wt}} \mid x_{\setminus i}\right)$$

A positive $\Delta_i$ means the model finds the mutant more plausible than the wild type in this sequence. Meier and colleagues tested this with ESM-1v on 41 deep mutational scanning datasets. It beat the alignment-based DeepSequence model on 17 of them, and a paired test could not tell the two apart, with no alignment needed.^[8]

ESM-C Mutation Scoring runs this calculation for every substitution at every position, and ProGen2 scores whole sequences by autoregressive likelihood.

ProteinGym later put many methods on the same footing: over 250 assays covering substitutions and indels, with more than 2.7 million mutated sequences. Autoregressive models such as Tranception and ProGen2 tended to beat masked models like ESM in the zero-shot setting.

The best overall method, TranceptEVE, combined an autoregressive model with an alignment-based prior, and the alignment-based GEMME led several categories, including viral and non-human eukaryotic proteins and shallow alignments.^[10]

A likelihood is not a measurement. It tells you how typical a variant looks given natural sequences, which correlates with stability and activity but is not either one, and it says nothing about the assay conditions or the binding partner you care about.

Contacts and structure prediction

Rao and colleagues showed that contact information sits in the attention maps of ESM-1b. Fitting a logistic regression on top of the maps, with only 20 training proteins, identified 102 of 660 attention heads as predictive of contacts and beat the alignment-based Gremlin method on short and medium range contacts at every alignment depth. On long-range contacts it won when alignments were shallow (fewer than 1,000 sequences) or very deep.^[9]

ESMFold took this further. It feeds ESM-2 representations to a folding module that outputs atom coordinates, so the whole prediction needs one sequence and one forward pass. The released esmfold_v1 pairs the 3-billion-parameter ESM-2 with a folding trunk of about 690 million parameters.^[16] On CAMEO targets ESMFold averaged a TM-score of 0.83 against 0.88 for AlphaFold2, and on CASP14 0.68 against 0.85.^[2]

Speed is the trade. A 384-residue protein took 14.2 seconds on one V100, six times faster than a single AlphaFold2 model, and up to about 60 times faster on shorter sequences, excluding the alignment search that AlphaFold2 needs.

Meta used that speed to predict more than 617 million metagenomic structures for the ESM Metagenomic Atlas, with over 225 million predicted at high confidence.^[2]

Read the output with pLDDT as for any predictor. The ESMFold authors also found that the language model's perplexity correlates with prediction quality (Pearson r of −0.55 on CAMEO and −0.67 on CASP14), so a sequence the model finds surprising is a warning sign. Biohub's newer ESMFold2 builds a diffusion structure generator on ESM C representations.^[17]

Generation and design

Sampling from a trained model produces sequences that look like its training data. ProGen fine-tuned on five lysozyme families generated enzymes with catalytic efficiencies similar to natural lysozymes, at sequence identities to natural proteins as low as 31.4%.^[5] PepMLM adapts ESM-2 to propose peptide binders from a target sequence, and EvoDiff and ProFam add motif scaffolding and family-conditioned sampling.

Structure-conditioned designers such as ESM-IF1 are a related but different thing: they take a backbone as input, so they sit under inverse folding rather than sequence-only language modeling. For the wider workflow, see protein sequence design. Whichever generator you use, the output is a hypothesis. A high model likelihood does not make a protein express, fold or function.

Protein language models versus alignment-based methods

AlphaFold2 gets its evolutionary signal at prediction time. It searches sequence databases for homologs, builds a multiple sequence alignment, and reads coevolution from the columns.^[14] A protein language model gets the same kind of signal from its weights, learned once during training. That is why ESMFold needs only a sequence, and also why it cannot do better than what training put into the weights.

When the authors removed alignments from AlphaFold2 and RoseTTAFold, both degraded below ESMFold, which suggests the language model had absorbed much of what an alignment provides. The setting is artificial, though, and with alignments and templates AlphaFold2 stays more accurate on both benchmarks.^[2]

Situation	Protein language model	Alignment-based method
Thousands of sequences to screen quickly	Fits: one forward pass, no database search	Alignment search dominates the cost
Highest accuracy on a single well-studied target	Behind AlphaFold2 on CASP14 (TM-score 0.68 vs 0.85)	Fits: AlphaFold2 with deep alignments
Mutation ranking without labels	Competitive; autoregressive models lead the masked ones	GEMME and EVE-based methods remain strong, and the best overall method combines both
Protein with a rich family of homologs	Works, but gains less from size	Alignment captures family-specific constraints directly
Designed or engineered sequence with no natural relatives	Still returns a prediction, but accuracy is limited for orphans	Alignment is shallow or empty, so there is little signal to read

For the best available accuracy, AlphaFold2 is still the reference, though its weights carry licence terms to check before commercial use. For single-sequence prediction, a language-model-based predictor is the practical route.

Known limits

Bias toward well-studied species

A language model learns the distribution of its training database, and UniProt is uneven. Ding and Steinhardt compared the likelihood that ProGen2 and ESM-2 assign to orthologs of the same protein across the tree of life. Species identity alone explained 50% of the variance in ProGen2-xlarge likelihoods, and 67% after controlling for protein type. Fruit fly orthologs scored higher than C. elegans orthologs 92% of the time.^[11]

The effect carries into design. In simulated likelihood-guided optimization, designs drifted toward the species the model favors: starting from thermophilic proteins, 63% of designs had lower predicted melting temperatures. Those are in-silico predictions, not experiments, but they show why maximizing likelihood is not the same as improving a property.^[11]

What scaling does and does not buy

Larger models are better at predicting structure, but the gains are uneven. In the ESM-2 scaling analysis, proteins with many related sequences in the training set improved quickly and saturated at smaller model sizes, while proteins with few relatives kept improving as models grew.^[2] EvolutionaryScale reports no diminishing returns up to 6 billion parameters for ESM C.^[15]

For transfer learning with limited labeled data, size matters less. Vieira and colleagues found that ESM-2 650M and ESM C 600M fell only slightly behind ESM-2 15B and ESM C 6B across multiple datasets, and that larger models do not necessarily win when data is limited.^[12] For most embedding tasks a medium model is the sensible default.

Orphan proteins

Orphan proteins have no detectable relatives, and they make up 5% to 30% of eukaryotic proteomes. A 2026 evaluation on orphan proteins from root-knot nematodes (Meloidogyne) found that AlphaFold2 performed poorly, and so did the single-sequence tools ESMFold, OmegaFold and ProtT5. Secondary structure was more reliable than tertiary structure, with predictors agreeing on about 70% of elements.^[13] A language model does not conjure information that was never in its training data.

Sequence is not the whole protein

A PLM sees one chain. It does not know about the ligand, the partner, the pH or the cell type, so a mutation that looks neutral in isolation can break an interface the model never saw.

Licences for commercial use

Licences differ by model and sometimes between code and weights, so Table 3 records what each official source states. It is a summary of those pages, not legal advice.

Model	Code	Weights, as stated
ESM-1b, ESM-1v, ESM-2, ESMFold	MIT (repository archived August 1, 2024)	No separate weights licence on the repository page; Atlas data under CC BY 4.0
ESM C, ESM3-open	MIT (Biohub repository, Chan Zuckerberg Biohub copyright)	ESM C announcement states academic and commercial use under MIT; the ESM3 paper describes ESM3-open as for academic research use, and legacy EvolutionaryScale model cards for ESM C list an additional "other" licence tag
ProtT5 and the ProtTrans models	MIT	Academic Free License v3.0
ProGen, ProGen2	BSD-3	BSD-3 ("code and models")
ProstT5	MIT	MIT (stated for the project as a whole)
EvoDiff	MIT	Not stated separately
ProFam-1	MIT	Not stated on the repository page

The ESM3 and ESM C history is the case to watch. The ESM3 paper describes the open weights as provided for academic research use,^[3] while the current Biohub repository and model card state an MIT licence.^[17]

The ESM C announcement also says that hosted APIs and services fall under Biohub's Terms of Use, which are separate from the licence on downloaded weights.^[15] If you rely on a specific checkpoint commercially, read the licence text shipped with it and keep a copy.

Protein language model tools on ProteinIQ

Embeddings and representations

• ESM-2: per-residue and mean-pooled embeddings from ESM-2 650M
• ESM C: embeddings at 300M, 600M and 6B parameters
• ProstT5: sequence to 3Di translation, inverse translation and embeddings

Mutation scoring

• ESM-C Mutation Scoring: masked substitution scores, residue entropy and heatmaps
• ProGen2: autoregressive likelihood scoring of a whole sequence
• EvoIF: variant ranking from a backbone, a homolog alignment and an inverse-folding profile

Structure prediction

• ESMFold: single-sequence structure prediction with pLDDT and PAE
• ESMFold2: ESM C-based structure and complex prediction
• AlphaFold2: alignment-based prediction for comparison

Sequence generation

• ProGen2: sampling from a sequence context
• EvoDiff: unconditional generation, motif scaffolding and inpainting
• ProFam: family-conditioned generation from related sequences
• PepMLM: peptide binder candidates from a target sequence

Frequently asked questions

Is a protein language model the same as a large language model like ChatGPT?

They share the transformer architecture and the idea of predicting tokens, but a protein language model trains on amino acid sequences, not text, so its vocabulary is about 20 residues plus a few special symbols. It does not read English prompts. ESM3 extends the input with structure and function tokens, but it is still prompted in tokens, not sentences.^[3]

Which protein language model should I use?

It depends on the task. For embeddings on a small labeled dataset, a medium model such as ESM-2 650M or ESM C 600M is usually enough.^[12] For zero-shot mutation ranking, ProteinGym found autoregressive models ahead of masked ones, and combining a language model with alignment information performed best.^[10] For generation, ProGen2, EvoDiff or ProFam fit different inputs: a context, a blank to fill, or a protein family.

Do protein language models need an alignment or a structure as input?

ESM-2, ESM C, ProtT5 and ProGen2 take only a sequence. ProFam takes a set of related sequences. Structure-conditioned models such as ESM-IF1 take a backbone, which is why they are grouped with inverse folding.

Can a protein language model replace AlphaFold?

Not on accuracy. ESMFold scored a TM-score of 0.68 on CASP14 against 0.85 for AlphaFold2, but it runs without an alignment search and is many times faster. For screening many sequences, or for sequences with no homologs, that trade can be the right one.^[2]

Can I use ESM, ProtT5 and ProGen2 commercially?

The code is under permissive licences (MIT or BSD-3), but the weights are the question. ProtT5 weights are under the Academic Free License v3.0, and the ESM3 paper describes its open weights differently from the current MIT statements on the Biohub repository. See the licence table above and check the checkpoint you use.