Technical on Genomics x AI

Beyond coverage tracks: fine-tuning AlphaGenome's splicing heads from scratch

Fri, 24 Jul 2026 00:00:00 +0000

Summary

DeepMind has released AlphaGenome’s code and model weights, and the community has since developed alphagenome_ft and alphagenome-pytorch to enable seamless fine-tuning in both JAX and PyTorch. These implementations support bigwig-based modalities such as RNA sequencing (RNA-seq) coverage tracks, but other RNA-seq-derived outputs, including splice site probabilities, splice site usage, and splice junctions, require additional work, from data preprocessing and loading to validating fine-tuning on samples not represented during AlphaGenome pretraining. What initially seemed like a straightforward extension became a useful exercise in understanding how large genomic models learn and how to debug new output heads.

In this post, we share that development process: preprocessing the data, building loaders, running sanity checks, and overfitting a single interval, including the bugs we found along the way and how we fixed them. We then scale up to full fine-tuning and evaluation on held-out genomic intervals and report the resulting performance for two fine-tuning strategies.

All code, model adaptations, and pipelines used for this blog post are available.

Motivation

This post describes how we fine-tuned and evaluated AlphaGenome’s splicing heads, covering data preprocessing, loading, debugging, and testing on held-out genomic intervals. Our broader goal is to develop a workflow that can adapt AlphaGenome to new RNA-seq samples that were not represented during the model’s original pretraining. The workflow learns sample-specific, multimodal transcriptomic patterns from those data and predicts them at genomic intervals held out during fine-tuning.

As a case study, we selected two RNA-seq samples from López-Oreja et al. (2023): one carrying the cancer-driver mutation K700E in SF3B1, a core component of the spliceosome, and one wild-type sample. Although K700E changes only a single amino-acid residue, it can affect splicing across the transcriptome. In particular, the mutation is known to promote the use of cryptic splice sites, making it a useful test case for the RNA-seq-derived modalities considered here.

What comes from where

Since much of this work sits between “using AlphaGenome as published,” “reproducing details the paper doesn’t fully specify,” and “our own implementation choices,” here’s a breakdown of which parts of the pipeline fall into each category:

Component	Category
Junction-head architecture and candidate splice-site conditioning	Inherited from AlphaGenome, unmodified
STAR/deepTools-based alignment and coverage processing	Reproduced from the AlphaGenome paper’s methods, best-effort (original preprocessing code is not public)
Splice site usage / junction count extraction scripts, on-the-fly per-modality normalization	New implementation choices, specific to this project
`rope_params` truncated-normal initialization, ratio-normalized junction loss	Debugging fixes for a mismatch between the paper’s pseudocode and the JAX/PyTorch implementations — not new methodological contributions

AlphaGenome’s splicing heads

AlphaGenome predicts four RNA-seq-derived modalities: coverage, splice site class, splice site usage, and splice junction counts.

The coverage head predicts read depth along the sequence. The splice site head classifies each position as donor, acceptor, or background for each strand. The splice site usage head predicts how strongly each splice site is used, and the junction head predicts counts for candidate donor–acceptor pairs.

Coverage, splice site usage, and junction counts use sample-specific output tracks, whereas the splice site head is shared across samples. The junction head also receives candidate donor and acceptor positions as part of its input and predicts a normalized count for each candidate pair.

From raw data to preprocessed training tracks

AlphaGenome’s data preprocessing code is not publicly available, though the key steps are described in the paper’s methods. To reproduce the pipeline as closely as possible, we wrote standardized Snakemake workflows that derive all four training tracks from a single STAR RNA-seq alignment BAM file. Reads are aligned with STAR following the AlphaGenome paper’s alignment settings, retaining only uniquely mapped reads on canonical chromosomes.

Per-base RNA-seq coverage (stranded or unstranded) is computed using deepTools’ bamCoverage. Output: bigwig files.
Splice site classes are derived on the fly during data loading from the union of splice sites present in the splice site usage files, so no separate file is needed.
Splice site usage is computed from the BAM using our custom script compute_ssu.py, equivalent to SpliSER. For each splice site supported by at least one junction, usage is estimated as the fraction of reads supporting that site relative to reads that skip it. Output: zstd-compressed parquet files.
Splice junction counts are available directly from STAR at alignment time, or can be extracted post-hoc from the BAM using our custom script get_star_junctions.py. Output: tab-separated files.

To make sure our custom implementations to compute splice junction counts and splice site usage matched how STAR and SpliSER calculate them respectively, we ran a comparison on chromosome 1 on our samples.

In both cases we observe a Pearson correlation of 1 (for splice junction counts from uniquely mapped reads and for splice site usage), confirming the implementations are correct.

Both implementations were also highly efficient in runtime, with get_star_junctions.py requiring ~10s and ~150 MB per sample, and compute_ssu.py requiring ~20s and ~200 MB per sample compared to SpliSER’s ~600s and ~100 MB.

Data loading and on-the-fly normalization

All four tracks are loaded jointly for each genomic interval and normalized on the fly before being passed to the model. Getting this right matters: RNA-seq coverage spans several orders of magnitude across genes and samples, splice junction counts are extremely sparse, and splice site labels are binary signals at a handful of positions in a million-base-pair window. Each modality needs a different normalization strategy to avoid one dominating the loss or presenting the model with uninformative targets.

RNA-seq coverage bigwigs are read at 1 bp resolution. The raw values are loaded as-is; normalization happens at training time. Just before the loss is computed, targets are scaled into model prediction space by dividing by a per-track non-zero mean (computed once from the training set), applying a power transform (x^0.75) to compress dynamic range, and smooth-clipping extreme values. Model predictions are converted back to data space using the inverse of these operations for evaluation.

Splice site classes are derived on the fly as a 5-class label array: donor on the positive strand, acceptor on the positive strand, donor on the negative strand, acceptor on the negative strand, and background. All positions default to background, and annotated splice sites from the union of sites in the splice site usage files are then overlaid.

Splice site usage values are loaded from the precomputed parquet files. For each genomic interval, only the splice sites overlapping that interval are retrieved, and usage values are arranged into a per-position array with two channels per sample (one per strand).

Splice junction counts from STAR are CPM-normalized using the total mapped reads per sample, clipped at the 99.99th percentile, and then mean-scaled so that the typical non-zero value is close to 1. Junctions are assembled into a donor × acceptor count matrix, with forward and reverse strand channels interleaved across samples.

First time never works

As a first sanity check, we verified that the model could overfit a single genomic interval using linear probing (freezing the backbone and training only the new heads). A model with enough capacity should memorize a single batch; failure to do so points to a bug rather than a generalization problem. To make the test representative, we selected an interval with median splice junction density among all training intervals (see this notebook for the selection details).

The test was useful precisely because it failed. Coverage and splice site usage overfit fine on the first attempt; the splice site classification head did not overfit, and the splice junction head did not learn at all. The next two sections trace what was wrong with each of those two heads and how we fixed it.

Why the splice site head could not overfit

The splice site classification head struggled to overfit when initialized randomly. This is not surprising: the purpose of fine-tuning AlphaGenome is not to re-learn the general splicing code from scratch. Unlike the splice site usage and junction heads, which are sample-specific (AlphaGenome exposes them per tissue/track), the splice site classification head captures a single, shared property across all samples: where donor and acceptor sites sit on each strand. There is one general splice site head, not a track to pick per sample or cell type, so it makes sense to start from its pretrained weights directly and let the sample-specific heads adapt instead.

We explored three factors that could affect overfitting on the single interval: (1) initializing the splice site classification head from pretrained rather than random weights, (2) augmenting the interval’s splice sites with all annotated sites from the GTF (to reduce label sparsity, since AlphaGenome was pretrained on many more samples and thus more splice sites), and (3) segmenting the loss computation to match the 8-segment sequence parallelism used during AlphaGenome pretraining. We implemented support for all three options via finetune.py flags: --pretrained-head-samples to initialize specific head weights from a pretrained track (e.g. splice_site:0), --gtf to supply a GTF or parquet file of canonical splice sites, and --num-segments to control how many segments the sequence is split into for loss computation (applies to all modalities, not just splicing).

Initializing the splice site classification head from pretrained weights enabled clean overfitting regardless of whether GTF sites or loss segmentation were used. When starting from random weights, loss segmentation had a secondary effect: higher overall loss but faster overfitting, with or without the GTF augmentation.

Why the junction head could not learn

In our first attempt, the splice junction head would not learn. Understanding why required looking at how the head works. For each genomic interval, the trunk produces 1 bp resolution embeddings across the full sequence. The junction head extracts embeddings only at the positions of known splice sites (up to 512; padded with -1 if fewer are present). Each extracted embedding is then linearly transformed with a learned per-sample scale and offset before RoPE positional encoding is applied. Junction counts for each donor-acceptor pair are then predicted as the softplus of the inner product between the corresponding donor and acceptor embeddings.

Inspecting the pretrained weights revealed the problem: the junction head’s rope_params, which contain the learned scale and offset, were non-zero and approximately truncated-normal distributed, whereas a newly initialized head set both to zero. With scale = 0 and offset = 0, the transformation scale * x + offset collapses every splice-site embedding to zero before RoPE is applied, preventing the head from learning properly. We confirmed the pretrained parameter distribution for predicting K562 polyA-plus RNA-seq stranded tracks, chosen only as an available stranded pretrained initialization rather.

Cross-referencing with the original JAX implementation and reaching out to the authors (see GitHub issue) confirmed this was a bug also in the original implementation. The fix was to initialize rope_params with a truncated normal distribution (std=0.1), controlled via the --rope-init flag in finetune.py (default: truncated_normal; zeros is kept only for ablation experiments).

With the fix in place, the randomly initialized junction head overfit the interval comparably to a head initialized from the pretrained K562 weights when splice-site positions were taken from the sample’s own annotated, STAR-called junctions, using --junction-position-source annotated (the default). This position source is sample-dependent rather than GTF-derived: it uses the junctions observed in the sample to define the candidate splice-site positions for which the model learns to predict counts. We used the K562 weights only as a pretrained reference initialization, not because K562 is biologically matched to our samples.

Using predicted positions from the splicing head (--junction-position-source predicted) exposed a further issue: here it’s the pretrained-weight initialization of the junction heads that collapsed, while random initialization still learned. A possible explanation is that the pretrained junction heads were optimized alongside a dense, tissue-diverse set of splice site positions from pretraining, suggesting that positions predicted by a freshly fine-tuned splice site head on just two samples are sparser and differently distributed, likely placing it outside its operating range. Looking at the loss values pointed to a related problem: the junction cross-entropy was going negative during training, which would destabilize any head but hit the pretrained one harder.

The root cause seemed to be a mismatch between the paper’s pseudocode and both the JAX and PyTorch implementations. The supplementary methods define multinomial_cross_entropy by normalizing both targets and predictions to ratios before computing the cross-entropy, so log(p_pred) <= 0 always and the loss is guaranteed non-negative. The JAX implementation instead uses a log-normalizer formulation (log_normalizer - log_likelihood), which is mathematically equivalent when junction counts are present but produces a negative contribution when a training window contains annotated splice sites with zero observed counts; common in sparse datasets like SF3B1 RNA-seq, where not every annotated site will have reads in a given sample. The PyTorch port (--junction-loss original, the default) inherited this behavior unchanged.

We opened another GitHub issue with the authors, who confirmed the discrepancy and introduced a ratio-normalized formulation: both targets and predictions are divided by their within-mask sums before computing -p_true * log(p_pred). We ported this as --junction-loss normalized. Since p_pred <= 1, its log is always non-positive and the loss is always non-negative. With this correction, training is stable, the loss no longer dips below zero on our window, and both random and pretrained-weight initialization can overfit the single interval regardless of whether annotated or predicted splice site positions are used.

What actually worked: practical recipe for full fine-tuning AlphaGenome on new samples across RNA-seq modalities

After resolving the failure points identified above, we scaled up to a first genome-wide fine-tuning run using linear probing, with the trunk frozen and only the new output heads trained. We used the settings that performed most reliably during single-interval debugging:

pretrained initialization for the splice site head;
annotated junction positions;
truncated-normal initialization for the junction head’s RoPE parameters;
the normalized, ratio-based junction loss.

Using annotated junction positions gives the junction head a reliable training signal. Positions predicted by a splice site head that is still learning seemed to introduce an additional source of error. This follows AlphaGenome’s pretraining strategy of first training on annotated positions and later distilling to predicted ones.

We also used eight-way loss segmentation. Although the single-interval experiments did not establish it as necessary, we retained it for consistency with AlphaGenome pretraining as well.

Performance on held-out intervals improved over training, alongside a modest but consistent generalization gap: at every epoch, we compared test-set performance against a same-sized random subsample of training intervals (train_sample). At epoch 10, held-out test performance trailed the training subsample by roughly 0.06–0.10 Pearson r across all three modalities (gene expression, splice site usage, splice junctions), for both linear-probe and LoRA. Reassuringly, that gap stayed roughly constant across epochs rather than widening — i.e. the models aren’t progressively overfitting the training intervals with more training, though they also aren’t closing the gap to training performance either.

Gene expression performance saturated early, so we asked whether LoRA fine-tuning - updating the trunk’s internal DNA representation via low-rank adapters, rather than only the new output heads — could do better, particularly on the splicing-specific modalities. Comparing linear-probing against LoRA at epoch 10 on the held-out test set:

Metric	Linear-probe	LoRA	Δ (LoRA − probe)
Gene expression (Pearson r)	0.899	0.893	−0.006
Splice site usage (Pearson r)	0.646	0.659	+0.013
Splice junctions (Pearson r)	0.788	0.811	+0.023

As expected, LoRA, by updating the trunk’s internal DNA representation instead of leaving it frozen, improved performance over linear probing on the splicing-specific metrics (splice site usage, splice junctions), while gene expression stayed essentially unchanged between the two (linear-probe even edges it out slightly there).

The two strategies also differ substantially in compute footprint, though less in runtime than in memory. Full genome fine-tuning took about a week on a single GPU for both strategies, since backpropagating through the frozen trunk to reach the low-rank adapters in LoRA still requires storing its activations end to end, even though the trunk’s own weights aren’t updated. Where the two diverge is memory: LoRA needed a full NVIDIA H100 (80 GB), while linear probing, which only needs gradients for the new heads, ran comfortably on NVIDIA Hopper-class GPUs with 64 GB. See Xinming’s GPU benchmarking post for a closer look at how these memory/compute tradeoffs play out across fine-tuning strategies.

The workflows for full fine-tuning and evaluation are available in the repository: workflows/05-full_finetuning and workflows/06-evaluation.

Limitations

The initial debugging (up to and including the loss-formulation fix) was performed on a single genomic interval with just two RNA-seq samples, so those specific observations may not generalize to larger or more diverse training sets — this is why we followed up with the full genome-wide fine-tuning run above. The preprocessing pipeline reproduces the main steps described in the AlphaGenome paper, but since the original code is not public, we cannot guarantee exact parity.

The splicing fine-tuning code currently lives in a dedicated fork of alphagenome-pytorch and is pending merge into the main branch. Support in the JAX-based alphagenome_ft will follow once that merge is complete. We’ll update this post once both merges land, with any code or figure references that change as a result.

This post focuses on getting the implementation correct and measuring held-out performance, not on biological interpretation. We do not yet relate these predictions to independent data or ask whether the fine-tuned heads capture SF3B1-specific splicing changes — that comparison, and the accompanying biological interpretation, is left for the follow-up post. We also only evaluate two RNA-seq samples (one WT, one K700E) and a single AlphaGenome fold; generalization to more samples, mutations, or cell types remains to be tested.

Conclusion

This process gave us a much better understanding of what it actually takes to fine-tune new transcriptomic heads on AlphaGenome. Extending the model to splicing modalities was not simply a matter of adding data loaders and output layers (as we hoped): higher label sparsity, head initialization, loss formulation, and the interaction between position sources and weight initialization all turned out to matter.

The most useful strategy was to build confidence at each step before scaling up, from preprocessing validation and target inspection to single-interval overfitting. Each stage exposed a different class of problem. The single-interval test in particular was worth its weight as the two loss bugs and a zero-initialization bug in the junction head would have been much harder to diagnose in a full training run. We hope that sharing these intermediate failures, alongside the fixes and the flags that control them, makes it easier for others to extend AlphaGenome to new RNA-seq modalities and biological questions.

The full fine-tuning results also suggest that gene expression and splicing tasks don’t need the same amount of compute: gene expression performance saturated early and didn’t benefit from LoRA, so linear probing on modest hardware is likely enough there, whereas the splicing modalities kept improving and gained the most from LoRA’s larger compute and memory footprint. If gene expression is your main target, linear probing is probably all you need; splicing tasks are where the extra compute marginally pays off.

In the end, this project became more than a simple port. It resulted in reusable preprocessing scripts, training pipelines, and a set of practical checks and warnings that we hope make fine-tuning RNA-seq–derived splicing modalities on AlphaGenome more transparent and reproducible. We are especially grateful to the DeepMind developers for openly sharing their code and model weights, and for the responsiveness of Tom Ward (@tomwardio) and Vincent Dutordoir (@vdutor) when we reported bugs and implementation issues; their feedback was instrumental in reaching the conclusions presented here. We hope these efforts help make fine-tuning splicing heads as seamless and accessible as possible for the broader community.

Reproducibility

The repository alphagenome_finetuning_rna contains all the necessary code, from data downloading to analysis and figures, to reproduce these results.

References

Avsec, Ž. et al. Advancing regulatory variant effect prediction with AlphaGenome., 649, Nature (2026). https://doi.org/10.1038/s41586-025-10014-0
López-Oreja, I. et al. SF3B1 mutation-mediated sensitization to H3B-8800 splicing inhibitor in chronic lymphocytic leukemia., 6, Life Sciences Alliance (2023). https://doi.org/10.26508/lsa.202301955
Dent, C.I. et al. Quantifying splice-site usage: a simple yet powerful approach to analyze splicing., 3, NAR Genomics and Bioinformatics (2021). https://doi.org/10.1093/nargab/lqab041
Ramírez, F. et al. deepTools2: a next generation web server for deep-sequencing data analysis., 44, Nucleic Acids Research (2016). https://doi.org/10.1093/nar/gkw257
Dobin, A. et al. STAR: ultrafast universal RNA-seq aligner., 29, Bioinformatics (2013). https://doi.org/10.1093/bioinformatics/bts635

Acknowledgements

Thanks to the Genomics x AI community and the Kundaje Lab at Stanford, where the AlphaGenome PyTorch port is being developed as well. We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer MareNostrum 5, hosted by the Barcelona Supercomputing Center (Spain) through an EuroHPC Development Access call. We acknowledge support of the Spanish Ministry of Science and Innovation through the Centro de Excelencia Severo Ochoa (CEX2020-001049-S, MCIN/AEI /10.13039/501100011033), and the Generalitat de Catalunya through the CERCA programme, and to the EMBL partnership. We are grateful to the CRG Core Technologies Programme for their support and assistance in this work.

Cherimoya: a lightweight genomic sequence-to-function model built for large-scale design and understanding

Mon, 13 Jul 2026 00:00:00 +0000

Summary

Cherimoya is a lightweight sequence-to-function (S2F) model for chromatin accessibility, adapting the Chrom/BPNet architecture into a far smaller and faster model through a redesigned convolutional block and custom GPU kernels. At 614K parameters (under 10% of ChromBPNet), it is small and fast enough to make the millions of forward passes that variant scoring, interpretation, and sequence design demand a practical proposition, rather than optimizing for benchmark scores alone.

Three things stand out:

Smaller and better. Despite the tiny parameter count, Cherimoya beats ChromBPNet on both accessibility and variant effect prediction, and closes over half the gap to massive generalist models like Borzoi and AlphaGenome on counts prediction.
Fast enough to change what is feasible. Custom GPU kernels make it roughly 6x faster than ChromBPNet at inference, and a single model trains in about 10 minutes on genome-wide data. Running ISM on CPU is faster than running Borzoi on a GPU.
Usable without code. Command-line tools and agentic skills take you from raw reads to trained models, attributions, and motif calls without writing a line of Python.

Code: cherimoya

Genomic sequence-to-function (S2F) models use nucleotide sequence to predict genomic modalities, such as protein binding, chromatin accessibility, transcription, and alternative splicing. In recent years, these models have grown significantly in scale and can now make predictions for thousands of tracks simultaneously across hundreds of thousands of base pairs [1,2]. Although the details of these models’ architecture and training strategies differ significantly, there is usually one primary performance measure: how accurately they predict experimental readouts on held-out regions of the genome. But, in our opinion, these sorts of predictions are the least interesting usage of S2F models. After all, most assays for genomic modalities provide genome-wide readouts and there is no more “secret held-out genome” that one could make predictions for in the same way an image classifier can make predictions for new images.

Rather, we believe that the true value of S2F models is in their ability to do pretty much everything else: predicting variant (substitution, indel, structural) effects on phenotype, fine-mapping a set of candidate mutations to nominate putative drivers, dissecting the cis-regulatory logic encoded within individual loci of interest or globally, designing DNA with precise quantitative properties, prioritizing hypotheses by asking “what if?” questions before running experiments, etc. Predictive accuracy on held-out portions of the genome likely correlates somewhat well with performance on these tasks, but there are important distinctions. Exacerbating this issue, Rafi et al. [3] have raised concerns that the classic cross-chromosomal splits used for evaluation of S2F models may include homology leaks that sharpen this distinction, as model memorization of the training set translates into higher predictive performance on homologous regions in the validation and test sets but worse performance on downstream tasks.

Another challenge emerges when considering that each of these downstream tasks can require running S2F models thousands to millions of times. As an example, running in silico saturation mutagenesis on ~100k peaks of length ~1000bp requires ~300M forward passes. A model that is too big and/or too slow gives up much of its practical usefulness when it cannot be easily used at this scale, regardless of its benchmark score. In a world where academics are usually GPU-limited, especially for those whose research is not computationally focused, it is likely infeasible to use AlphaGenome for more than modest tasks. Taken together, these concerns mean that predictive performance should not be the only thing we consider when developing new S2F models.

These concerns motivated us to develop Cherimoya: a lightweight S2F model inspired by Chrom/BPNet [4,5]. Despite having only 614K parameters – less than 10% that of ChromBPNet – Cherimoya performs better at predicting observed chromatin accessibility readouts and at variant effect prediction. Through a combination of fewer parameters and custom GPU kernels, Cherimoya can run several times faster at its peak performance than ChromBPNet, and even faster than that when just matching performance. A single Cherimoya model can take as little as 10 minutes to train on genome-wide data, even when such data includes hundreds of thousands of peaks. This small footprint has huge consequences in terms of the scale of problems that one can tackle, and how quickly one can investigate interesting hypotheses. Finally, through a set of command-line tools and agentic skills, one can easily train and use Cherimoya models without needing to write a single line of code.

Model architecture, Cheri blocks & model optimization

Cherimoya models mimic the architecture of Chrom/BPNet models: an initial convolution layer with a wide kernel to capture long patterns, a stack of dilated residual layers, and a branch point where one path applies another convolution with a wide kernel to convert the internal representations into profile predictions, and on the other path a global average pooling followed by a dense layer is used to predict the log of the total number of reads mapping to the region. This multi-task framework allows Cherimoya to predict both the shape and strength of the signal in the same forward pass.

The largest architectural difference between Cherimoya and Chrom/BPNet is that, instead of using dense dilated convolutions, Cherimoya introduces the Cheri Block, which is an adaptation of the ConvNeXt [6] architecture. The Cheri Block consists of a 3-tap dilated depthwise convolution, which pools information spatially but independently for each channel, a layer normalization that gets applied independently on each example in the batch, and a channel mixing MLP similar to that of the transformer layer. This can be viewed similarly to a factorization of the dense convolution, with information first aggregated spatially, and then subsequently aggregated across channels. Finally, a small fixed-scale residual multiplies each channel to keep deep stacks stable by holding the network near the identity at initialization, as the contribution from each Cheri Block is initially low. We found that despite being a fixed parameter, removal of this scaling operation almost always leads to catastrophic collapse in the training process. An important note is that, even though there are several operations performed in each Cheri block, there is only one non-linearity per block.

Although most individual PyTorch operations already have efficient CUDA implementations that are better than what anyone could write by hand, new architectural blocks offer sequences of operations that can benefit from custom GPU kernels. To explain at a high level, I/O is expensive for GPUs, and so one can get a big speedup when a GPU can perform multiple operations on the same data before writing the results back to memory. Sometimes, this involves reordering how computation is done to minimize I/O operations, and can result in unintuitive sequences of operations. We provide two such kernels for Cherimoya: a simple kernel that fuses the 3-tap dilated convolution with the subsequent layer-norm operation (in red), and a megakernel that fuses together the entire layer with minimal intermediary values (in yellow). The simple kernel is useful for training models because it caches intermediate values useful for speeding up the backward pass; the megakernel is only for inference and benefits explicitly by eliminating several I/O operations. In our timings, the simple kernel is around 3-4x faster than the PyTorch operations for those two layers, and the megakernel is around 2x faster at inference than a Cheri block with the simple kernel.

Several changes had to be made to the optimization of Cherimoya models to get the best performance out of these Cheri blocks.

First, instead of using a heuristic to set the balance between the log count loss and the profile loss, as introduced in the BPNet paper, we use Kendall-Gal [7] learned loss to automatically balance them. Second, instead of using a single Adam optimizer, we used (1) Muon [8] for the internal 2D matrices, (2) AdamW optimizer for the rest of the model parameters, and (3) SGD for the Kendall-Gal loss weights. Third, autoresearch was used to refine the precise architectural units and optimizer hyperparameters. Over 5,000 experiments were run across several sessions using Evo (not to be confused with the Evo genomic LLM model). An entire blog post could be written about effective use of autoresearch, but here are four key points that we found:

You need an objective fast enough for quick iteration. Cherimoya was ideal in this sense because training a model on a reasonably sized dataset took only 5-20 minutes.
Training deep learning models is noisy, and so running each idea a single time is not always a good estimate of the performance when small differences matter. We ran each experiment three times with different initializations and if the median was higher than the frontier, we ran an additional seven times (ten total) and only incorporated the idea if the median across all ten runs was better than the current frontier.
Autoresearch will overfit to whatever objective you give it, and almost all of the changes that lead to frontier performance on the autoresearch evaluation set will not generalize. This is where domain knowledge-driven skepticism plays a huge role in doubting weird choices, and where extensive external validation is necessary for confirming that the proposals generalize.
Even when the proposed changes generalize to larger datasets, many of them led to significantly worse performance on downstream tasks such as variant effect prediction. Potentially, this is related to the homology leakage issue pointed out by Rafi et al.

Nonetheless, autoresearch is invaluable when used carefully for converting free GPU time into incremental improvements. Even when proposals were unsuccessful, knowing that one is close to the frontier is important because, when developing a new model, one is sometimes left wondering if they are just one small change away from a big performance boost.

Cherimoya sets a new Pareto frontier of performance and parameters

Everyone in machine learning is talking about “Pareto frontiers” these days, which are the curves showing the best tradeoff between model complexity and performance, so we decided to do the same when comparing Cherimoya and ChromBPNet models. We compared the performance of Cherimoya models with an increasing number of filters per layer against the performance of officially released ChromBPNet models trained by the lab that developed the model. To get a better estimate of the frontier, we averaged across ten chromatin accessibility experiments.

Our findings were somewhat surprising: Cherimoya reaches the same predictive performance for both tasks at 32 filters, using only 0.65% of ChromBPNet’s parameters. Performance improvements seem to level off around 128 filters, still less than 10% of the parameters of ChromBPNet, and so we used 128 filters as the default for modeling chromatin accessibility experiments moving forward. Somewhat interestingly, profile predictions continue to improve with complexity past the point where the count predictions level off. This may have implications for readouts where signal shape is more complex or informative, such as transcription initiation (e.g., Pro-CAP) experiments.

Cherimoya approaches the performance of massive models

These initial results suggested that ChromBPNet models are undertrained given their parameter budget, potentially due to their use of dense operations and simple optimization strategies, and raise questions about whether other genomic models may also be undertrained. To investigate this, we wanted to compare Cherimoya’s performance against massive models like Borzoi and AlphaGenome. Although these models do use other computational layers and are known to outperform ChromBPNet, they are likely too massive to truly optimize the hyperparameters and architecture.

In an attempt to define a shared task, we evaluated all models on the common task of predicting the log counts in experimental peaks for 467 DNase experiments using sequence centered on that peak. For each model, we used the full receptive field (1M for AlphaGenome, ~500k for Borzoi). Additionally, we included an AlphaGenome model that only takes in 2kbp of input sequence so as to more closely mirror ChromBPNet/Cherimoya. For each peak in the Borzoi validation set (which corresponds to the AlphaGenome test set), we used the respective ChromBPNet/Cherimoya model where that peak was also in the validation set, to ensure that no model had seen those coordinates in their respective training sets. We consider the validation set as opposed to the test set here because the work is not yet complete, but anticipate little difference in trends between the two.

We observe the expected trends, with AlphaGenome (median performance 0.724) and Borzoi (0.736) significantly outperforming ChromBPNet (0.666), and with ChromBPNet outperforming the shrunken AlphaGenome (0.582). Perhaps surprisingly, we see that Cherimoya (0.704) closes over half of the gap between ChromBPNet and the larger models despite using over an order of magnitude fewer parameters. Interestingly, AlphaGenome struggled with several lung experiments (in pink) that the other models did not find challenging, and all models struggled to make predictions in HL-60. This suggests that one downside of massive multi-task models is that it is challenging to ensure that the models perform well on all tasks without a few dropping out.

In doing this evaluation, we also considered the size of the model, the memory usage, and the inference speed, to provide a more comprehensive picture. Each model was benchmarked on the same H200 GPU using bfloat16 precision and torch.compile(mode='max-autotune') to ensure maximum throughput, with a warmup period ensuring that the first few slow inference passes do not influence the results.

Model	Params	Checkpoint	Peak VRAM (b=1)	Inference (ex/s)
Cherimoya	0.6M	2.5 MB	0.14 GB	~40,600
ChromBPNet	6.6M	26 MB	0.19 GB	~6,900
Borzoi	186M	744 MB	17.9 GB	~48
AlphaGenome (1Mb)	450M	921 MB	117 GB	~10

At ~40K examples per second, Cherimoya is about 6x faster than ChromBPNet and hundreds to thousands of times faster than the foundation models. This is made possible, in part, by the megakernel that fuses the entire Cheri block for inference. But it is not the only component: a forward+backward step at batch 512 takes ~71 ms, against ChromBPNet’s ~244 ms.

Cherimoya is a strong predictor of variant effect

As mentioned, predictive accuracy is only the first step in evaluating a model. Although it is encouraging that such a small model can perform so well on held-out data, we next evaluated Cherimoya’s performance at variant effect prediction. There are several datasets available for this task, but we chose to initially focus on one that was provided as part of the ChromBPNet work because it was well documented and simple to understand. In this setting, chromatin accessibility quantitative trait loci (caQTLs) were identified in African LCLs. Models trained to predict chromatin accessibility in LCLs from the ENCODE data were then used to score these caQTLs. A signed Pearson correlation was calculated over the difference in chromatin accessibility predictions between the reference and alternative alleles for the statistically significant caQTLs, in the same manner described in the original work; we did this for both a model trained on DNase-seq and one trained on ATAC-seq.

We see similar results here as in the previous prediction task: Cherimoya outperforms ChromBPNet, and AlphaGenome (all folds) performs the best. Although Cherimoya is not closing the gap quite as strongly here between ChromBPNet and AlphaGenome, it is still outperforming Borzoi and Enformer and providing a nice small boost relative to ChromBPNet.

Cherimoya enables interpretation and design with a low compute budget

A motivation for developing lightweight models is to reduce the cost of training and using genomic S2F models in downstream tasks. Accordingly, we next set out to evaluate the speed of these models on downstream tasks.

The first such task is interpreting loci. A popular method for doing this is DeepLIFT/SHAP [9], which implements a correction to gradients to get interpretable saliency maps that more closely match Shapley values. However, DeepLIFT/SHAP requires custom correction operations for each non-linear operation and, as of the time of writing, there is no confirmed correction operation for layer normalization (though such work is currently underway). Consequently, we must use in silico saturation mutagenesis (ISM) for attribution with Cherimoya models, which calculates the importance of each nucleotide as the average difference in prediction if it were each of the other three nucleotide choices.

A common critique of running ISM is that it is costly. Interpreting a 1kb locus requires running 3k forward passes. Fortunately, due to the megakernel, forward passes are where Cherimoya excels. To evaluate these timings more thoroughly, we compare the speed at which several models can interpret a 1kbp window within their original receptive fields. Below, we have the median speed of running ISM across 10 loci for several models, with both the GPU (at half precision) and CPU (at full precision) timings.

model	cuda	cpu
cherimoya	0.0846s	18.9110s
chrombpnet	0.3785s	70.9334s
alphagenome2kb	1.2375s	284.7192s
borzoi	64.4582s
alphagenome	324.2994s

Cherimoya is roughly 4.5x faster than ChromBPNet, in line with previous results. But perhaps more interestingly, running Cherimoya on a CPU is ~3x faster than running Borzoi on an H200 GPU. The ~20 seconds it takes Cherimoya on a CPU to do 3,000 forward passes may not be ideal for interactive analysis, but if one instead focused on 100 bp windows, a ~2 second cost is very manageable and brings the ability to interpret loci to those without a powerful GPU. Such speed even opens up the possibility of serving models on websites and having the compute done locally.

Next, we performed a similar analysis but with various design algorithms. Here, we chose to exclude the full Borzoi and AlphaGenome models as they would be extremely slow. With the remaining three models, we considered design tasks to increase the accessibility of a region using greedy substitution, a beam-search version of greedy substitution, ledidi [10], which is a gradient-based design algorithm, and simple screening wherein sequences are randomly generated and the best-scoring ones are returned.

model	greedy	ledidi	beam	screen (1000 it)
cherimoya	0.0737	0.0068	0.2982	0.0097
chrombpnet	0.4353	0.0242	1.3781	0.0769
alphagenome2kb	2.5078	0.0530	11.8330	0.0559

We see similar results as before, with Cherimoya being several times faster than the other models. As Cherimoya and ledidi were both designed with speed as a primary concern, it is not surprising that Cherimoya + ledidi is 64x faster than ChromBPNet + greedy substitution, which is a more common design algorithm, and ~370x faster than AlphaGenome 2kbp + greedy substitution.

Python API and command-line interface

Now that we’ve convinced you that Cherimoya performs well, let’s talk about how to actually use it in practice.

There are two main ways to use Cherimoya models: the Python API and the command-line tools. The Python API largely focuses on defining the model architecture and training process, saving and loading models, while outsourcing the usage of the model after training to tangermeme [11], which is a general-purpose package for using S2F models. Although one can use the Python API directly to train models, our lab usually uses the command-line tools to get started and then loads the trained models into Python if we need to do bespoke analyses.

The command-line tools include the following commands:

cherimoya attribute: Runs in silico saturation mutagenesis on a set of loci
cherimoya evaluate: Evaluates a trained model against observed data
cherimoya fit: Fits a model to observed data
cherimoya marginalize: Runs in silico marginalizations given a motif database
cherimoya negatives: Identifies GC-matched negatives for a set of peaks, no model involvement
cherimoya seqlets: Calls seqlets on attributions (from any methods) using the recursive seqlet caller in tangermeme

Each can be run individually, but likely the simplest way to go from raw data to trained models and outputs is via the cherimoya pipeline command, which handles data processing, model training, interpretation, seqlet calling/annotation, TF-MoDISco, and marginalizations by running through the above commands. The command is extremely flexible: you can start with read/fragment files or with bigWig files, you can pass in a control track if you have it, you can pass in peaks or have them called for you using MACS3, and likewise you can pass in negatives or have them called using a GC matching strategy based on the peaks. This graphic illustrates the flexibility one has in getting to a trained model.

The biggest obstacle for using these command-line tools is that each command requires a JSON that specifies paths to the relevant data and includes important hyperparameters. Defaults for these can be found on GitHub. Because the pipeline command requires a massive JSON – including one internal JSON for each step – a helper utility, cherimoya pipeline-json, can take in links to the important data and parameters and produce a template JSON that one can subsequently manually refine before running the cherimoya pipeline command.

Quickstart guide

To demonstrate this in action, let’s look at an example of using the command-line tools to train and run Cherimoya models on an ATAC-seq experiment using the pipeline command. For this example, we will use the deeply sequenced K562 ATAC-seq experiment on the ENCODE Portal.

First, we need to download the relevant files: the reference genome, a motif database (for characterizing the model-identified motifs), and the reads in BAM format.

curl -Z -L -C - --fail --remote-name-all \
https://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/latest/hg38.fa.gz \
https://jaspar.elixir.no/download/data/2026/CORE/JASPAR2026_CORE_vertebrates_non-redundant_pfms_meme.txt \
https://encode-public.s3.amazonaws.com/2021/03/16/781bf763-825b-41df-9e58-c15a3e16f365/ENCFF534DCE.bam \
https://encode-public.s3.amazonaws.com/2021/03/16/d0607f6a-198b-4c38-94ba-9161359a5500/ENCFF128WZG.bam \
https://encode-public.s3.amazonaws.com/2021/03/16/e5948cab-16cc-4bfd-b1fb-93f192ca62d2/ENCFF077FBI.bam \
&& gunzip --keep -f hg38.fa.gz

After downloading these files, we use cherimoya pipeline-json to create the pipeline JSON.

cherimoya pipeline-json \
-s hg38.fa \
-i ENCFF534DCE.bam -i ENCFF128WZG.bam -i ENCFF077FBI.bam \
-m JASPAR2026_CORE_vertebrates_non-redundant_pfms_meme.txt \
-n k562_atac -o k562_atac.pipeline.json \
-ps 4 -ns -4 -u -pe

This will create a properly formatted pipeline JSON file that is ready to run. Before doing so, you should manually skim it over and ensure there is nothing else you would like to change (such as batch sizes or any other processing or model settings that may be relevant but not exposed). When you’re ready to run the pipeline, you can run cherimoya pipeline -p k562_atac.pipeline.json and the steps will be performed sequentially. As a final note, a "skip": true argument can be passed into any JSON if you want to skip the step, and a trained model can be provided to skip model training and just perform the other steps.

When the pipeline is complete, you will have several outputs: the trained model, attributions for the specified loci, annotated seqlet calls using tangermeme’s recursive seqlet caller and counts for the number of each annotation (i.e., how many motif instances of each type in a database were found highlighted by the attributions), in silico marginalizations for all motifs in a database, and TF-MoDISco results for de novo discovered motifs. For this example, the first few rows of the TF-MoDISco report should look like the following, and correspond to several motifs known to drive accessibility in K562.

Agentic tooling

The Cherimoya repository comes with built-in skills that use progressive disclosure to guide the agent in the right direction based on your prompt. We have found this to be immensely useful in creating scripts that we then refine, and when the agent wants to create small snippets to verify something. It is beautiful to see these agents go from writing long bespoke scripts that are challenging to audit to a small number of function calls from a package like Cherimoya or tangermeme. However, it is challenging to guarantee behavior from agents, especially as the underlying models are constantly changing. So, other than to mention that such skills exist and we have been happy with them internally so far, we will not go further here.

Wrap up

Cherimoya is a lightweight S2F model that achieves strong performance despite its small parameter count. We have adopted it internally in the Programmable Genomics Laboratory across a range of projects, and plan to continue expanding its scope, performance, and capabilities. To write like an AI: this is not just a one-off project; it’s a foundation for future work in the lab. As we scale up our design work from a small number of cell lines to targeting every cell type in a complex mammalian organ, such as the brain, lightweight models that can run quickly and fit easily in GPU memory are crucial.

Cherimoya is from Christian Ramirez, Achsah Marlene Aruva, Zhiping Weng, and Jacob Schreiber (UMass Chan Medical School). Model and CLI code live in the cherimoya repo; analysis tooling builds on tangermeme. Work is still in progress, and the final numbers and results will likely differ slightly from this blog post. Thanks to Anthropic for providing credits through their AI for Science program.

References

1. Linder, J., Srivastava, D., Yuan, H., Agarwal, V. & Kelley, D. R. Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation. Nat. Genet. 57, 949–961 (2025). https://doi.org/10.1038/s41588-024-02053-6

2. Avsec, Ž., Latysheva, N., Cheng, J., Novati, G., Taylor, K. R., Ward, T. et al. Advancing regulatory variant effect prediction with AlphaGenome. Nature 649, 1206–1218 (2026). https://doi.org/10.1038/s41586-025-10014-0

3. Rafi, A. M., Kiyota, B., Yachie, N. & de Boer, C. G. Characterizing homology-induced data leakage and memorization in genome-trained sequence models. bioRxiv (2026). https://doi.org/10.1101/2025.01.22.634321

4. Pampari, A., Shcherbina, A., Kvon, E., Kosicki, M., Nair, S., Kundu, S., Kathiria, A. S., Risca, V. I., Kuningas, K., Alasoo, K., Greenleaf, W. J., Pennacchio, L. A. & Kundaje, A. ChromBPNet: bias factorized, base-resolution deep learning models of chromatin accessibility reveal cis-regulatory sequence syntax, transcription factor footprints and regulatory variants. bioRxiv (2024). https://doi.org/10.1101/2024.12.25.630221

5. Avsec, Ž., Weilert, M., Shrikumar, A., Krueger, S., Alexandari, A., Dalal, K., Fropf, R., McAnany, C., Gagneur, J., Kundaje, A. & Zeitlinger, J. Base-resolution models of transcription-factor binding reveal soft motif syntax. Nat. Genet. 53, 354–366 (2021). https://doi.org/10.1038/s41588-021-00782-6

6. Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T. & Xie, S. A ConvNet for the 2020s. arXiv (2022). https://arxiv.org/abs/2201.03545

7. Kendall, A., Gal, Y. & Cipolla, R. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. arXiv (2017). https://arxiv.org/abs/1705.07115

8. Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L. & Bernstein, J. Muon: An optimizer for hidden layers in neural networks. (2024). https://kellerjordan.github.io/posts/muon/

9. Shrikumar, A., Greenside, P. & Kundaje, A. Learning Important Features Through Propagating Activation Differences. arXiv (2017). https://arxiv.org/abs/1704.02685

10. Schreiber, J., Lorbeer, F. K., Heinzl, M., Reiter, F., Rafanel, B., Lu, Y. Y., Stark, A. & Noble, W. S. Programmatic design and editing of cis-regulatory elements. bioRxiv (2025). https://doi.org/10.1101/2025.04.22.650035

11. Schreiber, J. tangermeme: A toolkit for understanding cis-regulatory logic using deep learning models. bioRxiv (2025). https://doi.org/10.1101/2025.08.08.669296

Benchmarking seq2func models on distal enhancer effects with CRISPRi screens

Thu, 25 Jun 2026 00:00:00 +0000

Summary

We benchmark four sequence-to-function genomic deep learning models — Enformer, Borzoi, NTv3, and AlphaGenome — zero-shot on two K562 CRISPRi enhancer-knockdown screens (Fulco et al., 2019 and Gasperini et al., 2019), extending the in-silico CRISPRi setup from Karollus et al., 2023.

Three things stand out:

AlphaGenome leads on both screens (Pearson’s r = 0.67 on Fulco et al., 0.45 on Gasperini et al.), with Borzoi a close second on Fulco et al. and a more distant second on Gasperini et al.
All four models systematically underpredict knockdown magnitude, and the gap to experimental measurements widens with enhancer-to-TSS distance — distal cis-regulatory element (CRE) effects remain difficult to predict.
AlphaGenome’s RNA-Seq head (with GENCODE-exon aggregation) beats its CAGE head on the larger screen — which output track you choose matters.

Code: seq2func_crispri_eval

Motivation

Sequence-to-function (seq2func) models keep getting bigger, longer-context, and more capable. AlphaGenome is the current state of the art, succeeding Enformer (2021) and Borzoi (2024). DNA language models like NTv3 now compete in the same space by post-training on the same functional genomic tracks.

AlphaGenome was launched as a substantial step forward, with most of its zero-shot evaluation focused on single-nucleotide variant (SNV) effects. Two related weaknesses are especially well-documented for this class of model, both originally demonstrated on Enformer: predicting expression across personalised genomes (e.g. Sasse et al.), and predicting distal CRE effect magnitudes (how much knocking down a far-away enhancer changes its target gene’s expression). Both involve sequence perturbations the model wasn’t directly trained on, and both are where seq2func models historically struggle. Performance on these tasks is what tells us whether a model’s apparent advance translates to the questions biologists actually care about.

The personalised genome question is how well a model predicts gene expression for a specific individual from their own genome. It’s hard because the variants that distinguish one individual from the reference are sparse and small in effect, easy to lose against the much stronger reference signal a sequence model is trained to predict. Tu, 2026 and Shen, 2025 recently revisited it: AlphaGenome improves over Enformer on this task, but still falls well short of useful accuracy. The needle moves, but there’s a long way to go.

The distal-CRE magnitude question — much bigger perturbations, disabling a whole regulatory element — has received little attention since Karollus et al., 2023. They defined an in-silico CRISPRi benchmark on Fulco 2019 and Gasperini 2019 and ran it on Enformer and Basenji2, finding both substantially undershoot real enhancer effects. Borzoi, NTv3, and AlphaGenome have all followed Enformer, claiming gains from longer contexts and broader predictive tasks — yet none have been evaluated on Karollus’s magnitude benchmark.

A related but easier task — enhancer-gene linking — has been tested more widely: in the original Enformer paper (Figure 1a), in AlphaGenome’s evaluation on the ENCODE-rE2G CRISPRi dataset (Figure 1b), and in the DNALONGBENCH benchmark. Linking is binary classification: does enhancer X regulate gene Y? A model succeeds by sorting interacting from non-interacting pairs above some threshold. Magnitude prediction is regression: how much does Y drop when X is disabled? It demands ranked and calibrated effect sizes across a continuous range, committing the model to a quantitative theory of enhancer-promoter regulation rather than a topological one. AlphaGenome leads on linking and notes that even it underestimates the impact of very distal enhancers — but doesn’t run the magnitude-prediction benchmark.

We aim to address this. We re-run the Karollus benchmark on AlphaGenome, Borzoi, and NTv3 — and on Enformer, to anchor against Karollus’s original numbers. The result is a measure of progress on distal-CRE prediction across the last four years of seq2func releases, not just a test of AlphaGenome’s generalisation abilities.

Side note: CRISPRi (CRISPR interference) uses a catalytically dead Cas9 fused to a repressive domain to silence a target locus without cutting DNA. When the target is a distal enhancer, you can read out which genes go down in expression — giving you a measured enhancer → gene effect size.

Side note: “seq2func” = sequence-to-function. A model that takes raw DNA as input and predicts functional genomic signals (RNA-seq, CAGE, ATAC, ChIP, etc.) as output. Enformer, Borzoi, and AlphaGenome were all trained end-to-end as seq2func models. NTv3 is a hybrid: first pretrained as a genomic language model (gLM) via masked language modeling — predicting hidden bases from their surrounding sequence across many species’ genomes — then post-trained on functional assays to produce seq2func outputs.

The benchmark

We test on two K562 CRISPRi enhancer screens:

Fulco et al., 2019 — ~60 validated enhancer–gene pairs from K562 CRISPRi-FlowFISH. Small but high-confidence.
Gasperini et al., 2019 — ~440 high-confidence significant pairs from pooled K562 CRISPRi at scale.

For each pair, the procedure is:

Build a sequence window of the model’s native context (196 kb for Enformer, 524 kb for Borzoi, 1 Mb for NTv3 and AlphaGenome), centred on the gene’s TSS.
Score the wild-type window.
Dinucleotide-shuffle a 2 kb slice covering the enhancer. Repeat N = 50 times, average the model’s prediction.
Aggregate the model output over K562 RNA-Seq tracks in a TSS-centred 640 bp window. For AlphaGenome we additionally use the mean signal across GENCODE exons of the target gene (their paper’s preferred RNA-Seq score).
Score pred_delta = (WT − mean(shuffle)) / WT per pair and correlate with the measured fractional knockdown.

Side note: This is a marginalisation procedure — by averaging predictions across many randomised backgrounds in which the enhancer’s motif content is destroyed, we isolate the enhancer’s contribution from the surrounding sequence context (the same idea underlies Global Importance Analysis (GIA); Koo et al., 2021). The background choice matters: N-replacement is out-of-distribution for the model and can produce strange predictions; drawing random bases changes local G+C content, biasing on composition rather than motif loss; even a plain base shuffle destroys dinucleotide frequencies like CpG counts, themselves a learned regulatory signal. Dinucleotide shuffling (Altschul–Erickson) destroys motif content while keeping both single-base and dinucleotide composition intact — the standard background for in-silico CRISPRi.

This protocol is a direct extension of Karollus 2023 — most of it is bit-for-bit faithful to theirs. We reuse their Fulco et al. evaluation tables (ziga_additional_columns.tsv + enhancer_knockdown_effects.tsv from their Zenodo release) with the same merge keys and validated-pair filtering, their TSS / enhancer / strand conventions, the same Enformer DeepMind checkpoint, and their K562 CAGE readout, central-bin aggregation, and Pearson/Spearman correlation framework. For Gasperini we apply the same protocol to the Cell 2019 high-confidence pairs (after hg19 → hg38 liftover). The model-specific adjustments — per-architecture aggregation conventions like 5×128 bp bins for Enformer, 20×32 bp for Borzoi, exon-mean for AlphaGenome RNA-Seq — are architecture-driven; the evaluation harness around them is the Karollus harness. Four deliberate departures from Karollus are flagged in Limitations.

Each model is evaluated using its canonical published inference setup, with no test-time augmentations applied (TTA is explored separately for AlphaGenome below). Inference compute per pair therefore isn’t uniform: Borzoi runs four forward passes (one per fold), while the other three run one — we compare models as their authors released them rather than artificially restrict Borzoi to a single fold. Enformer uses the lucidrains PyTorch port of the DeepMind weights — same weights as Karollus used, different framework wrapper. Borzoi predictions come from the Flashzoi community PyTorch port, evaluated as the original paper’s 4-fold ensemble (predictions averaged across the four published fold checkpoints). AlphaGenome uses the PyTorch port loaded with the all-fold distilled checkpoint, a single model trained to reproduce the multi-fold ensemble’s behaviour. NTv3 is the InstaDeepAI HuggingFace release.

Results

AlphaGenome leads, with Borzoi close behind on Fulco et al. Figure 2 shows predicted vs. measured fractional knockdown for all four models on each screen, with points coloured by enhancer-to-TSS distance.

The headline numbers (left two columns: each model on every pair its own receptive field can reach; right two columns: every model restricted to Enformer’s 196 kb receptive field, so all four score the same set of pairs):

Model	Fulco — full	Gasperini — full	Fulco — Enformer RF	Gasperini — Enformer RF
Enformer	0.60 / 0.31	0.30 / 0.27	0.60 / 0.31	0.30 / 0.27
Borzoi	0.66 / 0.49	0.34 / 0.33	0.66 / 0.49	0.34 / 0.35
NTv3	0.34 / 0.09	0.12 / 0.14	0.34 / 0.09	0.13 / 0.17
AlphaGenome	0.67 / 0.54	0.45 / 0.45	0.67 / 0.54	0.46 / 0.48

Values are Pearson’s r / Spearman’s ρ. N pairs in the “full” columns varies by model because larger-context models score extra distal pairs that Enformer can’t see (Fulco N = 62–63, Gasperini N = 352–438); in the “Enformer RF (Receptive Field)” columns N is uniform (Fulco et al. N = 62, Gasperini et al. N = 352), restricted to the enhancers Enformer predicts.

The matched-RF columns rule out the simple “they just see more sequence” explanation for AlphaGenome and Borzoi’s lead: Fulco et al. barely moves (only one distal pair drops), and on Gasperini et al. the rankings are unchanged — AlphaGenome even nudges up slightly (Pearson 0.45 → 0.46, Spearman 0.45 → 0.48).

A few takeaways:

AlphaGenome outperforms all other models on both screens, though essentially tied with Borzoi on Fulco et al. — see the small-sample caveat in Limitations.
NTv3 is the weakest on Gasperini et al. NTv3 has a 1 Mb context, the same as AlphaGenome, but still trails the leaders by a wide margin (Pearson’s r 0.13 on the matched-RF set) — so context length isn’t the explanation. Enformer, despite its shorter 196 kb context (which excludes ~20% of Gasperini et al. pairs), lands mid-pack on the pairs it can see (Pearson’s r 0.30), close to Borzoi (0.34).
On the same set of pairs, AlphaGenome still leads. Of Gasperini et al.’s pairs, 21 fall within AlphaGenome’s 1 Mb context but outside Borzoi’s 524 kb — so they contribute to AlphaGenome’s correlation but not Borzoi’s (Note, all of the tested Fulco et al. pairs are in Borzoi’s context). To make the comparison apples-to-apples, we restrict AlphaGenome to only the pairs Borzoi can also score: Pearson’s r 0.45 → 0.44, Spearman’s ρ 0.45 → 0.45 — giving essentially the same result.
The Gasperini et al. ceiling is low across the board. Even AlphaGenome leaves substantial variance unexplained. Some of this is likely measurement-side: Gasperini et al.’s scaled screen uses a high-MOI single-cell pooled design (median 28 gRNAs per cell), and the statistical and trans-perturbation challenges of high-MOI screens are well-documented (Barry et al., 2021). Fulco et al.’s CRISPRi-FlowFISH targets one enhancer at a time with a bulk-population readout, which we’d expect to give more precise per-pair effect sizes. We can’t separate measurement noise from model error from these correlations alone, so attributing the gap is interpretive — but it’s at least consistent with the per-screen difference we see. Either way, the headline framing is that AlphaGenome improves CRISPRi prediction but a large gap remains, especially for distal cis-regulatory elements (CREs).

Predictions shrink, knockdowns don’t

Figure 3 plots observed and predicted effect against enhancer-to-TSS distance, to more closely match Karollus et al.’s plotting style.

Two patterns hold across all four models:

Predictions are systematically smaller in magnitude than observed knockdowns. Even where the rank ordering is right (good Spearman), the predicted effect sizes are compressed.
The gap grows with distance. Models capture some of the distance-decay biology, but the slope is shallower than the data demands — most obviously for NTv3, less so for AlphaGenome.

A short aside: AlphaGenome’s RNA-Seq head, aggregated as the mean signal across the target gene’s GENCODE exons (their paper’s preferred approach), beats AlphaGenome’s CAGE head on Gasperini et al. (Pearson’s r 0.45 vs. 0.39). On Fulco et al. the difference is small (0.67 vs. 0.66). So if you’re benchmarking a new model on these screens, your output-aggregation choice should be considered carefully!

Scoring it two ways

A second aside: we computed the predicted effect two ways. Figure 2 used the linear pred_delta = (WT − mean(shuffle)) / WT. We also computed log₂(WT / mean(shuffle)) against −log₂(1 − y_delta) — a log fold-change scaling that de-emphasises a few high-end outliers. Figure 5 shows the log₂ version. Spearman is identical between the two scorings (rank-invariant); Pearson can shift noticeably, especially on Gasperini, and AlphaGenome and Borzoi narrowly swap order on Fulco — though the broad picture (AlphaGenome and Borzoi well above NTv3) is robust.

Test-time augmentation: stabilising AlphaGenome

Karollus et al.’s original benchmark applies test-time augmentations (TTA) — for each sequence, run the model on 3 small offsets (−43, 0, +43 bp) crossed with both orientations (forward + reverse-complement), then average the 6 predictions. The 43 bp shift isn’t arbitrary: Enformer’s output is binned at 128 bp, and 128 / 3 ≈ 43, so the three offsets sample the TSS at roughly even thirds within its bin, averaging out where exactly the landmark falls. Genomic seq2func models have small but real positional sensitivity — predictions wobble across shifts — and TTA reliably damps this. Past work has shown the same effect in adjacent settings (Toneyan et al., 2022).

Side note: Test-time augmentation (TTA) means running the model on slightly perturbed versions of the same input and averaging the outputs, so the final prediction is less dependent on incidental properties like exact bin alignment or input orientation.

The headline benchmark (Figures 2–5) doesn’t use TTA, so all four models are compared on equal footing. To see what stabilising AlphaGenome’s predictions buys on this task, we ran a separate analysis applying Karollus et al.’s 6-pass recipe to AlphaGenome alone. Figure 6 shows the result: TTA improves AlphaGenome’s headline correlations on both screens, lifting Pearson’s r on Fulco from 0.67 to 0.69 and Spearman on Gasperini from 0.45 to 0.47.

Does ensembling help?

On the matched-RF (Receptive Field) set used in the headline table, a simple equal-weight average of the four models’ predictions doesn’t beat the best single model on either screen: Pearson’s r is 0.66 on Fulco (just below AlphaGenome’s 0.67) and 0.42 on Gasperini (below AlphaGenome’s 0.46). On Gasperini, where AlphaGenome dominates, averaging in the weaker models clearly dilutes the predictive signal; on Fulco, where the models are more comparable, the average still lands fractionally below the leader rather than above it. Equal-weight ensembling, in other words, doesn’t buy you anything over just using AlphaGenome here.

Limitations

A few things to flag about our analysis.

Differences from Karollus 2023. We cross-checked our Enformer reimplementation against Karollus’s original code line-by-line with an LLM (Claude Opus 4.7). Our Enformer numbers don’t exactly match their published ones but the qualitative pattern is the same — Enformer underpredicts distal enhancer effects. Four deliberate departures from their approach may explain some of this gap:

Window construction. Karollus reads precomputed sequence_start/sequence_end from a fixed table, designed so both TSS and enhancer sit inside Enformer’s central crop. We build the window on-the-fly as strict TSS-centred so that all models were compared equally.
TTA only for AlphaGenome. Karollus et al. applies TTA (6 forward passes per sequence) uniformly across all evaluated models. Our headline four-model comparison (Figures 2–5) does not apply TTA to any model — each model is evaluated using its standard published setup: single forward pass for Enformer, NTv3, and AlphaGenome (the distilled checkpoint), and the 4-fold ensemble for Borzoi. We separately apply Karollus et al.’s 6-pass TTA recipe to AlphaGenome (Figure 6) to characterise the stability gain, but we don’t apply it to Enformer, Borzoi, or NTv3.
No N-replacement control. Karollus also runs an all-N enhancer replacement (replacing every base with the “N” wildcard) as a stronger knockout than shuffling. We deliberately don’t: while reference genomes do contain N regions (gaps, centromeres, hard-masked repeats), these are typically excluded or under-weighted in training pipelines for these models, so a 2 kb block of N embedded in an otherwise-normal genic context is anomalous input. Predictions there reflect how the model handles unfamiliar input, not how it responds to motif loss, which is the actual question. The dinucleotide shuffle keeps the model in-distribution and isolates the motif-loss signal cleanly.
Linear vs. log₂ observed effect. Karollus uses log₂(1 + fraction_change); we report the linear fractional change and the log₂ observed effect. Spearman is identical; Pearson’s r differs by a small log-vs-linear distortion.

These methodological differences may shift absolute Pearson values, though broad model rankings should be unaffected.

Small Fulco sample (n ≈ 63). Fulco et al.’s ~60 validated enhancer–gene pairs is small enough that correlation estimates carry wide confidence intervals — for r ≈ 0.65 at n = 63 the 95% CI spans roughly ±0.15 either side. Numerical differences within roughly ±0.05–0.10 (e.g., AlphaGenome 0.67 vs Borzoi 0.66 on Fulco Pearson) should be read as essentially tied, not as a meaningful ordering. Gasperini’s ~440 pairs give tighter estimates, so the model gaps there are more reliable.

K562 only. Both screens are in K562, and K562 is heavily represented in every model’s training data. None of these numbers say anything about how the models would do on cell types under-represented in training. We’d expect the gap between models, and between predictions and truth, to widen there.

In-silico ≠ real CRISPRi. Dinucleotide shuffling destroys motif content but doesn’t capture chromatin context changes, dCas9 occupancy, or 3D genome reorganisation. It’s a motif-loss proxy, not a full simulation. Karollus discusses this; worth keeping in mind when reading the numbers.

CRISPRi isn’t purely cis. Knocking down an enhancer can affect other nearby genes, or the target gene’s own regulatory partners, whose altered expression shifts the cellular context against which we measure the target. Part of any measured enhancer effect is downstream of these indirect (trans) effects, which a sequence-only model predicting from a TSS-centred window can’t capture.

Implications

The benchmark tests seq2func models on experimentally validated distal enhancer-promoter interactions. AlphaGenome’s gains over Enformer on distal CRE effect-magnitude prediction are substantial, and most pronounced on the larger Gasperini et al. screen. That’s real progress. But the headline finding — that even AlphaGenome underestimates the magnitude of distal-enhancer effects — converges with what AlphaGenome’s own paper observed on the easier enhancer-gene linking task on ENCODE-rE2G: distal CREs remain hard from two distinct evaluation angles. Fully benchmarking seq2func models on distal CRE effect magnitudes will require follow-up CRISPRi screens in cell lines beyond K562 — without them, we can’t tell which gains generalise.

Echoing Tu, 2026 and Shen, 2025 from a different angle: AlphaGenome improves CRISPRi effect prediction, but the remaining gap — especially for distal CREs — is still an open problem for the field.

The repo is set up so adding a fifth model is one new scripts/test_<dataset>_<newmodel>.py writing a CSV with y_delta and pred_delta columns. If you’ve got a model you want to test, we’d love to see it!

Code and links

Source code & evaluation scripts
Karollus et al., 2023 — the benchmark this extends
SequenceModelBenchmark Zenodo (Karollus tables)
Models: Enformer (PyTorch) · Borzoi (original) · Flashzoi (Borzoi PyTorch port, 4-fold ensemble — used here) · NTv3 · AlphaGenome PyTorch port

References

Avsec, Ž. et al. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods, 18, 1196–1203 (2021). https://doi.org/10.1038/s41592-021-01252-x
Linder, J. et al. Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation. Nature Genetics, 57, 949–961 (2025). https://doi.org/10.1038/s41588-024-02053-6
Boshar, S. et al. A foundational model for joint sequence-function multi-species modeling at scale for long-range genomic prediction. bioRxiv (2025). https://doi.org/10.64898/2025.12.22.695963
Avsec, Ž. et al. Advancing regulatory variant effect prediction with AlphaGenome. Nature, 649, 1206–1218 (2026). https://doi.org/10.1038/s41586-025-10014-0
Karollus, A., Mauermeier, T., Gagneur, J. Current sequence-based models capture gene expression determinants in promoters but mostly ignore distal enhancers. Genome Biology, 24, 56 (2023). https://doi.org/10.1186/s13059-023-02899-9
Fulco, C. P. et al. Activity-by-contact model of enhancer–promoter regulation from thousands of CRISPR perturbations. Nature Genetics, 51, 1664–1669 (2019). https://doi.org/10.1038/s41588-019-0538-0
Gasperini, M. et al. A genome-wide framework for mapping gene regulation via cellular genetic screens. Cell, 176, 377–390.e19 (2019). https://doi.org/10.1016/j.cell.2018.11.029
Tu, X. et al. A modality gap in personal-genome prediction by sequence-to-function models. bioRxiv (2026). https://doi.org/10.64898/2026.02.01.702969
Shen, L. AlphaGenome enhances personal gene expression prediction but retains key limitations. bioRxiv (2025). https://doi.org/10.1101/2025.08.05.668750
Sasse, A., Ng, B., Spiro, A.E. et al. Benchmarking of deep neural networks for predicting personal gene expression from DNA sequence highlights shortcomings. Nature Genetics, 55, 2060–2064 (2023). https://doi.org/10.1038/s41588-023-01524-6
Koo, P.K., Majdandzic, A., Ploenzke, M., Anand, P., Paul, S.B. Global importance analysis: An interpretability method to quantify importance of genomic features in deep neural networks. PLoS Computational Biology, 17(5), e1008925 (2021). https://doi.org/10.1371/journal.pcbi.1008925
Toneyan, S., Tang, Z., Koo, P.K. Evaluating deep learning for predicting epigenomic profiles. Nature Machine Intelligence, 4, 1088–1100 (2022). https://doi.org/10.1038/s42256-022-00570-9
Gschwind, A.R. et al. An encyclopedia of enhancer-gene regulatory interactions in the human genome. bioRxiv (2023). https://doi.org/10.1101/2023.11.09.563812
Cheng, W. et al. DNALONGBENCH: a benchmark suite for long-range DNA prediction tasks. Nature Communications, 16, 10108 (2025). https://doi.org/10.1038/s41467-025-65077-4
Barry, T., Wang, X., Morris, J.A. et al. SCEPTRE improves calibration and sensitivity in single-cell CRISPR screen analysis. Genome Biology, 22, 344 (2021). https://doi.org/10.1186/s13059-021-02545-2

Welcome to the Genomics × AI Blog

Thu, 19 Feb 2026 00:00:00 +0000

Welcome to the Genomics × AI blog.

We created this space for the community to use to share short, opinionated blog posts about how genomics and machine learning actually get done in real labs: what works, what doesn’t, and the ideas we’re still testing. Posts will be written in Git, reviewed internally within your lab, editorially reviewed in public, and published only when an editor is happy with them—so every article has a clear history and a clear home.

Note: This is to disseminate ideas, incremental results, negative results and tutorials quickly using blog posts as a more personalised, free-form format.

If you’d like to contribute, start by reading the Submission Guidelines and opening a discussion with us on GitHub Discussions.