I went foundations first, in this order.

  • Hashes, keys, accounts, slots and epochs
  • Sysvars and the bank
  • LiteSVM's call path, from send_transaction through sanitize, execute, the blockhash and history checks and into process_transaction

For each piece I worked out the concept and predicted what the code would do, then ran it to check. That led me deeper into the repo, and eventually to something worth investigating.

The helper that went nowhere

In crates/litesvm/tests/stake_program.rs, advance_epoch is meant to move the chain forward one epoch. This is what it looked like.

fn advance_epoch(svm: &mut LiteSVM) {
    refresh_blockhash(svm);
    let old_clock = svm.get_sysvar::<Clock>();
    let root_slot = old_clock.slot;
    let slots_per_epoch = svm.get_sysvar::<EpochSchedule>().slots_per_epoch;
    svm.warp_to_slot(root_slot + slots_per_epoch);
    let mut new_clock = old_clock;
    new_clock.epoch += 1;
    svm.set_sysvar::<Clock>(&new_clock)
}

It copies the Clock, then calls warp_to_slot, which moves svm's real Clock forward. Then it takes the copy from before the warp, adds one to the epoch and writes it back. set_sysvar replaces every field, so the old slot overwrites the warped one. The epoch moved and the slot never did.

Nobody noticed because the test calls it twice to get past epoch 2, guarding against an old StakeHistory bug, and everything it checks depends only on the epoch. It passed before the fix and it passes after.

slotepoch
before any call459,648,0000
after 2 calls, old helper459,648,0002
after 2 calls, fixed460,512,0002

459,648,000 is LiteSVM's default start slot, MAINNET_DEFAULT_SLOT, the slot of the latest mainnet feature activation, Alpenglow's. The fixed run ends two epochs of 432,000 slots later.

get_sysvar returns a copy

get_sysvar<T>(&self) -> T takes &self and hands back an owned value. set_sysvar<T>(&mut self, sysvar: &T) builds a new account and replaces the old one.

LiteSVM stores the Clock the way mainnet does, as an account with 40 bytes of data rather than a struct. get_sysvar decodes those bytes into a fresh Clock on every call. Changing it changes nothing in svm until you write it back with set_sysvar.

It's a classic lost update. A stale copy is still fine for comparing, and my tests in #452 keep the pre-warp Clock as before on purpose. The damage only comes from writing it back.

I got this wrong at first. I thought warp_to_slot didn't change Clock.slot and that the copy was a different clock. There's one Clock in svm. The warp moves it, and the copy is a snapshot that doesn't follow later changes.

The fix

Read once, change, write once.

fn advance_epoch(svm: &mut LiteSVM) {
    refresh_blockhash(svm);
    let slots_per_epoch = svm.get_sysvar::<EpochSchedule>().slots_per_epoch;
    let mut clock = svm.get_sysvar::<Clock>();
    clock.slot += slots_per_epoch;
    clock.epoch += 1;
    svm.set_sysvar::<Clock>(&clock)
}

To check it I added this line after the two calls,

println!("{:?}", svm.get_sysvar::<Clock>());

and ran

cargo test --features precompiles --test stake_program -- --nocapture

It printed

Clock { slot: 460512000, epoch_start_timestamp: 0, epoch: 2, leader_schedule_epoch: 0, unix_timestamp: 0 }

The same run showed that a number in my guide was out of date. The start slot had moved with a PR merged two days earlier. Trust the source over the notes.

Called isn't checked

warp_to_slot is documented to set only Clock.slot. Five existing tests call it. Two would notice if it stopped setting the slot, but none check that it leaves the rest of the Clock alone.

To show that, I broke it on purpose. With clock.epoch += 1; added inside warp_to_slot, compute_budget still passed all 9 tests, stake_program passed its 1, and the persistence roundtrip passed all 20, because it only asserts that the slot is 42. Only my two new warp_to_slot tests failed.

Grep tells you a function is called. Coverage tells you its lines ran. Only breaking it on purpose tells you whether anything checks it.

The new tests compare the whole Clock in one assert, since Clock derives PartialEq and Debug. One warps forward and one warps backward, because the warp sets the slot rather than advancing it.

assert_eq!(after, Clock { slot: target, ..before });

Under the mutation the failure prints both Clocks, the same slot with epoch: 1 against epoch: 0. It says exactly what broke.

Takeaways

This was my entry into open-source contribution on Solana. Respect to Aursen for building a test tool this fast that so many people rely on.

The key takeaway was patience. Understand what's actually going on, then verify it, because one small detail changes everything. The bug wasn't hidden. It sat in a test that passed, and reading slowly enough was all it took to see it.

← journal↑