Repository navigation
Conversation
The borsh and wincode deserializers for the prefixed vec types called `Vec::with_capacity(prefix)` with the length read from the input, before reading any items. A short input with a large prefix panicked with "capacity overflow" (`U64PrefixedVec`) or aborted on allocation failure (e.g. `U32PrefixedVec<u64>` with `u32::MAX`), instead of returning an error. Cap the initial capacity at 4096 bytes worth of items, like borsh does for `Vec<T>`, and let the vector grow as items are read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joncinque
left a comment
There was a problem hiding this comment.
Thanks for your contribution. If we want to limit this to any number, we should probably go with the maximum heap size allowed on-chain: https://github.com/anza-xyz/solana-sdk/blob/47e7cfc9c7563879eb32a5efac9b6f6243923407/message/src/versions/v1/mod.rs#L38
Or we can use the default.
What do you think @febo ?
|
Thanks! Note this only caps the initial |
Problem
The borsh and wincode deserializers generated by
prefixed_vec_type!(collections/src/vec.rs) callVec::with_capacity(prefix)with the length prefix read from the input, before any item is read. A short input with a large prefix therefore panics or aborts instead of returning an error:U64PrefixedVec::<u8>::try_from_slice(&u64::MAX.to_le_bytes())(8 bytes) panics withcapacity overflow, and so doeswincode::deserialize::<U64PrefixedVec<u8>>on the same bytes;U32PrefixedVec::<u64>::try_from_slice(&u32::MAX.to_le_bytes())(4 bytes) tries to allocate ~32 GiB and aborts the process (memory allocation of 34359738360 bytes failed);The
U*PrefixedStrtypes are built on these, so they're affected too. borsh's ownVec<T>impl avoids this by capping the initial capacity at 4096 bytes (borsh::de::hint::cautious), and wincode applies its preallocation limit, but these custom impls skip both.Summary of Changes
cautious_capacity::<T>(prefix)helper that caps the initial capacity at 4096 bytes worth of items (same as borsh), and use it in both the borsh and wincodePrefixedVecdeserializers. The read loop and wire format are unchanged; vectors over that size just grow as items are read.prefixed_vec_borsh_with_oversized_prefixandprefixed_vec_wincode_with_oversized_prefix. Both panic onmainand pass with this change.cargo test --all-featuresforspl-collections,cargo fmt --check, clippy with the Makefile's deny flags, andcargo checkwith no features /borshonly /wincodeonly are all clean.I used an AI assistant (Claude) while preparing this. I reproduced the issue and checked the fix and tests myself.
🤖 Generated with Claude Code