Repository navigation
3.10.22.x: gh-151497 tarfile pre-allocation; rename XML_SetHashSalt16Bytes - #122
Merged
Merged
Conversation
…rfile extended headers (pythonGH-151498) (pythonGH-151978) tarfile reads a member's extended header (a GNU long name/link or a pax header) with a single read sized by the header's size field: buf = tarfile.fileobj.read(self._block(self.size)) The size is taken from the archive and is not validated, so a ~512-byte crafted file can claim several gigabytes (or, via base-256 encoding, far more) and make read() pre-allocate that much memory -- on open/iterate, before any extraction filter runs. Read the extended-header data in bounded chunks instead, so an oversized or truncated header can no longer force a huge allocation. The bytes returned for valid archives are unchanged. (cherry picked from commit da99711) Co-authored-by: Shardul Deshpande <iamsharduld@users.noreply.github.com> (cherry picked from commit b83961a) 3.10.22.x: the test_tarfile conflict was context only (test_extractfile_attrs, pythongh-74468, isn't on this line); only the ReadSizeRecorder and ExtendedHeaderMemoryTest classes are added.
pyexpatns.h prefixes the bundled Expat's exported symbols with PyExpat_ so they cannot clash with a system libexpat loaded in the same process. Expat 2.9.0 added XML_SetHashSalt16Bytes, which pyexpat and _elementtree call, but the header did not rename it (upstream main does not either). The AllocTracker functions are already renamed on this line. Same change as 3.7.17.x 69ab1c8 (#117 review). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
ezequielp-activestate
approved these changes
Oct 29, 2026
ezequielp-activestate
left a comment
ezequielp-activestate
left a comment
There was a problem hiding this comment.
Approving.
- The pythongh-151497 port matches upstream 3.13 (
b83961ade8) line for line, apart from one blank line. Upstream has no follow-ups to it. - With this branch's
Lib/teston a python:3.10 image,test_tarfilepasses with this branch'starfile.pyand fails with the 3.10.22.x one. - The
pyexpatns.hchange addsXML_SetHashSalt16Bytes. EveryXML_*function declared in the bundledexpat.hnow has aPyExpat_rename.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two follow-ups from the 3.7.17.8 review (#117) that also apply to this line. For the next 3.10.22.x release.
What's in it
tarfileread it with oneread(size), so a 512-byte file could maketarfile.open()allocate gigabytes before any extraction filter runs. The header is now read in 1 MiB chunks. Cherry-picked from upstream 3.13 (b83961a); upstream only backported it to 3.13 and 3.14. Thetest_tarfileconflict was context only.pyexpatns.hrenamesXML_SetHashSalt16Bytes. Expat 2.9.0 added it and pyexpat and_elementtreecall it, but it kept its unprefixed name, so it could clash with a system libexpat in the same process. The AllocTracker functions are already renamed on this line. EveryXMLPARSEAPIfunction in the bundledexpat.his now renamed. Upstream main doesn't rename this one either.Testing
tarfileraisesMemoryErrortrying to allocate the claimed size; with the fix it raisesReadErrorwithout the allocation. The three newExtendedHeaderMemoryTestcases and the whole oftest_tarfilepass.XML_symbols inpyexpat, andPyExpat_XML_SetHashSalt16BytesreplacesXML_SetHashSalt16Bytes.test_pyexpat,test_xml_etree,test_xml_etree_c,test_sax,test_minidom,test_xml_dom_minicompatandtest_tarfilepass.